JP2011107829A - Sensor and image processing system - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a sensor, which reliably determines the presence of changes between picked-up images, resulting in appropriately using the determined results even without sufficient ambient illumination. <P>SOLUTION: The sensor 1 includes an optical sensor liquid crystal panel integrated with a photodiode group including at least a photodiode which responds to an infrared ray for each display pixel 111, wherein the photodiode group integrated into an optical sensor crystal liquid panel includes at least an infrared photodiode 113. The sensor includes: an image comparing part 103 for detecting a difference between a first image picked up by the infrared photodiode 113 and a second image picked up by the same photodiode as the infrared photodiode 113 which has picked up the first image after the lapse of an arbitrary time; and a difference detection part 104 for determining the presence/absence of the change of the first image and the second image from the difference obtained by the image comparing part 103. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to a sensor device including an optical sensor liquid crystal panel and an image processing system.

  Generally, a PC (personal computer) has a function of performing a certain operation and restoring the operation when the main body is in a power saving mode = standby mode.

  The return from the standby mode is executed by the user operating “mouse” and “keyboard” which are input devices in a general PC. When the PC is a node type, as a user's return operation, “open a liquid crystal panel”, “press any key”, “push a power button”, or the like is common.

  Furthermore, in recent years, power saving of PCs has been further strengthened, and PCs have been configured to shift to a power saving mode more finely. For example, it is configured to enter the power saving mode even when the user leaves the seat lightly during work by the PC. In such a case, it is necessary to improve so that the operation for returning from the power saving mode can be quickly performed. As one of the operations, for example, a node type PC is provided as one of input devices. It has been proposed to perform a return operation by touching the touchpad.

  For example, it is considered that a liquid crystal display device having a touch sensor disclosed in Patent Document 1, for example, is used as a touch pad. In a liquid crystal display device having a general touch sensor, a user touches a liquid crystal panel. In order to detect this, a process for acquiring position information touched by the user is necessary. The processing for acquiring this position information consumes a lot of power because it requires computation.

  Therefore, for example, as a liquid crystal panel disclosed in Patent Document 1, in addition to a liquid crystal matrix for display, photodiodes are arranged in a matrix and images are read by obtaining this information. It has been proposed to provide the above-mentioned touchpad with an “optical sensor liquid crystal”.

  In this case, without using power consumption processing, that is, processing for detecting coordinates, the scanner function of the optical sensor liquid crystal is used to compare the image when the user is away from the image when touched. It is possible to easily detect that the user “touched”.

  Specifically, it is as follows.

  The optical sensor liquid crystal is a so-called image pickup device. For example, a simple comparison is made between an image taken when leaving a seat and an image taken periodically in a power saving mode. In the case of the “image”, a “finger shadow” that is not in the former image is captured, and a large “difference” is obtained as a comparison result. By detecting this “difference”, the return process is triggered.

  For example, a sensor device as shown in FIG.

  As shown in FIG. 5, the sensor device includes an image capturing device 1100 and an optical sensor liquid crystal panel 1110.

  The image capturing device 1100 includes an image interface 1101 to which an image captured by the optical sensor liquid crystal panel 1110 is input, an image storage device 1102 for storing an image from the image interface 1101, and an image sent from the image interface 1101. And an image comparison unit 1103 that compares the image stored in the image storage device 1102 and a difference detection unit 1104 that detects a difference between images from the image comparison result by the image comparison unit 1103. The difference detection unit 1104 outputs the difference detection result 1105 to a device (not shown) as a power saving mode release signal.

  In FIG. 5, broken line breaks indicated on the optical sensor liquid crystal panel 1110 indicate display pixels 1111, and solid line breaks indicate sensor pixels 1112. Here, one sensor pixel 1112 corresponds to 4 × 4 display pixels 1111.

  In one sensor pixel 1112, a photodiode (hereinafter referred to as a visible light photodiode) 1114 that is sensitive to visible light is disposed.

  In the sensor device configured as described above, an image captured by the optical sensor liquid crystal panel 1110 is input to the image interface 1101 in the right image capturing device 1100 via the liquid crystal driver 1115. An image input to the image interface 1101 is an image captured by the visible light photodiode 1114.

  The image interface 1101 stores an image including pixels captured by the visible light photodiode 1114 from the input image in the image storage device 1102.

  The image comparison unit 1103 compares the image from the image interface 1101 with the image stored in the image storage device 102. The comparison result obtained by comparing the images in the image comparison unit 1103 is transferred to the subsequent difference detection unit 1104, and the difference between the images is detected.

  The difference detection unit 1104 not only detects the difference, but also determines whether or not two images are different depending on the detected difference, and the difference detection result 1105 including the determination result is illustrated as a power saving mode release signal. Output to devices that do not.

  That is, before the optical sensor liquid crystal panel 1110 enters the power saving mode, an image “only the background without a hand” is captured. This image is stored in the image storage device 1102.

  Next, during the power saving mode, an image is captured constantly or intermittently, and the image comparison unit 1103 sequentially compares the captured image with the image stored in the image storage device 1102.

  Furthermore, when “holding the hand”, the captured image is sufficiently different from the image stored in the image storage device 1102, and the image capturing device 1100 outputs a power saving mode release signal as the difference detection result 1105. To do. Thereby, when setting the power saving mode, the user can cancel the setting of the power saving mode only by an operation of holding the hand.

Japanese Patent Laying-Open No. 2005-275644 (released on October 06, 2005) Japanese Patent No. 4201191 (registered on October 17, 2008)

  By the way, in the case of a normal office or a living room in the home, there is sufficient ambient illuminance, and a sufficient “difference” is obtained between an image when the user leaves the seat and an image having a “finger shadow”. On the other hand, ambient illuminance is insufficient in a dark restaurant in a room or in a bedroom in a home with only a headlamp, and the “difference” between the two is not sufficient. In the experiment, when the ambient illuminance is from several tens of lux to one hundred lux or less, it is impossible to determine whether the “difference” between the two is correct.

  Therefore, as shown in FIG. 5, when only a photodiode that is sensitive to visible light is used as a photodiode, in an environment where the ambient illuminance is low, the operation should be resumed from the power saving mode by an operation of “holding hands”. However, there is a risk of causing a malfunction in the operation of a PC or the like that “cannot be restored”.

  From the above, when using the optical sensor liquid crystal to determine the presence or absence of changes between images captured with visible light, and using the determination result, sufficient ambient illuminance is required, and sufficient illumination illuminance If there is not, there arises a problem that the above determination result cannot be used appropriately.

  The present invention has been made in view of the above problems, and its purpose is to appropriately determine the determination result by reliably determining whether or not there is a change between captured images even if the ambient illuminance is not sufficient. It is in providing the sensor apparatus which can be utilized for.

  In order to solve the above problems, the sensor device of the present invention includes a photodiode for each display pixel or for each display pixel group including n × m (n and m are integers of 1 or more) display pixels. In the sensor device including the incorporated photosensor liquid crystal panel, the photodiode group incorporated in the photosensor liquid crystal panel includes at least a photodiode that is sensitive to infrared light, and is sensitive to the infrared light. A first image obtained by imaging with a photodiode and a second image obtained by imaging with the same photodiode as the photodiode that is sensitive to the infrared light that has captured the first image after an arbitrary time has elapsed. Difference detecting means for detecting a difference between the first image and a determination means for determining whether there is a change between the first image and the second image from the difference obtained by the difference detecting means. It is characterized by a door.

  According to the above configuration, if there is an infrared light irradiation environment in the surroundings, it is possible to determine whether or not there is a change between the first image and the second image even in an environment where the ambient illuminance is low.

  Accordingly, if there is an infrared light irradiation environment in the surroundings, it is possible to appropriately use the determination result of whether or not there is a change between the first image and the second image regardless of the ambient illuminance.

  Here, the ambient infrared light irradiation environment described above is an infrared light irradiation environment provided by an infrared light source provided outside the sensor device, or a photosensor liquid crystal panel inside the sensor device. Including irradiation environment of infrared light by infrared light backlight.

  The photodiode group incorporated in the optical sensor liquid crystal panel includes a photodiode sensitive to visible light and a photodiode sensitive to infrared light, the photodiode sensitive to visible light, and the above The photodiodes that are sensitive to infrared light are alternately arranged at the same ratio.

  In this case, photodiodes that are sensitive to visible light and photodiodes that are sensitive to infrared light are alternately arranged at the same ratio. The image is not affected by the arrangement position of the photodiode that is sensitive to infrared light, and the image has no distortion or very little distortion. Conversely, an image captured only with a photodiode sensitive to infrared light is not affected by the arrangement position of the photodiode sensitive to visible light (regular pixel omission) and has no distortion in shape, or The image is extremely distorted.

  In other words, if the photodiodes that are sensitive to visible light and the photodiodes that are sensitive to infrared light are not in the same ratio, the captured image has an influence on the arrangement position of the photodiodes that are not used for imaging (regularity of pixels). The image becomes distorted.

  In this way, when an image is picked up by extracting only pixels from one photodiode from images picked up by two types of photodiodes, a distorted image in which pixels are regularly removed is obtained. Therefore, in order to treat the image size of an image captured by one photodiode as an image of the original image size, the pixel (missing pixel) where the other photodiode is arranged is interpolated from one surrounding pixel. It needs to be generated. For example, there are the following interpolation methods.

  The photodiode group incorporated in the optical sensor liquid crystal panel includes a photodiode sensitive to visible light and a photodiode sensitive to infrared light, the photodiode sensitive to visible light, and the above It is characterized in that photodiodes sensitive to infrared light are alternately arranged in the horizontal direction.

  According to the above configuration, the photodiodes that are sensitive to visible light and the photodiodes that are sensitive to infrared light are alternately arranged in the horizontal direction, so that the nearest pixel to the interpolation target pixel. Are the left and right neighbors. In this case, interpolation processing can be performed using data of an image captured by a photodiode sensitive to visible light adjacent to the left and right in the horizontal direction of the photodiode sensitive to infrared light corresponding to the pixel to be interpolated. Therefore, no line memory is required for interpolation processing, and simple interpolation processing is sufficient.

  The photodiode group incorporated in the optical sensor liquid crystal panel includes a photodiode sensitive to visible light and a photodiode sensitive to infrared light. The photodiode sensitive to visible light, and the infrared It is characterized in that photodiodes sensitive to light are alternately arranged in the vertical direction.

  According to the above configuration, the photodiodes that are sensitive to visible light and the photodiodes that are sensitive to infrared light are alternately arranged in the vertical direction, so that the nearest pixel to the interpolation target pixel Is next to the top and bottom. In this case, interpolation processing can be performed using data of a captured image by a photodiode sensitive to visible light adjacent vertically in the vertical direction of the photodiode sensitive to infrared light corresponding to the pixel to be interpolated. Therefore, no line memory is required for interpolation processing, and simple interpolation processing is sufficient.

  The photodiode group incorporated in the optical sensor liquid crystal panel includes a photodiode sensitive to visible light and a photodiode sensitive to infrared light. The photodiode sensitive to visible light, and the infrared The light-sensitive photodiodes are arranged in a checkered pattern.

  According to the above configuration, the photodiodes that are sensitive to visible light and the photodiodes that are sensitive to infrared light are arranged in a checkered pattern, so that the nearest pixel is located above and below the pixel to be interpolated. There are four pixels on the left and right. In this case, since the interpolation processing can be performed using the data of the image captured by the photodiode sensitive to the visible light adjacent to the upper, lower, left and right of the photodiode sensitive to the infrared light corresponding to the pixel to be interpolated, A line memory is not required for the interpolation processing, and simple interpolation processing is sufficient. In addition, since the data of the image captured by the photodiode sensitive to visible light corresponding to the four adjacent pixels is used, the value of the pixel after the interpolation processing can be set to a more appropriate value.

  The optical sensor liquid crystal panel is provided with an infrared backlight that irradiates infrared light from a surface opposite to the image capturing surface.

  According to the above configuration, it is possible to capture an image with a photodiode that is sensitive to infrared light, even in the absence of an infrared light irradiation environment.

  Specifically, the imaging execution control unit includes a lighting control unit that performs lighting control of the infrared light backlight, and an imaging execution control unit that performs image imaging execution control using a photodiode that is sensitive to the infrared light. And means for performing image capturing so as to always perform image capturing with a photodiode sensitive to the infrared light when the lighting backlight is controlled so that the infrared backlight is always lit. It is characterized by.

  According to the above configuration, since the image capturing using the photodiode sensitive to infrared light is always performed, the difference determination between the first image and the second image can be quickly performed. That is, it is possible to quickly detect a change between the first image and the second image.

  In addition, a lighting control unit that performs lighting control of the infrared backlight, and an imaging execution control unit that performs execution control of image capturing by a photodiode that is sensitive to the infrared light, the imaging execution control unit includes: When the infrared light backlight is controlled to be intermittently lit by the lighting control means, the image is picked up by the photodiode sensitive to the infrared light at the timing when the infrared light backlight is lit. It is characterized by executing.

  According to the above configuration, since the image capturing by the photodiode sensitive to the infrared light is intermittently performed, it is not necessary to always perform the process for obtaining the difference between the captured images. As a result, the sensor The power consumption in the apparatus can be reduced.

  The lighting control means is characterized in that the infrared backlight is controlled to be lit for M frame periods (N and M are integers and N> M) in N frame periods.

  According to the above configuration, since the image capturing by the photodiode sensitive to the infrared light is periodically performed, it is not necessary to always perform the processing for obtaining the difference between the captured images. As a result, the sensor The power consumption in the apparatus can be reduced.

  Furthermore, a drive control means for controlling the driving of the difference detection means and the determination means is provided, and the drive control means stops driving the difference detection means and the determination means when the infrared backlight is turned off. May be.

  While the infrared backlight is off, no image is captured, so there is no need to detect the difference between the captured images or determine the difference.

  Therefore, as in the above configuration, by stopping the driving of the difference detection unit and the determination unit when the infrared backlight is turned off, it is possible to reduce the power consumed for the processing related to the difference detection and the difference determination. As a result, the power consumption of the entire sensor device can be reduced.

  The first image and the second image are picked-up images picked up using all photodiodes sensitive to infrared light.

  According to the above configuration, the first image and the second image are captured images captured using all of the photodiodes that are sensitive to infrared light. Finer control is possible. For example, there is an effect that a threshold value can be set with high accuracy to determine whether the change is caused by image noise or the change is caused by holding the hand.

  The first image and the second image are picked-up images picked up using a part of a photodiode sensitive to infrared light.

  According to the above configuration, the first image and the second image are captured images that are captured using a part of the photodiode that is sensitive to infrared light, so that the first image is stored. The storage capacity of the image storage device can be suppressed.

  Here, the presence / absence of “change” in the image is determined by performing statistical processing of “what percentage of the whole” the number of pixels whose pixel value has changed to some extent.

  For this reason, it is preferable that the first image and the second image are captured images using 1000 or more and 4096 or less of photodiodes sensitive to infrared light.

  The difference detection means detects an absolute value of a difference between pixel values of the first image and the second image as a difference.

  The determination means preferably has the following configuration.

  That is, the determination unit counts pixels in which the absolute value of the difference between pixel values as a difference detected by the difference detection unit exceeds a predetermined threshold, and the number of pixels is equal to or greater than a predetermined number. It is characterized in that it is determined that there is a change between the first image and the second image.

  The determination means counts pixels whose absolute value of the difference between the pixel values as the difference detected by the difference detection means exceeds a predetermined threshold, the number of pixels is equal to or greater than the first threshold, and When the threshold value is less than or equal to 2, it is determined that there is a change between the first image and the second image.

  Furthermore, it further includes an illuminance measuring means for visible light, and when the measurement result by the illuminance measuring means is not more than a predetermined illuminance, the lighting control means performs lighting control of the infrared light backlight, and the imaging The execution control means performs execution control of image capturing by the photodiode sensitive to the infrared light, and the drive control means performs drive control of the difference detection means and the determination means.

  The image processing system of the present invention receives a determination result by the sensor device having the above-described configuration and a determination unit of the sensor device, and is obtained by imaging with a photodiode sensitive to infrared light of the sensor device. And a second image obtained by imaging with the same photodiode as the photodiode that senses the infrared light obtained by imaging the first image after an arbitrary period of time, and the received first image And the second image are subjected to predetermined image processing to acquire specific information, and when the specific information by the image processing apparatus is not acquired for a certain period of time, the image processing operation is performed. If the image processing is stopped when the determination result received from the sensor device is a result indicating that there is a change between the first image and the second image, the operation of the image processing is stopped. It is characterized in that to return the.

  The optical sensor liquid crystal panel in the sensor device captures the first image at the timing of stopping the image processing operation of the image processing device.

  The specific information acquired by the image processing device is positional information within the screen of the optical sensor liquid crystal panel in the sensor device.

  The sensor device of the present invention includes a photosensor liquid crystal panel in which a photodiode is incorporated for each display pixel or for each display pixel group including n × m (n and m are integers of 1 or more) display pixels. In the sensor device, the photodiode group incorporated in the optical sensor liquid crystal panel includes at least a photodiode sensitive to infrared light, and is obtained by imaging with the photodiode sensitive to infrared light. The difference which detects the difference of the 1st image and the 2nd image obtained by image-capturing with the same photodiode as the photodiode which senses the infrared light which imaged the 1st image after arbitrary time passed Irradiation of infrared light to the surroundings by including detection means and determination means for determining whether or not there is a change between the first image and the second image from the difference obtained by the difference detection means ring If there is an effect that it is possible regardless of the ambient illumination, to appropriately use the determination result of the presence or absence of a change between the first and second images.

It is a schematic block diagram which shows an example of the image processing system to which the sensor apparatus of this invention is applied. It is a schematic sectional drawing of the optical sensor liquid crystal panel of the sensor apparatus shown in FIG. (A)-(b) is a top view which shows the example of arrangement | positioning of a photosensor in the optical sensor liquid crystal panel of the sensor apparatus shown in FIG. It is a schematic block diagram which shows the example of the other sensor apparatus of this invention. It is a schematic block diagram which shows the comparative example of a sensor apparatus.

An embodiment of the present invention will be described as follows.
(Overview of image processing system)
In the present embodiment, an image processing system including the sensor device of the present invention will be described.

  As shown in FIG. 1, the image processing system includes a sensor device 1, an image processing device 120, and a power control unit 130. Here, the setting of the power saving mode in the image processing system is performed by the image processing apparatus 120, and the cancellation of the power saving mode is performed by the sensor apparatus 1.

  The power control unit 130 controls power supply to the sensor device 1 and the image processing device 120.

  Accordingly, the power control unit 130 performs control to set or cancel the power saving mode in which the supply of power to the optical sensor liquid crystal panel 110 of the sensor device 1 is suppressed. Here, the power control unit 130 receives an instruction for setting the power saving mode, thereby performing power supply control so that the power saving mode is set for the image processing system, and canceling the setting. Power supply control is performed so as to cancel the power saving mode set for the image processing system.

The setting of the power saving mode for the image processing system is performed based on the result of image processing in the image processing apparatus 120, and the release of the set power saving mode is performed based on the detection result in the sensor apparatus 1. . In the following, first, the set power saving mode canceling process, which is a characteristic part of the present invention, will be described, and then the power saving mode setting process will be described.
(Explanation for canceling the set power saving mode)
A process for canceling the set power saving mode in the image processing system by the sensor device 1 will be described.

  The sensor device 1 includes an image imaging device (imaging execution control means) 100 and an optical sensor liquid crystal panel 110. In FIG. 1, the right side of the sensor device 1 is the image pickup device 100, and the left side is the optical sensor liquid crystal panel 110.

  The image capturing device 100 includes an image interface 101 to which an image captured by the optical sensor liquid crystal panel 110 is input, an image storage device 102 for storing an image from the image interface 101, and an image sent from the image interface 101. And an image comparison unit 103 that compares the image stored in the image storage device 102 and a difference detection unit (difference detection unit) 104 that detects a difference between images from the image comparison result by the image comparison unit 103. The difference detection unit 104 outputs the difference detection result 105 to the image processing device 120 and the power control unit 130.

  Note that not only the difference detection result 105 from the difference detection unit 104 but also an image from the image interface 101 is directly sent to the image processing apparatus 120.

  In the optical sensor liquid crystal panel 110, a liquid crystal driver 115 for driving the optical sensor liquid crystal panel 110 is mounted with COG (Chip On Glass). Data is sent.

  In FIG. 1, broken line breaks on the optical sensor liquid crystal panel 110 indicate display pixels 111, and solid line breaks indicate sensor pixels 112. Here, as shown in FIG. 1, one sensor pixel 112 corresponds to 4 × 4 display pixels 111. However, the number of display pixels 111 with respect to one sensor pixel 112 is not limited to the 4 × 4 described above. In other words, the plurality of display pixels 111 (display pixel group) may be configured to correspond to one sensor pixel 112, or the sensor pixel 112 may be provided for each display pixel 111.

  In one sensor pixel 112, a photodiode (hereinafter referred to as an infrared light photodiode) 113 that is sensitive to infrared light or a photodiode (hereinafter referred to as a visible light photodiode) 114 that is sensitive to visible light is disposed. Has been. Here, the ratio of the infrared light photodiode 113 to the visible light photodiode 114 in one photosensor liquid crystal panel 110 is 1: 1, and the infrared light photodiode 113 and the visible light photodiode 114 are shown in FIG. As shown, it is arranged in a checkered pattern. The ratio of the infrared light photodiode 113 to the visible light photodiode 114 is not limited to 1: 1, and the arrangement of the infrared light photodiode 113 and the visible light photodiode 114 is as follows. It is not limited to a checkered pattern.

As the configuration of the photosensor liquid crystal panel 110, a photodiode is incorporated for each display pixel 111 or for each display pixel group including n × m (n and m are integers of 1 or more) display pixels 111. Anything is fine. That is, the photosensor liquid crystal panel 110 is formed with a photodiode group including a plurality of photodiodes.
(Description of the structure of the optical sensor liquid crystal panel 110)
As shown in FIG. 2, the photosensor liquid crystal panel 110 includes an image display unit (active matrix liquid crystal display element) 10 and an image input unit 20.

  The image display unit 10 has a predetermined pattern of transparent electrodes (made of ITO (tin-added indium oxide)) 205 and 208 on the inner surfaces of a pair of transparent substrates 200 and 206 that sandwich the ferroelectric liquid crystal 207. Are formed respectively.

  A thin film transistor (TFT) 201 is formed for each pixel 220 on the inner surface of the transparent substrate 200. The TFT 201 includes a channel layer 216 made of polycrystalline silicon, a gate insulating film 217, a gate electrode 202, a source electrode 203, and a drain electrode (connected to the transparent electrode 205) 204. Here, the transparent electrode 205 constitutes a pixel electrode and is formed on the transparent substrate 200 via an insulating layer 118.

  In the image display unit 10, the gate electrode 202 of each TFT 201 is connected to the liquid crystal driver 115 via a gate line (not shown) and the source electrode 203 of each TFT 201 is connected to a data line (not shown). Are connected to the liquid crystal driver 115 described above.

  As shown in FIG. 2, the image input unit 20 includes a light receiving layer 209 that receives light and converts it into an electric signal on the outer surface of the transparent substrate 206, and a transparent layer (transparent protective layer) that protects the light receiving layer 209. A layer 231 and a thin glass (color filter layer) 232.

  The light receiving layer 209 includes three layers of a lower electrode 213 that can block light, a semiconductor layer 214 that performs photoelectric conversion, and a transparent upper electrode 215, and forms an infrared light photodiode 113. The visible light photodiode 114 has the same configuration as the infrared light photodiode 113. That is, the visible light photodiode 114 and the infrared light photodiode 113 are the same photodiode, that is, a photodiode sensitive to both visible light and infrared light. As shown below, the visible light photodiode 114 and the infrared light photodiode 113 are only distinguished for convenience.

  Here, as described above, the thin glass 232 functioning as a color filter layer is further selected from infrared light and visible light from irradiation light irradiated to the infrared light photodiode 113 and the visible light photodiode 114. A filter (not shown) is provided. This filter is a filter formed so as to have a function of shielding visible light in a portion corresponding to the sensor pixel 112 (FIG. 1) provided with the infrared light photodiode 113.

  As described above, when a filter having a function of selecting infrared light and visible light from incident light is used, a photodiode that is sensitive to both infrared light and visible light can be used. In other words, a photodiode that is sensitive to both infrared light and visible light is laid in common, and a filter is provided at the entrance to change the characteristics of the filter for each pixel. A pixel may be provided. In the former case, a visible light cut filter is provided, and in the latter case, an infrared cut filter is provided.

  Here, since it is important to distinguish (select) and respond to infrared light and visible light in the present invention, in order to distinguish between infrared light and visible light, as described above, infrared light and visible light are used. A method of selecting incident light using the above-described filter or the like using a photodiode sensitive to both light, or changing the characteristics of the photodiode itself, that is, a photodiode sensitive only to infrared light. In addition, infrared light and visible light may be selected using a photodiode sensitive only to visible light.

  As described above, since the optical sensor liquid crystal panel 110 integrally configures the image display unit 10 and the image input unit 20, it can be reduced in size as compared with the case where they are configured separately. Has merit.

  When performing image display, similarly to the conventional liquid crystal display device, light (so-called backlight) 210 is incident substantially vertically from the transparent substrate 200 side in the image display unit 10 shown in FIG. Transmission or blocking is performed for each pixel 220 in accordance with an applied voltage to the transparent electrodes 205 and 208. Thereby, the image display unit 10 performs image display.

  When inputting an image, as shown in FIG. 2, the finger 211 is brought close to the surface of the thin glass 232 of the image input unit 20. The light receiving layer 209 detects the light 212 reflected by the finger 211 among the light 210 transmitted by the image display unit 10 according to the voltage applied to the transparent electrodes 205 and 208. At this time, the light receiving layer 209 detects whether the reflected light 212 is the infrared light or the visible light selected by the selection function of the thin glass plate 232.

  The light detected by the light receiving layer 209 is taken out as an output voltage and sent to the liquid crystal driver 115 described above. The processing ahead of the liquid crystal driver 115 will be described later.

  When inputting an image, the image display unit 10 selects a portion corresponding to each pixel of the finger 211. Therefore, it is not necessary to separate the light receiving layer 209 of the image input unit 20 for each pixel.

  Further, when inputting an image, as shown in FIG. 2, since reading is performed by bringing the finger 211 close to the surface of the thin glass 232 of the image input unit 20, it is not necessary to provide an optical system such as a lens.

  As described above, the photosensor liquid crystal panel 110 having the above configuration includes the image input function of the image input unit 20 in addition to the image display function of the image display unit 10.

In the optical sensor liquid crystal panel 110, the image input function in the image input unit 20 refers to a function of capturing an image using the infrared light photodiode 113 and the visible light photodiode 114.
(Description of operation of sensor device 1)
The operation of the sensor device 1 will be described below with reference to FIG.

  An image captured by the optical sensor liquid crystal panel 110 is input to the image interface 101 in the image capturing apparatus 100 on the right side via the liquid crystal driver 115. The image input to the image interface 101 is an image in a state in which an image captured by the infrared light photodiode 113 and an image captured by the visible light photodiode 114 are mixed in a checkered pattern.

  The image interface 101 selects, from the input image, an image including only pixels captured by the infrared light photodiode 113, only pixels captured by the visible light photodiode 114, or both pixels. Is done. The selected image is stored in the image storage device 102 based on a certain trigger signal.

  Here, as an example of the trigger signal, a signal including an explicit instruction from a so-called “host” that controls the sensor device 1 / image processing system from the outside can be cited. That is, the so-called “set”, “system”, and “product” including the sensor device 1 / image processing system need to determine whether or not to enter the power saving mode, or accept an instruction from the user. Therefore, since preparation for recovery is necessary when shifting to the power saving mode, a trigger signal including a “store” instruction for storing the above-described image in the image storage device 102 from the host is generated. Will be emitted.

  Further, the captured and sorted images are transferred from the image interface 101 to the image comparison unit 103.

  The image comparison unit 103 compares an image from the image interface 101 (hereinafter referred to as a second image) with an image stored in the image storage device 102 (hereinafter referred to as a first image). . Here, the second image is an image obtained by imaging with the same photodiode as the photodiode that is sensitive to the infrared light obtained by imaging the first image after an arbitrary time has elapsed. The comparison result obtained by comparing the images in the image comparison unit 103 is transferred to the difference detection unit 104 in the subsequent stage, and the difference between the first image and the second image is detected.

  The difference detection unit 104 not only detects the difference but also determines whether or not the first image and the second image are different from each other by the detected difference, and the difference detection including the determination result. The result 105 is transferred to the image processing apparatus 120 and the power control unit 130.

  As described above, the difference detection unit 104 not only functions as a difference detection unit that detects a difference between the first image and the second image, but also based on the difference between the first image and the second image. It also has a function as determination means for determining the presence or absence of a change.

  Hereinafter, an example of specific processing in the image capturing apparatus 100 will be described.

  The image comparison unit 103 calculates an absolute value of a difference between pixel values corresponding to each of the second image input from the image interface 101 and the first image stored in the image storage device 102. Ask. Here, when the absolute value of the difference is equal to or larger than a predetermined first threshold value, a pixel number counter (not shown) provided in the image capturing apparatus 100 is incremented by one. This is performed for the number of pixels corresponding to the captured image.

  As a result of this series of processing, the value of the pixel number counter, that is, the number of pixels whose absolute difference value is equal to or larger than a predetermined threshold value becomes the value of the comparison result in the image comparison unit 103. This comparison result is transferred to the difference detection unit 104.

  The difference detection unit 104 counts pixels (the number of pixels is equal to or greater than a predetermined number) in which the absolute value of the difference between pixel values as a detected difference exceeds a predetermined threshold, and the number of pixels is equal to or greater than the first threshold If it is present and less than or equal to the second threshold, it is determined that there is a change between the first image and the second image.

  Further, the difference detection unit 104 determines that the transferred comparison result, that is, the number of pixels in which the absolute value of the difference is equal to or greater than a predetermined threshold is equal to or greater than a predetermined second threshold. In this case, it is determined that the difference between the two images (the first image and the second image) is sufficiently large (the two images are sufficiently different), and a value indicating “true” is set as the difference detection result 105 in the subsequent stage. The image is output to the image processing apparatus 120 and the power control unit 130.

  The operation of the image capturing apparatus 100 configured as described above is as described above. To summarize, the image captured at a certain time (the first image stored in the image storage device 102) and the image captured in real time are captured. If a sufficient difference is recognized between the detected image (the second image input from the image interface 101) and the image capturing apparatus 100 sets a value indicating “true” as the difference detection result 105 in the subsequent image processing. It is configured to output to the device 120 and the power control unit 130.

  In addition, the image capturing apparatus 100 is configured such that only the infrared light image captured by the infrared light photodiode 113, only the visible light image captured by the visible light photodiode 114, or the infrared light image and visible light. It is configured to be able to handle three patterns of images that are mixed with images. For this reason, the trigger signal for canceling the power saving mode without being influenced by the surrounding environment by properly using the three patterns of images as necessary according to the surrounding environment (illuminance, etc.) of the image processing system. (Difference detection result 105) can be output appropriately.

  Here, the characteristics of the operation of the sensor device 1 configured by combining the image pickup device 100 and the optical sensor liquid crystal panel 110 configured as described above will be described as follows from the viewpoint of application.

  First, before the optical sensor liquid crystal panel 110 enters the power saving mode, a first image “only the background without a hand” is taken. This first image is stored in the image storage device 102.

  Next, during the power saving mode, an image is captured constantly or intermittently to be a second image, and the image comparison unit 103 stores the second image and the first image stored in the image storage device 102. Are compared sequentially.

  Further, when “holding the hand”, the captured image (second image) is sufficiently different from the first image, and the present image capturing apparatus 100 performs image processing with “true” as the difference detection result 105. The data is output to the device 120 and the power control unit 130.

  Finally, the power control unit 130 cancels the setting of the power saving mode of the optical sensor liquid crystal panel 110 according to the value of the difference detection result 105. The details of the processing when the difference detection result 105 is sent to the image processing device 120 will be described later.

  Therefore, it is first possible to realize an “operation” of canceling the power saving mode by “operation” of holding the hand over the optical sensor liquid crystal panel 110. Furthermore, imaging and comparison can be performed selectively or simultaneously with visible light or infrared light, and if infrared light is used, even if it is completely dark when viewed from a human, It is possible to detect an “operation” of holding a hand.

In the image processing system having the above-described configuration, the photosensor liquid crystal panel 110 mounted on the sensor device 1 includes photodiodes having different light sensitive to the infrared light photodiode 113 and the visible light photodiode 114 in one sensor pixel 112. By being equipped, the set power-saving mode is ensured by detecting “operation” that the user holds his / her hand without being affected by the surrounding environment (mainly the environment where lighting is not sufficient). There is an effect that it can be canceled.
(Explanation of power saving mode setting process)
Next, the power saving mode setting process in the image processing system by the image processing apparatus 120 will be described.

  As described above, the image processing system includes the image processing device 120 necessary for setting the power saving mode in addition to the sensor device 1.

  The image processing apparatus 120 has a function of acquiring position information (position information in the screen) as specific information from at least an image input via the image capturing apparatus 100. In this case, for example, when the optical sensor liquid crystal panel 110 captures an image “touched by a finger”, the image processing device 120 identifies and detects the part of the finger pad touching the finger, Is stored in the form of information.

  A timer 121 is connected to the image processing apparatus 120. The timer 121 sends a signal to the image processing device 120 at regular time intervals, and the image processing device 120 that receives the signal monitors whether position information is detected within this time interval = “whether a finger touches”. .

  If the result of monitoring indicates that the finger has not been touched, it is determined that the position information detection function has not been used. In this case, the position information detection may not be used. The image processing apparatus 120 determines that it can be interpreted as being present, and sends a signal to the power control unit 130 to instruct the setting of the power saving mode of the entire image processing system, including the stop of its own operation.

  Therefore, when the specific information is not acquired for a certain period of time, that is, when the finger is not touched for a certain period of time, the image processing apparatus 120 is in a power saving state (power saving mode) with respect to the image processing system. A signal for shifting to is sent to the power control unit 130.

  When receiving the signal for shifting to the power saving mode, the power control unit (drive control means) 130 controls the power supply to the sensor device 1 and the image processing device 120 to shift to the power saving mode.

  The power control unit 130 preferably stops driving the difference detection unit 104 when the infrared backlight 140 is turned off.

  Here, as described in the explanation of the operation of the sensor device 1, the image capturing device 100 captures the first image at the timing of shifting to the power saving mode, that is, the timing of stopping the image processing operation of the image processing device 120. Take a picture. Then, after shifting to the power saving mode, the image capturing apparatus 100 performs a conventional process. The difference detection result 105 becomes “true (changed)” due to “hand held” on the optical sensor liquid crystal panel 110, and the image processing apparatus 120 is restarted based on this signal and the power control unit 130. Is transferred to the sensor device 1 and the image processing device 120, and the image processing system returns from the power saving mode to the operation mode. In this manner, a series of operations relating to the transition to the power saving mode and the return from now on can be performed, and the return process is executed by the operation of “holding a hand”.

  In the present embodiment, an example in which infrared photodiodes 113 and visible light photodiodes 114 incorporated in the optical sensor liquid crystal panel 110 are alternately arranged at the same ratio (1: 1) has been described.

  In this case, since the visible light photodiodes 114 and the infrared light photodiodes 113 are alternately arranged at the same ratio, for example, an image captured only by the visible light photodiode 114 is an infrared light photodiode. The image has no distortion as a shape, or has very little distortion without being affected by the arrangement position 113 (regular missing of pixels). On the contrary, an image picked up only by the infrared light photodiode 113 is not affected by the arrangement position of the visible light photodiode 114 (regular loss of pixels), and there is no distortion or extremely distortion. There are few images.

  In other words, if the visible light photodiode 114 and the infrared light photodiode 113 are not in the same ratio, the captured image has an influence on the arrangement position of the photodiodes that are not used for imaging (regular pixel omission). It tends to be a distorted image.

  As described above, when only the pixels from one of the photodiodes are extracted from the images captured by the two types of photodiodes, the image is captured in a distorted image in which the pixels are regularly missing. Therefore, in order to treat the image size of an image captured by one photodiode as an image of the original image size, the pixel (missing pixel) where the other photodiode is arranged is interpolated from one surrounding pixel. It needs to be generated. For example, depending on the arrangement relationship between the infrared light photodiode 113 and the visible light photodiode 114, there is the following interpolation processing method.

  With respect to the arrangement of the infrared light photodiode 113 and the visible light photodiode 114 shown in FIGS. 3A to 3C, respective interpolation processing methods will be described below.

  Here, FIG. 3A shows an example of a vertical stripe arrangement in which infrared photodiodes 113 and visible photodiodes 114 are alternately arranged in the horizontal direction, and FIG. FIG. 3C shows an example of a horizontal stripe arrangement in which the diodes 113 and the visible light photodiodes 114 are alternately arranged in the vertical direction, and FIG. 3C is the same as the arrangement shown in the sensor device 1 shown in FIG. An example of a checkered arrangement in which 113 and visible light photodiodes 114 are arranged in a checkered pattern is shown. In any example, the ratio of the infrared light photodiode 113 to the visible light photodiode 114 is 1: 1.

  In the case of the arrangement example shown in FIG. 3A, that is, in the case of a vertical stripe arrangement in which the infrared photodiodes 113 and the visible light photodiodes 114 are alternately arranged in the horizontal direction, the nearest pixel is left. Adjacent and right neighbors do not require a line memory for interpolation generation, and simple interpolation processing is possible.

  In the case of the arrangement example shown in FIG. 3B, that is, in the case of the horizontal stripe arrangement in which the infrared light photodiodes 113 and the visible light photodiodes 114 are alternately arranged in the vertical direction, the nearest pixel Is adjacent to the upper side and the lower side, so that no line memory is required for the generation of interpolation, and simple interpolation processing is possible.

  Thus, the arrangement as shown in FIG. 3B is simple and advantageous in the manufacturing process of the liquid crystal panel as shown in FIG. There is no feature on the image plane as shown in FIG. 3 or in the case shown in FIG.

  Here, the characteristics of the image plane are, for example, when considering the case where an infrared image is obtained by extracting only infrared light pixels, (1) whether the extraction process is easy, or (2) the image after generation. The so-called “score” is assigned to each photodiode arrangement with respect to the image quality, and so on, and this “score” is high or low.

  That is, the horizontal arrangement shown in FIG. 3A has a feature that the extraction process is easy because the line memory is not necessary when the trace of visible light extracted is filled with the interpolation value.

  In addition, in the case of the checkered state arrangement shown in FIG. 3C, the frequency characteristic “only” in the horizontal or vertical direction of the extracted image is deteriorated as shown in FIGS. 3A and 3B. There is no bias in deterioration, and the horizontal and vertical are deteriorated equally. Therefore, the image quality is the best.

  Furthermore, in the case of the arrangement example shown in FIG. 3C, that is, in the case of an arrangement in a checkered state in which the infrared light photodiode 113 and the visible light photodiode 114 are arranged in a checkered pattern, the nearest pixels are vertically and horizontally Thus, the effectiveness of the pixel value after interpolation generation is extremely high. That is, since the interpolation accuracy is higher than in the case of a vertical stripe state or a horizontal stripe state, a more natural image can be obtained.

  In the image processing system shown in FIG. 1, the ambient illumination environment for recognizing infrared light by the infrared light photodiode 113 is not particularly limited, and red light from an infrared light source provided outside the image processing system is not limited. It may be an external light irradiation environment, or may be an infrared light irradiation environment from an infrared backlight that irradiates infrared light as a backlight in the optical sensor liquid crystal panel 110. It is preferable that the optical sensor liquid crystal panel 110 is provided with an infrared backlight that is an infrared light source from the viewpoint that light irradiation can be reliably performed.

  FIG. 4 shows a sensor device 2 having a configuration in which an infrared backlight 140 and a backlight driver 141 are added to the sensor device 1 shown in FIG. The infrared backlight 140 is disposed so as to irradiate infrared light from the surface opposite to the image capturing surface of the photosensor liquid crystal panel 110.

  In the case of using an infrared light image in the sensor device 1 shown in FIG. 1, it is possible to detect an operation of “holding a hand” even in the darkness of human vision, but it is detected depending on where the infrared light source is provided. The certainty of is different. In other words, if an infrared light source is to be obtained from outside the image processing system, the above operation should always be performed near the infrared light source in order to detect the operation of holding the hand using the infrared light image. There is a need to do.

  However, as shown in FIG. 4, if the infrared light backlight 140 is provided in the sensor device 2 itself, it is possible to reliably detect the operation of “holding the hand” using the infrared light image. Thus, when using an infrared light image using the sensor device 2, it is necessary to turn on the infrared light backlight 140. In this case, the imaging of the first image and the imaging of the input image need to be performed with the infrared light backlight turned on.

  Here, it is necessary to turn on the infrared backlight 140 at the time of imaging. However, when the input image is taken intermittently, it is necessary to keep the infrared backlight 140 turned on during the non-imaging time. No. Since turning on the infrared light backlight 140 consumes electric power, the backlight should be actively turned off from the viewpoint of power saving. Accordingly, in order to perform execution control of image capturing in the image capturing apparatus 100, an instruction to turn on the infrared backlight 140 is sent to the backlight driver (lighting control means) 141 in synchronization with the timing of capturing the input image. Give it.

  As described above, when the sensor device 2 including the infrared light backlight 140 is used, if the visible light illuminance is sufficient in the surrounding environment in the image processing system, the image is captured using the visible light photodiode 114, and the visible light illuminance is obtained. If it is not sufficient, even if there is no infrared light source outside, it is possible to reliably perform image capturing using the infrared light photodiode 113 with the infrared light backlight 140 as the infrared light source.

  In the above example, since the infrared backlight should be actively turned off, the infrared backlight 140 is controlled to be turned on using the backlight driver 141 in synchronization with the timing of capturing the input image. However, for example, when the infrared backlight 140 is always lit, the following advantages can be obtained.

  That is, since the infrared light backlight 113 is always lit, the image pickup by the infrared light photodiode 113 is always performed, so that the difference determination between the first image and the second image can be quickly performed. Can be done. That is, there is an advantage that a change between the first image and the second image can be detected quickly.

  When the backlight driver 141 is controlled so that the infrared backlight 140 is intermittently turned on, the infrared light photodiode 113 causes the infrared backlight 140 to turn on. You may make it perform execution control of image pick-up.

  In this case, since the image capturing by the infrared light photodiode 113 is intermittently performed, it is not necessary to always perform a process for obtaining a difference between the captured images. As a result, the sensor device 1 ( It has the merit that the power consumption in 2) can be reduced.

  The backlight driver may control the infrared backlight 140 to light up for M frame periods (N and M are integers and N> M) in N frame periods.

  In this case, since the image pickup by the infrared light photodiode 113 that is sensitive to the infrared light is periodically performed, it is not necessary to always perform the process for taking the difference between the picked-up images. As a result, power consumption in the sensor device 1 (2) can be reduced.

  Further, the image pickup apparatus 100 further includes a visible light illuminance measuring unit (not shown), and when the measurement result by the illuminance measuring unit becomes a predetermined illuminance or less, the backlight driver as the lighting control unit. 141 performs lighting control of the infrared light backlight 140, performs execution control of image capturing by the infrared light photodiode 113 which is a photodiode sensitive to the infrared light, and performs power control as the drive control means. The unit 130 performs drive control of the difference detection unit 104 constituting the difference detection unit and the determination unit.

  In the above embodiment, when the first image and the second image are captured, the number of infrared photodiodes 113 included in the optical sensor liquid crystal panel 110 is not particularly limited, but the first The first image and the second image may be captured using all of the infrared light photodiode 113.

  In this case, it is possible to finely control the criterion for determining whether the change has occurred. For example, it is possible to set a threshold value for high accuracy with respect to whether it is “change” due to image noise or “change” due to “holding hands”.

  In the above configuration, the image storage device 102 for storing the first image is necessary. In order to suppress the storage capacity of the image storage device 102, the following may be performed.

  That is, the first image and the second image are picked up using a part of the infrared photodiode 113 included in the optical sensor liquid crystal panel 110.

  Here, the presence / absence of “change” in the image according to the present application is determined by performing a statistical process of “how many percent of the pixels” have a pixel value changed to some extent.

  Therefore, from the determination of the appropriateness of the change of the image, the first image and the second image are 1000 or more and 4096 or less of the infrared photodiodes 113 included in the optical sensor liquid crystal panel 110. It is preferable to take an image using the infrared light photo diode 113.

  For the purpose of the present invention, all of the photodiodes for image capturing included in the optical sensor liquid crystal panel 110 of the sensor device 1 (2) may be infrared photodiodes 113. That is, the ratio of the visible light photodiode 114 may be zero.

  The present invention is not limited to the above-described embodiments, and various modifications can be made within the scope shown in the claims, and embodiments obtained by appropriately combining technical means disclosed in different embodiments. Is also included in the technical scope of the present invention.

  INDUSTRIAL APPLICABILITY The present invention can be used for devices that are used in an environment where ambient illuminance is sufficiently low, such as devices that perform image pickup to recover from the power saving mode, and electronic devices such as notebook computers and mobile phones.

DESCRIPTION OF SYMBOLS 1 Sensor apparatus 2 Sensor apparatus 10 Image display part 20 Image input part 100 Image imaging device 101 Image interface 102 Image storage apparatus 103 Image comparison part 104 Difference detection part 105 Difference detection result 110 Photosensor liquid crystal panel 111 Display pixel 112 Sensor pixel 113 Red External light photodiode 114 Visible light photodiode 115 Liquid crystal driver 118 Insulating layer 120 Image processing device 121 Timer 130 Power control unit 140 Infrared light backlight 141 Backlight driver 200 Transparent substrate 201 TFT
202 Gate electrode 203 Source electrode 204 Drain electrode 205 Transparent electrode 206 Transparent substrate 207 Ferroelectric liquid crystal 208 Transparent electrode 209 Light receiving layer 213 Lower electrode 214 Semiconductor layer 215 Upper electrode 216 Channel layer 217 Gate insulating film 220 Pixel 231 Transparent protective layer 232 Thin plate Glass

Claims (20)

In a sensor device including an optical sensor liquid crystal panel in which a photodiode is incorporated for each display pixel or for each display pixel group including n × m (n and m are integers of 1 or more) display pixels,
The photodiode group incorporated in the optical sensor liquid crystal panel includes at least a photodiode sensitive to infrared light,
The first image obtained by imaging with the photodiode sensitive to the infrared light and the same photodiode as the photodiode sensitive to the infrared light after imaging the first image after an arbitrary period of time Difference detecting means for detecting a difference from the second image obtained by
A sensor device comprising: a determination unit that determines whether or not there is a change between the first image and the second image based on a difference obtained by the difference detection unit.   The photodiode group incorporated in the optical sensor liquid crystal panel includes a photodiode sensitive to visible light and a photodiode sensitive to infrared light, the photodiode sensitive to visible light, and the above 2. The sensor device according to claim 1, wherein the photodiodes sensitive to infrared light are alternately arranged at the same ratio.   The photodiode group incorporated in the optical sensor liquid crystal panel includes a photodiode sensitive to visible light and a photodiode sensitive to infrared light, the photodiode sensitive to visible light, and the above 2. The sensor device according to claim 1, wherein photodiodes sensitive to infrared light are alternately arranged in a horizontal direction.   The photodiode group incorporated in the optical sensor liquid crystal panel includes a photodiode sensitive to visible light and a photodiode sensitive to infrared light. The photodiode sensitive to visible light, and the infrared 2. The sensor device according to claim 1, wherein photodiodes sensitive to light are alternately arranged in a vertical direction.   The photodiode group incorporated in the optical sensor liquid crystal panel includes a photodiode sensitive to visible light and a photodiode sensitive to infrared light. The photodiode sensitive to visible light, and the infrared 2. The sensor device according to claim 1, wherein the photodiodes sensitive to light are arranged in a checkered pattern.   The infrared light backlight which irradiates infrared light from the surface on the opposite side to the imaging surface of an image is provided in the said optical sensor liquid crystal panel, The any one of Claims 1-5 characterized by the above-mentioned. The sensor device according to 1. Lighting control means for performing lighting control of the infrared light backlight;
Imaging execution control means for performing execution control of image imaging by the photodiode sensitive to the infrared light,
The imaging execution control means captures an image so as to always capture an image with a photodiode sensitive to the infrared light when the lighting control means is controlled so that the infrared backlight is always lit. The sensor device according to claim 6, wherein: Lighting control means for performing lighting control of the infrared light backlight;
Imaging execution control means for performing execution control of image imaging by the photodiode sensitive to the infrared light,
The imaging execution control unit converts the infrared light into the infrared light at a timing when the infrared backlight is turned on when the lighting control unit is controlled to intermittently turn on the infrared light backlight. The sensor device according to claim 6, wherein imaging is performed by a sensitive photodiode.   The lighting control means controls the backlight of the infrared light so as to be lit for M frame periods (N and M are integers and N> M) in N frame periods. 9. The sensor device according to 8. Drive control means for performing drive control of the difference detection means and determination means,
The sensor device according to claim 6, wherein the drive control unit stops driving the difference detection unit and the determination unit when the infrared backlight is turned off.   The sensor device according to claim 1, wherein the first image and the second image are picked-up images picked up using all of the photodiodes sensitive to infrared light.   The sensor device according to claim 1, wherein the first image and the second image are picked-up images picked up using a part of a photodiode sensitive to infrared light.   13. The sensor device according to claim 12, wherein the first image and the second image are captured images using 1000 or more and 4096 or less of photodiodes sensitive to infrared light. .   The sensor device according to claim 1, wherein the difference detection unit detects an absolute value of a difference between pixel values of the first image and the second image as a difference.   The determination means counts pixels in which the absolute value of the difference between pixel values as the difference detected by the difference detection means exceeds a predetermined threshold, and the number of pixels is equal to or greater than a predetermined number. The sensor device according to claim 1, wherein it is determined that there is a change between the first image and the second image.   The determination means counts pixels whose absolute value of a difference between pixel values as a difference detected by the difference detection means exceeds a predetermined threshold, the number of pixels is equal to or greater than a first threshold, and a second 2. The sensor device according to claim 1, wherein when it is equal to or less than the threshold value, it is determined that there is a change between the first image and the second image. Furthermore, a drive control means for performing drive control of the difference detection means and the determination means;
Comprising illuminance measuring means for visible light,
When the measurement result by the illuminance measuring means is below a predetermined illuminance,
The lighting control means performs lighting control of the infrared light backlight,
The imaging execution control means performs execution control of imaging with a photodiode that is sensitive to the infrared light,
9. The sensor device according to claim 7, wherein the drive control unit performs drive control of the difference detection unit and the determination unit. The sensor device according to any one of claims 1 to 17,
While receiving the determination result by the determination means of the sensor device, the first image obtained by imaging with a photodiode sensitive to the infrared light of the sensor device, and the first image after an arbitrary time elapses A second image obtained by imaging with the same photodiode as the photodiode that is sensitive to the captured infrared light is received, and the received first image and second image are subjected to predetermined image processing and specified. An image processing device for acquiring the information of
When the specific information by the image processing device is not acquired for a certain time or more, the operation of the image processing is stopped,
When the determination result received from the sensor device is a result indicating that there is a change between the first image and the second image, if the image processing is stopped, the operation of the image processing is returned. A featured image processing system.   The image processing system according to claim 18, wherein the optical sensor liquid crystal panel in the sensor device captures the first image at a timing when the image processing operation of the image processing device stops.   19. The image processing system according to claim 18, wherein the specific information acquired by the image processing apparatus is position information within a screen of the optical sensor liquid crystal panel in the sensor apparatus.

Images