JP2008098989A - Solid-state imaging element - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a solid-state imaging element which shortens a detection time by attaining high speed processing without using a large-scale arithmetic circuit, and suppresses detection of an optical flow in other directions. <P>SOLUTION: The invention relates to the solid-state imaging element which receives light to detect the optical flow and also has an array of pixels receiving the light along a predetermined optical flow spreading in a predetermined direction, wherein the respective pixels are arrayed along the predetermined optical flow at predetermined intervals and have the same area, and longitudinal/lateral ratios are varied along the predetermined optical flow. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to a solid-state image sensor, and more particularly to a solid-state image sensor suitable for use in optical flow detection.

  In general, as a technique for detecting the movement of an object (the direction and speed of movement) using a solid-state imaging device, a technique of motion detection using an optical flow is used.

  Here, the optical flow can be detected by comparing an image currently captured by the solid-state imaging device with an image captured in the past.

  More specifically, the optical flow represents a motion of an object in a temporally continuous image as a vector. Specifically, a feature point at a certain time t is extracted, a position where the feature point moves after dt time after the time t is detected, and a moving direction and a moving distance of the feature point are vectorized. The flow data indicating the optical flow is obtained by extracting only the detected vector. When the observer moves straight, the optical flow moves so as to spread outside the vanishing point of the image.

Conventionally, when detecting the optical flow described above, a solid-state imaging device in which pixels are arranged in a two-dimensional matrix is used, and two or more of time t and time t + dt captured by the solid-state imaging device are used. A vector representing an optical flow is detected by matching feature points between a number of images.

  That is, conventionally, as disclosed in, for example, Japanese Patent Application Laid-Open No. 2006-134035 presented as Patent Document 1 and Japanese Patent Application Laid-Open Publication No. 2006-72439 presented as Patent Document 2, all images currently captured are disclosed. A vector representing an optical flow is detected by searching a matching point with a past image for each place or block.

However, according to the above-described conventional method, the amount of information to be processed increases, and the optical flow can be processed by various complicated image processing (edge detection, Hough transform, pattern matching, statistical method prediction, etc.) after the image is acquired. Since it is necessary to perform detection, there has been a problem in that a large-scale arithmetic circuit is used for detecting an optical flow, and that the detection takes a long time.

  In addition, according to the above-described conventional method, there is a problem that an optical flow in another direction is also detected, and it is necessary to remove such an optical flow in post-processing.

JP 2006-134035 A JP 2006-72439 A

  The present invention has been made in view of the above-described various problems of the prior art, and its object is to increase the processing speed without using a large-scale arithmetic circuit. An object of the present invention is to provide a solid-state imaging device capable of shortening the detection time and enabling detection of optical flows in other directions.

  In order to achieve the above object, the present invention is an array specialized for only detection of a predetermined optical flow in which pixels of a solid-state imaging device spread in a predetermined direction.

  Therefore, according to the present invention, since only the optical flow in a predetermined direction is detected, the amount of information to be processed is greatly reduced as compared with the conventional method, and the optical flow in the predetermined direction is detected quickly and reliably. Will be able to.

  By using such a solid-state imaging device according to the present invention, an optical flow of a moving object having a relative speed (hereinafter appropriately referred to as “moving object” in the present specification) can be detected at high speed.

  Therefore, if the direction of the pixel array of the solid-state image sensor according to the present invention is set appropriately and only the optical flow in a specific direction is detected, the solid-state image sensor according to the present invention has a relative speed, Detection of a moving object that approaches or moves away from the solid-state imaging device in a specific direction can be performed at high speed and reliably.

  The solid-state imaging device according to the present invention can be manufactured with a structure such as a CCD or a CMOS.

Hereinafter, in order to facilitate understanding of the present invention, the principle of the present invention will be described in detail.

  FIG. 1 is an image of the observer traveling direction when the observer is moving straight, and an optical flow (vector) when there is no moving object in the image is indicated by an arrow.

  As shown in FIG. 1, when there is no moving object in the video, the optical flow of the stopped object such as the background is generated so as to spread from the vanishing point.

  Here, if there is a moving object in this image, an optical flow in a direction different from the optical flow of the stopped object such as the background shown in FIG. 1 occurs, and the movement of the moving object is detected by detecting the vector. The direction and size can be detected.

  When an object approaches the observer while moving in the horizontal direction, the size of the object increases with time, so the optical flow has a spread as shown in FIG.

  Therefore, when the objective is to detect the optical flow of a moving object that is approaching while moving in the horizontal direction, the present invention detects only the predetermined optical flow spreading in the predetermined direction shown in FIG. The pixels of the solid-state image sensor according to the above may be arranged.

  In this way, the solid-state imaging device according to the present invention detects only the optical flow shown in FIG. 3 and does not detect other optical flows. Therefore, the amount of information to be processed is drastically reduced when compared with the conventional method, and the optical flow shown in FIG. 3 can be detected at high speed and reliably.

That is, according to the first aspect of the present invention, in a solid-state imaging device that receives light and detects optical flow, pixels that receive light are arranged along a predetermined optical flow that spreads in a predetermined direction. It is what I did.

  According to a second aspect of the present invention, in the first aspect of the present invention, the pixels are arranged at predetermined intervals along the predetermined optical flow. It is a thing.

  According to a third aspect of the present invention, in the invention according to any one of the first or second aspect of the present invention, each of the pixels arranged along the predetermined optical flow is The aspect ratio is changed along the predetermined optical flow with the same area.

  Since the present invention is configured as described above, the processing speed can be increased without using a large-scale arithmetic circuit, and the detection time can be shortened. Thus, it is possible to suppress the detection of the optical flow in the other direction.

  Hereinafter, an example of an embodiment of a solid-state imaging device according to the present invention will be described in detail with reference to the accompanying drawings.

Here, FIG. 4 shows a schematic configuration explanatory diagram showing the arrangement of pixels on the imaging surface of the solid-state imaging device according to the first embodiment of the present invention, and FIG. 5 shows the second embodiment of the present invention. FIG. 6 is a schematic diagram illustrating the arrangement of pixels on the imaging surface of the solid-state imaging device according to the embodiment. FIG. 6 shows the arrangement of pixels on the imaging surface of the solid-state imaging device according to the third embodiment of the present invention. A schematic configuration explanatory diagram is shown, and FIG. 7 is a schematic configuration explanatory diagram showing an arrangement of pixels on an imaging surface of a solid-state imaging device according to a fourth embodiment of the present invention.

  These solid-state imaging device 10 according to the first embodiment of the present invention, the solid-state imaging device 20 according to the second embodiment of the present invention, the solid-state imaging device 30 according to the third embodiment of the present invention, and the first of the present invention. Each of the solid-state imaging devices 40 according to the fourth embodiment is a solid-state imaging device specialized for detecting only an optical flow in a specific direction shown in FIG.

  That is, the solid-state imaging devices 10, 20, 30, and 40 proceed from the left end to the right end of the image captured by the solid-state imaging devices 10, 20, 30, and 40 and spread in the vertical direction of the image. This is for detecting an optical flow.

First, the solid-state imaging device 10 will be described. The pixels 10a of the solid-state imaging device 10 are densely arranged along a predetermined optical flow that spreads in a predetermined direction shown in FIG. Each pixel 10a is arranged with a slight shift.

  In the above configuration, according to the solid-state imaging device 10, since the pixels 10a are arranged along the optical flow shown in FIG. 3, only the optical flow shown in FIG. 3 is detected by each pixel 10a, and the other opticals are detected. No flow is detected. For this reason, the amount of information to be processed is greatly reduced when compared with a conventional solid-state imaging device, and the optical flow shown in FIG. 3 can be detected at high speed and reliably.

Next, the solid-state imaging device 20 will be described. The solid-state imaging device 20 has a predetermined arrangement without densely arranging the pixels 20a in order to make it easier to detect the optical flow shown in FIG. They are arranged along the optical flow shown in FIG. 3 at intervals.

  Therefore, the solid-state imaging device 20 can be specialized by detecting the optical flow shown in FIG.

  In the solid-state imaging device 20, it is difficult to detect an optical flow that is slightly deviated from the arrangement of the pixels 20a.

Next, the solid-state image sensor 30 will be described. In the solid-state image sensor 30, the shape of each pixel 30a is different, but the area of each pixel 30a is set to be the same. Each will have the same number of photons.

  Further, since the pixel 30 a becomes a vertically long pixel as it goes to the right side of the solid-state imaging device 30 in FIG. 6, the tolerance of optical flow detection is higher than that of the solid-state imaging device 20.

Next, the solid-state image sensor 40 will be described. In the solid-state image sensor 40, the area of each pixel 40a is set to be the same as in the solid-state image sensor 30, but the arrangement of the pixels 40a is the same. Is different.

  The arrangement of the pixels 40a in the solid-state imaging device 40 is an example in which the pixels 40a are arranged in consideration of image distortion when light collected using a fisheye lens or the like is received.

Although not shown, the solid-state imaging device according to the present invention may be configured by arranging pixels so as to extend not only in the horizontal direction but also in the vertical direction.

FIG. 8 shows a block configuration diagram of a sensor system in which the solid-state image sensor according to the present invention shown as the solid-state image sensor 10, the solid-state image sensor 20, the solid-state image sensor 30 or the solid-state image sensor 40 is configured as a flow detection camera. ing. The sensor system 100 is mounted on a vehicle and used when detecting an object in relation to the vehicle.

This sensor system 100 includes a flow detection camera 102 using a solid-state image sensor according to the present invention, and an arithmetic circuit 104 that inputs flow data and video data generated and output by the solid-state image sensor constituting the flow detection camera 102. And a signal circuit 106 that outputs a control signal to the flow detection camera 102.

In the above configuration, in the sensor system 100, the flow detection camera 102 using the solid-state imaging device according to the present invention detects an optical flow from an input video by receiving light, and the flow data and video data are detected. Output to the arithmetic circuit 104.

  The arithmetic circuit 104 processes the flow data and the video data, calculates the arrival time of the object to the host vehicle on which the sensor system 100 is mounted and the possibility of collision, and outputs it as a signal for the subsequent control processing. Note that the control processing includes brake control and warning to the driver.

  Further, by inputting the information of the own vehicle as a control signal of the flow detection camera 102 by the signal circuit 106, it is possible to control as information that there is no collision from the own vehicle if the own vehicle is stopped.

FIG. 9 and subsequent figures show operation examples when the sensor system 100 including the solid-state imaging device according to the present invention is mounted on an automobile.

  Here, when the front of the host vehicle is monitored as an application for monitoring the front of the automobile, if there is no moving object in front of the host vehicle while the host vehicle is traveling, the optical flow is as shown in FIG.

  Therefore, the pixels of the solid-state imaging device according to the present invention used in the sensor system 100 mounted on an automobile are arranged so that the optical flow shown in FIG. 9 can be detected. The optical flow shown in FIG. 9 is such that the vanishing point located at the center of the image to be captured is sandwiched between the respective side edges of the image to be imaged, toward the center, and the image in the vertical direction toward the center. It becomes an optical flow that advances as it spreads.

  Thereby, the optical flow of the stopped object such as the background that can be seen from the moving object is not detected, and only the object that moves so as to hinder the moving direction of the host vehicle can be detected.

  Note that a moving object on the vanishing point is detected as a vector in the same direction as the stop object, and therefore cannot be detected at this angle of view. The pixel arrangement of the solid-state imaging device according to the present invention for detecting the optical flow shown in FIG. 9 is intended only to detect an object that may be an obstacle in the traveling direction of the host vehicle. is there.

Here, FIG. 10 is an explanatory diagram when the relationship between the traveling state of the host vehicle 200 and the other vehicle 202 ahead is viewed from above.

  In FIG. 10, the host vehicle 200 moves upward, and the other vehicle 202 ahead moves from right to left.

  Note that the flow detection camera 102 of the sensor system 100 mounted on the host vehicle 200 is attached to the headlamp, the bumper, the vehicle interior, the side mirror, or the like of the host vehicle 200 in order to detect the optical flow shown in FIG. It is assumed that the solid-state imaging device according to the present invention having an arrangement of the following pixels is used.

  At this time, images captured by the flow detection camera 102 mounted on the host vehicle 200 are as shown in FIGS. Note that FIG. 12 shows an image that has passed dt time from the image of FIG. 11 taken at a certain time t.

  First, when both the host vehicle 200 and the other vehicle 202 in front are moving, the size and position of the other vehicle 202 in front change as shown in FIGS.

  Accordingly, the optical flow shown in FIG. 13 is the result of detecting the optical flow from the other vehicle 202 ahead and the feature points of the road surface using FIG. 11 and FIG.

  For the other vehicle 202 in front of the movement, the movement distance from the right to the left and the change in size due to the approach of the host vehicle 200 are detected as an optical flow.

  Since the road surface marker has a downward optical flow, the flow detection camera 102 does not detect it.

  Therefore, the detection result of the optical flow by the flow detection camera 102 is as shown in FIG.

  Further, even when the host vehicle 200 described above is run in an urban area, the optical flow of the person 204 moving in the traveling direction of the host vehicle is detected as shown in FIG. 15, but the optical flow of the stopped person 206 is detected. Will not be.

  Furthermore, the image when the host vehicle 200 is stopped changes as shown in FIGS. For this reason, the optical flow is an optical flow only in the horizontal direction as shown in FIG. 18, and therefore this optical flow is not detected by the flow detection camera 102.

Next, the case of monitoring the rear from the host vehicle 200 will be considered. In this case, the flow detection camera 102 of the sensor system 100 mounted on the host vehicle 200 is used in the tail lamp, the bumper, or the passenger compartment of the host vehicle 200. It shall be attached to etc.

  Further, when the flow detection camera 102 is attached to the rear of the host vehicle 200, the direction of the optical flow detected by the solid-state imaging device according to the present invention is set as shown in FIG. Therefore, the pixels of the solid-state imaging device according to the present invention used for this are also arranged along the optical flow shown in FIG. 19 so as to detect only the optical flow shown in FIG.

  Here, FIG. 20 and FIG. 21 are images of the other vehicle 208 behind traveling toward the host vehicle 200, and FIG. 21 is an image after dt time from FIG. The other vehicle 208 behind becomes gradually larger as it approaches the host vehicle 200.

  The result of detecting the optical flow from the images of FIGS. 20 and 21 is shown in FIG. Looking at the optical flow detected in FIG. 22, the other vehicle 208 behind spreads out to the lower part of the image.

  On the other hand, since the street light 210 flows toward the vanishing point, it is not detected by the solid-state imaging device according to the present invention of pixels arranged along the optical flow shown in FIG.

  Therefore, the optical flow detected by the solid-state imaging device according to the present invention of the pixels arranged along the optical flow shown in FIG. 19 is as shown in FIG.

  From the detected optical flow (vector), the other vehicle 208 in the rear that may come to the own vehicle 200 is detected, the danger of the collision to the other vehicle 200 in the rear is notified, and the collision avoidance of the own vehicle 200 is avoided. Can encourage action.

Next, a case where the solid-state imaging device according to the present invention is used in a crime prevention system will be described with reference to FIGS.

  24, 25, 27, and 28 show an example in which a solid-state imaging device according to the present invention is attached to a security monitoring camera of a security system and images are taken.

  Here, FIGS. 24 and 25 show the case where the image is taken from the horizontal direction, and FIGS. 27 and 28 show the case where the image is taken from the upper direction.

  Even when the solid-state imaging device according to the present invention is used in a crime prevention system, the solid-state imaging device according to the present invention of pixels arranged along the optical flow is used in accordance with the optical flow to be detected.

  For example, in the case shown in FIG. 24, an optical flow from left to right can be detected, and in the case shown in FIG. 27, an optical flow from top to bottom can be detected.

  25 is an image after dt time has elapsed since imaging of FIG. 24, and FIG. 28 is an image after dt time has elapsed since imaging of FIG. 27.

  When there are a person 212 and a person 214 coming to the security surveillance camera, the position and size change as the security surveillance camera is approached, so that an optical flow as shown in FIGS. It is possible to detect visitors and visitors.

The embodiment described above can be modified as shown in the following (1) to (3).

  (1) The pixel arrangement of the solid-state imaging device according to the present invention shown in the above-described embodiment is merely an example for explanation, and it goes without saying that an arbitrary arrangement may be selected according to the application. is there.

  (2) In the above-described embodiment, the case where the solid-state imaging device according to the present invention is used in an automobile or a security system has been shown. However, the application of the solid-state imaging device according to the present invention is not limited to these. For example, it can be used for a sensor of a robot.

  (3) You may make it combine the above-mentioned embodiment and the modification shown in above-mentioned (1) thru | or (2) suitably.

  INDUSTRIAL APPLICABILITY The present invention can be used in various fields such as automobile periphery monitoring and collision avoidance control, robot motion control, or intruder detection in a crime prevention system and motion detection in a parking lot. It is.

FIG. 1 is an explanatory diagram showing an image of the observer traveling direction when the observer is moving straight ahead. FIG. 2 is an explanatory diagram showing an optical flow when an object approaches the observer while moving in the horizontal direction. FIG. 3 is an explanatory diagram showing an optical flow of a moving object approaching while moving in the horizontal direction. FIG. 4 is a schematic configuration explanatory diagram showing the arrangement of pixels on the imaging surface of the solid-state imaging device according to the first embodiment of the present invention, which is specialized for detecting only the optical flow in a specific direction shown in FIG. . FIG. 5 is an explanatory diagram of a schematic configuration showing an arrangement of pixels on the imaging surface of the solid-state imaging device according to the second embodiment of the present invention specialized for detecting only the optical flow in a specific direction shown in FIG. . FIG. 6 is an explanatory diagram of a schematic configuration showing an arrangement of pixels on the imaging surface of the solid-state imaging device according to the third embodiment of the present invention specialized for detecting only the optical flow in a specific direction shown in FIG. . FIG. 7 is a schematic configuration explanatory diagram showing the arrangement of pixels on the imaging surface of the solid-state imaging device according to the fourth embodiment of the present invention, which is specialized for detecting only the optical flow in a specific direction shown in FIG. . FIG. 8 is a block diagram of a sensor system in which the solid-state imaging device according to the present invention is configured as a flow detection camera. FIG. 9 is an explanatory diagram showing an optical flow that should be detected by the solid-state imaging device according to the present invention used in a sensor system mounted as a vehicle front monitoring application. FIG. 10 is an explanatory diagram for explaining the operation of the solid-state imaging device according to the present invention. FIG. 11 is an explanatory diagram illustrating an image captured by a flow detection camera mounted in front of the host vehicle when the host vehicle is running. FIG. 12 is an explanatory diagram illustrating an image captured by a flow detection camera mounted in front of the host vehicle when the host vehicle is running. FIG. 13 is an explanatory diagram showing an optical flow detected from feature points of other vehicles ahead and road surfaces using FIG. 11 and FIG. 12. FIG. 14 is an explanatory diagram showing an optical flow detected by the solid-state imaging device according to the present invention. FIG. 15 is an explanatory diagram showing an optical flow detected from an image when the host vehicle travels in an urban area. FIG. 16 is an explanatory diagram illustrating an image captured by a flow detection camera mounted in front of the host vehicle when the host vehicle is stopped. FIG. 17 is an explanatory diagram illustrating an image captured by a flow detection camera mounted in front of the host vehicle when the host vehicle is stopped. FIG. 18 is an explanatory diagram showing an optical flow detected from the images in FIGS. 16 and 17. FIG. 19 is an explanatory diagram showing an optical flow to be detected by the solid-state imaging device according to the present invention used in a sensor system mounted as a vehicle rear monitoring application. FIG. 20 is an explanatory diagram showing an image of the other vehicle behind traveling toward the host vehicle. FIG. 21 is an explanatory diagram showing an image of the other vehicle behind traveling toward the host vehicle. FIG. 22 is an explanatory diagram showing an optical flow detected from the images of FIGS. FIG. 23 is an explanatory diagram showing an optical flow detected by the solid-state imaging device according to the present invention. FIG. 24 is an explanatory diagram showing an image captured by attaching the solid-state imaging device according to the present invention to the security monitoring camera of the security system. FIG. 25 is an explanatory diagram showing an image captured by attaching the solid-state imaging device according to the present invention to the security monitoring camera of the security system. FIG. 26 is an explanatory diagram showing an optical flow detected from the images in FIGS. 24 and 25. FIG. 27 is an explanatory diagram showing an image captured by attaching the solid-state imaging device according to the present invention to the security monitoring camera of the security system. FIG. 28 is an explanatory diagram showing an image captured by attaching the solid-state imaging device according to the present invention to the security monitoring camera of the security system. FIG. 29 is an explanatory diagram showing an optical flow detected from the images shown in FIGS. 27 and 28.

Explanation of symbols

10, 20, 30, 40 Solid-state imaging device 10a, 20a, 30a, 40a Pixel 100 Sensor system 102 Flow detection camera 104 Arithmetic circuit 106 Signal circuit

Claims (3)

In a solid-state image sensor that receives light and detects optical flow,
A solid-state imaging device, wherein pixels that receive light are arranged along a predetermined optical flow that spreads in a predetermined direction. The solid-state imaging device according to claim 1,
The pixels are arranged at predetermined intervals along the predetermined optical flow. A solid-state imaging device, wherein the pixels are arranged at predetermined intervals. In the solid-state image sensor according to any one of claims 1 and 2,
Each of the pixels arranged along the predetermined optical flow has the same area and has an aspect ratio changed along the predetermined optical flow.