JP2022029717A - Imaging device - Google Patents

Abstract

To provide an imaging device that is an imaging camera or a solid-state imaging device which performs information output having little information amounts concerning a moving object, without outputting imaging information to the outside.SOLUTION: An imaging camera 1 generates a differential signal for pixel signals at different imaging times which are outputted from a solid-state imaging device 3; makes a movement-contour extracting part 4 extract information on a contour of a moving object on the basis of the differential signal; makes a movement-information extracting part 5 extract information on the moving object (a size, a framework point and a feature point) and movement information (a direction and a speed of movement) on the basis of the information on the contour; and makes an information output part 6 unite and output the movement information.EFFECT: The imaging camera can reduce considerable information amounts in comparison with image information, by outputting numerical information concerning movements that are not linked to pixel positions, unlike a standard imaging camera and a standard solid-state imaging device which output images linked to pixel positions. This can reduce calculation loads, so that calculation can be completed in parallel with reading-out of a pixel scanning line and the solid-state imaging device 3 can output the image directly.SELECTED DRAWING: Figure 1

Description

The present invention relates to an imaging device that extracts and outputs image information such as the size, direction, speed, and position of a moving object acquired by imaging.

In a solid-state image sensor or an image pickup device such as an image pickup camera using the same, it is common to output about 30 frame images per second in order to display an image. A frame image is an image captured and is displayed in time series on a display device such as a liquid crystal panel. What is a solid-state image sensor? Charges generated by photoelectric conversion units (hereinafter referred to as pixels) arranged in an area like CMOS area sensors and CCD area sensors are converted to voltage by the output circuit unit and either as they are or as color signals. It is processed and output from the solid-state image sensor in chronological order.

The image information output from the solid-state image pickup device is image-processed in the image pickup camera, and various image corrections such as camera shake correction, color correction, exposure correction, and contour enhancement are performed, and the image signal is output from the image pickup camera. .. In addition to simply outputting image signals, high-performance imaging cameras with additional functions such as motion contour display, motion detection, motion tracking, and automatic zoom are also provided for the imaging camera.

The image output from the solid-state image sensor and the image output from the image pickup camera are displayed on the display device or recorded on the recording device. The recorded image information is displayed on the display device as needed. As a display device, the number of vertical and horizontal display pixels is fixed like a TV monitor or a liquid crystal monitor. The image output from the solid-state image sensor or the image sensor is displayed corresponding to the pixels of this display device. The images output from the solid-state image sensor and the image pickup camera are output in chronological order along the horizontal scanning lines, and are displayed corresponding to the pixels at a predetermined position corresponding to the scanning lines of the display device.

In the video signal output from an image pickup device such as a solid-state image pickup device or an image pickup camera, video information is arranged in chronological order along the horizontal scan lines, and a horizontal blanking period is provided between adjacent horizontal scan lines. A vertical blanking period is provided between the end of the scanning line of a frame and the start of the scanning line of the next frame. The video signal includes a synchronization signal, and by displaying the video using this synchronization signal on the display side, the video synchronized with the camera can be displayed without distortion. Even when recording on a recording device, the image is normally displayed on the display device based on this synchronization signal. Since the image corresponding to the pixel position of the solid-state image sensor is displayed as the reproduced image corresponding to the pixel position of the display device, in the following description, for convenience in order to clarify the difference from the present application, such a conventional image pickup device is used. The image of is expressed as a pixel position link image.

In a color solid-state imager, there are cases where the pixels of the three primary colors red (R), green (G), and blue (B) are output as they are, and there are cases where a brightness signal and a color signal are output. A color signal is output corresponding to the position of the block (2 × 2 pixels in the Bayer arrangement), and the image corresponds to the pixel position. Even in this case, it is called a pixel position link image. Even in the case of external synchronization in which the synchronization signal generated externally is input to the camera and the timing of the camera is adjusted to this without using the synchronization signal held inside the camera, it is also expressed as a pixel position link image.

The solid-state image sensor not only captures an image, but may also detect the movement of a subject on the image pickup screen of the solid-state image sensor. As a method for detecting the movement of an image on such an image pickup screen, a method of detecting the movement by comparing the signal outputs of the same pixel units between the previous and next frames is common, and is shown in Patent Document 1. As described above, there is a method in which adjacent pixels are paired, the integration time is changed, the gain is adjusted, and the motion is detected by the difference signal output. Further, as shown in Patent Document 2, there is also a method in which adjacent pixels having different sensitivities are set as a pair, the integration time is changed, and the motion is detected by the difference signal output.

In the latter patent document 2, by directly outputting a difference signal for determining motion detection and outputting only contour information of a moving object (hereinafter referred to as a moving object), the step of extracting the contour of the moving object from the image is omitted, and Human-Machine is used. -It has been shown to have applications to Interface.
Even in the contour information of this moving object, a difference in the amount of charge photoelectrically converted according to the change in motion occurs in the pixel, and the difference is output as contour information corresponding to the pixel position, so this is also a pixel position link image. Express.

Patent Document 3 describes the concept of reducing the power consumption of the system by controlling the power ON / OFF of the subsequent system based on the result of motion detection, and controlling the image signal output from the solid-state imaging device for information. The concept of reducing the amount is shown. In the motion detection methods of Patent Documents 2 and 3, a case where motion detection is possible by pixel difference without using a frame memory is shown. The motion contour image in this case is also a pixel position link image.
When motion detection is performed using the frame memory, a contour image is acquired from the pixel output difference between the corresponding positions of the current pixel output and the pixel output of the previous frame. Even in this case, the contour image of the motion can be expressed as a pixel position link image because it is obtained corresponding to the pixel position where there is a change between the frames.

Patent Document 4 discloses a contour curve and a motion vector estimation method of a moving body existing in a moving image, and a method of acquiring feature points on the contour curve. Further, in Patent Document 5, feature points representing a predetermined feature are extracted from an image of one frame taken by a photographing device, and feature points are extracted from a region near the feature points of an image of a later frame in time. , A method of extracting a motion vector from both feature points with a smaller amount of calculation is shown.

Patent Document 4 and Patent Document 5 show block diagrams and flowcharts in which an image output from a camera is captured as an input signal, and then motion contour information, motion vector information, and feature point information are acquired by an image processing device. .. In both cases, a pixel position link image is output from the camera, and image signal processing is performed based on this image information. Specifically, in Patent Document 4, the contour image is a pixel position link image, and the feature point is the maximum point of the bending angle of the minute section acquired along the contour image. Therefore, the feature point is also the pixel position link image. be. Further, also in Patent Document 5, the feature points are the intersections of the pattern edges and the points on the curve where the curvature is maximized, and the feature points are also pixel position link images.
Other application examples of extracting required information by the image processing block also use the image information from the camera as an input, so that the image is similarly linked to the pixel position. That is, the calculation processing is performed based on the pixel position-linked image output from the image pickup device (solid-state image pickup device or image pickup camera), and the calculation result is also displayed as the pixel position-linked image.

Patent No. 3521109 Patent No. 5646421 Patent No. 5604700 Patent No. 3674084 Patent No. 6624841

In the conventional image information processing method, image information linked to pixel positions is output from a solid-state image pickup device or an image pickup camera, and this image information is used as an input in an image processing block in the subsequent stage to perform motion contour information and motion vector information. , The feature point information was calculated and output corresponding to the pixel position.

The amount of information in a pixel position-linked image (normal camera output image) is based on the amount of information determined by the number of pixels of the solid-state imaging device x the amount of information per pixel. Normally, the number of pixels is 2 million to 8 million pixels, and the amount of information per pixel is 8 bits = 1 byte, and the amount of information is simply 2 to 8 megabytes per frame. The amount of information on a still image on a smartphone is several megabytes / screen. Although the resolution is reduced for smartphone videos, it is still several hundred megabytes / minute. In this way, when trying to calculate a moving image in real time with the image processing block in the subsequent stage, ultra-high-speed calculation processing is required, and the power consumption is about several tens of watts. For this reason, battery drive was not practical, and the system was also large.

Patent Document 2 discloses a solid-state image pickup device that outputs a contour image of a moving object from the difference between pairs of pixels having different photoelectric conversion periods. Even if the number of pixels is 2 million pixels and the contour image (1 million pixels) with the difference is binarized, the amount of information per frame of the contour image is 1 bit / pixel x 1 million pixels / 8 bits = 125 kilobytes. Become. This binarized contour image information is also a pixel position link image because it is a difference output of each pixel pair.

Since the power consumption for calculating moving images in real time in the image processing block in the subsequent stage is large, there are restrictions on the installation location, and even if the camera that acquires the image linked to the pixel position is battery-powered and the restrictions on the installation location are removed, the camera It was necessary to connect the image processing block with a cable or send the image information via Wifi. In the former case, it was necessary to reduce the amount of information in consideration of the cable length and in the latter case, the transmission capacity of Wifi.

An object of the present invention is to directly output image information (shape, direction of movement, speed, position) of a moving object that is not linked to pixel positions from a solid-state imaging device or an image pickup camera, which is an image pickup device, so that information to be processed in a subsequent stage can be obtained. It is an object of the present invention to provide an image pickup apparatus in which the amount is significantly reduced and the calculation load is significantly reduced.

In order to solve the above problems, the solid-state image sensor or the image pickup camera outputs the necessary image information of the moving object by linking it with the necessary time. The necessary image information of the moving body is information on the shape, the direction of movement, the speed, and the position of the moving body. The required time is the time information to which the required image information should be linked. Since the necessary image information is not directly linked to the pixel position of the solid-state image sensor, it is called a pixel position non-link image and is expressed separately from the pixel position-linked image from a conventional image sensor (solid-state image sensor or image sensor). ..

The expressions of pixel position linked image and pixel position non-linked image are not general expressions, and a little more explanation will be added for the first use this time.
As a simple example, a 100 × 100 pixel solid-state image sensor will be described. From this solid-state image sensor, image information of 10,000 pixels corresponding to one frame is output in chronological order. There is a synchronization pulse at the beginning of the 1-frame image, and the amount of time delay from the synchronization pulse indicates which pixel signal is output. The same applies to an image pickup camera using this solid-state image sensor, and the amount of time delay from the synchronization pulse indicates which position the image signal of the pixel is output. In this way, in the conventional image pickup device (solid-state image pickup device or image pickup camera), the image information corresponding to the pixel position is linked in chronological order. This is called a pixel position link image.

On the other hand, in the present invention, attention is paid to a moving body, and the position and shape of the moving body, the direction of movement, and the velocity are the output information of the moving body. The position of the moving body is indicated by the representative point of the moving body, which does not correspond to the pixel position but is the coordinate information of X and Y when displaying the representative point. In addition, the size, direction, and speed of the moving object are numerical data and are not linked to the pixel position. Also, since the time information only needs to know the number of frame images and does not correspond to the pixel position, the amount of time delay from the synchronization pulse is meaningless. That is, it suffices if the numerical data of the above items of each moving object is output. This is called a pixel position non-link image. The expression "image" was used for comparison with the conventional example, but the numerical information obtained based on the image of the moving object is used for implementation.

Among the necessary image information of the moving body, the information on the shape and position of the moving body will be further explained. If the contour information is used as the shape of the moving body, the pixel position link image is obtained and the amount of information remains large. In order to solve this problem, the shape of the moving body is not accurately expressed but is roughly expressed by the skeleton. In order to acquire this skeleton information, a small number of representative points inside the moving body (referred to as skeletal points in the following description) are extracted based on the contour of the moving body, and a line group connecting the skeletal points (referred to as a skeletal pattern). ) Expresses the shape of the moving object. The representative point referred to here is a skeleton point representing a skeleton pattern such as an end portion, a bending portion, a branch portion, and a central portion of the skeleton pattern. Further, among the skeleton points of one moving body, the points representing the movement of the moving body are called feature points, and the representative position of the moving body is expressed by the coordinates of the feature points. The direction and speed of movement of the moving object shall be the direction and speed of movement of this feature point.

As described above, in the present proposal, among the captured images, the still images are not dealt with, but only the moving objects are dealt with. Furthermore, the amount of information is greatly reduced by expressing the shape of the moving object with a skeleton pattern (multiple skeleton points) and handling the moving object in the coordinates, velocity, and direction of the feature points. At this time, the skeleton points and feature points constituting the skeleton pattern are calculated and obtained based on the motion contour image, and are pixel position non-link images. If the skeleton pattern and the feature points are not displayed as an image and are treated by the coordinates of the skeleton points and the coordinates of the feature points constituting the skeleton pattern, the numerical information is obtained by the respective X and Y coordinates. Even if the shape of the moving body is expressed by the size instead of the skeleton pattern, the velocity and direction of the moving body are also numerical information attached to the feature points. That is, the necessary image information of the moving body is numerical information, and hereinafter referred to as numerical information of the moving body and pixel position non-link numerical information.

Conventional image pickup devices (solid-state image pickup devices, image pickup cameras) output light / dark information (brightness signal) corresponding to pixel positions and color information (color signals), that is, image information linked to pixel positions. On the other hand, in the image pickup device (solid-state image pickup device, image pickup camera) of the present proposal, only numerical information of the coordinates of the skeleton point of the moving object, the coordinates of the feature point, the direction of movement of the feature point, and the speed is output without corresponding to the pixel position. .. This is pixel position non-link numerical information.

In the image pickup device (solid-state image sensor, image pickup camera) of the present proposal, when displaying an image of a moving body skeleton pattern and feature points, a skeleton pattern is displayed on the display device by using a synchronization signal matching the video signal of the solid-state image pickup device. , It is easy to calculate and display the display position of the feature points from the coordinates. The size, direction of movement, and speed information of the moving object are displayed by superimposing a line segment and an arrow on the position of the displayed feature point, the size is indicated by the line segment length in the X and Y directions, and the direction and length are indicated by the direction of the arrow. Now just display the speed. This is convenient when moving objects appear frequently.

When the frequency of appearance of moving objects is low, when the pixel position non-link numerical information with zero output is output for each synchronization signal, information and power consumption are generated, albeit small, so the image sensor of the present proposal (solid-state image sensor, image pickup) It is good to output the pixel position non-link numerical information of the moving object only when there is a moving object from the camera). The information output here is numerical information of the X and Y coordinates of the skeleton points constituting the skeleton pattern of the moving body, the X and Y coordinates of the feature points, and the size, direction, and velocity of the moving body attached to the feature points. As the time information instead of the synchronization signal, the frame time information (numerical information) in which the numerical data of the moving object is generated is given by the time stamper.
The information sampling interval is determined from the speed of the moving object in the imaging field of view and the moving distance of the moving object.

To explain the method of time information described above a little more, there are two types of methods for linking the time required for pixel position non-link numerical information, and the first is to use the synchronization signal of a normal solid-state image sensor or image sensor. A method of linking with the numerical information of a moving object and outputting it, and displaying the numerical information of the moving object on a display device based on this synchronization signal. The second method is to add a time stamper when outputting numerical information of a moving object from a solid-state image sensor or an image pickup camera, and link the time information and the numerical information of the moving object to output as information.
In the former case, the synchronous pulse is continuously output even during the period when there is no moving object on the imaging screen, but the numerical information of the moving object is not output. As in Patent Document 3, if a function is added to control the ON / OFF of the output from the image sensor and the ON / OFF control of the subsequent circuit based on the motion detection, solid-state imaging is performed during the period when there is no moving object on the image pickup screen. No synchronization signal or necessary image information is output from the device or image sensor, and the amount of information and power consumption can be suppressed.
In the latter case, no information is output from the solid-state image sensor or the image pickup camera during the period when there is no moving object on the image pickup screen. During the period when there is a moving object, the time information and the numerical information of the moving object are output as a set from the solid-state image sensor or the image pickup camera.

To further explain the skeletal pattern and characteristic points of moving objects, cases where moving objects move independently; for example, traffic volume surveys on roads (cars, bicycles, pedestrians as moving objects) and applications to Human-Machine-Interface. In the case where a part of the body (arm) moves, the method of defining the feature points is different. In the former case where the moving body moves independently, the central point of the moving body may be used as a feature point, and it is common that the moving body moves in the same direction and at the same speed at any place of the moving body. In the latter case where a part of the body moves, the speed (movement amount per unit time) differs between the tip part (palm), middle part (elbow), and base part (shoulder) of the arm. In the application to Human-Machine-Interface, the skeletal point at the tip of the arm with the highest velocity is the feature point.

According to the present invention, a significant reduction in the amount of information and a significant reduction in power consumption are realized at the same time. Furthermore, by performing the calculation of the numerical information of these moving objects in parallel with the pixel output along the scanning line, the calculation load can be significantly reduced.

The effect of the present invention will be described quantitatively as compared with the prior art. A conventional example will be described using a solid-state image pickup device that outputs a contour image of a moving object from the difference between paired pixels having different photoelectric conversion periods shown in Patent Document 2 described above. This is the comparison target because it is a solid-state image sensor having a configuration that can reduce the amount of image information even in the conventional example.
If the number of pixels is 2 million pixels and the amount of information per pixel is 1 byte in the case of normal mode output, the amount of information is 2 megabytes per frame. When outputting the contour image of a moving object from the pair pixel difference, it is binarized to 125 kilobytes per frame, and the amount of information is 1/20. Both are pixel position link images.

On the other hand, in the case of the image pickup device that outputs the pixel position non-link image information according to the present invention, the numerical information of the moving body is the X and Y coordinates of the skeleton points constituting the skeleton pattern of the moving body, the X and Y coordinates of the feature points, and the feature points. Numerical information on the size, direction, and speed of the moving object associated with, and time information. The amount of this information depends on the number of moving objects and the number of skeletal points in the skeletal pattern, but for simplification, the case of one moving object and one skeletal point (in this case, this is a feature point) is described as follows on a numerical basis. Estimate to. In the traffic volume survey, there is only one skeletal point of the moving body, which is a characteristic point.

The number of pixels is 1 million pixels after the difference, and the aspect ratio of the display is 16: 9, which is 1330 x 750 pixels. Is 10 bits (1024) x 9 bits (512), and the amount of information is 19 bits / point. If the size of the moving body is covered by 5bit (32) for people (0.5m), bicycles (2m), private cars (4m), and buses (16m), and the direction of the moving body is 16 directions (22.5 ° intervals), it will be 4bit. Cover. The speed of moving objects is covered by 6bit (64) for humans (2-6km / hr.) And private cars (120km / hr.). As for the time to be time stamped, assuming that the numerical data acquisition interval of the moving object is 1 second or more and the data transmission is performed once / hour, 3600 seconds is covered by 12 bits (4096). Assuming that the feature point is 1p and the skeleton point is 2p, the amount of information of one moving object is 19 (coordinates) x 3 + 5 (size) + 4 (direction) + 6 (velocity) + 12 (time) = 84 bits = 10.5 bytes. The above-mentioned conventional 1-frame image information amount of 2 megabytes is 3 × 10 ^-6 , and the contour image information amount is 125 kilobytes of 4.8 × 10 ^-5 . 6 orders of magnitude smaller. This is a great merit when the image information output from the image pickup apparatus is processed by the post-stage processing circuit. Time information is eliminated by time stamping on the system based on the time output from the image pickup device without time stamping the information, and the resolutions of the X and Y coordinate points are set to 6bit (64) and 5bit (32), respectively. By reducing the number, the direction is 1 bit (2), the speed is 2 bit (4), and the size is 2 bit (4), the total is 16 bits, which can be suppressed to 2 bytes of information. Furthermore, if the number of vehicles in the traffic volume survey is 16 and the cover is covered, the amount of information is 4 bits.

Since the conventional image pickup camera performs various processing in the post-stage processing circuit, the amount of image information is large because the application field for outputting and processing the pixel position link image is general-purpose. On the other hand, in the present invention, since the amount of information is limited to a specific purpose (movement in the above description), the amount of information is not versatile, but the amount of information output from the image pickup apparatus can be significantly reduced.

FIG. 1 is a block diagram showing a configuration of a motion information extraction / output system according to the first embodiment of the present invention. FIG. 2 is a block diagram showing a schematic configuration of the motion contour extraction unit of FIG. FIG. 3 is a block diagram showing a schematic configuration of the motion information extraction unit of FIG. FIG. 4 is a block diagram showing a schematic configuration of the information output unit of FIG. FIG. 5 is a diagram illustrating an example of information output from the information output unit of FIG. 4 and a premise. FIG. 6 is a block diagram showing a schematic configuration of a pixel difference processing unit and a motion contour determination unit in FIG. 2. FIG. 7 is an explanatory diagram of a process of acquiring motion contour information and pixel difference code information in the block of FIG. FIG. 8 is a block diagram showing a schematic configuration of a pixel difference processing unit and a motion contour determination unit, which are different methods from those in FIG. FIG. 9 is a diagram illustrating a pixel configuration, a pixel output characteristic, and a separation circuit configuration of the solid-state image sensor of FIG. FIG. 10 is an explanatory diagram of another pixel configuration, pixel output characteristics, and separation circuit configuration of the solid-state image pickup device of FIG. FIG. 11 is a process explanatory diagram for acquiring motion contour information and pixel difference code information in the block of FIG. FIG. 12 is a block diagram of a moving body direction determination unit and an explanatory diagram of a process for acquiring moving body direction information. FIG. 13 is a block diagram of a moving body speed determination unit and an explanatory diagram of a process for acquiring moving body speed information. FIG. 14 is a block diagram of a moving body size determination unit and an explanatory diagram of a process for acquiring moving body size information. FIG. 15 is a block diagram of a moving body skeleton point determination unit and an explanatory diagram of a process for acquiring moving body skeleton point information. FIG. 16 is a block diagram of a moving body feature point determination unit and an explanatory diagram of a process for acquiring moving body feature point information. FIG. 17 is a block diagram of a moving body skeleton point determination unit of another method and an explanatory diagram of a skeleton point information acquisition method. FIG. 18 is a block diagram of a moving body feature point determination unit of another method and an explanatory diagram of a feature point information acquisition method. FIG. 19 is an explanatory diagram of a moving body skeleton point acquisition method and a feature point acquisition method in another case. FIG. 20 is a block diagram showing a configuration of a motion information extraction / output system according to a second embodiment of the present invention. FIG. 21 is a block diagram showing a schematic configuration of a solid-state image sensor according to a second embodiment of the present invention. FIG. 22 is a block diagram showing a schematic configuration of another solid-state image sensor according to the second embodiment of the present invention. FIG. 23 is a block diagram of a modified example of the schematic configuration of another solid-state image sensor according to the second embodiment of the present invention. FIG. 24 is a block diagram showing a schematic configuration of a simplified pixel difference processing unit and a motion contour determination unit. FIG. 25 is an explanatory diagram of a driving method of the pixel difference processing unit and the motion contour determination unit of FIG. 24. FIG. 26 is an explanatory diagram of a method of acquiring motion contour information in FIG. 25. FIG. 27 is a block diagram showing a schematic configuration of a motion information extraction unit using the motion contour information of FIG. 26. FIG. 28 is an explanatory diagram of a pixel configuration, a pixel output characteristic, and a separation circuit configuration of the solid-state image sensor of FIG. 24.

As an embodiment of the present invention, several selection cases are possible. The case of the combination is 1) the case of an image pickup camera or a solid-state image sensor as an image pickup device. 2) A case where the time link is performed by a synchronization signal, or a case where a time stamper is used. The former case of synchronization signal is further subdivided into a case where motion is detected and the synchronization signal and necessary image information are not output. 3) As a method of motion detection, there is a case where the frame difference is used, or a case where the pixel output difference is used without using the frame memory. 4) Necessary information of the moving body is the skeleton point coordinates, feature point coordinates, movement direction, and velocity of the moving body, and the acquisition of these information is performed using the frame memory or in parallel with the pixel output. Case. 5) Further image processing of 3 or more images.
Since the image pickup apparatus according to the embodiment of the present invention is divided into several cases as described above, each case will be described with reference to the drawings. There are combinations of these cases, but we will focus on each case alone and a typical combination.

In the following description, the case of the image pickup camera 1 of the first embodiment of the image pickup apparatus will be used, and the configuration of individual blocks of the present invention, the functions of each embodiment, and the output data will be described. The case of using the solid-state image pickup device as the second embodiment of the image pickup device will be finally performed after the description of the case of the image pickup camera 1 is completed.
In the following description, the same parts will be given the same reference numerals and processing names, the detailed description thereof will be given first, and the duplicated explanations of the same parts will be omitted.

<Embodiment 1 of the image pickup apparatus>
In FIG. 1, taking an image pickup camera as an example of the image pickup apparatus of the present invention, motion contour (size of moving object) information, motion vector (direction and speed of motion) information, and feature points (coordinates of moving object) are shown from the imaging screen according to the first embodiment. ) A block diagram showing a schematic configuration for extracting information is shown.
The block constituting the image pickup camera 1 of the present invention is composed of a solid-state image pickup device 3 provided with an image pickup lens 2, a motion contour extraction unit 4, a motion information extraction unit 5, an information output unit 6, and a clock 7. When an existing solid-state image sensor is used, image pickup information 8 is output from the solid-state image sensor 3.

FIG. 2 is a diagram showing the blocks constituting the motion contour extraction unit 4 and the flow of information related to them. The motion contour extraction unit 4 is composed of a pixel difference processing unit 9, a motion contour determination unit 10, a moving object light / dark determination unit 11, and a threshold value 12 for performing motion contour determination. As a flow of information, the imaging information 8 output from the solid-state imaging device 1 is input to the pixel difference processing unit 9, the difference calculation between pixels having different exposure times is performed, and the pixel difference information 34 is output.

When extracting the motion contour, it is necessary to delete the information of the stationary portion, and it is necessary to set the exposure time and the amplifier gain so that the output of the stationary portion can be erased by the difference between the pixels having different exposure times. The motion contour determination unit 10 outputs the motion contour information 14 from the pixel difference information 34 using the threshold value 12 as a determination reference. The motion contour determination unit 10 outputs the code information at the time of pixel difference as the pixel difference code information 15. Further, there is a moving body light / dark determination unit 11 for acquiring moving body light / dark information 16 for determining whether the moving body is bright or dark with respect to the surroundings from the image pickup information 8. For this reason, the motion contour determination unit 10 sends a motion object determination signal 13 indicating where the moving object is located to the motion object light / dark determination unit 11. Since the motion contour determination signal 13 can be substituted by the motion contour information 14, here, the motion contour information 14 is branched and input to the motion object light / dark determination unit 11 as the motion object determination information 13. This moving object light / dark information 16 is used together with the pixel difference code information 15 when the motion information extracting unit 5 calculates the direction of the motion of the moving object.

FIG. 3 is a diagram showing blocks constituting the motion information extraction unit 5 and the flow of information related to them. The motion information extraction unit 5 is composed of a moving body velocity determination unit 17, a moving body direction determination unit 18, a moving body size determination unit 19, a moving body skeleton point determination unit 20, and a moving body feature point determination unit 21. As the flow of information, the motion contour information 14, the pixel difference code information 15, and the moving object light / dark information 16 output from the motion contour extracting unit 4 are input to the moving body direction determination unit 18, and the motion contour information 14 is the moving body velocity determination unit 17. , The moving body size determination unit 19, the moving body skeleton determination unit 20, and the moving body feature point determination unit 21.

The moving body speed determination unit 17 outputs the moving body speed information 22 obtained by the method shown in FIG. The moving body velocity information 22 is sent to the moving body direction determination unit 18 and is useful for quantifying the direction. It is also sent to the moving body size determination unit 19 and is useful for size determination.
The moving body direction information 24 obtained by the method shown in FIG. 10 is output from the moving body direction determination unit 18. The intermediate calculation result in the moving body direction determination unit 18 is sent to the moving body speed determination unit 17 as the moving body direction intermediate information 23 and is useful for acquiring the moving body velocity information 22. The moving body direction information 24 is sent to the moving body size determination unit 19 and is useful for determining the moving body size.

The moving body size information 25 obtained by the method shown in FIG. 12 is output from the moving body size determining unit 19. The information obtained in the process of size extraction by the moving body size determination unit 19 is sent to the moving body skeleton point detecting unit 20 as skeleton point candidate information 26, and is useful for skeleton point extraction.
From the moving body skeleton point detection 20, the moving body skeleton point information 27 obtained by the method shown in FIG. 13 is output. In addition, feature point candidate information 26'is also output from the moving body skeleton point detection 20. The feature point candidate information 26'is sent to the moving body feature point determination unit 21, and the moving body feature point information 28 obtained by the method shown in FIG. 14 is output.

FIG. 4 is a diagram showing a flow of information related to the blocks constituting the information output unit 6. The constituent block is only the information synthesis / output unit 29. As the flow of information, the moving body velocity information 22, the moving body direction information 24, the moving body size information 25, the moving body skeleton point information 27, the moving body feature point information 28, and the time output from the clock 7 are output from the motion information extraction unit 5. The information 30 is input to the information synthesis / output unit 29 and output to the outside as the information output 31.

The moving body direction information 24, the moving body velocity information 22, the moving body size information 25, the moving body skeleton point information 27, and the moving body feature point information 28 acquired by the motion information extraction unit 5 shown in FIG. 3 are clocks in the image pickup camera 1 of FIG. Together with the time information 30 output from 7, the information is input to the information synthesis / output unit 29 of the information output unit 6 as shown in FIG. 4, synthesized internally, and output as the information output 31 from the information synthesis / output unit 29. This information output form will be described with reference to FIGS. 5A and 5B based on an information output example.

From the information synthesis / output unit 29 of the information output unit 6 described with reference to FIG. 4, the moving body direction information 24, the moving body velocity information 22, the moving body size information 25, the moving body skeleton point information 27, the moving body feature point information 28, and the time information 30 Is output, but as an example of this information output, the amount of information as shown in FIG. 5A is assumed. As a precondition, the accumulated information is sent once every hour, and since 1 hour is 3600 seconds, the time information is expressed in 12 bits. The moving body velocity information is represented by 6 bits with 64 types of velocity classification. The moving object direction information is expressed in 4 bits with 16 types of direction classification (in 22.5 degree increments). The moving body size information is classified into 32 types and is expressed in 5 bits. The moving object feature point coordinate information is expressed by one point of coordinate resolution of 10-bit notation equivalent to 1024 pixels for X coordinate and 9-bit notation equivalent to 512 pixels for Y coordinate. The moving body skeleton coordinate information is expressed in two points: a 10-bit notation in which the X coordinate corresponds to 1024 pixels and a 9-bit notation in which the Y coordinate corresponds to 512 pixels.

FIG. 5B shows an example of information output synthesis under the preconditions shown in FIG. 5A. The information block has 8 bits corresponding to 1 byte in the horizontal direction as a unit, and the number of bytes corresponding to the amount of information is shown in the vertical direction.
Under the preconditions shown in FIG. 5A, the time occupies 12 bits, the speed occupies 6 bits, the direction occupies 4 bits, and the size occupies 5 bits. The feature points occupy a total of 19 bits, 10 bits in the X coordinate and 9 bits in the Y coordinate. Since each of the skeleton points 1 and 2 also occupies a total of 19 bits of X coordinate 10 bits and Y coordinate 9 bits, it occupies 38 bits in total of the two skeleton points. The total amount of information is 12 + 6 + 4 + 5 + 19 + 38 = 84 bits, which is 10.5 bytes of information.
This information output synthesis example is exactly the pixel position non-link numerical information.

As an example of information output of the moving body shown in FIG. 5A, for a single moving body, feature point coordinate information indicating the position of the moving body, skeletal point coordinate information indicating the rough shape of the moving body, velocity of the moving body, and moving direction. , Size information, and time information output formats are shown, but this is the same even if there are multiple moving objects, and the same output format corresponds to each moving object. Further, it is not necessary to acquire and post all the information corresponding to this output format, and only the necessary information regarding the moving object may be output.

By directly outputting the image information (shape, moving direction, speed, position) of the moving object that is not linked to the pixel position of the present invention, the amount of information to be processed in the subsequent stage is greatly reduced, and the calculation load is greatly reduced. Provide the device. That is all for the whole image.

The functions and operations of each block in the image pickup camera 1 which is the first embodiment of the image pickup apparatus will be sequentially described below. The feature of the present application is to extract and output the motion information and the moving object information of the moving object based on the motion contour information. The motion information includes the motion contour, the direction, the velocity, and the size information, and the moving object information is the shape and the representative. Corresponding to the position, there is skeleton point and feature point information respectively. Depending on the field of application of the present application, the required information may be a single piece or a combination of a plurality of pieces of the information. Actually, the combination of the information acquisition blocks to be driven is changed to acquire the necessary information.

In the following description, the method of acquiring individual information will be described for each embodiment of information acquisition.
Distinguishing from the major classification of the embodiment of the imaging device, as a middle classification of the lower classification of the first embodiment of the imaging device, each information acquisition method is supported, for example, Embodiment 1; motion contour information, embodiment 2; moving object direction information. I will explain in sequence like this. Regarding the motion contour information of the first embodiment, since there are a method of using the frame memory and a method of not using the frame memory as a method of acquiring the contour information, a sub-classification is further used and described.

<Embodiment 1; motion contour information>
First, a method of acquiring pixel difference information 34 by the pixel difference processing unit 9 from the image pickup information 8 from the solid-state image pickup device 3 shown in FIG. 2, and motion contour information 14 by the motion contour determination unit 10 from the pixel difference information 34. A method of acquiring the pixel difference code information 15 will be described.
Regarding the motion contour information of the first embodiment, there are two types of contour information acquisition methods: a method using a frame memory and using a frame difference, and a method using a difference between pixels having different exposure times without using the frame memory. Therefore, the former is described as Embodiment 1-1, and the latter as Embodiment 1-2.

<Embodiment 1-1; Motion contour information; Frame memory used>
First, a method of acquiring pixel difference information 34 by the pixel difference processing unit 9 from the image pickup information 8 from the solid-state image pickup device 3 shown in FIG. 2, and motion contour information 14 by the motion contour determination unit 10 from the pixel difference information 34. The method of acquiring the pixel difference code information 15 will be described with reference to FIG.

FIG. 6 shows a case where the frame memory 32 is used. In this case, the pixel difference processing unit 9 is composed of a frame memory 32 and a difference circuit 33. By storing the imaging information 8 for one frame in the frame memory 32 and taking the difference from the captured image 8 of the current frame for each pixel by the difference circuit 33, two images with a time interval of one frame are taken. You can take the difference between the frames of. This is referred to as pixel difference information 34.
Since the pixel difference information 34 acquires the difference corresponding to the pixels, the static region becomes the difference of the same output value and becomes zero. In the area where the output value of the moving object changes when there is movement, it does not become zero when the difference is acquired. Specifically, when the light and darkness is uniform inside the moving body pattern, the inside becomes zero due to the difference between the frames, and if there is a change in the peripheral part of the moving body pattern, it does not become zero. That is, the contour pattern of the moving body remains.

The pixel difference information 34 output from the pixel difference processing 9 is converted into the motion contour information 14 by the motion contour determination unit 10. This method will also be described with reference to FIG.
The pixel difference information 34 is acquired by the pixel correspondence difference between the two frames, and includes the code information. This is referred to as pixel difference code information 15, and is output from the motion contour determination unit 10. After the pixel difference information 34 passes through the absolute value circuit 35, it becomes absolute value information, and the binarized motion contour information 14 acquired in comparison with the threshold value 12 by the motion contour determination circuit 36 is also used in the motion contour determination unit. Output from 10. As described above, the pixel difference information 34 includes the pixel difference code information 15 and the motion contour information 14 of the moving object, and is output from the motion contour determination unit 10.
In FIG. 2, the threshold value 12 is described outside the movement contour determination unit 10, but in the following description, the threshold value 12 will be described by putting it inside the movement contour determination unit 10.

From the image pickup information 8 from the solid-state image pickup device 3 shown in FIG. 6, the pixel difference processing unit 9 acquires the pixel difference information 34 using the frame memory 32, and the motion contour determination unit 10 further obtains the motion contour information 14 and pixels. The method of acquiring the difference code information 15 will be described with reference to FIGS. 7 (a) and 7 (b) in the case of a specific subject.

FIG. 7A0 shows a case where the pixel difference processing unit 9 uses the frame memory 32. The current image and the image one frame before are used as specific captured images, and the image one frame before in each image; the moving object image is 39'in FIG. 7 (a1), and the original image; a little forward in FIG. 7 (a2). The moving object image is shown by 39.
The pixel outputs on the same scanning line 40 in the captured image of FIG. 7 (a0) are shown in FIGS. 7 (b1) and 7 (b2). The pixel output on the scanning line 40 in the image one frame before FIG. 7 (a1) corresponds to the moving body 39', and similarly on the scanning line 40 in the current image of FIG. 7 (a2). The pixel output is shown in FIG. 7 (b2) corresponding to the moving body 39. The brightness of the moving objects 39, 39'and the center line 41 is shown in a case brighter than the background road 42, and as shown in FIGS. 7 (b1) and 7 (b2), the moving objects 39, 39'and the center line 41'and the center line 41 are shown. The pixel output of is larger than that of the road 42.

In FIG. 2, the moving object determination information 13 is output from the motion contour determination unit 10, is input to the moving object light / dark determination unit 11 together with the image pickup information 8, is processed, and the moving object light / dark information 16 is output. The moving body determination information 13 is the same as the moving contour information 14, and the light and darkness of the moving body inside the contour is output as the moving body light and dark information 16. In the examples of FIGS. 7 (b1) and 7 (b2), the moving object images 39 and 39'on the scanning line are recognized, and whether this is brighter or darker than the background (here, the road 42) (in FIGS. 7 (b1) and (b2)). The light) is determined, and the moving body light / dark information 16 is output from the moving body light / dark determination unit 11.

FIG. 7 (b3) shows the pixel difference sign information 15 obtained by outputting the pixel difference information 34 acquired by the difference circuit 33 from the pixel output on the scanning line 40 shown in FIGS. 7 (b1) and 7 (b2) as it is. In FIG. 7 (b3), the pixel difference code information 15 is output as the pixel difference code information 15', 15'at a location corresponding to the end of the moving body images 39, 39'. ,-.
FIG. 7 (b3) shows the pixel difference code information on the scanning line 40, but as a planar image, as shown in FIG. 7 (a3), the pixel difference code information 15'and 15 are formed around the moving object image 39. Is shown as an area.

The pixel difference information 34 becomes absolute value information via the absolute value circuit 35 in the motion contour determination unit 10, and the motion contour information 14 is obtained in comparison with the threshold value 12 in the motion contour determination circuit 36. The pixel difference code information 15'and 15 "on the scanning line 40 shown in FIG. 7 (b3) are converted into absolute value information and determined as a threshold value. In FIG. 7 (b4), the motion contour information 14 is the pixel difference code information 15'. , 15 "corresponds to, and is output as binarized motion contour information 14', 14".
FIG. 7 (b4) shows the motion contour information on the scanning line 40, but as a planar image, as shown in FIG. 7 (a4), the motion contour information 14'and 14 "are provided around the moving object image 39. It is shown as a region. From the movement contour information 14', 14 ", the approximate outer shape of the moving body can be known.

<Embodiment 1-2; Motion contour information; No frame memory used>
FIG. 8 shows a case where the frame memory is not used. Since the solid-state image sensor used in this case is composed of two types of pixels 43 having different exposure times as shown in FIGS. 9 and 10, the solid-state image sensor 3'and the solid-state image sensor 3 of FIG. 6 are distinguished from each other. write. In this case, the pixel difference processing unit 9'is composed of a separation circuit 37, an amplifier circuit 38, and a difference circuit 33. The image pickup information 8 output from the solid-state image pickup device 3'includes an image pickup STL which is image pickup information from a long-exposure pixel (referred to as STL in the following figures) and a short-time exposure pixel (in the following figures). The imaging STS, which is the imaging information from), is mixed. The separation circuit 37 separates these two types of image pickup information, the amplifier circuit 38 amplifies the image pickup STS of the short-exposure pixel, and the difference circuit 33 takes the difference from the image pickup STL of the long-exposure pixel to obtain the pixel difference information. Get 34'(distinguished from the frame memory case). The details of this operation will be described with reference to FIG.

The motion contour determination unit 10 shown in FIG. 8 for acquiring the motion contour information 14 and the pixel difference code information 15 from the pixel difference information 34'output from the pixel difference process 9'has a motion shown in FIG. 6 in both configuration and function. It is the same as the contour determination unit 10, and the description thereof will be omitted.

The configuration of the pixel 43 of the solid-state image sensor 3'shown in FIG. 8 is shown in FIG. 9A, the pixel output characteristic is shown in FIG. 9B, and the separation circuit 37 of the pixel difference processing unit 9'in FIG. The configuration of is shown in FIG. 9 (c).
As the pixel configuration of the solid-state image sensor 3'shown in FIG. 9A, a short-time exposure pixel STS and a long-time exposure pixel STL are arranged. Each image pickup signal is referred to as an image pickup STS and an image pickup STL to distinguish them. As shown in FIG. 9B, the pixel output characteristics of both pixels 43 show the same case where the pixel output increases with the passage of time (corresponding to the sensitivity), and the pixel output is commensurate with the exposure time. Is obtained and output as imaging information 8. In FIG. 9B, since the short-time exposure pixel STS and the long-time exposure pixel STL have the same sensitivity, the pixel output differs in proportion to the corresponding exposure time TS and TL. In FIG. 9A, it is assumed that each scanning line is output.

In the separation circuit 37 of the pixel difference processing unit 9'shown in FIG. 8, the image pickup STL and the image pickup STS which are the image pickup information from these pixels are separated, and the configuration of the separation circuit 37 is shown in FIG. 9 (c). As described above, the line memory 44 for delaying the imaging information 8 by one horizontal (1H) scanning period and the delay circuit 45 for delaying the imaging information 8 in pixel units are configured. When outputting for each scanning line in FIG. 9A, the line memory 44 is required in FIG. 9C.

The short-time exposure pixel STS and the long-time exposure pixel STL are operated with the exposure times TS and TL, respectively, and are read out line by line. The line memory 44 and the delay circuit 45 that delays each pixel are adjusted so that the phases of the pixel signals to be compared between the short-exposure pixel STS and the long-exposure pixel STL are the same. Therefore, the short-time exposure pixel output from the separation circuit 37, the image pickup STS and the image pickup STL which are the image pickup information from the long-time exposure pixel are adjacent to each other in the vertical direction in the pixel arrangement of FIG. 9A (1 horizontal line). It corresponds to the output of the short-exposure pixel STS and the long-exposure pixel STL (separated).

In the amplifier circuit 38 of the pixel difference processing unit 9'shown in FIG. 8, the imaging STS of the short exposure pixel is amplified by the gain of the exposure time ratio (TL / TS), which is equivalent to the signal level of the imaging STL of the long exposure pixel. It is processed so that it becomes. Pixel difference information 34'can be acquired by taking the difference between the image pickup STS and the image pickup STL by the difference circuit 33 of the pixel difference processing unit 9'.

The pixel difference processing unit 9 of FIG. 6 acquires the pixel difference information 34 by taking the difference between two frames having a time interval of one frame. On the other hand, in the solid-state image sensor 3'and the pixel difference processing unit 9'in FIG. 8, the pixel difference information 34 is acquired by the difference between the exposure times TL and TS in the same frame in the pixel output characteristics shown in FIG. 9 (b). There is. Even in the case shown in FIG. 9B, the static region is the difference of the same output value, so it becomes zero. If there is movement within one frame, it will not be zero when the difference is acquired in the region where the output value of the moving object changes.
Specifically, when the brightness is uniform inside the moving body pattern, the inside becomes zero and the contour pattern of the moving body remains. As described above, the pixel difference information 34'includes the contour information of the moving object.

The configuration of another pixel 43 of the solid-state image sensor 3'shown in FIG. 8 is shown in FIG. 10A, the pixel output characteristics are shown in FIG. 10B, and the configuration of the pixel difference processing unit 9'in FIG. Is shown in FIG. 10 (c).
As the pixel configuration of the solid-state image sensor 3'shown in FIG. 10A, short-time exposure pixels (indicated as STS in the figure) and long-time exposure pixels (indicated as STL in the figure) are arranged. The pixel size is different and the sensitivity is different. In the case where the sensitivity of the short-time exposure pixel is SS and the sensitivity of the long-exposure pixel is SL, and the sensitivity ratio of both; SS / SL is the inverse of the exposure time ratio of both; TS / TL, the pixel output characteristics are As shown in FIG. 10B, the pixel outputs of the short-time exposure pixel STS and the long-time exposure pixel STL are aligned.

FIG. 10C shows the configuration of the pixel difference processing unit 9'. The separation circuit 37 is the same as in FIG. 9 (c), and the image pickup STS and the image pickup STL, which are the image pickup information from the short-time exposure pixel and the long-time exposure pixel, are adjacent to each other in the vertical direction in the pixel arrangement of FIG. 10 (a). It corresponds to the output of the short-exposure pixel STS and the long-exposure pixel STL (one horizontal line apart). Here, it is assumed that both the short-exposure pixel STS and the long-exposure pixel STL lines are read out at the same time.
In the pixel output characteristics shown in FIG. 10B, since the pixel outputs of the short-time exposure pixel STS and the long-time exposure pixel STL are aligned, the amplifier circuit 38 shown in FIG. 10C does not need an amplification factor of 1. good. However, since only the exposure time ratio determined by SS / SL can be selected by the sensitivity ratio, it is better to have the amplifier circuit 38 even in the pixel configuration of FIG. 10A in order to increase the degree of freedom of the exposure time. No line memory is required because both lines are read at the same time. In FIG. 9A, the line memory 44 becomes unnecessary when the scanning lines of two adjacent lines are output at the same time.

From the image pickup information 8 from the solid-state image pickup device 3'shown in FIG. 8, the image pickup STL which is the image pickup information from the long-time exposure pixel STL and the image pickup information from the short-time exposure pixel STS by the pixel difference processing unit 9'. Pixel difference information 34'is acquired by separating the image pickup STS with the separation circuit 37, amplifying the image pickup STS of the short exposure pixel with the amplifier circuit 38, and taking the difference from the image pickup STL of the long exposure pixel with the difference circuit 33. Further, a method of acquiring the motion contour information 14 and the pixel difference code information 15 by the motion contour determination unit 10 will be described with reference to FIGS. 11 (a) and 11 (b) of a specific subject case. Since it is similar to FIGS. 7 (a) and 7 (b), overlapping parts will be omitted.

FIG. 11A0 shows a case where the pixel difference processing unit 9'uses a long-exposure pixel (STL) and a short-time exposure pixel (STS). As specific captured images, an image of STL pixels and an image of STS pixels are used. In FIG. 11 (a1), a moving object image in the image of STL pixels is 39', and in FIG. 11 (a2), an image of STS pixels is used. The moving object images are shown by 39, respectively. Here, the pixel configuration shown in FIG. 10A will be described, and the pixel outputs are complete.
As shown in FIGS. 9 (b) and 10 (b), the exposure time of the TL of the long-exposure pixel STL is longer than that of the TS of the short-time exposure pixel STS. The length is longer than the length of the moving object image 39 of the STS pixel. This is because the exposure time TL is longer than the TS, so the length of the moving object image 39'moving during that time becomes longer.

11 (b1) and 11 (b2) show the pixel outputs of the long-exposure pixel STL and the short-time exposure pixel STS on the scanning line 40 at the same timing in the captured image of FIG. 11 (a0). These are the pixel outputs on the scanning line 40 in the same frame, and are different from the pixel outputs shifted by one frame in FIGS. 7 (b1) and 7 (b2). The pixel output of the long-time exposure pixel STL on the scanning line 40 of FIG. 11A corresponds to the moving object image 39', and the short-time exposure pixel on the scanning line 40 of FIG. 11A is shown in FIG. 11B1. The pixel output of the STS is shown in FIG. 11 (b2) corresponding to the moving object image 39. Similar to FIG. 7, the brightness of the moving objects 39, 39'and the center line 41 is shown for a case brighter than the background road 42, and as shown in FIGS. 11 (b1) and 11 (b2), the moving object images 39, The pixel output of 39'and the center line 41 is larger than that of the road 42. With low-sensitivity STL pixels, the moving object image 39'is long, and the inclination of the pixel output at the end is gentle.

The pixel difference code information 15 obtained by outputting the pixel difference information 34 acquired by the difference circuit 33 as it is from the pixel output on the scanning line 40 shown in FIGS. 11 (b1) and 11 (b2) is shown in FIG. 11 (b3). In FIG. 11 (b3), the pixel difference code information 15 is output as the pixel difference code information 15', 15'at a location corresponding to the end of the moving body images 39, 39'. It becomes + and-.
FIG. 11 (b3) shows the pixel difference sign information on the scanning line 40, but as a planar image, as shown in FIG. 11 (a3), the pixel difference code information 15'and 15 are formed around the moving object image 39. Is shown as an area.

The pixel difference information 34 becomes absolute value information via the absolute value circuit 35 in the motion contour determination unit 10, and the motion contour determination circuit 36 compares with the threshold value 12 to obtain the motion contour information 14. The pixel difference code information 15'and 15 "on the scanning line 40 shown in FIG. 11 (b3) are converted into absolute value information and determined as a threshold value. In FIG. 11 (b4), the motion contour information 14 is the pixel difference code information 15'. , 15 ", and is output as binarized motion contour information 14', 14".
FIG. 11 (b4) shows the motion contour information on the scanning line 40, but as a planar image, as shown in FIG. 11 (a4), the motion contour information 14'and 14 "are provided around the moving object image 39. It is shown as a region. It is the same as in FIG. 7 (a4) that the approximate outer shape of the moving body can be known from the motion contour information 14', 14 ".

<Embodiment 2; Moving object direction information>
From the motion contour information 14, the pixel difference sign information 15, and the moving object light / dark information 16 output from the motion contour extracting unit 4 shown in FIG. 2, the moving object direction is determined by the moving object direction determining unit 18 of the motion information extracting unit 5 shown in FIG. The method of acquiring the information 24 will be described with reference to FIGS. 12 (a), (b), (c), (d), and (e).

As shown in FIG. 12A, the moving body direction determination unit 18 is composed of a moving body direction extraction unit 46 and a moving body direction calculation unit 47. The movement contour information 14, the pixel difference code information 15, and the moving body light / dark information 16 are input from the movement contour extraction unit 4 to the moving body direction extraction unit 46. The moving body direction extraction unit 46 outputs the moving body direction intermediate information 23 to the moving body speed determination unit 17, the same moving body speed determination unit 17 inputs the moving body velocity information 22 to the moving body direction calculation unit 47, and the moving body direction calculation unit 47 inputs the moving body speed information 22 to the moving body direction calculation unit 47. The moving body direction information 24 is output. The moving body direction information 24 is also output to the moving body size determination unit 19.

In FIG. 12 (b), a moving body image (current image) 39, a pixel difference code information 15, and a motion contour information 14 are shown in FIG. 12 (b1) corresponding to FIGS. 7 (a2), (a3), and (a4), respectively. ), (B2), and (b3). The pixel output of the moving body 39 on the scanning line 40 in the current image of FIG. 7 (a0) is shown in FIG. 12 (c1) corresponding to FIG. 7 (b2). The brightness of the moving body 39 is shown for a case brighter than the background road 42, and the moving body light / dark information 16 is a bright case. In FIG. 12 (c2) corresponding to FIG. 7 (b3), the pixel difference code information 15 is output as pixel difference code information 15', 15 "at a location corresponding to the end of the moving object image 39, and the code information is output. It becomes + and-, respectively.

The moving body direction extraction unit 46 determines the direction of movement using the movement contour information 14, the pixel difference code information 15, and the moving body light / dark information 16. First, the region where the movement has changed is grasped from the motion contour information 14, and whether the region with the motion contour information is at the front end or the rear end in the moving direction of the moving body is whether the moving body image 39 is brighter than the surroundings. Whether it is dark or not is determined by the moving body direction determination table of FIG. 12D from the two information of the moving body light / dark information 16 and the pixel difference code information 15. The information on the front end and the rear end of the moving body is used as the moving body direction intermediate information 23 and is used for determining the moving body velocity.

An example of this moving object direction determination table will be described with reference to the pixel difference code information 15'in FIG. 12 (c2). This is a pixel output region of the road 42 as shown in FIG. 7 (b1) in the image one frame before, and a moving body 39 brighter than the road 42 appears in the original image corresponding to FIG. 12 (c1). That is, this corresponds to the tip portion in the moving direction of the moving body. It is + in the pixel difference code information 15'that is obtained by subtracting the image one frame before from the current image. Corresponding to this, in the moving body direction determination table of FIG. 12 (d), the tip, the light, and the + correspond. Other combinations in the table can be obtained in the same way.

From the moving body direction determination table of FIG. 12 (d), it can be determined whether the movement contour information 14', 14 "around the moving body image (current image) 39 is the front end or the rear end of the moving body, but it advances in which angle direction. It has not been possible to quantify whether or not the image is present. The moving body direction extraction unit 43 outputs the moving body direction intermediate information 23 to the moving object velocity determination unit 17, but the moving object direction intermediate information 23 is information on whether the moving object is at the tip or the rear end. The moving body speed information 22 is input to the moving body direction calculation unit 47 from the moving body speed determination unit 17, and the moving body speed information 22 has speed components in the X direction (horizontal direction) and the Y direction (vertical direction). Vx0 and Vy0 are included, respectively. By taking the inverse tan of the ratio of Vx0 and Vy0, the direction can be displayed as an angle from the horizontal direction as shown in FIG. 12 (e). From the moving body direction calculation unit 47. The output moving body direction information 24 is information of this angle. These calculations are performed by the moving body direction calculation unit 47, and the moving body direction information 24 is output.

<Embodiment 3; moving body velocity information>
The motion contour information 14 output from the motion contour extraction unit 4 shown in FIG. 2 and the moving body direction intermediate information 23 output from the moving body direction determination unit 18 of the motion information extracting unit 5 shown in FIG. 3 are the moving body velocity determination unit 17. The method of acquiring the moving body speed information 22 from the moving body speed determination unit 17 will be described with reference to FIGS. 13 (a), (b), (c), and (d).

As shown in FIG. 13A, the moving body speed determination unit 17 is composed of a moving body speed extracting unit 48, a moving body speed calculation unit 49, and a conversion table 50. The motion contour information 14 is input to the moving body velocity extracting unit 48 from the motion contour extracting unit 4, and the moving body direction intermediate information 23 is input from the moving body direction extracting unit 46. The information of the moving body speed extraction unit 48 and the conversion table 50 is calculated by the moving body speed calculation unit 49, and the moving body speed information 22 is output.
The moving body velocity information 22 is also output to the moving body direction determination unit 18 and the moving body size determination unit 19.

In FIG. 13 (b1), the motion contour information 14'(front end) and 14 "(rear end) corresponding to FIG. 12 (b3) are shown in a planar image. The moving object direction intermediate information 23 is this. It is used to determine the selection of paired motion contour information 14'(front end) and 14 "(rear end). Further, FIG. 13 (b2) shows motion contour information 14 on the scanning line as shown in FIG. 7 (b4), and motion contour information 14'(tip), 14 "(rear) shown in FIG. 13 (b1). The output of the end) is shown. The moving body direction in the moving contour region is known from the moving body direction intermediate information 23 output from the moving body direction extraction unit 46.
In the moving body speed extraction unit 48, the width of the movement contour region (corresponding to the number of pixels) in the movement contour information 14', 14 "is obtained in the X direction and the Y direction. It is expressed as "x, and V'y, V" y. This width corresponds to the width (number of pixels) in which the moving object moves in one frame period.

The frequency distribution data of the number of motion speed pixels V'x, V "x, and V'y, V" y corresponding to the widths in the X and Y directions of the motion contour information 14'and 14 "in FIG. 13 (b1). These are shown in FIGS. 13 (c1) and 13 (c2), respectively. The frequency distribution of the number of motion speed pixels has a range depending on the shape of the moving body, but here, the number of speed pixels with the highest frequency is the representative speed in the X direction and the Y direction. The number of pixels is described as V'x0 and V'y0, respectively.

Motion speed of motion contour information 14', 14 "To obtain the velocity from the number of pixels V'x, V" x, and V'y, V "y, the optical system of the camera (distance to the subject, lens magnification, etc.) It is necessary to convert to the actual distance determined by). This is the conversion table 50, which is a conversion table between the number of pixels at each subject position on the screen and the actual length. The conversion coefficient differs depending on the pixel position, and the speed. When performing measurement, it is necessary to create a conversion table at each shooting location in advance. The actual length is obtained from the location of the contour area and the pixel width based on the conversion table 50, and the actual length is converted in one frame period. These calculations are performed by the moving body speed calculation unit 49, and the moving body speed information 22 is output.
Representative speeds The representative speeds in the X and Y directions obtained from the conversion table 50 from the number of pixels V'x0 and V'y0 are described in Vx0 and Vy0, respectively. In the following description, the length is given a dash'at the pixel number display stage, and is not given at the actual length. This applies not only to the speed shown in FIG. 13 but also to the length shown in FIG.

In FIG. 13D, Vx0 and Vy0, which are representative velocities in the X and Y directions of the moving body, and the square root of the sum of squares thereof are shown by V0. V0 is the representative velocity of the moving body. The moving body speed information 22 is output from the moving body speed calculation unit 49, and the contents include the representative speed V0 of the moving body and the representative speeds Vx0 and Vy0 in the X and Y directions. The moving body velocity information 22 is transferred to the moving body direction calculation unit 47 and is useful for calculating the direction of the moving body.

<Embodiment 4; moving body size information>
The motion contour information 14 output from the motion contour extraction unit 4 shown in FIG. 2, the motion velocity information 22 output from the motion velocity determination unit 17 of the motion information extraction unit 5 shown in FIG. 3, and the motion object direction determination unit 18 A method of inputting the moving body direction information 24 output from the above to the moving body size determination unit 19 and acquiring the moving body size information 25 from the moving body size determination unit 19 is described in FIGS. This will be described based on (d).

As shown in FIG. 14A, the moving body size determination unit 19 includes a moving body size extracting unit 51, a moving body size calculation unit 52, and a conversion table 50. The moving contour information 14 is input from the moving contour extracting unit 4, the moving body velocity information 22 is input from the moving body speed determining unit 17, and the moving body direction information 24 is input from the moving body direction determining unit 18 to the moving body size extracting unit 51. The information of the moving body size extracting unit 48 and the conversion table 50 is calculated by the moving body size calculation unit 52, and the moving body size information 25 is output. The information 25 is also output to the moving body skeleton point extracting unit 20 by the moving body size extracting unit 51. The information obtained in the process of size extraction by the moving body size extracting unit 51 is sent to the moving body skeleton point extracting unit 20 as skeleton point candidate information 26, and is useful for skeleton point extraction.

FIG. 14 (b1) shows motion contour information 14'(front end) and 14 "(rear end) corresponding to FIG. 12 (b3) as a planar image, and FIG. 14 (b2) shows. , The motion contour information 14 on the scanning line as shown in FIG. 7 (b4), and the output of the motion contour information 14'(front end) and 14 "(rear end) shown in FIG. 14 (b1) is shown. There is. The moving body direction information 24 is used for selection determination of the paired movement contour information 14'(front end) and 14 "(rear end).

In the moving body size extracting unit 51, the outer width of the movement contour region in the movement contour information 14', 14 "is obtained in the X direction and the Y direction. This outer width is on the same scanning line in the X direction. The outer width is obtained so as to cover both the motion contour information 14'and 14 ", which is called the motion size pixel number and is expressed by L'x. The external dimensions of either the motion contour information 14'or 14 "are excluded. As shown in FIG. 14 (b2), the motion size pixel number L'x in the X direction is such that the moving object moves in one frame period. Corresponding to the area width (number of pixels) to be covered, the speed pixels V'x0 and V "x0 are attached to both ends. Therefore, the actual number of motion size pixels is obtained by subtracting the number of velocity pixels V'x0 and V "x0 from the number of motion size pixels L'x. The Y direction is the same as the X direction.

In FIG. 14 (b1), the outer widths of the movement contour regions of the movement contour information 14'and 14 "are obtained in the X and Y directions as L'x and L'y, respectively. The pixel coordinate position of the point is indicated by a black circle in the figure. This coordinate position becomes the skeleton point candidate information 26, is output from the moving body size extraction unit 51, and is used by the skeleton point extraction unit 20.

The frequency distribution data of the number of motion size pixels L'x and L'y corresponding to the widths in the X and Y directions of the motion contour information 14'and 14 "of FIG. 14 (b1) are shown in FIGS. 14 (c1) and 14 (c2), respectively. ). The frequency distribution of the number of motion size pixels varies depending on the shape of the moving object. Here, the number of motion size pixels with the highest frequency is L'mx as the number of representative motion size pixels in the X and Y directions, respectively. , L'yp. As shown in FIG. 14 (b2), the representative size pixel number L'x0 of the size of the moving body in the X direction is L'x0 obtained by subtracting the representative speed pixel number V'x0 from L'xp. = L'xp-V'x0.

Representative size obtained from motion contour information 14', 14 "To acquire the actual moving object size from the number of pixels L'x0, L'y0, it is actually determined by the optical system of the camera (distance to the subject, lens magnification, etc.). This is the conversion table 50, which is the conversion table between the number of pixels at each subject position on the screen and the actual length, and is the same as the conversion table 50 used when calculating the speed. The operation is performed by the moving body size calculation unit 52, and the moving body size information 25 is output.

To obtain the actual size, subtract the representative speed pixels V'x0 and V'y0 from the representative motion size pixels L'xp and L'yp in the X and Y directions to obtain the actual size. The number of corresponding representative size pixels; L'x0 and L'y0 are obtained. As the next step, the moving body size information Lx0 and Ly0 in the X and Y directions obtained in the conversion table 50 are obtained.
As shown in FIG. 14D, the moving body size information Lx and Ly corresponding to the direction of movement of the moving body can be obtained by the moving body size information Lx = Lx0tanθ and Ly = Ly0tanθ in the X and Y directions. Since tan also takes a negative value depending on the value of θ, it is expressed as an absolute value.

<Embodiment 5; moving body skeleton point information (midpoint method)>
The motion contour information 14 output from the motion contour extraction unit 4 shown in FIG. 2 and the skeleton point candidate information 26 output from the motion object size determination unit 19 of the motion information extraction unit 5 shown in FIG. 3 are combined with the motion skeleton point. A method of inputting to the determination unit 20 and acquiring the moving body skeleton information 27 and the feature point candidate information 26'will be described with reference to FIGS. 15 (a), (b), (c), and (d).

As shown in FIG. 15A, the moving body skeleton point determination unit 20 is composed of a moving body skeleton point extraction unit 53 and a moving body skeleton point synthesis unit 54. The motion contour information 14 is input to the moving body skeleton point extraction unit 53 from the motion contour extraction unit 4, and the skeleton point candidate information 26 is input from the moving body size extraction unit 51. As shown in FIG. 14 (b1), the moving body size extracting unit 51 has already acquired the pixel coordinate position of the midpoint of the outer width of the motion contour area of the motion contour information 14', 14 "as the skeleton point candidate information 26. As shown in FIG. 15 (b1), the skeleton point candidate information 26 is acquired in each scanning line of the paired motion contour information 14'and 14'. In the figure, the broken line indicates a line segment (referred to here as a scanning line segment) corresponding to the outer width (L'x) of the contour region on the scanning line in the X direction, and the black circle indicates the midpoint thereof. The coordinate position group of this black circle is the skeleton point candidate information 26. There is a scanning line segment shown by a thick broken line in the figure, but this is at an intermediate position in the vertical direction in the scanning line segment group in the X direction in which the skeleton point candidate information 26 of the motion contour information 14'and 14 "exists. It is a certain scanning line segment, which becomes feature point candidate information 26'.

In the moving body skeleton point extraction unit 53, the moving body skeleton point is taken from the coordinate position group indicated by the black circle of the skeleton point candidate information 26x of the scanning line segment group in the X direction of the motion contour information 14'and 14 "shown in FIG. 15 (b1). As an extraction method, first, the angle φ formed by the skeleton candidate point information 26x ₁ and 26x ₂ at both ends, the extension line of the skeleton point candidate information 26, and the adjacent skeleton candidate point information is equal to or larger than a predetermined angle. Let the moving body skeleton point be _26x3 .
In the case of FIG. 15 (b1), there are _three moving skeleton points, 26 x ₁ 26 x ₂ and 26 x 3. The central scanning line segment is called feature point candidate information 26'x, and as shown in FIG. 15 (b2), the moving body skeleton extraction unit 53 outputs the feature point candidate information 26'to the moving body feature point determination unit 21. Will be done.

Similarly, the moving body skeleton points are first extracted from the coordinate position group indicated by the black circle of the skeleton point candidate information 26y of the scanning line segment group in the Y direction of the motion contour information 14'and 14 "shown in FIG. 15 (c1). The skeleton candidate point information 26y ₁ and 26y ₂ at both ends, the extension line of the skeleton point candidate information 26, and the moving skeleton point 26y ₃ at the point where the angle φ formed by the adjacent skeleton candidate point information is equal to or more than a predetermined value are extracted. This method of acquiring moving body skeleton points is called the midpoint method.
In the case of FIG. 15 (c1), there are three moving skeleton points, 26y ₁ , 26y ₂ , and 26y ₃ . Further, the scanning line segment in the center becomes the feature point candidate information 26'y, and as shown in FIG. 15 (c2), the feature point candidate information 26 is also transmitted from the moving body skeleton extraction unit 53 to the moving body feature point determination unit 21. 'It is output as.

The moving body skeleton points 26x ₁ 26x ₂ , 26x ₃ and 26y ₁ 26y ₂ , 26y ₃ extracted by the moving body skeleton point extracting unit 53 are synthesized by the moving body skeleton point synthesizing unit 54 and are as shown in FIG. 15 (d). The pixel coordinate position information of these moving body skeleton point groups is output to the outside from the moving body skeleton point synthesizing unit 51 as the moving body skeleton point information 27.
The feature point candidate information 26'x and 26'y shown in FIGS. 15 (b2) and 13 (c2) are output from the moving body skeleton point extraction unit 53 as feature point candidate information 26', respectively, and the moving body feature point determination unit 22. Is entered in.

<Embodiment 6; Moving object feature point information (midpoint method)>
A method of inputting the feature point candidate information 26'output from the moving body skeleton point determination unit 20 shown in FIG. 15A into the moving feature point determination unit 21 and acquiring the moving body feature point information 28 is described in FIG. This will be described based on (a) and (b).

As shown in FIG. 16A, the moving body feature point determination unit 20 is composed of only the moving body feature point extraction unit 55. Feature point candidate information 26'is input from the moving body skeleton point extraction unit 53 to the moving body feature point extraction unit 55.
In the moving body skeleton point extraction unit 53, the feature point candidate information 26'x is characterized as shown in FIG. 16 (b2) in the X scanning line segment direction as shown in FIG. 16 (b1). The point candidate information 26'y has already been acquired as the feature point candidate information 26'. In the moving body feature point extraction unit 55, the pixel coordinate position information of the intersection of the feature point candidate information 26'x and the feature point candidate information 26'y is output as the moving body feature point information 28. This moving object feature point acquisition method is also called the midpoint method.

<Embodiment 7; moving body skeleton point information (center of gravity method)>
FIG. 17 (a) shows a method of inputting only the motion contour information 14 output from the motion contour extraction unit 4 shown in FIG. 2 into the moving body skeleton point determination unit 20 and acquiring the moving body skeleton information 27 and the feature point candidate information 26'. ) And (b) will be described.

As shown in FIG. 17A, the moving body skeleton point determination unit 20 is composed of only the moving body skeleton point extraction unit 53. The motion contour information 14 is input from the motion contour extraction unit 4 to the moving body skeleton point extraction unit 53.
The method of extracting the moving body skeleton point is referred to as the midpoint method in the fifth embodiment, and the pixel coordinates of the midpoint of the scanning line segment corresponding to the outer width (L'x) of the motion contour region in FIG. 15A. With the position as the skeleton point candidate information 26, the method of acquiring the skeleton points 26x ₁ , 26x ₂ , 26x ₃ and 26y ₁ 26y ₂ and 26y ₃ has been described in FIGS. 15 (b1) and 15 (b2).
On the other hand, in the seventh embodiment shown in FIG. 17 (b), it is referred to as a center of gravity method, and the paired motion contour information 14', 14 "is divided into regions, and the pixel coordinate position of the center of gravity position of each region is skeleton point information. Acquire as 27.

The method of dividing the motion contour information 14'and 14 "regions is to divide the motion contour regions 14'at the points where the number of the motion contour regions 14" changes in the process of scanning with the scanning line extending in the X direction. The scanning lines corresponding to the locations are scanning lines 56 _x 1 and scanning lines 56 _{x 4} in FIG. 17 (b). The motion contour regions 14 "and 14'are divided at locations where the signs change. The scanning lines corresponding to these locations are scanning lines 56 _{x 2} and scanning lines 56 _x 3 in FIG. 17 (b).

In FIG. 17B, the pixel coordinate position of the center of gravity of the region divided by the scanning lines 56 _x1 , the scanning lines 56 _x2 , the scanning lines 56 _x3 , and the scanning lines 56 _x4 is set as the skeleton point candidate information 26, and FIG. The moving body skeleton points are shown as 26x ₁ , 26x ₂ , 26x ₃ , 26x ₄ , and 26x ₅ , 26x ₆ , 26x ₇ , 26x ₈ . Here, the method of acquiring the center of gravity is to calculate the average coordinates of the pixel coordinate positions for each region divided by the scanning lines 56 _x1 , the scanning lines 56 _x2 , the scanning lines 56 _x3 , and the scanning lines 56 _x4 of the contour information 14', 14 ". Since it is not performed with the scanning line extending in the Y direction, the moving body skeleton point synthesizing unit 54 in FIG. 15A becomes unnecessary, and the pixel coordinate position of the skeleton point candidate information 26 is used as the moving body skeleton point information 27. It is output.
In the case of FIG. 17B, the moving body skeleton points 26x ₁ , 26x ₂ , 26x ₃ , 26x ₄ , 26x ₅ , 26x ₆ , 26x ₇ , and 26x ₈ are obtained from the moving body skeleton point extraction unit 53 as feature point candidate information 26'. It is output and input to the moving object feature point determination unit 22.
The advantage of the center of gravity method is that there is no calculation on the scanning line in the Y direction, and the calculation proceeds along the scanning line in the X direction.

<Embodiment 8; Moving object feature point information (center of gravity method)>
FIG. 18 relates to a method of inputting feature point candidate information 26'output from the moving body skeleton point determination unit 20 shown in FIG. 17A into the moving feature point determination unit 21 and acquiring moving body feature point information 28. (A) will be described with reference to FIG. 18 (b).

FIG. 18 (a) has already been described with reference to FIG. 16 (a) of the sixth embodiment. In the present embodiment 8 (center of gravity method), in the moving body feature point extraction unit 55, the moving body skeleton points 26x ₁ , 26x ₂ , 26x ₃ , 26x ₄ , 26x ₅ , and 26x ₆ are used as the feature point candidate information 26'as they are. .. The moving body feature point extraction unit 55 calculates and acquires the average coordinate position of the pixel coordinate positions of the moving body skeleton points 26x ₁ , 26x ₂ , 26x ₃ , 26x ₄ , 26x ₅ , and 26x ₆ , and outputs the moving body feature point information 28. To.

The average coordinate position is obtained by weighting the pixel coordinate positions of the moving body skeleton points 26x ₁ , 26x ₂ , 26x ₃ , 26x ₄ , 26x ₅ , and 26x ₆ (the number of pixels when acquiring the center of gravity; corresponding to the area of the area). However, it may be used as the moving object feature point information 28. In this calculation, in parallel with the calculation of the moving body skeleton point for each area, the position of the center of gravity can be obtained by acquiring the average coordinates of the pixel coordinate positions over the entire movement contour information 14', 14 "of the moving body. These moving body feature point acquisition methods are also called the center of gravity method.
The advantage of the center of gravity method is that there is no calculation on the scanning line in the Y direction, and the calculation proceeds along the scanning line in the X direction.

<Embodiment 9; Non-uniform velocity moving body skeleton point information (midpoint method)>
In the above-described embodiments 5, 6, 7, and 8, a method of acquiring skeleton point candidate information 27 and moving body feature point information 28 of an integral moving body moving at the same speed and in the same direction has been described, but other moving bodies are used. In some cases, the speed and direction may differ from place to place, such as the movement of the arm. This moving body is referred to as a non-uniform velocity moving body, and this case will be described with reference to FIGS. 19 (a), (b), (c), (d), and (e).

The movement of the human arm is shown in FIGS. 19 (a1), (a2), and (a3). Here, it is assumed that only the arm 57 held up moves as shown by the arrow. The movement is a reciprocating motion from side to side. The motion contour information 14'and 14 "for this motion are as shown in FIGS. 19 (b1) and 19 (b2). FIG. 19 (b1) is a diagram of FIGS. 19 (a1) and 19 (a2), and FIG. 19 (b2) is a diagram. It is shown in the example obtained by the frame difference of 19 (a2) and (a3), respectively. Here, the width (corresponding to the speed) of the movement contour information 14'and 14 "different depending on the location of the arm 57. Specifically, in FIG. 19 (b1), the combined speed V0 of Vx0 and Vy0 corresponding to the speeds in the X and Y directions is different between the tip portion (hand) and the base portion (shoulder portion) of the arm 57, and naturally the tip. The part is fast.

From the motion contour information 14'and 14 "shown in FIG. 19 (b1), the skeleton point candidate information 26 obtained at the pixel coordinate position of the midpoint of the outer width of the motion contour region is shown in FIG. 19 (c1) with a black dot. As shown in FIG. 15 (b1), the skeleton point candidate information 26 is acquired in each scanning line of the paired motion contour information 14'and 14'. In the figure, the broken line indicates the scanning line segment corresponding to the outer width (L'x) of the contour region on the scanning line in the X direction, and the black circle indicates the midpoint thereof. The coordinate position group of this black circle is the skeleton point candidate information 26x.

The method of extracting the moving skeleton point from the skeleton point candidate information 26x indicated by the black circle in FIG. 19 (c1) is the same as in FIG. 15 (b1), and both ends 26x ₁ , 26x as shown in FIG. 19 (d1). ₃ and the inflection point _26x2 are obtained.
In the case of FIG. 19 (c1), the moving body skeleton points 26x ₁ , 26x ₂ , and 26x ₃ are directly output as feature point candidate information 26'from the moving body skeleton point extraction unit 53 and input to the moving body feature point determination unit 22.

<Embodiment 10; Non-uniform velocity moving body feature point information (midpoint method)>
The method of inputting the feature point candidate information 26'output from the moving body skeleton point determination unit 20 to the moving feature point determination unit 21 and acquiring the moving body feature point information 28 will be described with reference to FIG. 19 (d1). ..

In the case of a non-uniform velocity moving body, the motion information at the place where the moving body velocity is the fastest is important as the feature point, and the moving body feature point extraction unit 55 has the same feature point candidate information 26x as the skeleton point of FIG. 19 (d1). Of ₁ , _26x2 and 26x3 _, the pixel coordinate position of _26x1 having the fastest speed is output as moving object feature point information 28 as shown in FIG. 19 (e1).

<Embodiment 2 of the image pickup apparatus>
Up to now, the image sensor 1 has been described as the image sensor as the first embodiment of the image sensor of the major category, but thereafter, the image sensor of the image sensor of the major category has been described as the second embodiment, and the solid state shown in FIG. 20 as the image sensor. The image sensor 3 "will be described.

In the image pickup camera 1 of FIG. 1, the image pickup information 8 output from the solid-state image pickup device 3 of FIG. 6 and the solid-state image pickup device 3'of FIG. After processing with, it was output to the outside as information output 31. On the other hand, the solid-state image sensor 3 "shown in FIG. 20 includes a pixel unit 58, a motion contour extraction unit 4, a motion information extraction unit 5, an information output unit 6, and a clock 7 inside. FIG. 20 is a block diagram thereof. This is referred to as the solid-state image pickup device 3 ", and is distinguished from the solid-state image pickup device 3 of FIG. 6 and the solid-state image pickup device 3'of FIG. The pixel unit 58 is composed of two types of pixels 43 having different exposure times as shown in FIGS. 9 or 10. The functions of the other blocks are the same as the functions of the blocks of the image pickup camera 1 of FIG. 1, and the description of each block will be omitted.

FIG. 21 is a detailed block configuration diagram of the solid-state image sensor 3 ". The solid-state image sensor 3" has a sensor chip (corresponding to a normal solid-state image sensor 3') and a pixel difference corresponding to a motion contour extraction unit 4. The processing unit 9', the motion contour determination unit 10, the moving object light / dark determination unit 11, the motion information extraction unit 5, the information output unit 6, and the clock 7 are incorporated and configured.

In the solid-state imaging device 3 ”of FIG. 21, the motion contour information 14 and the pixel difference code information 15 output from the pixel difference processing unit 9 ′ motion contour determination unit 10 and the moving object light / dark information 16 output from the moving object light / dark determination unit 11. Is input to the motion information extraction unit 5, and after acquiring information about the moving object, the information output unit 6 incorporates the time information of the clock 7 and outputs the information output 31 to the outside.

Explaining the pixel portion 58 of the solid-state image sensor 3 "in FIG. 21, a Bayer-arranged color filter is formed in each pixel for color imaging. In the figure, green (Gr, Gb), red (R), and red (R) are formed. Shown in blue (B).
Based on the master clock generated by the timing generation circuit 59, exposure control with pixels, vertical scanning, and horizontal scanning are performed. As an exposure control method, the exposure time TL and TS of the long-exposure pixel STL and the short-time exposure pixel STS described with reference to FIGS. 9 and 10 are generated by the exposure time generation circuit 60, and the vertical scanning circuit 61 is formed. The pixel unit 58 is controlled via the device.

In the vertical parallel control circuit 62, by turning on a plurality of pixel drive lines arranged in the vertical direction at the same time, pixel signals of the same color arranged vertically can be vertically combined (averaged). At the top of the pixel unit 58, the circuit that simultaneously reads the pixel signals of a plurality of vertical signal lines output from the pixel unit 58 is a horizontal scanning circuit 63 that scans the reading in the horizontal direction, and receives the vertical signal line and signals in the horizontal direction. It is composed of a horizontal synthesis circuit 64, which synthesizes the above. The horizontal composition circuit 64 can horizontally combine (average) pixel signals of the same color. This pixel composition of the same color is performed between pixels having the same exposure time, and corresponds to the short-time exposure pixel STS and the long-time exposure pixel STL, and becomes an image pickup STS and an image pickup STL.

As shown in FIGS. 9 and 10, the short-time exposure pixel STS and the long-time exposure pixel STL are alternately repeated for each scanning line, but the synthesis (averaging) of the pixel signals of the same color with the same exposure time is performed. Since it is performed with a plurality of scanning lines, long-time exposure information and short-time exposure pixel information in almost the same area can be acquired, and the pixel difference information 34'obtained by difference in the difference circuit 33 is compared in almost the same area. There is an advantage that the accuracy of motion information is improved.
This is because if the difference between the short-exposure pixel STS and the long-exposure pixel STL is taken for each pixel, there will be a problem (edge noise) that the difference will not be zero in the contour even in the case of a stationary object, but by averaging. Edge noise in the contour portion can be significantly suppressed, which is an advantage.

The synthesis (averaging) of pixel signals of the same color with the same exposure time depends on the number of pixels to be synthesized in the vertical and horizontal directions, but has the advantage that the amount of information can be significantly reduced. For example, the amount of information can be reduced to two digits (1/100) by averaging 10 pixels each in the vertical and horizontal directions.
In the conventional motion analysis, in order to process a huge amount of image information output from a solid-state image sensor in real time on a pixel-by-pixel basis, a large image processing load is required in the post-stage processing circuit, and it is difficult to drive the battery.
In this method, the amount of information corresponding to the number of pixels is significantly reduced (two digits in the above example), and by processing the binarized motion contour information 14, the amount of information in pixel units is also significantly reduced (for example, 8 bits). ⇒ 1 bit) is also a big advantage.

Since it is sufficient to process the binarized data in which the amount of pixel number information is significantly reduced (for example, 2 digits), the calculation load in the motion information extraction unit 5 can be lightened. Therefore, it is possible to output the information output 31 from the solid-state image sensor 3 "of the present application at the speed (frame rate) at which the image information is output from the conventional solid-state image sensor.

<Embodiment 2-1 of the image pickup apparatus>
As the image pickup device of the second embodiment of the major classification image pickup device, there is also a form of the solid-state image pickup device 3 "'shown in FIG. 22. The difference from the solid-state image pickup device 3" of FIG. A readout circuit and an output circuit for performing color imaging are added, and a function for determining whether to output video based on information output is also added. A different configuration operation added to the second embodiment of the solid-state image sensor 3 of FIG. 21 will be described with reference to FIG.

In the solid-state image sensor 3 "'" of FIG. 22, a column type noise canceling circuit (CDS) 65 and a column type in which pixel signals of a plurality of vertical signal lines output from the pixel unit 58 are input to the lower part of the pixel unit 58. An analog-to-digital converter circuit (AD conversion) 66, a line memory 67, and a horizontal pixel scanning circuit 68 are added. Multiple line pixels of a Bayer arrangement (2 × 2 pixels) (for this reason, a line memory 67 is required). Based on this, the color processing is performed by the color processing unit 69.

Further, based on the information from the information output unit 6, the information output determination unit 70 determines whether or not to output the color video information 71, and when outputting a video signal, the horizontal pixel scanning circuit 68 and the color processing unit 69. Is driven, and the color video information 71 determined to be necessary is added to the information output 31 by the addition addition circuit 72, and is output from the video output circuit 73. The color video information 71 may be a still image or a moving image. In the case of a moving image, the information output 31 may be superimposed during the vertical blanking period.

In FIG. 22, when the information output determination unit 70 determines to output the color image information 71, a signal for driving the horizontal pixel scanning circuit 68 and the color processing unit 69 is output. The horizontal pixel scanning circuit 68 and the color processing unit 69 may be always driven, although the power consumption increases. In this case, the color video information 71 determined to be necessary by the addition addition circuit 72 is added and added, and the color video information 71 is output from the video output circuit 73.

FIG. 23 shows a solid-state image sensor 3 "'in the case where the information output determination unit 70 is omitted and the color image information 71 is always output independently of the information output 31. Both the power consumption and the information amount are conventional. The information output 31 is added as in the case of the solid-state image sensor.

As mentioned in [Paragraph 46], the features of the embodiments described above can be used in combination with each other. That is, as the information output of the moving object, it is not necessary that the time information, the speed, the direction, the size, the feature point coordinate information, and the skeleton point coordinate information used in the above explanation are prepared, and the information output is independent information. It may be a combination of multiple pieces of information. Further, the feature point information and the skeleton point information do not have to be coordinate information, but may be a quantity. That is, as the feature point information, in addition to the coordinate information of each moving object, the quantity of feature points also becomes the feature point information, which corresponds to the number of moving objects.

For the purpose of grasping the congestion status of the road, the number of moving objects corresponds to the number of cars on the screen. In this case, the number of vehicles is usually about 5 bits (32 units), and if there is a clock on the information receiving side of the image pickup device of the present application, the time stamper can be performed on the information receiving side, so that the image is output from the image pickup device of the present application. The amount of information is 1 byte or less.
By counting the number of moving objects linked to the direction, it is possible to count the number of vehicles in the up lane and the down lane on the highway. This amount of information can be reduced to 8 bits (256) or less, and an image pickup device having an information output amount of 1 byte can be realized.

<Embodiment 11; Motion contour information>
In the above description, motion contour information is extracted using two images having different shooting times, and the direction of the moving object is determined as described in FIGS. 12 (a) to 12 (d). In addition to the motion contour information 14, the moving body direction extracting unit 46 needs pixel difference code information 15 and moving body light / dark information 16, and the moving body direction extracting unit 46 is based on the moving body direction determination table shown in FIG. 12 (d). It was necessary to determine the moving direction of the moving object.
Next, a method of extracting the moving object direction without the pixel difference code information 15 and the moving object light / dark information 16 by using three images having different shooting times will be described.

Before extracting the motion direction, a method of acquiring motion contour information that changes a little will be described. The motion contour extraction unit 4'in the embodiment 11 is as shown in FIG. 24, and the motion contour extraction unit 4 of FIG. 16 can be omitted, and the configuration is simplified.
As for the motion contour information of the eleventh embodiment, as in the first embodiment, as a method of acquiring the contour information, a method of using a frame difference using a frame memory and a difference between pixels having different exposure times without using the frame memory. Since there are two types of methods using the above, the former is referred to as the embodiment 11-1, and the latter is referred to as the embodiment 11-2. In the following description, the same description as in the first and second embodiments will be omitted.

<Embodiment 11-1; Motion contour information; Frame memory used>
First, the pixel difference processing unit 9'acquires the pixel difference information 34'from the image pickup information 8 shown in FIG. 24, and the motion contour determination unit 10'acquires the motion contour information 14 from the pixel difference information 34'. The method of doing the above will be described with reference to FIGS. 25 (a) and 25 (b).

FIG. 25A shows a case where the frame memory 32 is used, and the configuration of the pixel difference processing FIG. 9'is the same as that of the pixel difference processing FIG. 9 of FIG. As for the configuration of the motion contour determination unit 10', the frame memory 32'and the composition unit 74 are added to the motion contour determination unit 10 of FIG. The combined motion contour information 14+ is output from the synthesizing unit 74.

A method of acquiring motion contour information 14+ will be described with reference to FIG. 25 (b). The continuous frame images (imaging information 8) acquired by the solid-state image sensor 3 are referred to as frames 1 to 6. As the processing sequence, the frame 1 is stored in the frame memory 32, and the difference circuit 33 acquires the pixel difference information 34 by taking the difference from the current captured image; frame 2 for each pixel. Here, the frame 1 is subtracted from the frame 2, and it is expressed as pixel difference information 2-1. Similarly, the frame 2 is sequentially stored in the frame memory 32, and at the next timing, the difference circuit 33 takes the difference from the next captured image; frame 3 and acquires the pixel difference information 3-2.

The pixel difference information 2-1 passes through the absolute value circuit 35 of the motion contour determination unit 10', and stores the motion contour information 14 acquired in comparison with the threshold value 12 by the motion contour determination circuit 36 in the frame memory 32'. The motion contour information 14 is based on the pixel difference information 2-1 and is referred to as motion contour information (2-1) for convenience of explanation.
At the next timing, the pixel difference information 3-2 comes out as the motion contour information (3-2) via the absolute value circuit 35 and the motion contour determination circuit 36. Here, the motion contour information is generated by the synthesis unit 74. (2-1), the motion contour information 14+ that combines the motion contour information (3-2) is acquired.

The specific case of the subject shown in FIG. 7 (a0) will be described with reference to FIGS. 26 (a1) to 26 (a5). As specific captured images, the current image corresponds to the frame 3, the image one frame before corresponds to the frame 2, and the image two frames before corresponds to the frame 1, and corresponds to the moving body images 39, 39', 39 ”, respectively. Images 39, 39', 39 "are slightly displaced as shown in FIG. 26 (a1) due to the difference in time of each frame.

FIG. 26 (a2) is the pixel difference information (2-1) obtained from the frame 2 and the frame 1. Pixel difference information 15', 15 "with different symbols is output at a location corresponding to the end of the moving body image 39', 39". The pixel difference information 15', 15 "becomes absolute value information via the absolute value circuit 35, is compared with the threshold value 12 by the motion contour determination circuit 36, and becomes a binarized" 1 "shown in FIG. 26 (a3). The motion contour information (2-1) indicated by the corresponding 14'and 14'is obtained.
FIG. 26 (a4) shows the pixel difference information (3-2) obtained from the frame 3 and the frame 2, and the pixel difference information 15', 15'with different codes at the locations corresponding to the end portions of the moving body images 39, 39'. Is output. Pixel difference information 15', 15 "becomes absolute value information, is compared with the threshold value 12, and is indicated by 14', 14" corresponding to the binarized "1" shown in FIG. 26 (a5). The motion contour information (3-2) is obtained in the same steps.

The motion contour information (2-1) is stored in the frame memory 32'of the motion contour determination unit 10'in FIG. 25 (a), and the motion contour information (3-2) comes out at the next timing. Here, the compositing unit 74 synthesizes the motion contour information (2-1) and the motion contour information (3-2) corresponding to the same location of the frames 1, 2, and 3, and acquires the motion contour information 14+.
The result synthesized by the synthesis unit 74 is shown in FIG. 26 (b). Corresponds to 0 or 1 of motion contour information (2-1) and motion contour information (3-2), and motion contour information 14+ corresponds to (0,0), (1,0), (0,1) at each location. ) And (1,1).

The plane pattern of the four classified motion contour information 14+ is shown in FIG. 26 (c). Each region is classified into four categories (0,0), (1,0), (0,1), and (1,1). Here, the regions (0,0), (1,0), (0,1), and (1,1) correspond to no movement → nothing, existence → nothing, nothing → existence, and existence → existence, respectively.
Here, (1,0) is the rear end of the moving body, and (0,1) is the tip of the moving body, so that the direction of the moving body can be known. It does not matter whether the moving object as shown in FIG. 12 (d) is brighter or darker than the background.
Also, the widths of regions (1,0), (0,1) are related to velocity and correspond to the distance traveled in time between frames. When the velocity is decomposed in the X direction and the Y direction, the angle component of the direction can be obtained by the method shown in FIG. 12 (e).
The external dimensions of the motion contour information are related to the size of the moving object. Since the external size includes the speed, the size can be obtained by subtracting the speed.

FIG. 27 is a diagram showing blocks constituting the motion information extraction unit 5'that extracts motion information using the contour information 14+ obtained in FIGS. 25 (a) and 25 (b) and the flow of information related to them. Is. Compared with FIG. 3, a part of the input and the feedback line can be omitted.
The method of obtaining the direction, speed, and size is the same as that of FIGS. 12 (e), 13 (b), (d), and 14 (b). The method for obtaining skeletal points and feature points is also the same.

In FIG. 25B, when the pixel difference information is acquired, the difference between consecutive frames is taken, but even if each difference is acquired between the thinned frames; for example, three frames 1, 3 and 5. good. It is also possible to perform finer movement grasping by performing with 4 or more frames.

<Embodiment 11-2; Motion contour information; No frame memory used>
28 (a), (b), and (c) show a method of acquiring motion contour information of a case where the frame memory is not used. As shown in FIG. 28A, the solid-state image sensor used has a configuration in which pixels 43 having three different types of exposure times are arranged. Pixels having these three types of exposure times are represented by short-time exposure pixels STS, medium-time exposure pixels STM, and long-time exposure pixels STL.
Although not explained, there is also an advanced form that performs detailed motion analysis with pixels having four or more types of exposure times.

As for the pixel output characteristics, as shown in FIG. 28 (b), the three types of pixels (STS, STM, STL) all show the case of pixels having the same sensitivity. Pixel output corresponding to the exposure time (TS, _TM , T _{L )} is obtained, and is output as image pickup information 8. Here, assuming that pixels with three types of exposure times are output at the same timing (simultaneous reading of three scanning lines), a case will be described in which the line memory required in FIG. 9C is not used.

FIG. 28 (c) shows a pixel difference processing unit 9'and a motion contour determination unit 10', which are components of the motion contour extraction unit 4. As a function of the pixel difference processing unit 9', the imaging information 8'from three types of pixels (STS, STM, STL) having different exposure times passes through the separation circuit 37', and the respective imaging information (imaging STS, imaging STM) is performed. , Imaging STL). The amplifier circuit 38'corrects the difference in each exposure time, the difference circuit 33'takes the difference, and the image pickup STS-imaging STM is the pixel difference information 34', and the image pickup STM-imaging STL is the pixel difference information 34'.

As a function of the motion contour determination unit 10'in FIG. 28 (c), the pixel difference information 34'and 34 "becomes absolute value information via the absolute value circuit 35', and the threshold value 12'in the motion contour determination circuit 36'. The motion contour information 14'and 14 "are obtained, respectively. These are simultaneously input to the synthesis unit 74'. In the synthesis unit 74', both are synthesized, and the result of the synthesis as shown in FIG. 28 (d) is obtained. Based on the motion contour information 14'and the motion contour information 14', the area corresponds to 0 or 1, and in the motion contour information 14+, each place is (0,0), (1,0), (0,1). , (1,1) are classified into four categories.

In the 11-2 embodiment, the motion contour information 14+ in FIG. 28D can be obtained by simultaneously reading out three adjacent scanning lines in which three types of pixels having different exposure times are arranged without using a frame memory. It can be acquired in real time and output in conjunction with the reading of scanning lines by a solid-state image sensor.
This motion contour information 14+ includes information on the direction, velocity, size, skeleton point, and feature point, which are the main information of the moving object, and the extraction method does not require complicated calculation as described above. Is.

That is, the tip portion in the moving direction of the moving body only displays the area (0, 1), and the velocity of the moving body only needs to be converted into the width of the movement contour information. The size of the moving body can be replaced by the size of the movement contour information. In addition, the feature points need only be averaged of the coordinate points of the motion contour information. That is, the above information can be acquired together with the reading scan of the scanning line by the solid-state image sensor. In the conventional method, it is necessary to perform complicated arithmetic processing in the post-stage processing circuit after capturing the captured image of a huge amount of image data, but the arithmetic is completed in parallel with the reading of the scanning line. When pixel signals are combined (averaging) to reduce noise, the number of pixels is reduced and the calculation becomes easier.
Therefore, as shown in FIG. 21, the above-mentioned processing circuit of the present application can be mounted on the same substrate as the solid-state imaging device 3 "', and the system can be greatly simplified and the power consumption can be reduced.

The image pickup device of the present application had a clock, but if the system outside the image pickup device has a clock function, the time function of the external system is used to output information, and the inside of the image pickup device of the present application is used. It is possible to omit the clock.
What is the clock function of the system outside the image pickup device? When the image pickup device of the present application is controlled by an external synchronization pulse, the delay time from the synchronization pulse can be substituted.

If a memory for temporarily storing information output is added to the image pickup apparatus of the present application, it is possible to temporarily store moving object information including time information in the memory and output memory data at regular time intervals.

In the image pickup apparatus of the present application, the optical imaging means using the image pickup lens 2 has been described, but the optical image formation means may use a method used for a pinhole camera (see Patent Publication No. 6051399) in addition to the image pickup lens. good.

1 Imaging camera 2 Imaging lens 3, 3', 3 "3", 3 "'Solid imager 4 Motion contour extraction unit 5 Motion information extraction unit 6 Information output unit 7 Clock 8 Imaging information 9, 9'Pixel difference processing unit 10, 10 'Motion contour determination unit 11 Motion light / dark determination unit 12 Threshold 13 Motion determination information 14, 14', 14', 14 + Motion contour information 15, 15', 15' Pixel difference sign information 16 Motion brightness information 17 Motion speed determination unit 18 Moving body direction judgment unit 19 Moving body size judgment unit 20 Moving body skeleton point judgment unit 21 Moving body feature point judgment unit 22 Moving body speed information 23 Moving body direction intermediate information 24 Moving body direction information 25 Moving body size information 26 Skeletal point candidate information 26'Characteristic point candidate information 27 Moving body skeleton point information 28 Moving body feature point information 29 Information synthesis / output unit 30 Time information 31 Information output 32, 32'Frame memory 33, 33'Difference circuit 34, 34'Pixel difference information 35, 35'Absolute value circuit 36, 36 'Movement contour determination circuit 37 Separation circuit 38 Amplifier circuit 39, 39' Moving object image 40 Scanning line 41 Center line 42 Road 43 Pixel 44 Line memory 45 Delay circuit 46 Moving object direction extraction unit 47 Moving object direction calculation unit 48 Moving object speed extraction unit 49 Moving object Speed calculation unit 50 Conversion table 51 Dynamic body size extraction unit 52 Dynamic body size calculation unit 53 Dynamic skeleton point extraction unit 54 Dynamic skeleton point synthesis unit 55 Dynamic feature point extraction unit 56 Scanning line 57 Arm 58 Pixel unit 59 Timing generation circuit 60 Exposure time generation Circuit 61 Vertical scanning circuit 62 Vertical parallel control circuit 63, 63'Horizontal scanning circuit 64 Horizontal synthesis circuit 65 CDS circuit (column type noise canceling circuit)
66 ADC circuit (column type analog-to-digital converter circuit)
67 Line memory 68 Horizontal pixel scanning circuit 69 Color processing unit 70 Information output judgment unit 71 Color video information 72 Addition addition circuit 73 Video output circuit 74, 74'Synthesis unit

In FIG. 13 (b1), the motion contour information 14'(front end) and 14 "(rear end) corresponding to FIG. 12 (b3) are shown in a planar image. The moving object direction intermediate information 23 is this. It is used to determine the selection of paired motion contour information 14'(front end) and 14 "(rear end). Further, FIG. 13 (b2) shows motion contour information 14 on the scanning line as shown in FIG. 7 (b4), and motion contour information 14'(tip), 14 "(rear) shown in FIG. 13 (b1). The output of the end) is shown. The moving body direction in the moving contour region is known from the moving body direction intermediate information 23 output from the moving body direction extraction unit 46.
In the moving body speed extraction unit 48, the width of the movement contour region (corresponding to the number of pixels) in the movement contour information 14', 14 "is obtained in the X direction and the Y direction. It is expressed as "x, and V'y, V" y. This width corresponds to the width (number of pixels) in which the moving object moves in one frame period.
In the above description, the moving speed is acquired while the direction of the moving body is grasped by using the moving body direction intermediate information 23, but the moving object direction intermediate information 23 is not always necessary for acquiring the moving speed. ..

In the moving body size extracting unit 51, the outer width of the movement contour region in the movement contour information 14', 14 "is obtained in the X direction and the Y direction. This outer width is on the same scanning line in the X direction. The outer width is obtained so as to cover both the motion contour information 14'and 14 ", which is called the motion size pixel number and is expressed by L'x. The external dimensions of either the motion contour information 14'or 14 "are excluded. As shown in FIG. 14 (b2), the motion size pixel number L'x in the X direction is such that the moving object moves in one frame period. Corresponding to the area width (number of pixels) to be covered, the speed pixels V'x0 and V "x0 are attached to both ends. Therefore, the actual number of motion size pixels is obtained by subtracting the number of velocity pixels V'x0 and V "x0 from the number of motion size pixels L'x. The Y direction is the same as the X direction.
In the above description, the motion size is acquired while the direction of the moving object is grasped by using the moving object direction information 24, but the moving object direction information 24 is not always necessary for acquiring the motion size.

Claims (18)

A pixel unit having a plurality of photoelectric conversion elements that convert an optical signal imaged by an optical imaging means into an electric signal,
A means for generating a difference signal between at least two shooting signals with different shooting times shot by the pixel unit and extracting contour information of a moving object based on the difference signal.
A means for extracting information on a moving object that is not linked to a pixel position based on the contour information of the extracted moving object,
An information output means for synthesizing and outputting information on a moving object,
An imaging device characterized by having.
As a method of generating a difference signal between at least two shooting signals with different shooting times taken by the pixel unit, a frame memory is used, and the image difference between frames with different shooting times is performed. The imaging device according to claim 1.
As a method of generating a difference signal between at least two shooting signals with different imaging times taken by the pixel portion, the exposure time is controlled by dividing the pixels arranged in two dimensions into at least two and making the length of the exposure time different. The image pickup apparatus according to claim 1, wherein the image pickup apparatus is realized by a control means and a difference signal generation means for generating a difference signal between the pixel signals having different exposure times.
As a means for extracting information on a moving object based on the contour information of the extracted moving object, information on the direction of movement is obtained from pixel difference code information which is the basis of the contour information and light / dark information on the background of the moving object. The image pickup apparatus according to claim 1, wherein the image is extracted.
As a means for extracting information on a moving object based on the extracted contour information of the moving object, it is characterized in that information on the direction of movement is extracted from the contour information of the moving object and the direction information of the moving object. The imaging device according to claim 1.
As a means for extracting information on a moving object based on the extracted contour information of the moving object, information on the motion size is obtained from the contour information of the moving object, the direction information of the moving object, and the velocity information of the moving object. The imaging device according to claim 1, wherein the image is extracted.
As a means for extracting information on a moving object based on the extracted contour information of the moving object, the skeleton of the moving object is obtained from the contour information of the moving object and the size information of the moving object, or only from the contour information of the moving object. The imaging device according to claim 1, wherein the point information is extracted.
As a means for extracting information on a moving object based on the extracted contour information of the moving object, the characteristics of the moving object are obtained from the contour information of the moving object and the size information of the moving object, or only from the contour information of the moving object. The imaging device according to claim 1, wherein the point information is extracted.
The imaging according to claim 1, wherein as a means for extracting information on a moving object based on the contour information of the extracted moving object, information on the time of the moving object is extracted from a clock in an image pickup apparatus. Device.
As the information output means for synthesizing and outputting the information of the moving object, the information of the moving object alone or in any combination including all of the information of the moving object described in claims 4 to 9 is selected. The image pickup apparatus according to claim 1, wherein the information is synthesized and output from the image pickup apparatus.
The imaging device according to claim 7, wherein the information on the skeleton points of the moving object is the coordinate position of the skeleton points.
The image pickup apparatus according to claim 8, wherein the information on the feature points of the moving object is the coordinate position of the feature points.
The imaging device according to claim 8, wherein the information on the feature points of the moving object is the number of feature points.
The image pickup device according to claim 1, wherein the image pickup device is an image pickup camera.
The image pickup device according to claim 1, wherein the image pickup device is a solid-state image pickup device.
The image pickup apparatus according to claim 1, wherein a means for outputting a captured image is added in addition to an information output means for synthesizing and outputting information on a moving object.
The imaging device according to claim 1, wherein the optical imaging means is an imaging lens.
The imaging device according to claim 1, wherein the optical imaging means is a pinhole camera system.

Images