JP4992462B2 - Imaging device, imaging apparatus, imaging system, imaging method, motion data generation system, motion data generation program, and motion data generation method - Google Patents

Description

  The present invention relates to an imaging device capable of reading out charges from a photoelectric conversion element by using both a destructive readout method and a non-destructive readout method, and a motion data generation system using the imaging device.

2. Description of the Related Art Conventionally, in the field of movies, games, and the like that require CG (Computer Graphics) characters with realistic motion, for example, motion capture that captures human physical movements into a computer is often used. There are several methods of motion capture, but the relatively popular method is optical. In the optical type, a target object (such as a human) with a marker is photographed by an imaging device (camera), the image data obtained by this photographing is image-processed to detect the position of the marker, and the motion (motion of the target object) This is a method for acquiring (generating) data. In the optical system, an imaging device (camera) having a high frame rate is used. This is because if the frame rate is high, the amount of movement of the marker per frame is small, and the accuracy of position detection is improved. However, when a camera with a high frame rate is used, the exposure time is shortened, so that the amount of light is insufficient and it is difficult to detect the position of the marker from the captured image. Therefore, when capturing the motion of the target object in the motion capture device, for example, an auxiliary light device (for example, a strobe) and a marker that reflects light are combined to prevent the amount of light from becoming insufficient (see Patent Document 1). .
JP 2003-106812 A

  However, in the prior art disclosed in Patent Document 1, if the target object is imaged at a high frame rate in order to increase the accuracy of position detection, the target object is captured even if a marker with a large amount of light can be accurately captured. It is impossible to photograph the appearance and background of the camera accurately (finely). For this reason, when images such as the detailed appearance and background of the target object are required together with the motion data of the target object, the target is separately set at a frame rate (normal frame rate) that provides a detailed appearance and background image of the target object. It is necessary to shoot an object.

  Therefore, the present invention has been made paying attention to such an unsolved problem of the conventional technology, and is an image having a normal frame rate corresponding to a predetermined exposure time and a plurality of types of images having a predetermined exposure time or less. Imaging device, imaging device, imaging system and imaging method capable of simultaneously imaging high-speed frame rate image corresponding to exposure time, motion data generation system using the imaging device, imaging system or imaging method, motion data generation program, and The object is to provide a method for generating motion data.

[Embodiment 1] In order to achieve the above object, an image pickup device according to Embodiment 1 includes:
A plurality of photoelectric conversion elements that convert the exposed light into electric charges and store the photoelectric conversion units arranged in a matrix; and
First readout means for scanning each pixel constituting the photoelectric conversion element, and reading out the charge when exposed for a predetermined exposure time from each scanned pixel by a destructive readout method with reset of accumulated charge;
During the predetermined exposure time, scanning is performed a plurality of times for each pixel constituting the photoelectric conversion element, and a plurality of types of exposure times equal to or less than the predetermined exposure time from each pixel in each scanning. A second reading means for reading out charges when exposed by a non-destructive reading method without resetting accumulated charges;
A first output that is read out by the first reading means and that sequentially stores pixel data composed of charges when exposed for the predetermined exposure time in a memory, and sequentially outputs the pixel data stored in the memory in the order of storage. Channel,
A second output channel for sequentially storing pixel data read by the second reading means in a memory for each scan in the plurality of scans, and sequentially outputting the pixel data stored in the memory in the stored order; Prepared,
The second reading unit is characterized in that the plurality of scans are performed on the remaining pixels obtained by thinning a predetermined pixel from a pixel group formed by the plurality of photoelectric conversion elements.

  With such a configuration, the first readout means scans each pixel constituting the photoelectric conversion element, and reads out the charge when exposed from the scanned pixel for a predetermined exposure time. The destructive readout method with reset can be performed, and the second readout means scans each pixel constituting the photoelectric conversion element a plurality of times during the predetermined exposure time, It is possible to read out charges from the respective pixels in scanning when exposure is performed for a plurality of types of exposure times less than or equal to the predetermined exposure time by a nondestructive readout method without resetting accumulated charges.

  Further, pixel data composed of charges when exposed at the predetermined exposure time, which is read by the first reading means, is sequentially stored in the memory by the first output channel, and the pixel data stored in the memory is stored in the memory. The pixel data read by the second reading means is sequentially stored in the memory for each scan in the plurality of scans and stored in the memory by the second output channel. Pixel data can be sequentially output in the stored order.

Still further, the second reading means can perform the plurality of scans on the remaining pixels obtained by thinning a predetermined pixel from a pixel group constituting the plurality of photoelectric conversion elements.
That is, the first readout unit performs destructive readout of charges from each pixel of the photoelectric conversion unit by exposure for a predetermined exposure time, while the second readout unit performs the exposure for a predetermined exposure time. The remaining pixels obtained by thinning out the predetermined pixels among the pixels of the photoelectric conversion unit are scanned a plurality of times, and the non-destructive readout of the charges of the pixels exposed at a plurality of types of exposure times less than the predetermined exposure time is performed. It can be carried out. Further, the pixel data when the exposure is performed with the predetermined exposure time scanned and read by the first reading means is output from the first output channel, and the exposure data is a plurality of types of exposure times less than the predetermined exposure time scanned and read by the second reading means. Since pixel data at the time of exposure is output from the second output channel, pixel data read by the destructive readout method and pixel data read by the non-destructive read method can be output independently.

  In the second reading means, since the remaining pixels obtained by thinning out the predetermined pixels are scanned, scanning can be performed at a higher speed than when scanning is performed for all the pixels. Furthermore, the readout of charges from each pixel is performed by a nondestructive readout method, and scanning is performed a plurality of times with a plurality of types of exposure times less than or equal to a predetermined exposure time during the exposure period of a predetermined exposure time. Charges can be read out at various frame rates. As a result, it is possible to follow the high-speed movement of the subject and capture a subject image during operation (in the middle of position displacement) that cannot be captured within a predetermined exposure time. Further, by reading out charges by the non-destructive readout method, it is possible to maintain the accumulated charge amount of each pixel even if scanning is performed a plurality of times during the exposure period of a predetermined exposure time.

  For example, when imaging an imaging target composed of a relatively bright and moving subject (for example, a motion capture marker) and a relatively low (dark) background, read by the first reading means A plurality of scans that can obtain a clear image in which the subject and the background are exposed with a sufficient exposure time from pixel data composed of charges at the time of exposure with a predetermined exposure time, and that are read by the second reading means. From the pixel data composed of charges at the time of exposure with a plurality of types of exposure times less than or equal to the predetermined exposure time, a plurality of images that follow the movement of the subject and accurately capture the movement of the subject can be obtained.

That is, there is an effect that a captured image in which the imaging target is clearly captured and a plurality of captured images in which the movement of the subject is accurately captured can be obtained simultaneously in the exposure period of the predetermined exposure time.
Further, by changing the number of pixels to be thinned out, an effect that the frame rate can be easily changed (speeded up) can be obtained.

  Here, the “photoelectric conversion unit” is configured using, for example, CMOS technology, and a non-destructive image sensor that uses CMOS technology is a threshold modulation type image sensor (for example, VMIS (Threshold Voltage). Modulation Image Sensor)). Hereinafter, the same applies to the imaging device of mode 2, the imaging devices of modes 3 and 6, the imaging system of modes 13 and 14, and the imaging methods of modes 15 and 16.

  In addition, the “destructive reading and reading method” empties the charge accumulated in the photoelectric conversion element when reading out the charge (pixel signal) from the photoelectric conversion element constituting the image pickup element (photoelectric conversion unit) of the image pickup apparatus. This is accompanied by reset processing. Hereinafter, the same applies to the imaging device of mode 2, the imaging devices of modes 3 and 6, the imaging system of modes 13 and 14, and the imaging methods of modes 15 and 16.

  Further, the “non-destructive readout method” is to read out charges (pixel signals) from the photoelectric conversion elements while keeping the accumulated state without emptying the charges accumulated in the photoelectric conversion elements. . In other words, since no reset process is involved when reading the charge, the charge can be read any number of times during the charge accumulation until the set exposure time is reached. Hereinafter, the same applies to the imaging device of mode 2, the imaging devices of modes 3 and 6, the imaging system of modes 13 and 14, and the imaging methods of modes 15 and 16.

[Embodiment 2] In order to achieve the above object, an image pickup device according to Embodiment 2 includes:
A plurality of photoelectric conversion elements that convert the exposed light into electric charges and store the photoelectric conversion units arranged in a matrix; and
First readout means for scanning each pixel constituting the photoelectric conversion element, and reading out the charge when exposed for a predetermined exposure time from each scanned pixel by a destructive readout method with reset of accumulated charge;
A plurality of scans performed on each pixel are divided into N groups in a predetermined combination, and the divided scans are divided into the same time during the predetermined exposure time. A non-destructive readout method in which charges are read out from each pixel in each scan during exposure at a plurality of types of exposure times less than or equal to the predetermined exposure time without resetting accumulated charges. Second reading means performed in
A first output that is read out by the first reading means and that sequentially stores pixel data composed of charges when exposed for the predetermined exposure time in a memory, and sequentially outputs the pixel data stored in the memory in the order of storage. Channel,
The pixel data read by the second reading means is sequentially stored in the memory for each scan, and the pixel data stored in the memory is sequentially output in the stored order. The N pieces are independent for each group (N is 2). The second to (N + 1) th output channels of the above natural number),
The second reading unit is characterized in that the scanning is performed on the remaining pixels obtained by thinning a predetermined pixel from a pixel group formed by the plurality of photoelectric conversion elements.

  With such a configuration, the first readout means scans each pixel constituting the photoelectric conversion element, and reads out the charge when exposed from the scanned pixel for a predetermined exposure time. The destructive readout method with reset can be performed, and the second readout means scans each pixel constituting the photoelectric conversion element a plurality of times during the predetermined exposure time, It is possible to read out charges from the respective pixels in scanning when exposure is performed for a plurality of types of exposure times less than or equal to the predetermined exposure time by a nondestructive readout method without resetting accumulated charges.

  Further, pixel data composed of charges when exposed at the predetermined exposure time, which is read by the first reading means, is sequentially stored in the memory by the first output channel, and the pixel data stored in the memory is stored in the memory. The pixel data read by the second reading means are sequentially stored in the memory for each scan by the second to (N + 1) output channels independent for each set. The pixel data stored in the memory can be sequentially output in the stored order.

Still further, the second reading means can perform the respective scans on the remaining pixels obtained by thinning a predetermined pixel from a pixel group formed by the plurality of photoelectric conversion elements.
That is, the first readout unit performs destructive readout of charges from each pixel of the photoelectric conversion unit by exposure for a predetermined exposure time, while the second readout unit performs the exposure for a predetermined exposure time. The remaining pixels obtained by thinning out the predetermined pixels among the pixels of the photoelectric conversion unit are scanned for each group, and each pixel is exposed independently for each group with a plurality of types of exposure times equal to or shorter than the predetermined exposure time. The non-destructive readout of the charges can be performed. Further, the pixel data when the exposure is performed with a predetermined exposure time scanned and read by the first reading means is output from the first output channel, and a plurality of types of data less than the predetermined exposure time of each set scanned and read by the second reading means. Since the pixel data at the time of exposure is output from the second to (N + 1) th output channels, the pixel data read by the destructive readout method and the pixel data of each set read by the non-destructive read method are used. Each can be output independently.

  In the second reading means, since the remaining pixels obtained by thinning out the predetermined pixels are scanned, scanning can be performed at a higher speed than when scanning is performed for all the pixels. Furthermore, the readout of charges from each pixel is performed by a nondestructive readout method, and scanning is performed a plurality of times with a plurality of types of exposure times less than or equal to a predetermined exposure time during the exposure period of a predetermined exposure time. Charges can be read out at various frame rates. As a result, it is possible to follow the high-speed movement of the subject and capture a subject image during operation (in the middle of position displacement) that cannot be captured within a predetermined exposure time. Further, by reading out charges by the non-destructive readout method, it is possible to maintain the accumulated charge amount of each pixel even if scanning is performed a plurality of times during the exposure period of a predetermined exposure time. Further, it is possible to easily change the frame rate by changing the number of pixels to be thinned out. Further, by physically increasing the number of output channels, it is possible to easily increase the number of scans, and thus it is possible to easily increase the number of times of readout of charges for the same pixel during a predetermined exposure period.

  For example, when imaging an imaging target composed of a relatively bright and moving subject (for example, a motion capture marker) and a relatively low (dark) background, read by the first reading means A plurality of scans that can obtain a clear image in which the subject and the background are exposed with a sufficient exposure time from pixel data composed of charges at the time of exposure with a predetermined exposure time, and that are read by the second reading means. From the pixel data composed of charges at the time of exposure with a plurality of types of exposure times less than or equal to the predetermined exposure time, a plurality of images that follow the movement of the subject and accurately capture the movement of the subject can be obtained.

That is, there is an effect that a captured image in which the imaging target is clearly captured and a plurality of captured images in which the movement of the subject is accurately captured can be obtained simultaneously in the exposure period of the predetermined exposure time.
Further, by increasing the number of thinned pixels and the number of output channels, it is possible to easily read out charges at a higher frame rate.

[Mode 3] On the other hand, in order to achieve the above object, an imaging apparatus according to mode 3
A plurality of photoelectric conversion elements that convert the exposed light into electric charges and store the photoelectric conversion units arranged in a matrix; and
First readout means for scanning each pixel constituting the photoelectric conversion element, and reading out the charge when exposed for a predetermined exposure time from each scanned pixel by a destructive readout method with reset of accumulated charge;
During the predetermined exposure time, scanning is performed a plurality of times for each pixel constituting the photoelectric conversion element, and a plurality of types of exposure times equal to or less than the predetermined exposure time from each pixel in each scanning. A second reading means for reading out charges when exposed by a non-destructive reading method without resetting accumulated charges;
The pixel data composed of the charges when exposed for the predetermined exposure time, read by the first reading means, are sequentially stored in the memory, and the pixel data stored in the memory are sequentially output in the stored order} An output channel,
Pixel data read by the second reading means is sequentially stored in a memory for each scan in the plurality of scans, and the pixel data stored in the memory is sequentially output in the stored order;
Normal captured image data generating means for generating captured image data when exposed for a predetermined exposure time based on the pixel data output from the first output channel;
High-speed captured image data generating means for generating captured image data for each scan in the plurality of scans based on pixel data output from the second output channel,
The second reading unit is characterized in that the plurality of scans are performed on the remaining pixels obtained by thinning a predetermined pixel from a pixel group formed by the plurality of photoelectric conversion elements.

  With such a configuration, the first readout means scans each pixel constituting the photoelectric conversion element, and reads out the charge when exposed from the scanned pixel for a predetermined exposure time. The destructive readout method with reset can be performed, and the second readout means scans each pixel constituting the photoelectric conversion element a plurality of times during the predetermined exposure time, It is possible to read out charges from the respective pixels in scanning when exposure is performed for a plurality of types of exposure times less than or equal to the predetermined exposure time by a nondestructive readout method without resetting accumulated charges.

  Further, pixel data composed of charges when exposed at the predetermined exposure time, which is read by the first reading means, is sequentially stored in the memory by the first output channel, and the pixel data stored in the memory is stored in the memory. The pixel data read by the second reading means is sequentially stored in the memory for each scan in the plurality of scans and stored in the memory by the second output channel. Pixel data can be output sequentially in the stored order, and captured image data at the time of exposure is generated with a predetermined exposure time based on the pixel data output from the first output channel by the normal captured image data generation means Images output from the second output channel by the high-speed captured image data generation means. Based on the data, the captured image data for each scan of the plurality of times of scanning can be generated.

Still further, the second reading means can perform the plurality of scans on the remaining pixels obtained by thinning a predetermined pixel from a pixel group constituting the plurality of photoelectric conversion elements.
That is, the first readout unit performs destructive readout of charges from each pixel of the photoelectric conversion unit by exposure for a predetermined exposure time, while the second readout unit performs the exposure for a predetermined exposure time. The remaining pixels obtained by thinning out the predetermined pixels among the pixels of the photoelectric conversion unit are scanned a plurality of times, and the non-destructive readout of the charges of the pixels exposed at a plurality of types of exposure times less than the predetermined exposure time is performed. It can be carried out. Further, pixel data at the time of exposure with a predetermined exposure time scanned and read by the first reading means is output from the first output channel, and exposure is performed with a plurality of types of exposure times less than or equal to the predetermined exposure time scanned and read by the second reading means. Since the pixel data at the time is output independently from the second output channel for each scan, the pixel data read by the destructive readout method and the pixel data read by the non-destructive read method are output independently. can do.

  In the second reading means, since the remaining pixels obtained by thinning out the predetermined pixels are scanned, scanning can be performed at a higher speed than when scanning is performed for all the pixels. Furthermore, the readout of charges from each pixel is performed by a nondestructive readout method, and scanning is performed a plurality of times with a plurality of types of exposure times less than or equal to a predetermined exposure time during the exposure period of a predetermined exposure time. Charges can be read out at various frame rates. As a result, it is possible to follow the high-speed movement of the subject and capture a subject image during operation (in the middle of position displacement) that cannot be captured within a predetermined exposure time. Further, by reading out charges by the non-destructive readout method, it is possible to maintain the accumulated charge amount of each pixel even if scanning is performed a plurality of times during the exposure period of a predetermined exposure time. Further, it is possible to easily change the frame rate by changing the number of pixels to be thinned out.

For example, when imaging an imaging target composed of a relatively bright and moving subject (for example, a motion capture marker) and a relatively low (dark) background, read by the first reading means Since the normal captured image data is generated from the pixel data composed of the charges at the time of exposure with a predetermined exposure time, it is possible to obtain a clear image in which the subject and the background are exposed with a sufficient exposure time. Further, high-speed captured image data for each scan is generated from the pixel data composed of charges at the time of exposure with a plurality of types of exposure times less than or equal to a predetermined exposure time by a plurality of scans read by the second readout means. As a result, it is possible to obtain a plurality of images that follow the movement of the subject and clearly capture the subject.
That is, there is an effect that a captured image in which the imaging target is clearly captured and a plurality of captured images in which the movement of the subject is accurately captured can be obtained simultaneously in the exposure period of the predetermined exposure time.

[Mode 4] Furthermore, the imaging device of mode 4 is the imaging device of mode 3,
The second output channel is configured to output the pixel data for each line of the pixels,
The second reading means is characterized in that the plurality of scans are performed on pixels of the remaining lines obtained by thinning out pixels of a predetermined line from the pixel group.
With such a configuration, since predetermined pixels can be thinned out in units of lines, it is easy to adjust the number of lines to be scanned, and the frame rate can be simplified by adjusting the number of lines of pixels to be scanned. The effect that it can be changed to is obtained.

[Mode 5] Further, the imaging device of mode 5 is the imaging device of mode 3 or 4,
The high-speed captured image data generation means includes a pixel value of pixel data output from the second output channel and one of the scans from the second output channel for each scan in the plurality of scans. A difference value from the pixel value of the pixel data output in the previous scan is calculated, and captured image data corresponding to each of the scans is generated based on the difference value.

With such a configuration, it is possible to calculate the difference value between the pixel data read during each scan in the plurality of scans and the pixel data read during the previous scan. It becomes possible to remove or reduce fixed pattern noise that is a problem in the nondestructive readout method from pixel data corresponding to each scan.
Therefore, since the captured image data can be generated from the pixel data from which the fixed pattern noise is removed or reduced, an effect that the image quality deterioration due to the fixed pattern noise can be prevented or reduced can be obtained.

  Here, “fixed pattern noise” includes dark current shading, which becomes a problem during long exposure, and noise generated due to a difference in sensor sensitivity for each pixel. Hereinafter, the same applies to the imaging device of mode 2, the imaging devices of modes 3 and 6, the imaging system of modes 13 and 14, and the imaging methods of modes 15 and 16.

[Mode 6] On the other hand, in order to achieve the above object, an imaging apparatus according to mode 6
A plurality of photoelectric conversion elements that convert the exposed light into electric charges and store the photoelectric conversion units arranged in a matrix; and
First readout means for scanning each pixel constituting the photoelectric conversion element, and reading out the charge when exposed for a predetermined exposure time from each scanned pixel by a destructive readout method with reset of accumulated charge;
A plurality of scans performed on each pixel are divided into N groups in a predetermined combination, and the divided scans are divided into the same time during the predetermined exposure time. A non-destructive readout method in which charges are read out from each pixel in each scan during exposure at a plurality of types of exposure times less than or equal to the predetermined exposure time without resetting accumulated charges. Second reading means performed in
A first output that is read out by the first reading means and that sequentially stores pixel data composed of charges when exposed for the predetermined exposure time in a memory, and sequentially outputs the pixel data stored in the memory in the order of storage. Channel,
The pixel data read by the second reading means is sequentially stored in the memory for each scan, and the pixel data stored in the memory is sequentially output in the stored order. The N pieces are independent for each group (N is 2). The second to (N + 1) th output channel of the above natural number),
Normal captured image data generating means for generating captured image data when exposed for a predetermined exposure time based on the pixel data output from the first output channel;
High-speed captured image data generation means for generating captured image data for each scan in the plurality of scans based on pixel data output from the second to (N + 1) th output channels,
The second reading unit is characterized in that the scanning is performed on the remaining pixels obtained by thinning a predetermined pixel from a pixel group formed by the plurality of photoelectric conversion elements.

  With such a configuration, the first readout means scans each pixel constituting the photoelectric conversion element, and reads out the charge when exposed from the scanned pixel for a predetermined exposure time. The destructive readout method with reset can be performed, and the second readout means scans each pixel constituting the photoelectric conversion element a plurality of times during the predetermined exposure time, It is possible to read out charges from the respective pixels in scanning when exposure is performed for a plurality of types of exposure times less than or equal to the predetermined exposure time by a nondestructive readout method without resetting accumulated charges.

  Further, pixel data composed of charges when exposed at the predetermined exposure time, which is read by the first reading means, is sequentially stored in the memory by the first output channel, and the pixel data stored in the memory is stored in the memory. The pixel data read by the second reading means are sequentially stored in the memory for each scan by the second to (N + 1) output channels independent for each set. It is possible to sequentially output the pixel data stored in the memory in the order in which they are stored, and the normal captured image data generating means generates a predetermined exposure time based on the pixel data output from the first output channel. Captured image data can be generated, and the second to (N + 1) th output channel is generated by the high-speed captured image data generation means. Based on the pixel data output from the captured image data for each scan of the plurality of times of scanning can be generated.

Still further, the second reading means can perform the respective scans on the remaining pixels obtained by thinning a predetermined pixel from a pixel group formed by the plurality of photoelectric conversion elements.
That is, the first readout unit performs destructive readout of charges from each pixel of the photoelectric conversion unit by exposure for a predetermined exposure time, while the second readout unit performs the exposure for a predetermined exposure time. The remaining pixels obtained by thinning out the predetermined pixels among the pixels of the photoelectric conversion unit are scanned for each group, and each pixel is exposed independently for each group with a plurality of types of exposure times equal to or shorter than the predetermined exposure time. The non-destructive readout of the charges can be performed. Further, the pixel data when the exposure is performed with a predetermined exposure time scanned and read by the first reading means is output from the first output channel, and a plurality of types of data less than the predetermined exposure time of each set scanned and read by the second reading means. Since the pixel data at the time of exposure is output from the second to (N + 1) th output channels, the pixel data read by the destructive readout method and the pixel data of each set read by the non-destructive read method are used. Each can be output independently.

  In the second reading means, since the remaining pixels obtained by thinning out the predetermined pixels are scanned, scanning can be performed at a higher speed than when scanning is performed for all the pixels. Furthermore, the readout of charges from each pixel is performed by a nondestructive readout method, and scanning is performed a plurality of times with a plurality of types of exposure times less than or equal to a predetermined exposure time during the exposure period of a predetermined exposure time. Charges can be read out at various frame rates. As a result, it is possible to follow the high-speed movement of the subject and capture a subject image during operation (in the middle of position displacement) that cannot be captured within a predetermined exposure time. Further, by reading out charges by the non-destructive readout method, it is possible to maintain the accumulated charge amount of each pixel even if scanning is performed a plurality of times during the exposure period of a predetermined exposure time. Further, it is possible to easily change the frame rate by changing the number of pixels to be thinned out. Further, by physically increasing the number of output channels, it is possible to easily increase the number of scans, and thus it is possible to easily increase the number of times of readout of charges for the same pixel during a predetermined exposure period.

  For example, when imaging an imaging target composed of a relatively bright and moving subject (for example, a motion capture marker) and a relatively low (dark) background, read by the first reading means Since the normal captured image data is generated from the pixel data composed of the charges at the time of exposure with a predetermined exposure time, it is possible to obtain a clear image in which the subject and the background are exposed with a sufficient exposure time. Further, high-speed captured image data for each scan is generated from the pixel data composed of charges at the time of exposure with a plurality of types of exposure times less than or equal to a predetermined exposure time by a plurality of scans read by the second readout means. As a result, it is possible to obtain a plurality of images that follow the movement of the subject and clearly capture the subject.

That is, there is an effect that a captured image in which the imaging target is clearly captured and a plurality of captured images in which the movement of the subject is accurately captured can be obtained simultaneously in the exposure period of the predetermined exposure time.
Further, by increasing the number of thinned pixels and the number of output channels, it is possible to easily read out charges at a higher frame rate.

[Mode 7] Furthermore, the imaging device of mode 7 is the imaging device of mode 6,
The second to (N + 1) th output channels output the pixel data for each line of the pixels,
The second reading means is characterized in that the plurality of scans are performed on pixels of the remaining lines obtained by thinning out pixels of a predetermined line from the pixel group.
With such a configuration, since predetermined pixels can be thinned out in units of lines, it is easy to adjust the number of lines to be scanned, and the frame rate can be simplified by adjusting the number of lines of pixels to be scanned. The effect that it can be changed to is obtained.

[Embodiment 8] Furthermore, the imaging apparatus of Embodiment 8 is the imaging apparatus of Embodiment 6 or 7,
The high-speed captured image data generation means includes a pixel value of pixel data output from the second to (N + 1) th output channels and each of the second to (N + 1) th times of the plurality of scans. A difference value with respect to the pixel value of the pixel data output in the previous scan of each of the scans from the output channel is calculated, and the captured image data corresponding to each of the scans based on the difference value It is characterized by generating.
With such a configuration, since predetermined pixels can be thinned out in units of lines, it is easy to adjust the number of lines to be scanned, and the frame rate can be simplified by adjusting the number of lines of pixels to be scanned. The effect that it can be changed to is obtained.

[Mode 9] On the other hand, in order to achieve the above object, the motion data generation system of mode 9
A motion data generation system that generates motion data indicating the operation content of an operating body that performs a predetermined operation,
An imaging apparatus according to any one of claims 3 to 8, comprising:
The imaging device reads pixel data at the time of exposure with the predetermined exposure time for the imaging target including the operating body with a marker for position detection by the first reading unit, and at the plurality of types of exposure times. Read pixel data at the time of exposure by the second reading means,
Position detecting means for detecting the position of the marker based on the captured image data generated by the high-speed captured image data generating means in the imaging device;
Motion data generating means for generating the motion data based on the detection result of the position detecting means.

If it is such a structure, in the imaging device of any one of Claim 3 thru | or 8, it will expose with the predetermined exposure time with respect to the imaging target containing the said operation body to which the marker for position detection was attached | subjected. Pixel data at the time of exposure and pixel data at the time of exposure with a plurality of types of exposure times can be read.
Furthermore, it is possible to detect the position of the marker based on the captured image data generated by the high-speed captured image data generation unit in the imaging apparatus by the position detection unit, and the position detection by the motion data generation unit The motion data can be generated based on the detection result of the means.

  Therefore, since the charge can be read out at a high frame rate with respect to the operating body provided with the high-intensity marker, the movement of the marker that is incapable of being caught in a predetermined exposure time following the movement of the operating body. An image can be acquired. And since the position of a marker can be detected from the image of such a marker, it is possible to perform highly accurate position detection with respect to the motion of the moving body, and thereby The effect that more accurate motion data can be generated is obtained. In addition, from the captured image data corresponding to the predetermined exposure time, clear background image (or other motion object image) data and clear motion object image data can be acquired. It can be used for purposes other than the generation of.

  Here, this system may be realized as a single device, terminal, or other device, or may be realized as a network system in which a plurality of devices, terminals, or other devices are communicably connected. . In the latter case, each component may belong to any one of a plurality of devices and the like as long as they are connected so as to communicate with each other.

[Mode 10] Furthermore, the motion data generation device of mode 10 is the motion data generation device of mode 9,
The image pickup apparatus includes a texture data extraction unit that extracts texture data of the operating body from the captured image data generated by the normal captured image data generation unit in the imaging apparatus.

With such a configuration, it is possible to extract the texture data of the operating body from the captured image data generated by the normal captured image data generation unit in the imaging apparatus by the texture data extraction unit.
Therefore, motion data can be generated from captured image data captured at a high frame rate, and texture data can be generated from captured image data obtained by clearly capturing an action object. Therefore, it is possible to save the trouble of separately capturing the captured images separately.

[Mode 11] In order to achieve the above object, a motion data generation program according to mode 11
A motion data generation program for generating motion data indicating the operation content of an operating body that performs a predetermined operation,
9. The captured image data for the imaging target including the operating body with the position detection marker generated by the high-speed captured image data generation unit in the imaging apparatus according to claim 3. Based on the position detection step of detecting the position of the marker,
A program for causing a computer to execute processing including a motion data generation step for generating the motion data based on a detection result of the position detection step is included.
With such a configuration, when the program is read by the computer and the computer executes processing according to the read program, the same operation and effect as those of the motion data generation system according to mode 9 can be obtained.

[Mode 12] In order to achieve the above object, the motion data generation method according to mode 12 includes:
A motion data generation method for generating motion data indicating an operation content of an operating body that performs a predetermined operation,
9. The captured image data for the imaging target including the operating body with the position detection marker generated by the high-speed captured image data generation unit in the imaging apparatus according to claim 3. Based on the position detection step of detecting the position of the marker,
And a motion data generation step of generating the motion data based on the detection result of the position detection step.
Thereby, an effect equivalent to that of the motion data generation system according to mode 9 is obtained.

[Mode 13] On the other hand, in order to achieve the above object, the imaging system of mode 13
A plurality of photoelectric conversion elements that convert the exposed light into electric charges and store the photoelectric conversion units arranged in a matrix; and
First readout means for scanning each pixel constituting the photoelectric conversion element, and reading out the charge when exposed for a predetermined exposure time from each scanned pixel by a destructive readout method with reset of accumulated charge;
During the predetermined exposure time, scanning is performed a plurality of times for each pixel constituting the photoelectric conversion element, and a plurality of types of exposure times equal to or less than the predetermined exposure time from each pixel in each scanning. A second reading means for reading out charges when exposed by a non-destructive reading method without resetting accumulated charges;
A first output that is read out by the first reading means and that sequentially stores pixel data composed of charges when exposed for the predetermined exposure time in a memory, and sequentially outputs the pixel data stored in the memory in the order of storage. Channel,
Pixel data read by the second reading means is sequentially stored in a memory for each scan in the plurality of scans, and the pixel data stored in the memory is sequentially output in the stored order;
Normal captured image data generating means for generating captured image data when exposed for a predetermined exposure time based on the pixel data output from the first output channel;
High-speed captured image data generating means for generating captured image data for each scan in the plurality of scans based on pixel data output from the second output channel,
The second reading unit is characterized in that the plurality of scans are performed on the remaining pixels obtained by thinning a predetermined pixel from a pixel group formed by the plurality of photoelectric conversion elements.

Thereby, the same operation and effect as those of the imaging device of aspect 3 can be obtained.
Here, the present system may be realized as a single device, terminal, or other device (in this case, equivalent to the imaging device of form 3), or may communicate with a plurality of devices, terminals, and other devices. You may make it implement | achieve as a network system connected to. In the latter case, each component may belong to any one of a plurality of devices and the like as long as they are connected so as to communicate with each other.

[Embodiment 14] In order to achieve the above object, an imaging system of embodiment 14 includes:
A plurality of photoelectric conversion elements that convert the exposed light into electric charges and store the photoelectric conversion units arranged in a matrix; and
First readout means for scanning each pixel constituting the photoelectric conversion element, and reading out the charge when exposed for a predetermined exposure time from each scanned pixel by a destructive readout method with reset of accumulated charge;
A plurality of scans performed on each pixel are divided into N groups in a predetermined combination, and the divided scans are divided into the same time during the predetermined exposure time. A non-destructive readout method in which charges are read out from each pixel in each scan during exposure at a plurality of types of exposure times less than or equal to the predetermined exposure time without resetting accumulated charges. Second reading means performed in
A first output that is read out by the first reading means and that sequentially stores pixel data composed of charges when exposed for the predetermined exposure time in a memory, and sequentially outputs the pixel data stored in the memory in the order of storage. Channel,
The pixel data read by the second reading means is sequentially stored in the memory for each scan, and the pixel data stored in the memory is sequentially output in the stored order. The N pieces are independent for each group (N is 2). The second to (N + 1) th output channel of the above natural number),
Normal captured image data generating means for generating captured image data when exposed for a predetermined exposure time based on the pixel data output from the first output channel;
High-speed captured image data generation means for generating captured image data for each scan in the plurality of scans based on pixel data output from the second to (N + 1) th output channels,
The second reading unit is characterized in that the scanning is performed on the remaining pixels obtained by thinning a predetermined pixel from a pixel group formed by the plurality of photoelectric conversion elements.

Thereby, the same operation and effect as those of the imaging device of aspect 6 can be obtained.
Here, this system may be realized as a single device, terminal, or other device (in this case, equivalent to the imaging device according to mode 6), or can communicate with a plurality of devices, terminals, and other devices. You may make it implement | achieve as a network system connected to. In the latter case, each component may belong to any one of a plurality of devices and the like as long as they are connected so as to communicate with each other.

[Mode 15] On the other hand, in order to achieve the above object, the imaging method of mode 15 includes:
An imaging method used in an imaging device having a photoelectric conversion unit in which a plurality of photoelectric conversion elements that convert exposed light into charges and store the matrix are arranged in a matrix,
A first reading step of scanning each pixel constituting the photoelectric conversion element, and reading out charges when exposed for a predetermined exposure time from each of the scanned pixels by a destructive reading method with reset of accumulated charges;
During the predetermined exposure time, scanning is performed a plurality of times for each pixel constituting the photoelectric conversion element, and a plurality of types of exposure times equal to or less than the predetermined exposure time from each pixel in each scanning. A second reading step in which reading of the charge at the time of exposure is performed by a non-destructive reading method without resetting the accumulated charge;
The first output that sequentially reads out the pixel data composed of the charges when exposed for the predetermined exposure time, and sequentially outputs the pixel data stored in the memory in the stored order, which is read out in the first readout step. A normal captured image data generation step for generating captured image data when exposed for a predetermined exposure time based on pixel data output from the channel;
The pixel data read in the second reading step is sequentially stored in the memory for each scan in the plurality of scans, and the pixel data stored in the memory is output from the second output channel that sequentially outputs in the stored order. A high-speed captured image data generation step for generating captured image data for each scan in the plurality of scans based on the pixel data,
The second reading step is characterized in that the plurality of scans are performed on the remaining pixels obtained by thinning a predetermined pixel from a pixel group constituting the plurality of photoelectric conversion elements.
Thereby, the same operation and effect as those of the imaging device of aspect 3 can be obtained.

[Mode 16] On the other hand, in order to achieve the above object, an imaging method according to mode 16 includes:
An imaging method used in an imaging device having a photoelectric conversion unit in which a plurality of photoelectric conversion elements that convert exposed light into charges and store the matrix are arranged in a matrix,
A first reading step of scanning each pixel constituting the photoelectric conversion element, and reading out charges when exposed for a predetermined exposure time from each of the scanned pixels by a destructive reading method with reset of accumulated charges;
A plurality of scans performed on each pixel are divided into N groups in a predetermined combination, and the divided scans are divided into the same time during the predetermined exposure time. A non-destructive readout method in which charges are read out from each pixel in each scan during exposure at a plurality of types of exposure times less than or equal to the predetermined exposure time without resetting accumulated charges. A second reading step performed in
The first output that sequentially reads out the pixel data composed of the charges when exposed for the predetermined exposure time, and sequentially outputs the pixel data stored in the memory in the stored order, which is read out in the first readout step. A normal captured image data generation step for generating captured image data when exposed for a predetermined exposure time based on pixel data output from the channel;
The pixel data read in the second reading step is sequentially stored in the memory for each scan, and the pixel data stored in the memory is sequentially output in the stored order. A high-speed captured image data generation step for generating captured image data for each scan in the plurality of scans based on pixel data output from the second to (N + 1) output channels of the above (natural number). ,
In the second reading step, each of the scans is performed on the remaining pixels obtained by thinning a predetermined pixel from a pixel group formed by the plurality of photoelectric conversion elements.
Thereby, the same operation and effect as those of the imaging device of aspect 6 can be obtained.

[First Embodiment]
Hereinafter, an embodiment (first embodiment) of an imaging device, an imaging device, an imaging system, and an imaging method according to the present invention will be described with reference to the drawings. 1 to 8 are diagrams showing a first embodiment of an imaging device, an imaging apparatus, an imaging system, and an imaging method according to the present invention.
Hereinafter, a schematic configuration of the imaging apparatus 1 according to the present invention will be described with reference to FIG. Here, FIG. 1 is a block diagram showing a schematic configuration of the imaging apparatus 1 according to the present invention.

  As shown in FIG. 1, the imaging apparatus 1 uses an imaging device 100 (described later) composed of photoelectric conversion elements and the like, and a destructive readout method and a nondestructive readout method for pixel signals from the imaging device 100. Readout, image processing unit 10 that outputs pixel data of the read pixel signal, high frame rate image data is generated based on pixel data read out from image processing unit 10 by a nondestructive readout method, etc. The image processing unit 12 that performs the image processing, the normal scanning pixel data and the thinned scanning image data output from the imaging processing unit 10, and various image data such as the high frame rate image data generated by the image processing unit 12 are stored. And a frame memory 14.

Further, the imaging apparatus 1 according to the present embodiment controls and executes various processes such as a thinning scanning process described later based on various control signals from an external apparatus (system controller) (not shown).
The imaging processing unit 10 includes an imaging device 100, a first AFE (Analog Front End) 102, and a second AFE 104.

The image sensor 100 condenses light from a subject on a sensor cell array 56 (described later) with an imaging lens (not shown), and accumulates charges corresponding to the amount of light collected in each pixel of the sensor cell array 56.
The image sensor 100 sequentially scans each pixel column of the sensor cell array 56 with a normal exposure time (can be set to any time by a user or the like) based on a drive signal output from a drive pulse generator 52 (described later). (Hereinafter referred to as normal scanning). Then, the charge group accumulated in each scanned pixel column is converted into a voltage group and sequentially read out, and the accumulated charge in each pixel column is reset (accumulated charge is emptied). Further, the read voltage group (pixel signal) is sequentially output to the first AFE 102 via a first output channel 58 (described later) including the first line memory.

  In addition, the image sensor 100 selects a predetermined number of pixels according to the thinning amount designated by the system controller based on the drive signal output from the drive pulse generator 52 (described later), according to the thinning amount specified by the system controller. Scanning is performed while thinning out the columns (hereinafter referred to as thinning scanning). Then, a voltage group (pixel signal) is read from each scanned pixel column (at this time, the accumulated charge is not reset). Note that thinning scanning is performed a plurality of times during the exposure period of the normal exposure time. That is, in one exposure period, the charge group accumulated at each scan in the plurality of thinning scans is sequentially converted into a voltage group and read, and the read voltage group (pixel signal) is read for each scan. The data is sequentially output to the second AFE 104 via a second output channel 60 (described later) including a line memory.

  As described above, in the present embodiment, by performing charge readout from each pixel by using both the destructive readout method and the non-destructive readout method, in one exposure period (normal exposure time), The pixel signal at the time of exposure at the normal exposure time of each pixel row is read out by the destructive readout method, and the pixel signal at the time of exposure is exposed at an exposure time less than the normal exposure time of each pixel row by thinning scanning multiple times. Read with.

Further, the internal configuration of the image sensor 100 will be described with reference to FIG. Here, FIG. 2 is a diagram illustrating an internal configuration of the image sensor 100.
As shown in FIG. 2, the image sensor 100 includes a reference timing generator 50, a drive pulse generator 52, a scanning line scanner 54, a sensor cell array 56, a first output channel 58, and a second output channel 60. It is comprised including.

The reference timing generator 50 generates a reference timing signal based on a vertical synchronization signal 0 and a horizontal synchronization signal from a timing control unit 12b (described later) of the image processing unit 12.
The drive pulse generator 52 generates a drive pulse based on the reference timing signal from the reference timing generator 50 and the reset line selection signal and readout line selection signal from the scanning line scanner 54 and supplies them to the sensor cell array 56.

  The scanning line scanner 54 selects the reset line position with respect to the pixel column group of the sensor cell array 56 based on the thinning amount information (drive control signal) from the communication device 12a (described later) of the image processing unit 12. In addition to generating a signal, a read line selection signal is generated by selecting the position of the read line for normal scanning and thinning scanning. These generated selection signals are output to the drive pulse generator 52. Also, a vertical synchronization signal 1 which is a reference synchronization signal for the thinning scanning is generated and output to the drive pulse generator 52 and a timing control unit 12b (described later) of the image processing unit 12, respectively.

  In the sensor cell array 56, each pixel is configured using CMOS technology, and each pixel column is exposed with a normal exposure time based on the driving pulse supplied from the driving pulse generator 52, and the pixel column is exposed by the exposure. The accumulated charge group is sequentially read out by the destructive readout method for each pixel column. That is, the stored charge is also reset after reading the charge group. Further, the pixel signal immediately after reset for each pixel column is read by the nondestructive reading method and sequentially output to the first output channel 58. In addition, the sensor cell array 56 performs thinning-out scanning a plurality of times during the exposure period of the normal exposure time, and for each scan, the charge group accumulated in each pixel column is sequentially non-applied for each scanned pixel column. The data is read out by the destructive readout method and sequentially output to the second output channel 60.

  The first output channel 58 stores pixel signal data corresponding to the normal exposure time read from the sensor cell array 56 in the first line memory S constituting the first line memory for each pixel column, and immediately after the reset. The pixel signal data is stored in the first line memory N constituting the first line memory for each pixel column. Then, the stored normal exposure time and pixel signal data immediately after reset are output to the differential amplifier 62 (shown in FIG. 6).

The differential amplifier 62 performs a subtraction process on the pixel signal data immediately after the normal exposure time and reset, and outputs the pixel signal after the subtraction process to the first AFE 102.
The second output channel 60 stores pixel signal data corresponding to the exposure time at each scan read from the sensor cell array 56 for each scan in the second line memory for each scan and for each pixel column. To do. Then, the stored pixel signal data is output to the second AFE 104.

Further, a detailed configuration of the sensor cell array 56 will be described with reference to FIG. Here, FIG. 3 is a diagram showing a detailed configuration of the sensor cell array 56.
As shown in FIG. 3, the sensor cell array 56 has a plurality of sensor cells (pixels) 56a configured using CMOS arranged in a matrix, and for each pixel column, the sensor cell 56a constituting each pixel column is arranged. Thus, the address line, the reset line, and the readout line are connected in common, and various drive signals are transmitted to the sensor cells 56a constituting each pixel column via the three control lines. When the address line and the readout line are activated, the accumulated charge is transferred to the first or second output channel 58 or 60 via the signal line shown in FIG. With such a configuration, a pixel column for which a reset operation or a read operation is performed by an address line is validated (selected), and a reset operation is performed on each sensor cell 56a of the pixel column selected by the selection signal. Inputs a signal for instructing the reset operation via the reset line, and when reading out the pixel signal, inputs a signal for instructing the transfer of the accumulated charge via the readout line. Each sensor cell 56a selected by the selection signal performs a reset operation when a signal instructing a reset operation is input, and when a signal instructing a transfer of stored charge is input, the first sensor cell 56a is connected via a signal line. Alternatively, the stored charge is transferred to the second output channel 58 or 60.

Furthermore, the internal configuration of the scanning line scanner 54 will be described with reference to FIG. Here, FIG. 4 is a diagram showing an internal configuration of the scanning line scanner 54.
As shown in FIG. 4, the scanning line scanner 54 includes a normal scanning counter 54a, a normal scanning address decoder 54b, a thinning scanning counter 54c, a thinning scanning address decoder 54d, and an OR logic 54e.

The normal scanning counter 54a repeats the count-up operation based on the vertical synchronization signal 0 and the horizontal synchronization signal from the reference timing generator 50. Here, the value of the counter corresponds to the line number of each pixel column at the time of normal scanning, and this line number is output to the normal scanning address decoder 54b.
The normal scanning address decoder 54b validates the pixel row having the line number from the normal scanning counter 54a as a “read line” and invalidates the other pixel rows. Further, a read line control signal indicating validity / invalidity of each pixel column is output to the OR logic 54e, and this read line control signal is output to the drive pulse generator 52 as a reset line selection signal.

  The thinning scanning counter 54c generates the vertical synchronizing signal 1 based on the horizontal synchronizing signal from the reference timing generator 50 and the information indicating the thinning amount from the communication device 12a, and counts based on the generated vertical synchronizing signal 1. Repeat the up operation. Here, the value of the counter corresponds to the line number of the pixel column at the time of thinning scanning, and this line number is output to the thinning scanning address decoder 54d. The vertical synchronization signal 1 is also output to the timing control unit 12 b of the image processing unit 12.

The thinning scan address decoder 54d validates the pixel row having the line number from the thinning scan counter 54c as a “read line”, and invalidates the other pixel rows. Further, a read line control signal indicating validity / invalidity of each pixel column is output to the OR logic 54e.
The OR logic 54e performs an OR operation for each line on the basis of the read line control signal from the normal scan address decoder 54b and the read line control signal from the thinning scan address decoder 54d to obtain the final read line for the normal scan. A selection signal is generated, and a final read line selection signal for the thinning scan is generated. These generated read line selection signals are output to the drive pulse generator 52.

Further, thinning scanning will be described with reference to FIG. Here, FIG. 5A is a diagram illustrating an example of the thinning scan, and FIG. 5B is a graph illustrating the transition of the level (accumulated charge amount) of the pixel signal output from the sensor cell 56a by the thinning scan of FIG. FIG.
In the example shown in FIG. 5A, thinning scanning is performed four times in the first line, the fifth line, the ninth line, and the thirteenth line in the exposure period of the normal exposure time, and the accumulated charge ( Pixel signal). By performing thinning scanning in this manner, the number of pixel columns to be scanned is reduced to ¼. In this way, by shortening the scanning time for one scan (one frame), the number of scans in the normal exposure time can be quadrupled, so the frame rate is four times faster than in normal scan. Can be

  Here, paying attention to “a pixel” in the first line of FIG. 5A, as shown in FIG. 5B, the accumulated charge amount of the pixel is 2 as compared with the first scanning. It increases (with the passage of time) as the scanning proceeds, the third time, the fourth time, and the fourth time. This is because the charge is read from each pixel by the non-destructive readout method in the thinning scan, and the accumulated charge is not reset. Therefore, the accumulated charge of each pixel can be read out many times during the exposure period. The amount is maintained. That is, even if the charge is read at the scanning timings (1) to (4) by the thinning scanning shown in FIG. 5A, as shown in FIG. The amount is maintained as it is, and charge accumulation (exposure) is continued.

  In order to perform such decimation scanning, the decimation scanning counter 54c performs a count-up operation based on a drive control signal that specifies the decimation amount output from the communication device 12a, and decimates and scans the count value corresponding to the decimation amount. The data is output to the address decoder 54d. In the example of FIG. 5A, the thinning amount is “4”, and the count value of the thinning scanning counter 54c is incremented by “4” by 1, 5, 9, 13 and the count value is 13 ( When the maximum value (which varies depending on the thinning amount and the total number of lines of pixels) is reached, the count value is returned (reset) to “1” (corresponding to the start line number), and the vertical synchronization signal 1 is activated. To do. That is, the thinning scanning counter 54c activates the vertical synchronization signal 1 every time the count value reaches the maximum value.

Further, based on FIG. 6, an exposure operation and a pixel signal readout operation for each pixel column in the sensor cell array 56 of the image sensor 100 will be described. Here, FIG. 6 is a diagram illustrating an example of exposure and pixel signal readout operations for each pixel column in the sensor cell array 56 of the image sensor 100.
As shown in FIG. 6, first, a first readout line for performing readout of a pixel signal at the time of exposure with a normal exposure time of each pixel column in the sensor cell array 56 and resetting accumulated charge after readout of the pixel signal by normal scanning. In addition to the selection, a second readout line for performing non-destructive readout of the pixel signal at the time of exposure is selected for the pixel row to be scanned in the sensor cell array 56 by the exposure time at the time of scanning by a plurality of thinning scans. Then, in one exposure period (normal exposure time), pixel signals are read at the normal exposure time for each pixel column by normal scanning and reset of accumulated charges, and pixels to be scanned by a plurality of thinning scans. The non-destructive readout of the pixel signal at the time of exposure is performed independently with the exposure time at the time of each scanning for the column.

  That is, as shown in FIG. 6, the first readout line and the second readout line have charges corresponding to the normal exposure time in any one of these pixel columns (for example, the first to twelfth lines). Is stored, the first readout line is selected to read out the pixel signal of the pixel column and to reset the accumulated charge. In addition, the scanning timing of the pixel column whose accumulated charge has been reset is controlled so that when the first readout line is scanned again after scanning the other pixel column, the exposure for just the normal exposure time is performed. The Note that the scanning of each pixel column by the normal scanning is sequentially performed while looping through all the pixel columns (for example, the first to twelfth lines) for each frame. In the example of FIG. 6, scanning is sequentially performed for each line from the first line (bottom line) to the twelfth line (top line), and when the scanning reaches the twelfth line, one frame period. The scanning for the next frame is again performed from the first line.

  On the other hand, in the thinning-out scanning, the pixel signal of each pixel column to be scanned is non-destructively in the exposure time during each scanning while the charge corresponding to the normal exposure time is accumulated after the accumulated charge is reset by the first readout line. The second read lines are selected so as to read sequentially. Accordingly, when the accumulated charge in the pixel column to be scanned is reset by the first readout line, the pixel of the pixel exposed in the exposure time at each scanning in the second readout line with respect to the reset pixel column. Nondestructive readout of signals is performed sequentially in each scanning order. In the thinning scanning, scanning is performed while thinning out a predetermined number of pixel rows in accordance with the thinning amount designated by the system controller, and when the pixel row of the highest number is reached among the pixel rows to be scanned, the pixels to be scanned Returning to the lowest-numbered pixel row in the row, scanning is sequentially performed while looping, as in the normal scanning. However, in the thinning scanning, the loop is repeated by the number of scans that can be executed (or set in advance) in one frame of the normal exposure time.

  In the present embodiment, as shown in FIG. 6, the pixel signal (analog data) at the time of exposure with the normal exposure time read by the normal scan is the first output channel 58 (hereinafter referred to as CH1). On the other hand, the pixel signal immediately after the reset is read out to the first line memory N of CH1. Then, as shown in FIG. 6, these read pixel signals are output to a differential amplifier 62 provided on the output side of CH1, and in the differential amplifier 62, they correspond respectively before and after resetting. The pixel level is subtracted to detect the signal level and remove noise. The pixel signal after the subtraction processing is output to the first AFE 102 where it is converted into digital data (pixel data). On the other hand, the pixel signal for each scan read by a plurality of thinning scans is read for each scan to the second line memory of the second output channel (hereinafter referred to as CH2) and sent to the second AFE 104. The data is output and converted into digital data (pixel data).

  Further, in this embodiment, when the pixel signal readout of the first readout line and the pixel signal readout of the second readout line are batting (for example, when both perform readout in the normal exposure time). For example, the readout period set by the horizontal synchronization signal is divided into two periods, the pixel signal is read out to the first line memory of CH1 by the first readout line in one period, and by the second readout line in the other period. This is avoided by reading out the pixel signal to the second line memory of CH2.

Further, the internal configuration of the image processing unit 12 will be described with reference to FIG. Here, FIG. 7 is a diagram illustrating an internal configuration of the image processing unit 12.
As shown in FIG. 7, the image processing unit 12 includes a communication device 12a, a timing control unit 12b, a high frame rate image generation unit 12c, a memory access unit 12d, and an output reading unit 12e. .

The communication device 12a is a means for the image sensor 100 and the image processing unit 12 to perform various data communications. The communication device 12a receives information on a thinning amount for thinning scanning of the sensor cell array 56 from an external system controller (not shown). The received thinning amount information is transmitted to the scanning line scanner 54 of the image sensor 100.
The timing control unit 12 b generates a drive signal (pixel clock, horizontal synchronization signal, vertical synchronization signal 0) for the image sensor 100 and outputs this to the reference timing generator 50 of the image sensor 100. The timing control unit 12b also outputs a pixel position (pixel column (line) number, pixel number) in the sensor cell array 56 of the image sensor 100 from the horizontal synchronization signal and the vertical synchronization signal 0 to the pixel signal output from CH1 of the imaging processing unit 10. Since the pixel number is known, the pixel column (line) number and the pixel number (hereinafter also referred to as “address information”) are generated, and the address information is output to the memory access unit 12d. In addition, the timing control unit 12b also outputs a pixel position (in the sensor cell array 56 of the image sensor 100) of the pixel signal output from the horizontal synchronization signal, the vertical synchronization signal 1 from the scanning line scanner 54, and CH2 of the imaging processing unit 10. Since the pixel column (line) number and pixel number) are known, the address information is generated and the address information is output to the high frame rate image generation unit 12c.

  First, the high frame rate image generation unit 12c stores image data (hereinafter referred to as “thinning-out scanning”) stored in the frame memory 14 and output from the second AFE 104 of the imaging processing unit 10 for each scanning in a plurality of thinning-out scannings. Imaging) and imaging in the previous scan (hereinafter referred to as pre-scan) corresponding to the thinned-out scan image data for each scan (hereinafter referred to as current scan). The thinned-out scanned image data output from the second AFE 104 of the processing unit 10 is read out via the memory access unit 12d. Next, each read out pixel data constituting the current scan thinned scan image data and each pixel data constituting the previous scan thinned scan image data at the same position as each pixel data corresponding to each pixel data A difference value is calculated, and corresponding image data (hereinafter referred to as high frame rate image data) is generated for each thinning scan based on the difference value. Then, the generated high frame rate image data is stored (written) in the frame memory 14 via the memory access unit 12d.

  The memory access unit 12d accesses each image data stored in the frame memory 14 based on image data write / read commands from the imaging processing unit 10, the high frame rate image generation unit 12c, and the output reading unit 12e. That is, the access request from the three systems is arbitrated, and image data is read out and written into the frame memory 14. In the present embodiment, the memory access unit 12d receives the normal scan image data output from the first AFE 102 and the thinned scan image data output from the second AFE 104 of the imaging processing unit 10 in the frame memory 14. Write to. In addition, in response to a read command from the high frame rate image generation unit 12c, each thinning scan and pre-scanning corresponding thinning scan image data are read from the frame memory 14 or from the high frame rate image generation unit 12c. High frame rate image data is written into the frame memory 14 in response to a write command.

  The output reading unit 12e outputs the image data in the frame memory 14 in synchronization with synchronization signals (pixel clock, horizontal synchronization signal, vertical synchronization signal 0) from an external output device (not shown). Specifically, first, a request for reading out high frame rate image data and normal scan image data (including a read command) is made to the memory access unit 12d to acquire the high frame rate image data and normal scan image data. Then, the high frame rate image data and the normal scan image data are output to an external output device in synchronization with the synchronization signal from the output device. The output device executes predetermined processing such as display processing and analysis processing based on the output data.

  As shown in FIG. 7, the frame memory 14 includes normal scan image data output via CH1, each thinning scan output via CH2, thinning scan image data corresponding to these thinning scans, and high frame rate. The memory has a capacity capable of storing the high frame rate image data generated by the image generation unit 12c. When there is a read request from the memory access unit 12d, the image data indicated by the request is read. Further, when there is a write request from the memory access unit 12d, the frame memory 14 writes the image data indicated by the write request.

Next, the actual operation of the imaging apparatus 1 of the present embodiment will be described based on FIG. Here, FIG. 8A is a diagram illustrating an example of normal scanning and thinning scanning, and FIG. 8B is a diagram illustrating the transition of the accumulated charge amount for a certain pixel on the tenth line.
For convenience of explanation, it is assumed that the sensor cell array 56 includes 15 pixel columns of the first to fifteenth lines.

When the communication device 12a of the image processing unit 12 receives information on a thinning amount from a system controller (not shown) that is an external device, the imaging device 1 captures a drive control signal that specifies the thinning amount. The data is output to the scanning line scanner 54 of the processing unit 10. On the other hand, the imaging processing unit 10 sets a normal exposure time for the first readout line corresponding to CH1.
In addition, the image processing unit 12 generates a drive signal (pixel clock, horizontal synchronization signal, vertical synchronization signal 0) in the timing control unit 12b and outputs it to the image sensor 100.

  When the drive signal is supplied, the image sensor 100 starts a count-up operation in the normal scanning counter 54a of the scanning line scanner 54, and outputs the count value to the normal scanning address decoder 54b. The normal scan address decoder 54b enables the line having the line number output from the normal scan counter 54a as the “first readout line”, and disables the other lines.

For example, when the line number 1 is validated as the first readout line, the remaining lines other than the line number 1 (second line to 15th line) are invalidated. Further, a first read line control signal indicating that the line number 1 is valid is output to the OR logic 54e.
On the other hand, the thinning-out scanning counter 54c starts a count-up operation based on a driving signal (horizontal synchronization signal) from the timing control unit 12b and a driving control signal that designates a thinning amount from the communication device 12a, and thins out the count value. The data is output to the address decoder 54d. Further, the vertical synchronizing signal 1 which is a reference synchronizing signal for thinning scanning is generated and output to the timing control unit 12b.

  Here, if the designated thinning-out amount is “3” and the start number of the count value is “1”, the thinning-out scanning counter 54c counts up as 1, 4, 7, 10, 13. . That is, the count-up is performed with the designated thinning-out amount “3” as an increase amount at the time of count-up. Since the sensor cell array 56 has 15 pixel columns, when the count value reaches “13” (maximum value), the count value is returned to “1” (reset) and the vertical synchronization signal 1 is activated. Again, counting up is performed with the thinning amount “3” as an increase amount from the count value “1”. The thinning scanning counter 54c repeats such a count-up operation according to the number of thinning scans in the exposure period of the normal exposure time. Here, thinning scanning is performed three times during the exposure period of the normal exposure time.

Similar to the normal scanning address decoder 54b, the thinning scanning address decoder 54d validates the line having the line number output from the thinning scanning counter 54c as the “second readout line” and invalidates the other lines. Then, the second read line control signal is output to the OR logic 54e.
The OR logic 54e performs an OR operation for each line position in the exposure region based on the first readout line control signal from the normal scanning address decoder 54b and the second readout line control signal from the thinning scanning address decoder 54d. A final readout line selection signal for normal scanning and thinning scanning is generated. The generated first read line selection signal and second read line selection signal are output to the drive pulse generator 52.

  As a result, as shown in FIG. 8A, the destructive readout of the pixel signal at the time of exposure is performed for all the pixel columns of the sensor cell array 56 during the normal exposure time by scanning the first readout line. Non-destructive readout of pixel signals by thinning-out scanning of the second readout line is performed on the first line whose accumulated charge has been reset by destructive readout. Note that the amount of charge accumulated in each pixel column differs depending on the start timing of each thinning scan. In the example of FIG. 8A, the first thinning scan is started at the time t4, and similarly, the second and third thinning scans are started at the times t9 and t14. The second thinning scan is started after the first thinning scan is completed, and the third thinning scan is started after the second thinning scan is completed.

The first output channel 58 and the second output channel 60 output the read pixel signals to the first AFE 102 and the second AFE 104, respectively, via an internal line memory. The first AFE 102 and the second AFE 104 convert the pixel signal into pixel data that is digital data, and output the pixel data to the memory access unit 12 d of the image processing unit 12.
On the other hand, the memory access unit 12d outputs the normal scan image data and the thinned scan image data output via the first AFE 102 and the second AFE 104 based on the address information from the timing control unit 12b. The information is held and written in a predetermined area of the frame memory 14.

  In this way, for example, when the thinned scanning image data of the current scan and the thinned scanned image data of the previous scan are stored in the frame memory 14, the high frame rate image generation unit 12c A request to read out the thinned scan image data stored in the current scan and the previous scan is output. In response to a read request from the high frame rate image generation unit 12c, the memory access unit 12d reads out the thinned scan image data of the current scan and the previous scan from the frame memory 14, and uses the read thinned scan image data as the high frame rate image. It outputs to the production | generation part 12c.

  On the other hand, the high frame rate image generation unit 12c calculates a difference value between pixel values indicated by the respective pixel data of the thinned scan image data of the current scan and the previous scan acquired via the memory access unit 12d. Specifically, the pixel value indicated by each pixel data of the pre-scanned thinned-out image data at the same position as each pixel data is subtracted from the pixel value indicated by each pixel data of the thinned-out scanned image data of the current scan. Then, high frame rate image data is generated based on each pixel value of the subtraction result. For example, as shown in FIG. 8A, focusing on the tenth line, in the second thinning scan, the thinned scanning image data read at time t12 is thinned out in the pre-scanning (first thinning scanning). The thinned-out scanned image data read at time t17 becomes the thinned-out scanned image data for the current scan (second thinned-out scan). Therefore, the difference value is obtained by subtracting each pixel data of the thinned scanning image data corresponding to each pixel data from each pixel data of the thinned scanning image data at time t17. On the other hand, as shown in FIG. 8A, in the fourth thinning scan for the tenth line (first thinning scan for the second normal scanning), the thinned scanning image data read at time t22 is the previous one. The thinned-out scanned image data is scanned (third thinned-out scanning), and the thinned-out scanned image data read at time t27 becomes the thinned-out scanned image data in the current scanning (fourth thinned-out scanning). Therefore, each pixel data of the thinned scanning image data at time t22 corresponding to each pixel data is subtracted from each pixel data of the thinned scanning image data at time t27 to obtain a difference value. However, in this case, the third thinning scan is performed during the exposure period of the first normal exposure time, and the fourth thinning scan is performed during the exposure period of the second normal exposure time. As shown in FIG. 8B, the amount of charge accumulated in each pixel of the tenth line is greater than the amount of charge accumulated in the fourth thinning scan, as shown in FIG. Less. Therefore, the difference value becomes a negative value. In such a case, in the present embodiment, the difference value is invalidated and the pixel value of the high frame rate image is set to “0”.

The high frame rate image data generated in this way is stored in a predetermined area of the frame memory 14 via the memory access unit 12d.
Further, when the memory access unit 12d receives an output request for normal scan image data from an external device, the memory access unit 12d reads the normal scan image data stored in the frame memory 14, and synchronizes the read normal scan image data from the external device. Output in synchronization with signals (pixel clock, horizontal synchronizing signal and vertical synchronizing signal). Similarly, when an output request for high frame rate image data from an external device is received, the high frame rate image data stored in the frame memory 14 is read, and the read high frame rate image data is read from the synchronization signal ( (Pixel clock, horizontal synchronizing signal, and vertical synchronizing signal).

  As described above, in the imaging apparatus 1 of the present embodiment, normal scanning image data composed of pixel data during exposure with the normal exposure time, and a high-speed frame by a plurality of thinning scans in the same exposure period as the normal exposure time. A plurality of high-speed frame rate image data composed of pixel data read at a rate can be obtained simultaneously. That is, a captured image that is exposed with an appropriate exposure time (normal exposure time) and the imaging target is captured clearly, and a plurality of captured images that are captured in a state where a moving subject is clear and the motion content is understood. Can be obtained at the same time.

In addition, since the high-speed frame rate image data is generated by taking the difference between the thinned-out scanned image data of the current scan and the thinned-out scanned image data of the previous scan, it is possible to remove fixed pattern noise. Is possible.
In the first embodiment, the normal scanning image data is generated in the normal captured image data generation unit of any one of modes 3, 13 and 15, or the normal captured image data generation step of mode 15. The high frame rate image data corresponding to the processed image data is the image data generated by the high-speed captured image data generation means of any one of modes 3, 5, 13 and 15, or the high-speed captured image data generation step of mode 15. The sensor cell array 56 corresponds to any one of the photoelectric conversion units of the first, third, thirteenth, and fifteenth forms, and corresponds to the reference timing generator 50 in the image pickup device 100 of the image pickup processing unit 10. Charge breakage during exposure in the normal exposure time from the sensor cell array 56 by the scanning line scanner 54 and the drive pulse generator 52 The reading process corresponds to the first reading means of any one of the first, third, and thirteenth forms or the first reading step of the fifteenth form, and the first output channel 58 is any one of the first, third, thirteen, and fifteenth forms. Corresponding to the first output channel of the image processing unit 10, the non-destructive charge is obtained by thinning out a plurality of thinning scans from the sensor cell array 56 by the reference timing generator 50, the scanning line scanner 54, and the drive pulse generator 52 in the imaging device 100 of the imaging processor 10. The reading process corresponds to the second reading means of any one of the first, third, fourth and thirteenth forms or the second reading step of the fifteenth form, and the second output channel 60 has the first, third, fifth, thirteenth and fifteenth forms. Corresponds to any one of the second output channels.

In the first embodiment, the generation processing of the high frame rate image data by the timing control unit 12b, the high frame rate image generation unit 12c, and the memory access unit 12d in the image processing unit 12 is the same as in the third and fifth embodiments. Corresponding to any one of the 13 high-speed captured image data generation means or the high-speed captured image data generation step of Form 15, the image processing unit 12 writes the normal scan image data into the frame memory 14 by the memory access unit 12d. This corresponds to the normal captured image data generation means of form 3 or 13, or the normal captured image data generation step of form 15.
[Second Embodiment]
Hereinafter, an embodiment (second embodiment) of a motion data generation system, a motion data generation program, and a motion data generation method according to the present invention will be described with reference to the drawings. 9 to 14 are diagrams showing a second embodiment of the motion data generation system, the motion data generation program, and the motion data generation method according to the present invention.

Here, FIG. 9 is a block diagram showing a schematic configuration of the motion data generation system 3 according to the present invention.
The motion data generation system 3 is configured to include the imaging device 1 and the motion data generation device 2 in the first embodiment, and an operation body (human, animal) to which a marker is attached by the imaging device 1. The motion data generating device 2 generates motion data and extracts texture data from the image data output from the imaging device 1. Note that the imaging device 1 and the motion data generation device 2 are connected to each other so as to be able to perform data communication with each other by wire or wirelessly. The connection form may be any form such as a direct connection using a cable, a connection via a LAN, a WAN, or the Internet.

  As described above, the imaging apparatus 1 has the same basic configuration as that of the first embodiment, and an operating body (human, animal, etc.) with a reflective marker as an imaging target is used as the first embodiment. The normal scan image data obtained by the normal scan and the normal scan image data obtained by the thin scan and the high frame rate image data generated from the thin scan image data obtained by the thin scan are generated by the normal scan and the thin scan described in the embodiment. Output to device 2. Note that only one imaging device 1 is shown in FIG. 10, but when capturing the motion of the moving object and acquiring three-dimensional motion information (optical motion capture), there are two or more configurations. (Usually about 6 to 10 units), and it is desirable that the moving bodies are imaged from different angles so that all the markers are always imaged by two or more imaging devices.

In the present embodiment, the imaging device 1 has a light for the reflective marker to reflect and emit light. Thereby, even if the reflective marker is imaged at a high frame rate, it is possible to prevent imaging defects such as blackening of the marker image due to insufficient light quantity.
The motion data generation device 2 generates motion data indicating the operation content of the operating body based on the high frame rate image data generated in the imaging device 1, while the operating body is generated from the normal scanning image data output from the imaging device 1. Extract texture data.

Hereinafter, based on FIG. 10, the detailed structure of the motion data generation apparatus 2 is demonstrated. Here, FIG. 10 is a block diagram showing a detailed configuration of the motion data generation device 2.
As shown in FIG. 10, the motion data generation device 2 includes an image data acquisition unit 2a that acquires various image data from the imaging device 1, and an image data storage unit that stores the various image data acquired by the image data acquisition unit 2a. 2b, a position detection unit 2c that detects the position of the marker from the high frame rate image data, a motion data generation unit 2d that generates motion data based on the detection result of the marker, and texture data of the operating body from the normal scanning image data The configuration includes a texture data extraction unit 2e for extracting, and a motion data storage unit 2f for storing motion data and texture data.

The image data acquisition unit 2a supplies drive signals (pixel clock, horizontal synchronization signal, vertical synchronization signal 0) to the image pickup apparatus 1, and normal scan image data sent from the image pickup apparatus 1 in synchronization with these drive signals and Acquire high frame rate image data. Then, the acquired normal scanning image data and high frame rate image data are stored in the image data storage unit 2b.
The image data storage unit 2b stores the normal scan image data and the high frame rate image data acquired by the image data acquisition unit 2a. Then, in response to a request from the position detection unit 2c or the texture data extraction unit 2e, high frame rate image data or normal scan image data is read and output to the position detection unit 2c or the texture data extraction unit 2e, respectively.

The position detection unit 2c acquires high frame rate image data from the image data storage unit 2b in response to a motion data generation instruction from a processor (not shown), and performs image processing on the acquired high frame rate image data. To detect the position (coordinates) of the marker in the image. Then, the detection result is output to the motion data generating unit 2d.
The motion data generation unit 2d generates motion data for each marker corresponding to each attachment position based on the detection result input from the position detection unit 2c. Then, the generated motion data is stored in the motion data storage unit 2f.

The texture data extraction unit 2e acquires normal scan image data from the image data storage unit 2b in response to a texture data extraction instruction from a processor (not shown), and extracts texture data from the acquired normal scan image data. The extraction of the texture data may be performed manually through an operation unit (not shown). It may be performed automatically by image processing.
In addition, the texture data extraction unit 2e stores the extracted texture data in the motion data storage unit 2f for each type of operating body.

The motion data storage unit 2f stores the motion data generated by the motion data generation unit 2d and the texture data extracted by the texture data extraction unit 2e. Each piece of data is classified and stored, for example, for each type of operating body or each marker attachment position.
In this embodiment, the motion data generation device 2 includes a processor (not shown), a RAM (Random Access Memory), and a ROM (Read Only Memory) in which a dedicated program is stored. By executing the program, the functions of the above-described units are achieved. Here, each of the above-mentioned units is a mixture of a unit that fulfills its function only by a dedicated program, a unit that fulfills its function by controlling hardware by a dedicated program.

  Furthermore, a flow of motion data generation processing and texture data extraction processing in the motion data generation device 2 will be described with reference to FIG. Here, FIG. 11 is a flowchart showing motion data generation processing and texture data extraction processing. In addition, the motion data generation process and the texture data extraction process are performed by imaging an operating body with a marker in the imaging apparatus 1 and are sufficient for the execution of each process in the image data storage unit 2b of the motion data generation apparatus 2. Execution is possible by storing a certain amount of image data.

As shown in FIG. 11, in the motion data generation process and the texture data extraction process, the process first proceeds to step S100, where the position detection unit 2c determines whether or not there is a motion data generation instruction from the processor. If it is determined that there is an instruction (Yes), the process proceeds to step S102, and if not (No), the process proceeds to step S114.
When the process proceeds to step S102, the position detection unit 2c acquires the high frame rate image data corresponding to the instruction from the image data storage unit 2b, and the process proceeds to step S104.

In step S104, the position detection unit 2c detects the position of the marker image in each high frame rate image based on the high frame rate image data acquired in step S102, and proceeds to step S106.
In step S106, the position detection unit 2c outputs the detection result of step S104 to the motion data generation unit 2d, and the process proceeds to step S108.

In step S108, the motion data generation unit 2d generates motion data based on the marker position detection result input from the position detection unit 2c, and the process proceeds to step S110.
In step S110, the motion data generation unit 2d stores the motion data generated in step S108 in the motion data storage unit 2f, and the process proceeds to step S100.

On the other hand, in step S100, when there is no instruction to generate motion data and the process proceeds to step S112, the texture data extraction unit 2e determines whether there is an instruction to extract texture data, and it is determined that there is an instruction to extract. If yes (Yes), the process proceeds to step S114. If not (No), the process proceeds to step S100.
When the process proceeds to step S114, the texture data extraction unit 2e acquires the normal scan image data corresponding to the instruction from the image data storage unit 2b, and the process proceeds to step S116.

In step S116, the texture data extraction unit 2e extracts texture data from the normal scan image data acquired in step S116, and the process proceeds to step S118.
In step S118, the texture data extracting unit 2e stores the texture data extracted in step S116 in the motion data storage unit 2f, and the process proceeds to step S100.

  Next, the actual operation of the motion data generation system 3 according to the present embodiment will be described with reference to FIGS. Here, FIGS. 12A and 12B are diagrams illustrating an example of a normal scan image and a high frame rate image. FIG. 13 is an explanatory diagram of imaging results based on differences in frame rate. FIG. 14 is a diagram illustrating an example of a method for generating motion data.

  First, in the imaging device 1 (here, one unit is used for convenience of explanation), an action body (for example, a human) with a marker to be imaged is imaged. As in the first embodiment, the imaging apparatus 1 irradiates the operating body with the attached light, and images the operation pair irradiated with the light by the normal scanning and the thinning-out scanning a plurality of times (the pixel signal). Read). Then, the normal scan image data and the thinned scan image data obtained thereby are stored in the frame memory 14 via the memory access unit 12d. Further, in the high frame rate image generation unit 12c, the imaging device 1 uses the current scan (current frame) thinned scan image data and the previous scan (previous frame) thinned scan image in the same manner as in the first embodiment. High frame rate image data is generated from the data, and the generated high frame rate image data is stored in the frame memory 14 via the memory access unit 12d.

Further, the imaging device 1 outputs the normal scan image data and the high frame rate image data stored in the frame memory 14 to the motion data generation device 2 in synchronization with the drive signal from the motion data generation device 2.
On the other hand, the motion data generation device 2 acquires the normal scanning image data and the high frame rate image data from the imaging device 1 via the image data acquisition unit 2a, and the acquired normal scanning image data and the high frame rate image data. Is stored in the image data storage unit 2b.

  When the motion data generation apparatus 2 receives a motion data generation instruction from the processor after a predetermined amount of image data is stored in the image data storage unit 2b ("Yes" branch in step S100), the position detection is performed. In the unit 2c, high frame rate image data corresponding to the instruction is acquired from the image data storage unit 2b (step S102). Then, the marker position is detected based on the acquired high frame rate image data.

The reason why the marker position is detected from the high frame rate image data will be described below.
When the moving body moves quickly, for example, as shown in FIG. 12A, in the image of the normal scanning image data, the position of the marker moves as shown in the current frame and one frame thereafter. It will be. On the other hand, since the high frame rate image data is thinned out three times during the period of one frame of normal scanning, as shown in FIG. 12B, the image content of the current frame of normal scanning image data ( This is an image in which the movement of the marker up to (marker position) is captured in three stages.

That is, since the frame rate of the high frame rate image data is three times that of the normal scan image data, the thinning scan is performed rather than the change amount of the marker position between frames of the normal scan image data shown in FIG. The amount of change in the marker position between each frame can be reduced. Thereby, it is possible to accurately capture the movement of the marker.
For example, assume that the rectangular marker image in the state shown in FIG. 13A is rotated 90 ° clockwise in the current frame one frame after the normal exposure time. In this case, if the frame rate is higher than the frame rate of the normal exposure time, an image in the middle of rotation can be obtained as shown in FIG. 13B, so that it can be determined that the marker has rotated. On the other hand, as shown in FIG. 13C, an image captured at the frame rate of the normal exposure time is an image after being rotated by 90 °, and thus it cannot be determined that the marker has rotated. Therefore, in this embodiment, in order to accurately capture the movement of the marker, it is desirable to increase the number of thinning scans as much as possible.

  When the position detection of the marker is completed for the high frame rate image data acquired from the image data storage unit 2b, the position detection unit 2c outputs the detection result to the motion data generation unit 2d (step S106). The marker position detection result includes, for example, marker type (attached position) information and its coordinate information. The motion data generation unit 2d generates motion data based on the coordinate information for each marker type based on the detection result input from the position detection unit 2c (step S108). Here, as shown in FIG. 14, the amount of change in position can be calculated from the coordinates of the marker in the previous frame and the coordinates of the marker in the current frame, so here, for each type of marker, Motion data including coordinate information and position change information is generated. When the motion data is generated in this way, the motion data generation unit 2d stores the generated motion data in the motion data storage unit 2f (step S110).

  In addition, when an instruction to extract texture data is sent from the processor (“Yes” in step S112), the motion data generation device 2 responds to the instruction from the image data storage unit 2b in the texture data extraction unit 2e. The normal scanning image data is acquired (step S114). Then, texture data is extracted from the acquired normal scanning image data (step S116).

  Here, as shown in FIG. 12A, the image of the normal scanning image data is a clear image of both the operating body and the marker, while the image of the high frame rate image data is shown in FIG. As shown in b), since the exposure time is shortened as the frame rate is increased, the image other than the marker having a large light amount is crushed in black. That is, texture data used for texture mapping in three-dimensional computer graphics cannot be obtained from an image captured at a high frame rate. In the imaging apparatus 1 according to the present invention, image data at the time of exposure with the normal exposure time is obtained together with the high frame rate image data, so that texture data that can be used for texture mapping is extracted from the normal scan image data. be able to.

The extraction of the texture data may be automatically performed by a program, or may be manually performed by an operation of an operation unit (not shown) by a user.
When the texture data extraction is completed, the texture data extraction unit 2e stores the extracted texture data in the motion data storage unit 2f (step S118).
As described above, in the motion data generation system 3 according to the present embodiment, in the imaging apparatus 1, the normal scanning image data composed of the pixel data at the time of exposure with the normal exposure time for the operating body with the marker, In the same exposure period as the normal exposure time, it is possible to simultaneously obtain a plurality of high-speed frame rate image data composed of pixel data read at a high frame rate by a plurality of thinning scans.

  In the motion data generation system 3 of the present embodiment, the motion data generation device 2 detects marker position information from the high-speed frame rate image data, and generates highly accurate motion data for the marker movement from the detection result. It is possible to extract texture data from normal scanned image data. That is, it is possible to generate both motion data and texture data from one-time imaging data for the moving object.

  In the second embodiment, the normal scanning image data is the image data generated by the normal captured image data generation unit of any one of modes 3, 10, 13 and 15, or the normal captured image data generation step of mode 15. The high frame rate image data corresponding to the image data generated in the above is image data or form generated by the high-speed captured image data generating means of any one of forms 3, 5, 8, 9, 11, 12 and 13. The sensor cell array 56 corresponds to the photoelectric conversion unit of any one of Embodiments 1, 3, 13, and 15, and corresponds to the image data generated in the 15 high-speed captured image data generation step. 100 at the time of normal exposure from the sensor cell array 56 by the reference timing generator 50, the scanning line scanner 54, and the drive pulse generator 52. The charge destructive readout process at the time of exposure corresponds to the first readout means of any one of Embodiments 1, 3, 9, and 13, or the first readout step of Embodiment 15, A plurality of reference timing generators 50, scanning line scanners 54, and drive pulse generators 52 from the sensor cell array 56 corresponding to any one of the first output channels 3, 13, and 15. The non-destructive readout process of charge by thinning-out scanning corresponds to the second readout means of any one of Embodiments 1, 3, 4, 9, and 13, or the second readout step of Embodiment 15, and the second output channel 60 is , Corresponding to the second output channel of any one of forms 1, 3, 5, 13 and 15.

  In the second embodiment, the generation processing of the high frame rate image data by the timing control unit 12b, the high frame rate image generation unit 12c, and the memory access unit 12d in the image processing unit 12 is performed in the third, fifth, 9 corresponds to the high-speed captured image data generation means of any one of 9, 10, 12, and 13 or the high-speed captured image data generation step of form 15, and the frame memory of the normal scan image data by the memory access unit 12d in the image processing unit 12 The writing process to 14 corresponds to the normal captured image data generation unit of any one of modes 3, 10 and 13, or the normal captured image data generation step of mode 15.

In the second embodiment, the position detector 2c corresponds to the position detector of form 9, the motion data generator 2d corresponds to the motion data generator of form 9, and the texture data extractor 2e. Corresponds to the texture data extraction means of form 10.
In the second embodiment, steps S100 to S106 correspond to the position detection step of form 11 or 12, and steps S108 to S110 correspond to the motion data generation step of form 11 or 12.

  In the first and second embodiments, the image pickup apparatus 1 performs nondestructive reading of a pixel signal in a plurality of thinning scans, and one first output channel for destructively reading out a pixel signal during normal exposure. However, the present invention is not limited to this, and a plurality of output channels for performing non-destructive reading by thinning scanning are provided according to the number of times of thinning scanning. It is good also as a structure.

  The configuration of the imaging apparatus when the number of output channels for nondestructive reading is increased by 1 will be described below with reference to FIGS. Here, FIG. 15 is a diagram illustrating a schematic configuration of the imaging device 4 having two non-destructive output channels. FIG. 16 is a diagram illustrating an example of exposure and pixel signal readout operations for each pixel column in the imaging apparatus 4. FIG. 17 is a diagram illustrating an example of thinning scanning using two non-destructive output channels.

The imaging apparatus 4 having a configuration in which a third output channel for performing nondestructive readout is added to the imaging apparatus 1 in the first and second embodiments has an added third output channel as shown in FIG. The third AFE 106 for converting the pixel signal output from the digital signal 64 into digital pixel data is added.
Specifically, the third output channel 64 outputs the pixel signal read by the scanning line scanner 54 to the third AFE 106 via the internal line memory. The third AFE 106 converts the input pixel signal into pixel data that is digital data, and outputs the pixel data to the memory access unit 12 d of the image processing unit 12.

On the other hand, when the third output channel 64 is added, the exposure and pixel signal readout operations for each pixel column in the sensor cell array 56 of the image sensor 100 are as shown in FIG.
As shown in FIG. 16, first, a first readout line for performing readout of a pixel signal at the time of exposure at a normal exposure time of each pixel column in the sensor cell array 56 and resetting accumulated charge after readout of the pixel signal by normal scanning. In addition to the selection, the thinning scan is divided into a plurality of times (for example, 8 times in the normal exposure time (T1 to T16)), and divided into the first set (T1, T5, T9, and T13 four times) and the second set (T3). , T7, T11, and T15), the pixel signal to be scanned in the sensor cell array 56 for the first set is subjected to non-destructive readout of the pixel signal during exposure with the exposure time during each scan. 2 read lines are selected, and non-destructive readout of pixel signals at the time of exposure is performed for the second set of pixel rows to be scanned in the sensor cell array 56 at the exposure time at each scan. To select the cormorant third read line.

  Then, in one exposure period (normal exposure time), the pixel signal is read out at the normal exposure time for each pixel column by the normal scan, the accumulated charge is reset, and the thinning scan is performed a plurality of times in the first set. Non-destructive readout of the pixel signal at the time of exposure with the exposure time at the time of each scanning with respect to the pixel row to be scanned, and exposure at the time of exposure at the time of each scanning with respect to each pixel row to be scanned by multiple thinning scanning in the second set The non-destructive readout of the pixel signals is independently performed.

  That is, as shown in FIG. 16, the first readout line, the second readout line, and the third readout line are usually connected to any one of these pixel columns in all pixel columns (for example, the first to twentieth lines). When the charge for the exposure time is accumulated, the first readout line is selected to read out the pixel signal of the pixel column and reset the accumulated charge. In addition, the scanning timing of the pixel column whose accumulated charge has been reset is controlled so that when the first readout line is scanned again after scanning the other pixel column, the exposure for just the normal exposure time is performed. The In addition, the scanning of each pixel column by the normal scanning is sequentially performed while looping all the pixel columns (for example, the first to twentieth lines) for each frame. In the example of FIG. 16, scanning is sequentially performed for each line from the first line (bottom line) to the twentieth line (top line), and when the scanning reaches the twentieth line, one frame period is reached. The scanning for the next frame is again performed from the first line.

  On the other hand, in the thinning-out scanning, after the accumulated charge is reset by the first readout line, while the charge for the normal exposure time is accumulated, the exposure of each pixel column to be scanned is performed at the exposure time for each scan with respect to the first set. The second readout lines are selected so as to sequentially read out the pixel signals in a non-destructive manner. Furthermore, while the charge for the normal exposure time is accumulated, the third readout lines respectively read out the pixel signals of the respective pixel columns to be scanned sequentially in the exposure time for each scan with respect to the second set. Selected.

At this time, in order to align the exposure time, each readout line is selected so that the scanning interval (time interval) between the second readout line and the third readout line is the same.
As a result, when the accumulated charge of the pixel column to be scanned is reset by the first readout line, the pixel row after the reset is exposed with the exposure time at each scanning in the second readout line for the first set. Nondestructive readout of the pixel signals of the pixels that have been performed is sequentially performed in the order of each scanning.

Further, when scanning by the second readout line is started, exposure is performed at the exposure time at each scanning in the third readout line with respect to the second set so that the scanning interval becomes the exposure time T as shown in FIG. Nondestructive readout of the pixel signals of the pixels that have been performed is sequentially performed in the order of each scanning.
In the thinning scanning, scanning is performed while thinning out a predetermined number of pixel rows in accordance with the thinning amount designated by the system controller, and when the pixel row of the highest number is reached among the pixel rows to be scanned, the pixels to be scanned Returning to the lowest-numbered pixel row in the row, scanning is sequentially performed while looping, as in the normal scanning. However, in the thinning scanning, the loop is repeated by the number of scans that can be executed (or set in advance) in one frame of the normal exposure time.

  Then, as shown in FIG. 16, the pixel signal (analog data) at the time of exposure with the normal exposure time read by the normal scan is read to the first line memory S of the first output channel 58 (CH1), On the other hand, the pixel signal immediately after the reset is read out to the first line memory N of CH1. On the other hand, the pixel signal for each scan read by the plurality of thinning scans for the first set is read for each scan to the second line memory of the second output channel (CH2) and sent to the second AFE 104. The data is output and converted into digital data (pixel data). In addition, the pixel signal for each scan read by the plurality of thinning scans for the second set is read for each scan to the third line memory of the third output channel (CH3) and sent to the third AFE 106. The data is output and converted into digital data (pixel data).

  In the case of the imaging device 4 having the above-described configuration, for example, as shown in FIG. 17, the third output channel (CH 3) in the same exposure period apart from the four thinning scans for the second output channel (CH 2). Therefore, the pixel signal can be read out at a frame rate that is 8 times the frame rate that is four times higher than that in the normal scan with CH2 alone. As described above, the frame rate can be easily increased by adding a physical output channel.

The imaging device 4 may be configured such that the thinning scan is not performed in each set of scans performed at the same scan interval when a plurality of scans are divided into a plurality of groups. In this case, since thinning is not performed, the pixel signal can be read out at a frame rate that is twice the number of channels for performing nondestructive reading.
In the imaging apparatus 4, the number of channels for nondestructive readout is two CH2 and CH3. However, the configuration is not limited to this, and the number of channels may be three or more.

  In the second embodiment, the imaging apparatus 1 according to the present invention is applied to motion capture technology. However, the imaging apparatus 1 according to the present invention converts high frame rate image data into, for example, an image in machine vision. A system that requires both a high frame rate image to be imaged and a normal scan image, such as for use in an analysis process such as analysis, and the normal scan image data is used for an application that requires a clear image such as a display application. As long as it is applicable to any system.

  For example, when the imaging apparatus 1 according to the present invention is applied to a videophone camera, normal scanning image data is used for display of the other party's videophone, and high frame rate image data is used for analysis of the operation of the speaker on this side. Each of them can be applied to a videophone system that realizes an interface capable of operating a videophone in response to the movement of the speaker (for example, the movement of the fingertip of the speaker).

  In addition, in an automobile or the like, the imaging apparatus 1 according to the present invention is applied to a camera for imaging other automobiles, a traveling road surface, a guard rail, or the like, and the normal scan image data is displayed for displaying the captured image to the driver. The high frame rate image data is used for image processing for detecting danger and the like, and can be applied to a monitoring system for monitoring danger during traveling. In addition, by applying the imaging device 1 according to the present invention to a camera that captures a monitoring target, and using normal scanning image data for display and high frame rate image data for detection of dangerous objects such as intruders, It can be applied to various monitoring systems.

1 is a block diagram illustrating a schematic configuration of an imaging apparatus 1 according to the present invention. It is a block diagram which shows the internal structure of the image pick-up element 100 which concerns on this invention. 2 is a diagram illustrating a detailed configuration of a sensor cell array 56. FIG. 2 is a diagram showing an internal configuration of a scanning line scanner 54. FIG. (A) is a figure which shows an example of a thinning scanning, (b) is a figure which shows transition of the level (accumulated electric charge amount) of the pixel signal output from the sensor cell 56a by the thinning scanning of (a). 4 is a diagram illustrating an example of exposure and pixel signal read operations for each pixel column in the sensor cell array 56 of the image sensor 100. FIG. 2 is a diagram illustrating an internal configuration of an image processing unit 12. FIG. (A) is a figure which shows an example of a normal scanning and a thinning-out scanning, (b) is a figure which shows transition of the accumulated charge amount with respect to a certain pixel of a 10th line. It is a block diagram which shows schematic structure of the motion data generation system 3 which concerns on this invention. 3 is a block diagram illustrating a detailed configuration of a motion data generation device 2. FIG. It is a flowchart which shows the production | generation process of motion data, and the extraction process of texture data. (A) And (b) is a figure which shows an example of a normal scanning image and a high frame rate image. It is explanatory drawing about the imaging result by the difference in a frame rate. It is a figure which shows an example of the production | generation method of motion data. It is a figure which shows schematic structure of the imaging device 4 which has two nondestructive output channels. 6 is a diagram illustrating an example of an exposure operation and a pixel signal readout operation for each pixel column in the imaging apparatus 4. FIG. It is a figure which shows an example of the thinning scanning using two nondestructive output channels.

Explanation of symbols

1 is an imaging device, 2 is a motion data generation device, 3 is a motion data generation system, 10 is an imaging processing unit, 12 is an image processing unit, 14 is a frame memory, 100 is an imaging device, 102 is a first AFE, and 104 is Second AFE, 50 is a reference timing generator, 52 is a drive pulse generator, 54 is a scanning line scanner, 56 is a sensor cell array, 58 is a first output channel, 60 is a second output channel, 54a is a normal scan counter, 54b is a normal scanning address decoder, 54c is a thinning scanning counter, 54d is a thinning scanning address decoder, 54e is an OR logic, 12a is a communication device, 12b is a timing control unit, 12c is a high frame rate image generation unit, and 12d is a memory access unit. , 12e is an output reading unit, 2a is an image data acquisition unit, and 2b is an image data recording unit. Parts, 2c position detection unit, 2d is motion data generating unit, 2e texture data extraction unit, 2f is motion data storage unit

Claims (16)

A plurality of photoelectric conversion elements that convert the exposed light into electric charges and store the photoelectric conversion units arranged in a matrix; and
First readout means for scanning each pixel constituting the photoelectric conversion element, and reading out the charge when exposed for a predetermined exposure time from each scanned pixel by a destructive readout method with reset of accumulated charge;
During the predetermined exposure time, a plurality of scans performed with respect to each pixel at a plurality of types of exposure times less than or equal to the predetermined exposure time, and a scanning interval of each pixel by the plurality of types of exposure time is the same time interval made as performs the reading of the electric load from each of the pixels in each round of scanning for each exposure time of the plurality of types of exposure time, the second read performed by non-destructive readout schemes without resetting the accumulated charge Means,
A first output that is read out by the first reading means and that sequentially stores pixel data composed of charges when exposed for the predetermined exposure time in a memory, and sequentially outputs the pixel data stored in the memory in the order of storage. Channel,
The pixel data read by said second reading means, said plurality of sequentially stored in the memory for each scan of each round in the scan, and a second output channel for sequentially outputting pixel data stored in the memory in the order in which the stored With
The second output channel is configured to output the pixel data for each line of the pixels,
The second reading unit is configured to perform the plurality of scans on the pixels of the remaining lines obtained by thinning out pixels of a predetermined line from a pixel group formed by the plurality of photoelectric conversion elements. An image pickup device , wherein a plurality of scans for each pixel are performed, and a scan for each pixel is performed for the next time after the previous scan for each pixel is completed . A plurality of photoelectric conversion elements that convert the exposed light into electric charges and store the photoelectric conversion units arranged in a matrix; and
First readout means for scanning each pixel constituting the photoelectric conversion element, and reading out the charge when exposed for a predetermined exposure time from each scanned pixel by a destructive readout method with reset of accumulated charge;
During the predetermined exposure time, each of the pixels based on the N types of exposure times is scanned N times with respect to each of the pixels with N types of exposure times (N is a natural number of 2 or more) less than the predetermined exposure time. together with the scanning interval performed to have the same time interval, the N type of the reading of the electric load from each pixel in each round of scanning for each exposure time of the exposure time, without resetting the accumulated charge non A second reading means for performing destructive reading;
A first output that is read out by the first reading means and that sequentially stores pixel data composed of charges when exposed for the predetermined exposure time in a memory, and sequentially outputs the pixel data stored in the memory in the order of storage. Channel,
The pixel data read by said second reading means is stored in the sequential memory for each scanning of the times, and sequentially outputs the pixel data stored in the memory in the order in which the stored, N pieces of the independent for each scan of each round includes second to the (N + 1) th output channel, a,
The second output channel is configured to output the pixel data for each line of the pixels,
The second reading unit is configured to perform the N scans on the pixels of the remaining lines obtained by thinning out pixels of a predetermined line from the pixel group formed by the plurality of photoelectric conversion elements. An image pickup device , wherein each pixel is scanned N times, and after each previous pixel is scanned, each pixel is scanned the next time . A plurality of photoelectric conversion elements that convert the exposed light into electric charges and store the photoelectric conversion units arranged in a matrix; and
First readout means for scanning each pixel constituting the photoelectric conversion element, and reading out the charge when exposed for a predetermined exposure time from each scanned pixel by a destructive readout method with reset of accumulated charge;
During the predetermined exposure time, a plurality of scans performed with respect to each pixel at a plurality of types of exposure times less than or equal to the predetermined exposure time, and a scanning interval of each pixel by the plurality of types of exposure time is the same time interval made as performs the reading of the electric load from each of the pixels in each round of scanning for each exposure time of the plurality of types of exposure time, the second read performed by non-destructive readout schemes without resetting the accumulated charge Means,
Said read by the first reading means, the predetermined pixel data composed of a charge upon exposure exposure time stored sequentially in the memory, the first pixel data stored in the memory sequentially output in the order in which the stored An output channel,
The pixel data read by said second reading means, said plurality of sequentially stored in the memory for each scan of each round in the scan, and a second output channel for sequentially outputting pixel data stored in the memory in the order in which the stored ,
Normal captured image data generating means for generating captured image data when exposed for a predetermined exposure time based on the pixel data output from the first output channel;
High-speed captured image data generating means for generating captured image data for each scan in the plurality of scans based on pixel data output from the second output channel,
The second output channel is configured to output the pixel data for each line of the pixels,
The second reading unit is configured to perform the plurality of scans on the pixels of the remaining lines obtained by thinning out pixels of a predetermined line from a pixel group formed by the plurality of photoelectric conversion elements. An imaging apparatus , wherein a plurality of scans for each pixel are performed, and a scan for each pixel is performed the next time after the previous scan for each pixel is completed . The high-speed captured image data generation means includes a pixel value of pixel data output from the second output channel and one of the scans from the second output channel for each scan in the plurality of scans. The difference value with the pixel value of the pixel data output in the previous scan is calculated, and captured image data corresponding to each of the scans is generated based on the difference value. 3. The imaging device according to 3 .   The imaging apparatus according to claim 4, wherein the high-speed captured image data generation unit sets the difference value to 0 when the difference value becomes a negative value. A plurality of photoelectric conversion elements that convert the exposed light into electric charges and store the photoelectric conversion units arranged in a matrix; and
First readout means for scanning each pixel constituting the photoelectric conversion element, and reading out the charge when exposed for a predetermined exposure time from each scanned pixel by a destructive readout method with reset of accumulated charge;
During the predetermined exposure time, each of the pixels based on the N types of exposure times is scanned N times with respect to each of the pixels with N types of exposure times (N is a natural number of 2 or more) less than the predetermined exposure time. together with the scanning interval performed to have the same time interval, the N type of the reading of the electric load from each pixel in each round of scanning for each exposure time of the exposure time, without resetting the accumulated charge non A second reading means for performing destructive reading;
A first output that is read out by the first reading means and that sequentially stores pixel data composed of charges when exposed for the predetermined exposure time in a memory, and sequentially outputs the pixel data stored in the memory in the order of storage. Channel,
The pixel data read by said second reading means is stored in the sequential memory for each scanning of the times, and sequentially outputs the pixel data stored in the memory in the order in which the stored, N pieces of the independent for each scan of each round second to (N + 1) and output channels,
Normal captured image data generating means for generating captured image data when exposed for a predetermined exposure time based on the pixel data output from the first output channel;
Based on the pixel data output from said second to (N + 1) output channel, and an high-speed image data generating means for generating respective image data of each round of each scan in the scanning of the N times,
The second to (N + 1) th output channels output the pixel data for each line of the pixels,
The second reading unit is configured to perform the N scans on the pixels of the remaining lines obtained by thinning out pixels of a predetermined line from the pixel group formed by the plurality of photoelectric conversion elements. An image pickup apparatus , wherein each pixel is scanned N times, and after each previous pixel is scanned, each pixel is scanned the next time . The high-speed captured image data generation means includes a pixel value of pixel data output from the second to (N + 1) th output channels and each of the second to (N + 1) th times of the plurality of scans. A difference value with respect to the pixel value of the pixel data output in the previous scan of each of the scans from the output channel is calculated, and the captured image data corresponding to each of the scans based on the difference value The imaging apparatus according to claim 6, wherein:   8. The imaging apparatus according to claim 7, wherein the high-speed captured image data generation unit sets the difference value to 0 when the difference value becomes a negative value. A motion data generation system that generates motion data indicating the operation content of an operating body that performs a predetermined operation,
An imaging apparatus according to any one of claims 3 to 8, comprising:
The imaging apparatus reads out pixel data at the time of exposure with the predetermined exposure time for the imaging target including the operating body provided with a position detection marker by the first reading unit, and at the N types of exposure times. Read pixel data at the time of exposure by the second reading means,
Position detecting means for detecting the position of the marker based on the captured image data generated by the high-speed captured image data generating means in the imaging device;
And a motion data generation unit that generates the motion data based on a detection result of the position detection unit.   The motion data generation system according to claim 9, further comprising a texture data extraction unit that extracts texture data of the operating body from the captured image data generated by the normal captured image data generation unit in the imaging apparatus. A motion data generation program for generating motion data indicating the operation content of an operating body that performs a predetermined operation,
9. The captured image data for the imaging target including the operating body with the position detection marker generated by the high-speed captured image data generation unit in the imaging apparatus according to claim 3. Based on the position detection step of detecting the position of the marker,
A motion data generation program, comprising: a program for causing a computer to execute a process including a motion data generation step of generating the motion data based on a detection result of the position detection step. A motion data generation method for generating motion data indicating an operation content of an operating body that performs a predetermined operation,
9. The captured image data for the imaging target including the operating body with the position detection marker generated by the high-speed captured image data generation unit in the imaging apparatus according to claim 3. Based on the position detection step of detecting the position of the marker,
And a motion data generation step of generating the motion data based on a detection result of the position detection step. A plurality of photoelectric conversion elements that convert the exposed light into electric charges and store the photoelectric conversion units arranged in a matrix; and
First readout means for scanning each pixel constituting the photoelectric conversion element, and reading out the charge when exposed for a predetermined exposure time from each scanned pixel by a destructive readout method with reset of accumulated charge;
During the predetermined exposure time, a plurality of scans performed with respect to each pixel at a plurality of types of exposure times less than or equal to the predetermined exposure time, and a scanning interval of each pixel by the plurality of types of exposure time is the same time interval made as performs the reading of the electric load from each of the pixels in each round of scanning for each exposure time of the plurality of types of exposure time, the second read performed by non-destructive readout schemes without resetting the accumulated charge Means,
Said read by the first reading means, the predetermined pixel data composed of a charge upon exposure exposure time stored sequentially in the memory, the first pixel data stored in the memory sequentially output in the order in which the stored An output channel,
The pixel data read by said second reading means, said plurality of sequentially stored in the memory for each scan of each round in the scan, and a second output channel for sequentially outputting pixel data stored in the memory in the order in which the stored ,
Normal captured image data generating means for generating captured image data when exposed for a predetermined exposure time based on the pixel data output from the first output channel;
High-speed captured image data generating means for generating captured image data for each scan in the plurality of scans based on pixel data output from the second output channel,
The second output channel is configured to output the pixel data for each line of the pixels,
The second reading unit is configured to perform the plurality of scans on the pixels of the remaining lines obtained by thinning out pixels of a predetermined line from a pixel group formed by the plurality of photoelectric conversion elements. An imaging system , wherein each pixel is scanned a plurality of times, and after each previous pixel is scanned, each pixel is scanned the next time . A plurality of photoelectric conversion elements that convert the exposed light into electric charges and store the photoelectric conversion units arranged in a matrix; and
First readout means for scanning each pixel constituting the photoelectric conversion element, and reading out the charge when exposed for a predetermined exposure time from each scanned pixel by a destructive readout method with reset of accumulated charge;
During the predetermined exposure time, each of the pixels based on the N types of exposure times is scanned N times with respect to each of the pixels with N types of exposure times (N is a natural number of 2 or more) less than the predetermined exposure time. together with the scanning interval performed to have the same time interval, the N type of the reading of the electric load from each pixel in each round of scanning for each exposure time of the exposure time, without resetting the accumulated charge non A second reading means for performing destructive reading;
A first output that is read out by the first reading means and that sequentially stores pixel data composed of charges when exposed for the predetermined exposure time in a memory, and sequentially outputs the pixel data stored in the memory in the order of storage. Channel,
The pixel data read by said second reading means is stored in the sequential memory for each scanning of the times, and sequentially outputs the pixel data stored in the memory in the order in which the stored, N pieces of the independent for each scan of each round second to (N + 1) and output channels,
Normal captured image data generating means for generating captured image data when exposed for a predetermined exposure time based on the pixel data output from the first output channel;
Based on the pixel data output from said second to (N + 1) output channel, and an high-speed image data generating means for generating respective image data of each round of each scan in the scanning of the N times,
The second to (N + 1) th output channels output the pixel data for each line of the pixels,
The second reading unit is configured to perform the N scans on the pixels of the remaining lines obtained by thinning out pixels of a predetermined line from the pixel group formed by the plurality of photoelectric conversion elements. An imaging system comprising: N scans for each pixel, and scans for each pixel of the next time after the previous scan for each pixel is completed . An imaging method used in an imaging device having a photoelectric conversion unit in which a plurality of photoelectric conversion elements that convert exposed light into charges and store the matrix are arranged in a matrix,
A first reading step of scanning each pixel constituting the photoelectric conversion element, and reading out charges when exposed for a predetermined exposure time from each of the scanned pixels by a destructive reading method with reset of accumulated charges;
During the predetermined exposure time, a plurality of scans performed with respect to each pixel at a plurality of types of exposure times less than or equal to the predetermined exposure time, and a scanning interval of each pixel by the plurality of types of exposure time is the same time interval made as performs the reading of the electric load from each of the pixels in each round of scanning for each exposure time of the plurality of types of exposure time, the second read performed by non-destructive readout schemes without resetting the accumulated charge Steps,
The read in the first reading step, the predetermined pixel data composed of a charge upon exposure exposure time stored sequentially in the memory, the first pixel data stored in the memory sequentially output in the order in which the stored A normal captured image data generation step for generating captured image data when exposed for a predetermined exposure time based on pixel data output from the output channel;
The pixel data read by said second reading step, from the times of sequentially stored in the memory for each scan, a second output channel for sequentially outputting pixel data stored in the memory in the order in which the stored in the plurality of scans based on the pixel data output, anda high-speed image data generation step of generating each captured image data for each scan of each round in the plural scans,
The second output channel is configured to output the pixel data for each line of the pixels,
In the second reading step, the plurality of scans are performed on the pixels of the remaining lines obtained by thinning out pixels of a predetermined line from the pixel group constituting the plurality of photoelectric conversion elements, and the plurality of scans are performed on each pixel. In the imaging method, the scanning of each pixel of the next time is performed after the scanning of each pixel of the previous time is completed . An imaging method used in an imaging device having a photoelectric conversion unit in which a plurality of photoelectric conversion elements that convert exposed light into charges and store the matrix are arranged in a matrix,
A first reading step of scanning each pixel constituting the photoelectric conversion element, and reading out charges when exposed for a predetermined exposure time from each of the scanned pixels by a destructive reading method with reset of accumulated charges;
During the predetermined exposure time, each of the pixels based on the N types of exposure times is scanned N times with respect to each of the pixels with N types of exposure times (N is a natural number of 2 or more) less than the predetermined exposure time. together with the scanning interval performed to have the same time interval, the N type of the reading of the electric load from each pixel in each round of scanning for each exposure time of the exposure time, without resetting the accumulated charge non A second reading step performed by a destructive reading method;
The first output that sequentially reads out the pixel data composed of the charges when exposed for the predetermined exposure time, and sequentially outputs the pixel data stored in the memory in the stored order, which is read out in the first readout step. A normal captured image data generation step for generating captured image data when exposed for a predetermined exposure time based on pixel data output from the channel;
The pixel data read by the second read step stored in the sequential memory for each scanning of the times, and sequentially outputs the pixel data stored in the memory in the order in which the stored, N pieces of the independent for each scan of each round based on the second to (N + 1) pixel data outputted from the output channel includes a high-speed image data generation step of generating each captured image data for each scan of each round in the N scans,
The second to (N + 1) th output channels output the pixel data for each line of the pixels,
The second reading step, the N times of scanning, performs a pixel of the remaining lines from the pixel group constituting thinned out pixels of a predetermined line of said plurality of photoelectric conversion elements, N times for each pixel In the imaging method, the scanning of each pixel of the next time is performed after the scanning of each pixel of the previous time is completed .

Images