JP7100590B2 - Image sensor - Google Patents

Description

This technique relates to an image sensor. More specifically, the present invention relates to an image sensor for controlling a controlled target device using captured image data.

Conventionally, there is known an image processing device that captures images of an object at predetermined intervals and generates an image representing the trajectory of the object based on those images. For example, there has been proposed an image processing device that obtains a locus of an object between the time when the first image is taken and the time when the second image is taken and interpolates the locus using the found locus (for example, a patent). See Document 1).

Japanese Unexamined Patent Publication No. 2015-08794

In the above-mentioned conventional technique, interpolation processing is performed in order to obtain the trajectory of an object from an image of the object taken at predetermined intervals. Generally, a processing speed of about 30 to 120 fps (frames / second) is sufficient for displaying an image, but that is not enough for advanced control. Therefore, although it is possible to draw the locus of the obtained object, it is difficult to use it for controlling other devices. That is, it is difficult to control other devices in real time based on the trajectory of the object generated by the interpolation process.

This technique was created in view of such a situation, and aims to control the device based on the trajectory of an object by high-speed image data different from low-speed image data for display.

The present technology has been made to solve the above-mentioned problems, and the first aspect thereof is an image pickup element that images an object and generates a frame of image data arranged in a time series, and the above frame. The difference between the binarization processing unit that performs binarization processing for each and generates the binarization frame and the binarization frame adjacent to the time series is generated and included in the binarization frame. A tracking processing unit that tracks changes in the position of the object, a moment generation unit that calculates the moment of the object included in the binarization frame based on the result of the tracking processing unit, and a moment generation unit. Based on the center of gravity position generation unit that generates the center of gravity position of the object included in the binarization frame based on the moment generated by, and the center of gravity position of the object generated by the center of gravity position generation unit. An image including a locus generation unit that predicts and generates a locus of the object, and a control signal supply unit that supplies a control signal to the control target device based on the locus of the object generated by the locus generation unit. It is a sensor. This has the effect of controlling the controlled object device based on the predicted trajectory of the object.

Further, in the first aspect, the locus generation unit predicts a locus of a distance to the object, and the control signal supply unit predicts the control signal when the predicted distance satisfies a predetermined condition. May be supplied. This has the effect of controlling the controlled device according to the predicted distance.

Further, in the first aspect, the control signal may be a control signal for controlling the operation of the cushioning member. This has the effect of controlling the operation of the cushioning member based on the predicted trajectory of the object.

Further, in the first aspect, the control signal may be a control signal for controlling the operation of the warning device. This has the effect of controlling the operation of the warning device based on the predicted trajectory of the object.

Further, in the first aspect, the trajectory generation unit predicts the trajectory of the trajectory of the object, and the control signal supply unit supplies the control signal for displaying the predicted trajectory. You may do it. This has the effect of displaying the predicted trajectory of the object.

Further, in the first aspect, the locus generation unit predicts the flight distance of the object, and the control signal supply unit supplies the control signal for displaying the predicted flight distance. You may do it. This has the effect of displaying the predicted flight distance.

Further, in the first aspect, the locus generation unit predicts the arrival position of the object, and the control signal supply unit supplies the control signal for displaying the predicted arrival position. You may do it. This has the effect of displaying the predicted arrival position.

Further, in the first aspect, the control signal supply unit may supply the control signal for outputting the audio signal according to the predicted arrival position. This has the effect of outputting an audio signal according to the predicted arrival position.

Further, in the first aspect, the image pickup device may be further provided with an optical element for supplying an image containing the object, and the control signal may be a control signal for controlling the operation of the optical element. This has the effect of controlling the operation of the optical element based on the predicted trajectory of the object.

Further, in the first aspect, the control signal may be a control signal in which a part of the frame is replaced with a predetermined image and displayed. This has the effect of replacing the image based on the predicted trajectory of the object.

According to the present technique, it is possible to obtain an excellent effect that the device can be controlled based on the locus of an object by high-speed image data different from low-speed image data for display. The effects described herein are not necessarily limited, and may be any of the effects described in the present disclosure.

It is a figure which shows one configuration example of the control system in embodiment of this technique. It is a figure which shows the example of the control of a cushioning member in the 1st application example of embodiment of this technique. It is a figure which shows the other example of the control of a cushioning member in the 1st application example of embodiment of this technique. It is a flow chart which shows the processing procedure example of the control of a cushioning member in the 1st application example of embodiment of this technique. It is a figure which shows the example of the sphere simulation in the 2nd application example of the Embodiment of this technique. It is a figure which shows the example of the locus of a sphere in the 2nd application example of embodiment of this technique. It is a flow chart which shows the processing procedure example of the sphere simulation in the 2nd application example of embodiment of this technique. It is a figure which shows the example of the announcement in the field in the 3rd application example of the embodiment of this technique. It is a figure which shows the mode of predicting the invasion of a ball into the audience seat in the 3rd application example of embodiment of this technique. It is a figure which shows the system configuration example in the 3rd application example of the Embodiment of this technique. It is a flow chart which shows the processing procedure example of the announcement in the field in the 3rd application example of embodiment of this technique. It is a figure which shows the example of the volcanic bomb warning in the 4th application example of the Embodiment of this technique. It is a flow chart which shows the processing procedure example of the volcanic bomb warning in the 4th application example of the Embodiment of this technique. It is a figure which shows the example of the follow-up imaging in the 5th application example of the Embodiment of this technique. It is a figure which shows the mode of the follow-up imaging in the 5th application example of embodiment of this technique. It is a figure which shows the other aspect of the follow-up imaging in the 5th application example of embodiment of this technique. It is a flow chart which shows the processing procedure example of the follow-up imaging in the 5th application example of embodiment of this technique. It is a figure which shows the example of image superimposition in the 6th application example of embodiment of this technique. It is a figure which shows the timing example of image superimposition in the 6th application example of embodiment of this technique. It is a flow chart which shows the processing procedure example of image superimposition in the 6th application example of embodiment of this technique. It is a figure which shows the example of the distance calculation in the 7th application example of the Embodiment of this technique. It is a figure which shows the aspect of the object tracking in the 7th application example of the Embodiment of this technique. It is a figure which shows the mode of distance calculation in the 7th application example of embodiment of this technique. It is a flow chart which shows the processing procedure example of the distance calculation in the 7th application example of embodiment of this technique.

Hereinafter, embodiments for carrying out the present technology (hereinafter referred to as embodiments) will be described. The explanation will be given in the following order.
1. 1. Embodiment (configuration example of control system)
2. 2. First application example (example of control of cushioning member)
3. 3. Second application example (example of sphere simulation)
4. Third application example (example of announcement in the hall)
5. Fourth application example (example of volcanic bomb warning)
6. Fifth application example (example of follow-up imaging)
7. Sixth application example (example of image superimposition)
8. Seventh application example (example of distance calculation)

<1. Embodiment>
FIG. 1 is a diagram showing a configuration example of a control system according to an embodiment of the present technology. The image processing system in this embodiment includes an image sensor 100, a display device 320, and a controlled object device 330.

The image sensor 100 captures an image of a subject including an object, predicts the trajectory of the object, and controls the controlled object device 330 based on the trajectory of the object. The operation input device 310 receives an operation input from the outside. The display device 320 displays the information obtained by the image sensor 100. The controlled object device 330 is a device controlled by the image sensor 100.

The image sensor 100 controls the image pickup unit 110, the filter processing unit 120, the binarization processing unit 130, the tracking processing unit 140, the moment generation unit 150, the center of gravity position generation unit 160, and the aggregation processing unit 210. A unit 220 and an interface 230 are provided.

The image pickup unit 110 is an image pickup device that captures an image of a subject including an object. The image pickup unit 110 generates frames of image data arranged in time series at a predetermined frame rate. Here, as the frame rate, a high frame rate of 1000 frames (1000 fps) or more per second is assumed. It is not necessary that all the frames of the image data captured by the image pickup unit 110 be supplied to the outside of the image sensor 100. The high frame rate image data is intended for the control described below, and a lower frame rate is sufficient for display. That is, by limiting the image data having a high frame rate to the reference in the image sensor 100, it is possible to effectively utilize the bandwidth of the image sensor 100. The image pickup unit 110 is an example of the image pickup device described in the claims. In addition, the object is an object that includes not only living things such as humans and animals but also non-living things.

The filter processing unit 120 applies a filter process to each of the frames of the image data captured by the image pickup unit 110. As the filter processing in the filter processing unit 120, for example, noise removal processing by a moving average filter, a median filter, etc., contour detection processing by a Sobel filter, edge detection by a Laplacian filter, etc. are assumed. It is also possible to calculate the number of objects included in the image by obtaining the Euler number of the image by the filter processing unit 120. The Euler number is the number of components minus the number of holes.

The binarization processing unit 130 performs binarization processing on each of the frames filtered by the filter processing unit 120. The binarization processing unit 130 binarizes the image data based on the histogram information of the brightness and the color included in the image data of each frame, and generates a binarized frame composed of the binarized data.

The tracking processing unit 140 detects an object included in the binarization frame by generating a difference between frames adjacent to each other in the time series for the binarization frame generated by the binarization processing unit 130. , Tracks changes in the position of the object. When detecting an object, it is possible to specify a specific area in the image as a measurement target.

The moment generation unit 150 calculates the moment of the two-variable function in the binarization frame based on the result of the tracking processing unit 140. The 0th moment represents the amount of change in the area of the object included in the binarized frame, and is a value that is invariant with respect to rotation and enlargement / reduction of the image.

The center of gravity position generation unit 160 generates the center of gravity position of the object included in the binarization frame based on the moment generated by the moment generation unit 150. The value obtained by dividing each of the first moments in the horizontal direction and the vertical direction by the zeroth moment represents the position of the center of gravity.

The aggregation processing unit 210 performs aggregation processing based on various data obtained by the image sensor 100 to generate a locus of an object. As shown in the following application example, the aggregation processing unit 210 performs necessary processing according to the operating application. The control unit 220 controls the operation of each unit of the image sensor 100. The interface 230 controls the interface with the outside. In this example, the interface 230 is connected to the control target device 330 to supply a control signal to the control target device 330. Further, the interface 230 is connected to the display device 320, and the information obtained by the image sensor 100 is displayed on the display device 320. The aggregation processing unit 210 is an example of the locus generation unit described in the claims.

In this figure, the route in which the output from the center of gravity position generation unit 160 is supplied to the aggregation processing unit 210 and the control unit 220 is clearly shown, but various units of the image sensor 100 to the aggregation processing unit 210 are various. A route for supplying data may be provided as needed.

With this configuration, it is possible to predict the trajectory of the object and control the controlled object device based on the predicted trajectory. In addition, it is not necessary to process the image in the subsequent stage, and control can be performed in real time using only the calculation result.

<2. First application example>
FIG. 2 is a diagram showing an example of control of a cushioning member in the first application example of the embodiment of the present technique. In order to control the shock absorber using the control system in the present embodiment, an image sensor 100 is provided around the protected object to perform image pickup, the trajectory of the object is predicted, and the predicted trajectory is used. The cushioning member is controlled based on this. As a result, the protected object can be protected by the cushioning member against the danger of approaching the protected object.

In a in FIG. 2, the protected object 511 is, for example, a building, and a cushioning member 512 is provided on a fence around the building. The fence is provided with an image sensor 513 as an example of the image sensor 100. When a moving object 514 such as a motorcycle or an automobile approaches the protected object 511, the image sensor 513 detects the moving object 514 as a collision prediction target and determines whether or not the moving object 514 collides with the protected object 511. Make an identification.

When it is determined from the moving speed and the moving direction of the moving object 514 that there is a danger of collision, a warning is given to apply the brake to the moving object 514 at the position P1. For example, a warning lamp is provided on the fence and the warning lamp is turned on to give a warning. Further, when it is determined that the moving speed does not slow down and the collision cannot be physically avoided, the cushioning member 512 is controlled to be inflated and put into a defensive state at the timing when the moving object 514 reaches the position P2.

In b in FIG. 2, the protection target 515 is a human being. In this example, it is assumed that a human carries a cushioning member 516 and an image sensor 517 on the back. At this time, when the moving object 518 approaches from the back surface of the protected object 515 and the image sensor 517 predicts a collision, the cushioning member 516 is controlled to be inflated to be in a defensive state.

On the other hand, ordinary automobile airbags are designed to operate when they are actually impacted by a collision, and cannot be protected in advance. There is also the possibility of injury due to the sudden bulge itself. According to the cushioning member of this example, such inconvenience can be avoided. Further, since the cushioning member in this example is not an acceleration sensor but an image sensor, it can be installed on a protected object that does not move.

FIG. 3 is a diagram showing another example of control of the cushioning member in the first application example of the embodiment of the present technique. In this example, the protection target is a human and the collision prediction target is the ground.

In a in FIG. 3, it is assumed that a human holds the cushioning member 521 in front and attaches an image sensor 522 that images the front to the head. At this time, when the person to be protected falls forward and the image sensor 522 predicts a collision with the ground, the cushioning member 521 is controlled to be inflated to be in a defensive state.

In b in FIG. 3, it is assumed that a human carries a cushioning member 523 on his back and wears an image sensor 524 that captures an image of the rear on his head. At this time, when the person to be protected falls backward and the image sensor 524 predicts a collision with the ground, the cushioning member 523 is controlled to be inflated to be in a defensive state.

In c in FIG. 3, it is assumed that a cushioning member is spread on the ground and an image sensor 526 for photographing the human is arranged in a place where a human is working at a high place. At this time, when the image sensor 526 predicts a collision with the ground when a person to be protected falls, the cushioning member 525 at the predicted fall position is inflated and controlled to be in a defensive state.

As described above, in this example, the image sensor may be attached to the protected object itself, or may be arranged at a position where the movement of the protected object can be observed. Similarly, the cushioning member may be attached to the protected object itself, or may be arranged at a position where a collision of the protected object is predicted.

FIG. 4 is a flow chart showing an example of a processing procedure for controlling a cushioning member in the first application example of the embodiment of the present technique.

First, image data for predicting danger is acquired (step S811). This image data may be one in which the collision prediction target is observed from the side of the protection target, or may be one in which both the protection target and the collision prediction target are observed. Any image is captured by the image pickup unit 110, and image data is acquired. The acquired image data constitutes a time-series frame. As described above, the frame rate is assumed to be a high frame rate of 1000 fps or more, which makes it possible to accurately predict the collision between the protection target and the collision prediction target.

Noise is removed (noise reduction) from each acquired frame by the filter processing unit 120 (step S812).

After that, the process is selected according to the instruction from the operation input device 310 (step S813). When a singular target is instructed (step S813: No), the binarization processing unit 130 performs binarization by the histogram of the image in the frame, and the target is detected (step S815). Then, the difference between the frames adjacent to each other in the time series is generated by the tracking processing unit 140, and the detected target is tracked (target tracking) (step S816). Further, a moment calculation is performed by the moment generation unit 150 for the target (step S817).

Based on the calculated moment, the movement amount of the target is calculated by the center of gravity position generation unit 160 (step S818). Further, the totaling processing unit 210 calculates the moving direction of the target based on the calculated moment (step S818). The calculated movement amount and movement direction of the target are displayed on the display device 320 via the interface 230 (step S819).

When a plurality of targets are instructed (step S813: Yes), the distance calculation by the plurality of parallax is performed (step S814). At this time, the coordinate positions of each target are acquired, and the target detection in step S815 is performed.

Based on the calculated movement amount and movement direction of the target, a calculation is performed to predict the trajectory of the distance between the protection target and the collision prediction target and determine whether or not a collision occurs between the two (the calculation is performed). Step S821). If the result of the calculation exceeds a predetermined expected value, a collision determination is displayed on the display device 320 via the interface 230, and the cushioning member is controlled to be inflated to be in a defensive state (step S825). Further, when the expected value is not exceeded but is close to the expected value, the display device 320 is displayed to the effect of the advance warning determination via the interface 230, and is controlled to generate, for example, a warning sound (step S824). ). Further, if it does not approximate the expected value and does not exceed the expected value, such a warning is not given (step S823).

These processes are repeated for each of the frames of the image data arranged in chronological order.

As described above, in this first application example, assuming a high frame rate of 1000 fps or more as the frame rate, the locus of the distance between the protection target and the collision prediction target is determined based on the movement amount and the movement direction of the target. By predicting, the cushioning member can be controlled. Further, the warning display can be controlled by the same judgment.

<3. Second application example>
FIG. 5 is a diagram showing an example of a sphere simulation in a second application example of the embodiment of the present technique. In order to perform a sphere simulation using the control system in the present embodiment, an image of the sphere to be observed is imaged by the image sensor 100, and the movement amount and the movement direction of the sphere are calculated. This makes it possible to predict the trajectory of the sphere.

For example, in the case of simulation golf, the state in which the player 531 hits the ball 532 is imaged by the mobile terminal 533. The mobile terminal 533 includes an image sensor 100, and an image pickup is performed by the image pickup unit 110 thereof. Target tracking and moment calculation are performed based on the captured image, and the movement amount and movement direction of the ball 532 are calculated. As a result, the locus of the sphere is predicted, and the flight distance of the ball 532 is predicted. The flight distance predicted in this way is displayed as an estimated flight distance 534 on the display unit of the mobile terminal 533.

In this example, an example of displaying the estimated flight distance has been described, but for example, the speed and rotation direction of the ball 532 may be displayed. In order to measure the rotation of the ball 532, a marker is attached to the ball 532, and the state of rotation can be observed by tracking the marker.

FIG. 6 is a diagram showing an example of a trajectory of a sphere in a second application example of the embodiment of the present technique.

For example, when a sphere is thrown at a speed of 150 km / h, when observing the sphere at 30 fps like a conventional camera, the distance between the adjacent frames is about 140 cm, and it is difficult to predict the trajectory of the sphere. be. On the other hand, in this embodiment, by assuming a high frame rate of 1000 fps or more, the distance between the spheres between adjacent frames is about 4.2 cm, and the trajectory of the spheres can be predicted precisely.

That is, it is possible to determine a highly accurate moving speed at the initial stage of pitching and predict the trajectory of the sphere due to the initial motion. Further, by determining the presence or absence of the movement amount in the vertical direction in the middle of the locus, it is possible to recognize whether or not the apex has been reached and predict that the vehicle will descend thereafter.

In addition to the above-mentioned simulation golf, this example of the sphere simulation can be used, for example, for measuring baseball batting, measuring tennis serve, and measuring soccer free kick. By assuming a high frame rate of the present embodiment, it is possible not only to confirm the form but also to simultaneously observe the speed of the hit ball, the estimated flight distance, the rotation direction, and the like from the side of the player.

FIG. 7 is a flow chart showing an example of a processing procedure for a sphere simulation in a second application example of the embodiment of the present technique.

First, image data is acquired by imaging a sphere to be measured by the imaging unit 110 (step S831). The acquired image data constitutes a time-series frame. Noise is removed from each of the acquired frames by the filtering unit 120 (step S832).

Then, the binarization processing unit 130 performs binarization based on the color and brightness of the image in the frame (step S833). Here, it is assumed that a marker is attached to the sphere, and the marker is detected. If no marker is attached, the entire sphere is detected.

An area is set for the detected marker (step S834), and the difference between frames adjacent to each other in the time series in the area is generated by the tracking processing unit 140. As a result, marker tracking (target tracking) is performed in the set area (step S835). Further, a moment calculation is performed by the moment generation unit 150 for the marker (step S836).

Based on the calculated moment, the movement amount of the marker is calculated by the center of gravity position generation unit 160 (step S837). Further, the totaling processing unit 210 calculates the moving direction of the marker based on the calculated moment (step S837). The calculated movement amount and movement direction of the marker are displayed on the display device 320 via the interface 230 (step S838).

Further, the process is selected according to the instruction from the operation input device 310 (step S839), and when the turnover rate is calculated, the turnover rate of the marker is calculated by the aggregation processing unit 210 (step S841). The calculated rotation rate of the marker is displayed on the display device 320 via the interface 230 (step S842).

Then, a calculation for predicting the trajectory of the marker and the estimated flight distance is performed based on the calculated movement amount and movement direction of the marker (and further, if the rotation rate is calculated, the rotation rate) (step S843).

These processes are repeated for each of the frames of the image data arranged in chronological order.

As described above, in this second application example, the estimated flight of the sphere is estimated by predicting the trajectory of the sphere based on the movement amount and the movement direction of the sphere, assuming a high frame rate of 1000 fps or more as the frame rate. You can predict the distance.

<4. Third application example>
FIG. 8 is a diagram showing an example of an in-field announcement in a third application example of the embodiment of the present technique. In order to make an announcement in the hall using the control system in the present embodiment, a camera 541 including an image sensor 100 is installed in various places in the venue (ballpark), and a speaker 542 is installed in the vicinity of the camera 541. When the camera 541 tracks the whereabouts of the ball and it is predicted that the ball that has become a home run or a foul ball will enter the audience seat, the speaker 542 near the drop point is controlled to output an announcement to call attention.

FIG. 9 is a diagram showing an aspect of predicting the intrusion of a ball into the audience seat in the third application example of the embodiment of the present technique. In the stadium, a fence 543 is provided between the ground and the audience seats. If the position is lower than the fence 543, the ball hits the fence 543 and falls to the ground. However, if the ball exceeds the fence 543, it may fall into the audience seats and hit the spectators, injuring them.

In the present embodiment, the camera 541 tracks the whereabouts of the ball at a high frame rate, predicts the trajectory of the ball, and quickly catches the ball that is likely to cross the fence 543 before crossing the fence 543. This makes it possible to send a warning announcement to the audience and avoid accidental injuries. Since the speaker 542 that plays the announcement can be limited to the vicinity of the drop point, it is possible to narrow down the range and give appropriate attention. Further, this makes it possible to avoid unnecessarily raising the fence 543 and suppress a decrease in the sense of presence in the stadium.

FIG. 10 is a diagram showing a system configuration example in the third application example of the embodiment of the present technology. In this example, cameras 541 including the image sensor 100 are installed in various places on the upper part of the fence 543, and the information obtained by the image sensor 100 is supplied to the control device 545. Then, when it is predicted that the ball will enter the audience seat, a control signal is supplied from the control device 545 to the speaker 542 to output a warning announcement.

It is desirable that the camera 541 be provided with a protective cover 544 in order to loosen the impact when the ball hits the camera 541.

FIG. 11 is a flow chart showing an example of a processing procedure for an announcement in the field in a third application example of the embodiment of the present technique.

First, image data is acquired by imaging the ball to be measured by the imaging unit 110 (step S851). The acquired image data constitutes a time-series frame. Noise is removed from each of the acquired frames by the filter processing unit 120 (step S852).

Then, the binarization processing unit 130 performs binarization based on the color and brightness of the image in the frame, and the target ball is detected (step S853).

For the detected ball, the difference between the frames adjacent to each other in the time series is generated by the tracking processing unit 140, and the ball is tracked (target tracking) (step S854). Further, a moment calculation is performed on the ball by the moment generation unit 150 (step S855). As a result, when the motion is detected, the following processes after step S857 are performed, and when the motion is not detected, the processes after step S851 are repeated.

The distance between the camera 541 and the ball is calculated in the aggregation processing unit 210 based on the calculated moment (step S857). In addition, the aggregation processing unit 210 determines the moving direction and locus of the ball (step S858). Further, depending on the position of the ball, the aggregation processing unit 210 determines whether the ball is a foul ball or a home run (step S859). Based on this information, a control signal is supplied from the control unit 220 to the speaker 542 via the interface 230 to output a warning announcement, and the automatic announcement is reproduced (step S861).

These processes are repeated for each of the frames of the image data arranged in chronological order.

As described above, in this third application example, a high frame rate of 1000 fps or more is assumed as the frame rate, the trajectory of the ball is predicted, the falling point of the ball is estimated, and an announcement in the field is made according to the situation. It can be performed.

<5. Fourth application example>
FIG. 12 is a diagram showing an example of a volcanic bomb warning in the fourth application example of the embodiment of the present technique. In order to issue a volcanic bomb warning using the control system in the present embodiment, the crater of the volcano to be observed is imaged by the image sensor 100 as shown in the figure. Then, when a volcanic bomb occurs, the whereabouts of the volcanic bomb are traced, the trajectory of the volcanic bomb is predicted, and the drop point is estimated. Warnings such as evacuation advisories will be given with this estimated fall point as a dangerous area.

With a normal telephoto camera, it is possible to recognize the eruption from the observed image, but it is difficult to predict the reach of the volcanic bomb. In the example of this embodiment, it is possible to capture a volcanic bomb at a high frame rate, measure the size and moving speed of the volcanic bomb, and estimate its reach.

FIG. 13 is a flow chart showing an example of a procedure for processing a volcanic bomb warning in the fourth application example of the embodiment of the present technique. When the image sensor 100 is installed, calibration is performed so that appropriate imaging can be performed.

Image data is acquired by imaging the crater of the volcano to be observed by the image pickup unit 110 (step S871). The acquired image data constitutes a time-series frame. Each of the acquired frames is corrected for contrast and the like by the filter processing unit 120 (step S872). This makes it possible to mitigate the effects of weather and time.

Then, the binarization processing unit 130 performs binarization by the color and brightness of the image in the frame, the difference between the frames adjacent to each other in the time series is generated by the tracking processing unit 140, and the moving object is extracted (step). S873). For the extracted moving body, a moment calculation is performed by the moment generation unit 150, and the area and the vector of the movement amount are measured (step S874).

From the measured vector of the area of the moving body and the amount of movement, it is determined whether or not a volcanic bomb has occurred (step S875). If it is determined that a volcanic bomb has occurred (step S875: Yes), the processing after step S876 is performed. If it is determined that no volcanic bombs have occurred (step S875: No), the processes after step S871 are repeated.

Based on the measured area of the moving object, the mass of the volcanic bomb is estimated in the aggregation processing unit 210 (step S876). In addition, the reach of the volcanic bomb is estimated based on the measured area of the moving object and the vector of the amount of movement (step S877). Then, with respect to this estimated reach, warning information is generated (step S878), and the warning information is notified to related organizations (step S879).

These processes are repeated for each of the frames of the image data arranged in chronological order.

As described above, in this fourth application example, the fall point of the volcanic bomb is estimated by predicting the trajectory of the volcanic bomb, assuming a high frame rate of 1000 fps or more as the frame rate, and caution regarding the reachable range is obtained. Information can be generated.

<6. Fifth application example>
FIG. 14 is a diagram showing an example of follow-up imaging in the fifth application example of the embodiment of the present technique. In order to perform tracking imaging using the control system in the present embodiment, a lens drive actuator 552 is provided in the lens portion of the camera 550 including the image sensor 100. Further, an automatic tracking button 551 for controlling the lens drive actuator 552 is provided. While the automatic tracking button 551 is pressed, the lens drive actuator 552 is controlled so as to fix and capture the object detected as the target in the screen. Since the camera 550 is gripped by the photographer, the position of the target is often not constant due to camera shake or the like, but according to this embodiment, such a target is captured at a fixed position and imaged. Can be done.

FIG. 15 is a diagram showing a mode of follow-up imaging in the fifth application example of the embodiment of the present technique. Here, as an example, an example of catching the ball as a target during a soccer match is shown.

When the normal imaging mode is changed to the sports mode, the detection frame 553 is displayed around the ball. This indicates that the ball can be recognized by the image sensor 100. When the automatic tracking button 551 is pressed while the detection frame 553 is displayed, the movement of the ball is tracked by the tracking processing unit 140, and the moment calculation is performed by the moment generation unit 150. Then, the movement amount of the ball is calculated by the center of gravity position generation unit 160 based on the calculated moment. Further, the totaling processing unit 210 calculates the moving direction 554 of the ball based on the calculated moment. The aggregation processing unit 210 predicts the trajectory of the ball from the calculated movement amount and direction of the ball.

The control unit 220 supplies control information to the lens drive actuator 552 according to the predicted trajectory of the ball. The lens drive actuator 552 is driven by this control information to move the lens. At this time, when the camera moves, the position of the center of gravity of the camera shifts, so that the control information is further fed back from the control unit 220 to the lens drive actuator 552. As a result, the lens is moved by the drive of the lens drive actuator 552, and imaging is performed so that the position of the ball is a fixed position in the screen. The lens is an example of the optical element described in the claims.

In this example, it is assumed that the display position of the ball is maintained when the automatic tracking button 551 is pressed, but the display position of the ball is moved to the center when the automatic tracking button 551 is pressed, and then the display position of the ball is moved to the center. , May be maintained in its central position. Further, a low-pass filter may be applied in the filter processing unit 120 so that the trajectory of the ball does not change sharply. Further, if the tracking of the ball is momentarily deviated, the trajectory may be predicted for several frames.

FIG. 16 is a diagram showing another aspect of follow-up imaging in the fifth application example of the embodiment of the present technique. Here, a zoom mode is assumed as the imaging mode. In this zoom mode, the face is detected as well as the ball. In the figure, it is assumed that the detection frame 558 is displayed around the ball and the detection frame 559 is displayed around the face on the screen 555.

When the detection frames 558 and 559 are inserted in the zoom-in frame 557, the enlargement (zoom-in) operation is performed. On the other hand, when at least one of the detection frame 558 or 559 is removed from the zoom-out frame 556, the reduction (zoom-out) operation is performed. As a result, the zoom-in or zoom-out operation is performed so that both the detection frames 558 and 559 always enter the screen 555.

FIG. 17 is a flow chart showing an example of a follow-up imaging processing procedure in the fifth application example of the embodiment of the present technique. In the figure, the flow chart on the left side shows the processing procedure of the camera 550 main body, and the flow chart on the right side shows the processing procedure of the image sensor 100.

When the image pickup is started in the camera 550, the image sensor 100 is activated. When the power of the image sensor 100 is turned off (step S731: Yes), the operation of the image sensor 100 is not performed. When the power of the image sensor 100 is turned on (step S731: No), the camera 550 waits until a target is specified (step S732).

The camera 550 performs imaging and display in the monitoring mode (step S711). A target to be tracked is specified in this monitoring mode (step S712). For example, a ball is designated as a target in a soccer match. When the detection of the target starts (step S713: Yes), the coordinates of the target are notified to the image sensor 100 (step S714). If the target tracking fails (step S715: No), the target is detected again (step S713).

When the target is designated and its coordinates are notified (step S732: Yes), the image sensor 100 detects the target. If the detection is successful (step S733: Yes), the image sensor 100 returns the detection response to the camera 550 (step S734). Further, even if the detection is successful, if the subsequent target tracking fails (step S735: No), a notification indicating that the tracking has been removed is returned to the camera 550 (step S738), and the target is detected again. (Step S733).

When the camera 550 receives the notification of target detection or the notification of deviation from tracking from the image sensor 100 (step S716: Yes), the camera 550 waits for the automatic tracking button 551 to be pressed. Then, when the automatic tracking button 551 starts to be pressed (step S717: Yes), the image sensor 100 is instructed to control the tracking (step S718). When the pressing of the automatic tracking button 551 is completed (step S719: Yes), the image sensor 100 is instructed to control the end of tracking (step S721).

When the image sensor 100 receives control regarding tracking from the camera 550 (step S736: Yes), the image sensor 100 predicts the trajectory of the ball and controls the lens drive actuator 552 based on the predicted trajectory (step S737). Meanwhile, target tracking is repeated (step S735).

The camera 550 repeats target tracking (step S715) until mode switching is performed (step S722). When the mode is switched (step S722: Yes), the process returns to the target designation unless the power is turned off (step S712).

As described above, in this fifth application example, the lens drive actuator 552 is driven based on the predicted locus by predicting the locus of the target assuming a high frame rate of 1000 fps or more as the frame rate. Control. This makes it possible to take an image so that the position of the ball is a fixed position on the screen.

<7. Sixth application example>
FIG. 18 is a diagram showing an example of image superimposition in the sixth application example of the embodiment of the present technique. In order to perform image superimposition using the control system in the present embodiment, the original image is captured by the image sensor 100. Then, a portion of the original image to be superimposed is detected, and another image is superimposed on that portion. This makes it possible to supply superimposed images as needed.

In a in FIG. 18, as an example, an example in which the advertisement portion described in the player's uniform is replaced with another advertisement during a soccer match is shown. It is assumed that the company name 561 of "P Shoji" is described as an advertisement on the uniforms of the players in the field. Since this is physical, it does not change in itself. The on-site camera captures the situation as it is.

The captured images are broadcast to various places through the equipment of the broadcasting station. At that time, the company name or brand name according to the advertising contract is superimposed on the advertising portion of the captured image. For example, as shown in b in FIG. 18, the company name of "Q industry" is superimposed on "P trading", and the superimposed image is broadcast. As a result, the player's uniform displayed on the television appears as if the company name 562 of "Q industry" is described as an advertisement.

FIG. 19 is a diagram showing an example of image superimposition timing in the sixth application example of the embodiment of the present technique. Here, 30 fps is assumed as the frame rate of the broadcast camera. On the other hand, 1000 fps is assumed as the frame rate of the high-speed camera provided with the image sensor 100 of the present embodiment.

In this case, target tracking is performed with the advertisement portion to be superimposed in the image captured by the high-speed camera as the tracking target. As a result, the coordinate position and shape of the advertisement portion are recognized. Then, a new advertisement texture is generated, and the generated texture is attached to the advertisement part of the frame of the broadcasting camera. Since the frame rate of the high-speed camera is sufficiently faster than the frame rate of the broadcast camera, it is possible to superimpose the frame of the high-speed camera on the frame of the broadcast camera after performing these processes.

FIG. 20 is a flow chart showing an example of an image superimposition processing procedure in the sixth application example of the embodiment of the present technique.

First, image data is acquired by the image sensor 100 (step S881). The acquired image data constitutes a time-series frame. Noise is removed from each of the acquired frames by the filtering unit 120 (step S882).

Then, the binarization processing unit 130 performs binarization based on the color and brightness of the image in the frame, and detects the marker to be superimposed (step S883). An area is set for the detected marker (step S884), and the difference between frames adjacent to each other in the time series in the area is generated by the tracking processing unit 140. As a result, marker tracking (target tracking) is performed in the set area (step S885). Further, a moment calculation is performed by the moment generation unit 150 for the marker (step S886).

Further, the process is selected according to the instruction from the operation input device 310 (step S887). When the superimposed video is output, the superimposed video is output via the interface 230 after the superimposition processing is performed by the aggregation processing unit 210 (step S888) (step S888). On the other hand, when the original video is output, the original video is output via the interface 230 without performing the superimposition processing (step S889).

These processes are repeated for each of the frames of the image data arranged in chronological order.

As described above, in this sixth application example, by assuming a high frame rate of 1000 fps or more as the frame rate and predicting the locus of the superimposed object, the image at the position of the superimposed object is superimposed and replaced. Can be done.

<8. 7th application example>
FIG. 21 is a diagram showing an example of distance calculation in the seventh application example of the embodiment of the present technique. Conventionally, object tracking used in motion capture and the like has been performed using a plurality of cameras in order to accurately measure movement in three dimensions. Therefore, the system was large and expensive. Therefore, in order to calculate the distance using the control system in the present embodiment, the image of the object to be measured is imaged by the image sensor 100, and the distance of the object is calculated.

In order to calculate the distance in this example, it is necessary to calibrate in advance. That is, the object 571 to be tracked is placed at a predetermined coordinate position (0,0), and the distance and area at that time are recorded as the reference distance and the reference area. After that, tracking is performed on the object 571, the position of the center of gravity (x, y) and the area s of the object 571 are calculated, and the distance z is calculated from the area and the calibration result by the following equation.
Distance z = reference distance x (area s / reference area) ^1/2

FIG. 22 is a diagram showing an aspect of object tracking in the seventh application example of the embodiment of the present technique. In this example, since a high frame rate of 1000 fps or more is assumed as the frame rate imaged by the image sensor 100, the amount of movement in each frame is very small. On the premise of this, by localizing the detection range and obtaining the area and the position of the center of gravity in that range, tracking can be performed by a simple mechanism.

Centering on the center of gravity position 573 of the object 572 in the previous frame, the vicinity thereof is set as the detection target frame 574. At this time, one side (width and height) of the detection range is expressed by the following equation. However, α is a search range coefficient.
One side of the detection range = (0th moment of the previous frame) ^1/2 x α

When the tracking processing unit 140 detects the object 575 in the next frame in the detection target frame 574, the binarization processing unit 130 performs the binarization processing. As a result, "1" is indicated as the bitmap I (x, y) at the position (x, y) of the object 575, and "0" is indicated at the other positions. Based on this bitmap I (x, y), the moment generation unit 150 performs a moment calculation as shown in the following equation.
0th moment (area) = ΣΣI (x, y)
Horizontal first moment of area = ΣΣx × I (x, y)
Vertical first moment of area = ΣΣy × I (x, y)
However, ΣΣ in the above equation represents the summation operation in the horizontal direction and the vertical direction.

Based on the moment generated in this way, the coordinates of the center of gravity position are generated in the center of gravity position generation unit 160 by the following equation.
Center of gravity x coordinate = horizontal primary moment / 0th order moment Center of gravity y coordinate = vertical primary moment / 0th order moment

FIG. 23 is a diagram showing a mode of distance calculation in the seventh application example of the embodiment of the present technique. As described above, in this example, the distance can be calculated from the area obtained by the camera 561 by calibrating the relationship between the distance and the area at the reference position in advance.

Here, the area A0 at the reference position is used as the reference area, and the distance D0 at that time is used as the reference distance. If the area A1 is obtained when object tracking is performed, the distance D1 is calculated by the following equation.
D1 = D0 × (A1 / A0) ^1/2

Similarly, when the area A2 is obtained when the object tracking is performed, the distance D2 is calculated by the following equation.
D2 = D0 × (A2 / A0) ^1/2

FIG. 24 is a flow chart showing an example of a processing procedure for distance calculation in the seventh application example of the embodiment of the present technique. It is assumed that calibration has been performed as a premise for calculating this distance.

First, image data is acquired by the image sensor 100 (step S891). The acquired image data constitutes a time-series frame. Noise is removed from each of the acquired frames by the filter processing unit 120 (step S892).

Then, the binarization processing unit 130 performs binarization based on the color and brightness of the image in the frame, and detects the marker to be superimposed (step S893). An area is set for the detected marker (step S894), and the difference between frames adjacent to each other in the time series in the area is generated by the tracking processing unit 140. As a result, marker tracking (target tracking) is performed in the set area (step S895).

Further, a moment calculation is performed by the moment generation unit 150 for the marker (step S896). Based on the calculated moment, the movement amount of the marker is calculated by the center of gravity position generation unit 160 (step S897).

Further, the process is selected according to the instruction from the operation input device 310 (step S898), and when the distance is calculated, the distance is calculated by the aggregation processing unit 210 (step S899). The calculated distance is displayed on the display device 320 via the interface 230, for example.

These processes are repeated for each of the frames of the image data arranged in chronological order.

As described above, in this seventh application example, the distance to the object can be calculated by predicting the trajectory of the object assuming a high frame rate of 1000 fps or more as the frame rate.

As described above, according to the embodiment of the present technique, the trajectory of the object is predicted by imaging the object at a high frame rate and performing image processing in the image sensor 100, and the prediction result thereof. The controlled device 330 can be controlled based on the above.

It should be noted that the above-described embodiment shows an example for embodying the present technology, and the matters in the embodiment and the matters specifying the invention within the scope of claims have a corresponding relationship with each other. Similarly, the matters specifying the invention within the scope of claims and the matters in the embodiment of the present technology having the same name have a corresponding relationship with each other. However, the present technique is not limited to the embodiment, and can be embodied by applying various modifications to the embodiment without departing from the gist thereof.

Further, the processing procedure described in the above-described embodiment may be regarded as a method having these series of procedures, or as a program for causing a computer to execute these series of procedures or as a recording medium for storing the program. You may catch it. As the recording medium, for example, a CD (Compact Disc), MD (MiniDisc), DVD (Digital Versatile Disc), memory card, Blu-ray Disc (Blu-ray (registered trademark) Disc) and the like can be used.

It should be noted that the effects described in the present specification are merely examples and are not limited, and other effects may be obtained.

The present technology can have the following configurations.
(1) An image sensor that captures an object and generates frames of image data arranged in chronological order.
A binarization processing unit that performs binarization processing on each of the frames to generate a binarization frame, and a binarization processing unit.
A tracking processing unit that generates a difference between the binarized frames adjacent to the time series and tracks a change in the position of the object included in the binarized frame.
A moment generation unit that calculates the moment of the object included in the binarization frame based on the result of the tracking processing unit, and a moment generation unit.
A center of gravity position generation unit that generates the center of gravity position of the object included in the binarization frame based on the moment generated by the moment generation unit.
A locus generation unit that predicts and generates a locus of the object based on the center of gravity position of the object generated by the center of gravity position generation unit.
An image sensor including a control signal supply unit that supplies a control signal to a control target device based on the trajectory of the object generated by the trajectory generation unit.
(2) The locus generation unit predicts a locus of a distance from the object and predicts the locus.
The image sensor according to (1), wherein the control signal supply unit supplies the control signal when the predicted distance satisfies a predetermined condition.
(3) The image sensor according to (2) above, wherein the control signal is a control signal that controls the operation of the shock absorber.
(4) The image sensor according to (2) above, wherein the control signal is a control signal that controls the operation of the warning device.
(5) The trajectory generation unit predicts the trajectory of the trajectory of the object, and predicts the trajectory of the object.
The image sensor according to any one of (1) to (4), wherein the control signal supply unit supplies the control signal for displaying the predicted locus.
(6) The locus generation unit predicts the flight distance of the object and determines the flight distance.
The image sensor according to (5), wherein the control signal supply unit supplies the control signal for displaying the predicted flight distance.
(7) The locus generation unit predicts the arrival position of the object and predicts the arrival position of the object.
The image sensor according to (5), wherein the control signal supply unit supplies the control signal for displaying the predicted arrival position.
(8) The image sensor according to (7), wherein the control signal supply unit supplies the control signal for outputting an audio signal according to the predicted arrival position.
(9) The image pickup device is further provided with an optical element for supplying an image containing the object.
The image sensor according to any one of (1) to (8) above, wherein the control signal is a control signal for controlling the operation of the optical element.
(10) The image sensor according to any one of (1) to (9) above, wherein the control signal is a control signal in which a part of the frame is replaced with a predetermined image and displayed.

100 Image sensor 110 Imaging unit 120 Filter processing unit 130 Binarization processing unit 140 Tracking processing unit 150 Moment generation unit 160 Center of gravity position generation unit 210 Aggregation processing unit 220 Control unit 230 Interface 310 Operation input device 320 Display device 330 Control target device 511 515 Protected object 512, 516, 521, 523, 525 Cushioning member 513, 517, 522, 524, 526 Image sensor 514, 518 Moving object 531 Player 532 Ball 533 Mobile terminal 534 Estimated flight distance 541, 550, 561 Camera 542 Speaker 543 Fence 544 Protective cover 545 Control device 551 Automatic tracking button 552 Lens drive actuator

Claims (10)

An image sensor that captures an object and generates frames of image data arranged in chronological order at a frame rate of 1000 fps or higher .
A binarization processing unit that performs binarization processing on each of the frames to generate a binarization frame, and a binarization processing unit.
A tracking processing unit that generates a difference between the binarized frames adjacent to the time series and tracks a change in the position of the object included in the binarized frame.
A moment generation unit that calculates the moment of the object included in the binarization frame based on the result of the tracking processing unit, and a moment generation unit.
A center of gravity position generation unit that generates the center of gravity position of the object included in the binarization frame based on the moment generated by the moment generation unit.
A locus generation unit that predicts and generates a locus of the object based on the center of gravity position of the object generated by the center of gravity position generation unit.
An image sensor including a control signal supply unit that supplies a control signal to a control target device based on the trajectory of the object generated by the trajectory generation unit. The locus generation unit predicts a locus of a distance from the object and predicts the locus.
The image sensor according to claim 1, wherein the control signal supply unit supplies the control signal when the predicted distance satisfies a predetermined condition. The image sensor according to claim 2, wherein the control signal is a control signal that controls the operation of the cushioning member. The image sensor according to claim 2, wherein the control signal is a control signal that controls the operation of the warning device. The trajectory generation unit predicts the trajectory of the trajectory of the object and predicts the trajectory of the object.
The image sensor according to claim 1, wherein the control signal supply unit supplies the control signal for displaying the predicted locus. The locus generation unit predicts the flight distance of the object and
The image sensor according to claim 5, wherein the control signal supply unit supplies the control signal for displaying the predicted flight distance. The locus generation unit predicts the arrival position of the object and predicts the arrival position of the object.
The image sensor according to claim 5, wherein the control signal supply unit supplies the control signal for displaying the predicted arrival position. The image sensor according to claim 7, wherein the control signal supply unit supplies the control signal for outputting an audio signal according to the predicted arrival position. The image pickup device is further provided with an optical element that supplies an image containing the object.
The image sensor according to claim 1, wherein the control signal is a control signal that controls the operation of the optical element. The image sensor according to claim 1, wherein the control signal is a control signal in which a part of the frame is replaced with a predetermined image and displayed.

Images