JPWO2018105290A1 - Image sensor - Google Patents

Abstract

The apparatus is controlled based on the trajectory of the object based on the high-speed image data different from the low-speed image data for display. The imaging device captures an object and generates frames of image data arranged in time series. The binarization processing unit performs binarization processing on each of the frames to generate a binarized frame. The tracking processing unit generates a difference between the binarized frames adjacent to each other in time series and tracks the change in the position of the object included in the binarized frame. The moment generator calculates the moment of the binarized frame. The centroid position generation unit generates the centroid position of the object included in the binarized frame. The trajectory generation unit predicts and generates a trajectory of the object. The control signal supply unit supplies a control signal to the control target device based on the trajectory of the target object.

Description

  The present technology relates to an image sensor. Specifically, the present invention relates to an image sensor for controlling a control target device using captured image data.

  2. Description of the Related Art Conventionally, an image processing apparatus that captures an image of an object at predetermined intervals and generates an image representing the locus of the object based on those images is known. For example, there has been proposed an image processing apparatus that obtains a trajectory of an object from when the first image is photographed until the second image is photographed, and interpolates the trajectory using the obtained trajectory (for example, a patent). Reference 1).

Japanese Patent Application Laid-Open No. 2015-087904

  In the above-described prior art, interpolation processing is performed in order to obtain the locus of an object from the image of the object photographed at predetermined intervals. In order to display an image, a processing speed of about 30 to 120 fps (frames / second) is generally sufficient, but that is not sufficient for advanced control. Therefore, even though the trajectory of the obtained object can be drawn, it is difficult to use it for controlling other devices. That is, it is difficult to control other devices in real time based on the trajectory of the object generated by the interpolation process.

  The present technology has been created in view of such a situation, and an object of the present technology is to control an apparatus based on a trajectory of an object based on high-speed image data different from low-speed image data for display.

  The present technology has been made to solve the above-described problems. The first aspect of the present technology is an imaging device that captures an object and generates frames of image data arranged in time series, and the above-described frame. Generate a difference between a binarization processing unit that performs binarization processing on each to generate a binarized frame and the binarized frame adjacent to the time series, and includes the difference in the binarized frame A tracking processing unit that tracks a change in the position of the target object, a moment generation unit that calculates a moment of the target object included in the binarized frame based on a result of the tracking processing unit, and the moment generation unit A centroid position generation unit that generates a centroid position of the object included in the binarized frame based on the moment generated by A trajectory generator that predicts and generates a trajectory of the object based on the position of the center of gravity of the object, and a control that supplies a control signal to the control target device based on the trajectory of the object generated by the trajectory generator An image sensor including a signal supply unit. This brings about the effect | action of controlling a control object apparatus based on the locus | trajectory of the estimated target object.

  In the first aspect, the trajectory generation unit predicts a trajectory of the distance to the object, and the control signal supply unit includes the control signal when the predicted distance satisfies a predetermined condition. May be supplied. This brings about the effect | action of controlling a control object apparatus according to the estimated distance.

  In the first aspect, the control signal may be a control signal for controlling the operation of the buffer member. This brings about the effect | action of controlling operation | movement of a buffer member based on the locus | trajectory of the estimated target object.

  In the first aspect, the control signal may be a control signal for controlling the operation of the warning device. This brings about the effect | action of controlling operation | movement of a warning device based on the locus | trajectory of the estimated target object.

  In the first aspect, the trajectory generation unit predicts a trajectory of the trajectory of the object, and the control signal supply unit supplies the control signal for displaying the predicted trajectory. It may be. This brings about the effect | action that the locus | trajectory of the estimated target object is displayed.

  In the first aspect, the trajectory generation unit predicts a flight distance of the object, and the control signal supply unit supplies the control signal for displaying the predicted flight distance. It may be. This brings about the effect | action of displaying the predicted flight distance.

  In the first aspect, the trajectory generation unit predicts the arrival position of the object, and the control signal supply unit supplies the control signal for displaying the predicted arrival position. It may be. As a result, the predicted arrival position is displayed.

  In the first aspect, the control signal supply unit may supply the control signal for outputting an audio signal corresponding to the predicted arrival position. This brings about the effect | action that the audio | voice signal according to the estimated arrival position is output.

  Further, in the first aspect, an optical element that supplies an image including the object to the imaging element may be further provided, and the control signal may be a control signal that controls an operation of the optical element. This brings about the effect | action of controlling operation | movement of an optical element based on the locus | trajectory of the estimated target object.

  In the first aspect, the control signal may be a control signal that is displayed by replacing a part of the frame with a predetermined image. This brings about the effect | action of replacing an image based on the locus | trajectory of the estimated target object.

  According to the present technology, it is possible to achieve an excellent effect that the apparatus can be controlled based on the trajectory of the object based on the high-speed image data different from the low-speed image data for display. Note that the effects described here are not necessarily limited, and may be any of the effects described in the present disclosure.

It is a figure showing an example of 1 composition of a control system in an embodiment of this art. It is a figure showing an example of control of a buffer member in the 1st example of application of an embodiment of this art. It is a figure showing other examples of control of a buffer member in the 1st example of application of an embodiment of this art. It is a flowchart which shows the example of a process sequence of control of the buffer member in the 1st application example of embodiment of this technique. It is a figure which shows the example of the sphere simulation in the 2nd application example of embodiment of this technique. It is a figure which shows the example of the locus | trajectory of the sphere in the 2nd application example of embodiment of this technique. It is a flowchart which shows the process sequence example of the spherical body simulation in the 2nd application example of embodiment of this technique. It is a figure which shows the example of the announcement in the field in the 3rd application example of embodiment of this technique. It is a figure which shows the aspect which estimates the penetration | invasion to the passenger seat of the ball | bowl in the 3rd application example of embodiment of this technique. It is a figure which shows the system configuration example in the 3rd application example of embodiment of this technique. It is a flowchart which shows the process sequence example of the announcement in a field in the 3rd application example of embodiment of this technique. It is a figure showing an example of a volcanic bomb warning in the 4th example of application of an embodiment of this art. It is a flowchart which shows the process sequence example of the volcanic bomb warning in the 4th application example of embodiment of this technique. It is a figure which shows the example of the tracking imaging in the 5th application example of embodiment of this technique. It is a figure which shows the aspect of the tracking imaging in the 5th application example of embodiment of this technique. It is a figure which shows the other aspect of the tracking imaging in the 5th application example of embodiment of this technique. 16 is a flowchart illustrating an example of a tracking imaging process procedure according to the fifth application example of the embodiment of the present technology. It is a figure showing an example of image superposition in the 6th example of application of an embodiment of this art. It is a figure showing the example of timing of image superposition in the 6th example of application of an embodiment of this art. 18 is a flowchart illustrating an example of an image superimposition processing procedure in a sixth application example of the embodiment of the present technology. It is a figure showing an example of distance calculation in the 7th example of application of an embodiment of this art. It is a figure which shows the aspect of the object tracking in the 7th application example of embodiment of this technique. It is a figure which shows the aspect of the distance calculation in the 7th application example of embodiment of this technique. 24 is a flowchart illustrating an example of a distance calculation processing procedure according to a seventh application example of the embodiment of the present technology.

Hereinafter, modes for carrying out the present technology (hereinafter referred to as embodiments) will be described. The description will be made in the following order.
1. Embodiment (configuration example of control system)
2. First application example (example of buffer member control)
3. Second application example (sphere simulation example)
4). Third application example (example of on-site announcement)
5. Fourth application example (example of volcanic bomb warning)
6). Fifth application example (example of tracking imaging)
7). Sixth application example (example of image superimposition)
8). Seventh application example (distance calculation example)

<1. Embodiment>
FIG. 1 is a diagram illustrating a configuration example of a control system according to an embodiment of the present technology. The image processing system in this embodiment includes an image sensor 100, a display device 320, and a control target device 330.

  The image sensor 100 captures an image of a subject including the object, predicts the locus of the object, and controls the control target device 330 based on the locus of the object. The operation input device 310 receives an operation input from the outside. The display device 320 displays information obtained by the image sensor 100. The control target device 330 is a device controlled by the image sensor 100.

  The image sensor 100 includes an imaging unit 110, a filter processing unit 120, a binarization processing unit 130, a tracking processing unit 140, a moment generation unit 150, a centroid position generation unit 160, a totalization processing unit 210, and a control. A unit 220 and an interface 230.

  The image capturing unit 110 is an image sensor that captures an image of a subject including an object. The imaging unit 110 generates frames of image data arranged in time series at a predetermined frame rate. Here, a high frame rate of 1000 frames per second (1000 fps) or more is assumed as the frame rate. It is not necessary for all the frames of image data captured by the imaging unit 110 to be supplied to the outside of the image sensor 100. The image data with a high frame rate is intended for the control described below, and a frame rate lower than this is sufficient for display. In other words, it is possible to effectively utilize the bandwidth of the image sensor 100 by keeping high frame rate image data as a reference in the image sensor 100. The imaging unit 110 is an example of an imaging element described in the claims. In addition, the object is an object that widely includes non-living objects as well as living things such as people and animals.

  The filter processing unit 120 performs filter processing on each frame of image data captured by the imaging unit 110. As the filter processing in the filter processing unit 120, for example, noise removal processing using a moving average filter or median filter, contour detection processing using a Sobel filter, edge detection using a Laplacian filter, or the like is assumed. Further, the number of objects included in the image can be calculated by obtaining the Euler number of the image by the filter processing unit 120. The Euler number is the number of components minus the number of holes.

  The binarization processing unit 130 performs binarization processing on each of the frames subjected to the filter processing by the filter processing unit 120. The binarization processing unit 130 binarizes the image data based on luminance and color histogram information included in the image data of each frame, and generates a binarized frame including the binarized data.

  The tracking processing unit 140 detects an object included in the binarized frame by generating a difference between frames adjacent in time series for the binarized frame generated by the binarization processing unit 130. , To track changes in the position of the object. When detecting an object, it is possible to designate a specific region in the image as a measurement target.

  The moment generation unit 150 calculates the moment of the two-variable function in the binarized frame based on the result from the tracking processing unit 140. The 0th-order moment represents the amount of change in the area of the object included in the binarized frame, and is a value that is invariant to image rotation and enlargement / reduction.

  The center-of-gravity position generation unit 160 generates the center-of-gravity position of the target object included in the binarized frame based on the moment generated by the moment generation unit 150. A value obtained by dividing the respective primary moments in the horizontal direction and the vertical direction by the zeroth moment represents the position of the center of gravity.

  The aggregation processing unit 210 performs an aggregation process based on various data obtained by the image sensor 100 to generate a trajectory of the object. As shown in the application example below, the totalization processing unit 210 performs necessary processing according to the application that operates. The control unit 220 performs operation control on each unit of the image sensor 100. The interface 230 serves as an interface with the outside. In this example, the interface 230 is connected to the control target device 330 and supplies a control signal to the control target device 330. The interface 230 is connected to the display device 320 and causes the display device 320 to display information obtained by the image sensor 100. The aggregation processing unit 210 is an example of a trajectory generation unit described in the claims.

  In this figure, the output from the center-of-gravity position generation unit 160 clearly shows the route through which the output is supplied to the totalization processing unit 210 and the control unit 220. A route for supplying data may be provided as necessary.

  By comprising in this way, the locus | trajectory of a target object can be estimated and a control object apparatus can be controlled based on the estimated locus | trajectory. Further, it is not necessary to process the image in the subsequent stage, and control can be performed in real time using only the calculation result.

<2. First application example>
FIG. 2 is a diagram illustrating an example of control of the buffer member in the first application example of the embodiment of the present technology. In order to control the buffer member using the control system according to the present embodiment, the image sensor 100 is provided around the protection target, imaging is performed, the trajectory of the target is predicted, and the predicted trajectory is obtained. Based on this, the buffer member is controlled. Thereby, the protection object can be protected by the buffer member against the danger approaching the protection object.

  In a in FIG. 2, the protection target 511 is, for example, a building, and a buffer member 512 is provided on the surrounding fence. An image sensor 513 as an example of the image sensor 100 is provided on the fence. When a moving object 514 such as a motorcycle or an automobile approaches the protection target 511, the image sensor 513 detects the moving object 514 as a collision prediction target, and determines whether or not the moving object 514 collides with the protection target 511. Identify.

  If it is determined that there is a risk of collision based on the moving speed and moving direction of the moving object 514, a warning is given to brake the moving object 514 at the position P1. For example, a warning lamp is provided on the fence and the warning is performed by lighting the warning lamp. Further, when it is determined that the moving speed does not decelerate and the collision cannot be physically avoided, control is performed to inflate the buffer member 512 at the timing when the moving object 514 reaches the position P2 to enter the defense state.

  In FIG. 2b, the protection target 515 is a human. In this example, it is assumed that a human bears the buffer member 516 and the image sensor 517 on the back. At this time, when the moving object 518 approaches from the back surface of the protection target 515, if the image sensor 517 predicts a collision, the buffer member 516 is controlled to be inflated to be in a protective state.

  On the other hand, a normal automobile airbag is activated when it is actually subjected to an impact due to a collision, and cannot be protected in advance. There is also the possibility of injury from the sudden bulge itself. According to the buffer member of this example, such inconvenience can be avoided. Moreover, since the buffer member of this example is not an acceleration sensor but an image sensor, it can be installed on a protected object that does not move.

  FIG. 3 is a diagram illustrating another example of the control of the buffer member in the first application example of the embodiment of the present technology. In this example, the protection target is a human and the collision prediction target is the ground.

  In FIG. 3 a, it is assumed that a human holds the buffer member 521 in the front and the image sensor 522 that captures the front is attached to the head. At this time, when the person to be protected falls forward, if the image sensor 522 predicts a collision with the ground, the buffer member 521 is controlled to be inflated to be in a protective state.

  In FIG. 3 b, it is assumed that a person carries the shock-absorbing member 523 on the back, and an image sensor 524 that captures the rear is attached to the head. At this time, when the human being to be protected falls backward, if the image sensor 524 predicts a collision with the ground, the buffer member 523 is controlled to be inflated to be in a protective state.

  In FIG. 3 c, it is assumed that a buffer member is spread on the ground and an image sensor 526 for capturing the person is arranged at a place where a person is working at a high place. At this time, when a human being to be protected falls, if the image sensor 526 predicts a collision with the ground, control is performed so that the buffer member 525 at the predicted drop position is inflated to be in a protective state.

  Thus, in this example, the image sensor may be attached to the protection target itself, or may be disposed at a position where the movement of the protection target can be observed. Similarly, the buffer member may be attached to the protection target itself, or may be disposed at a position where a collision of the protection target is predicted.

  FIG. 4 is a flowchart illustrating an example of a processing procedure for controlling the buffer member in the first application example of the embodiment of the present technology.

  First, image data for predicting danger is acquired (step S811). This image data may be the observation of the collision prediction target from the protection target side, or may be the observation of both the protection target and the collision prediction target. One of the images is picked up by the image pickup unit 110, and image data is acquired. The acquired image data constitutes a time-series frame. As described above, a high frame rate of 1000 fps or higher is assumed as the frame rate, and thereby, a collision between the protection target and the collision prediction target can be accurately predicted.

  Noise is removed (noise reduction) from the acquired frames by the filter processing unit 120 (step S812).

  Thereafter, a process is selected in accordance with an instruction from the operation input device 310 (step S813). When a single target is instructed (step S813: No), the binarization processing unit 130 performs binarization based on the histogram of the image in the frame, and detects the target (step S815). Then, the difference between frames adjacent in time series is generated by the tracking processing unit 140, and the detected target is tracked (target tracking) (step S816). Further, moment calculation is performed on the target by the moment generator 150 (step S817).

  Based on the calculated moment, the moving amount of the target is calculated by the gravity center position generation unit 160 (step S818). Further, based on the calculated moment, the total processing unit 210 calculates the target moving direction (step S818). The calculated amount and direction of movement of the target are displayed on the display device 320 via the interface 230 (step S819).

  When a plurality of targets are instructed (step S813: Yes), a distance calculation using a plurality of parallaxes is performed (step S814). At this time, the coordinate position of each target is acquired, and target detection in step S815 is performed.

  Based on the calculated amount and direction of movement of the target, a trajectory of the distance between the protection target and the collision prediction target is predicted, and an operation is performed to determine whether or not a collision occurs between them ( Step S821). As a result of the calculation, when a predetermined expected value is exceeded, the fact of collision determination is displayed on the display device 320 via the interface 230, and control is performed to inflate the buffer member to enter the protective state (step S825). If the expected value is not exceeded, but approximates the expected value, the display device 320 displays a notice of prior warning determination via the interface 230 and is controlled to generate a warning sound, for example (step S824). ). Further, when the value does not approximate the expected value and does not exceed the expected value, such warning is not performed (step S823).

  These processes are repeated for each frame of image data arranged in time series.

  Thus, in the first application example, assuming a high frame rate of 1000 fps or more as the frame rate, the trajectory of the distance between the protection target and the collision prediction target is calculated based on the movement amount and movement direction of the target. By predicting, the buffer member can be controlled. Further, warning display can be controlled by the same determination.

<3. Second application example>
FIG. 5 is a diagram illustrating an example of a sphere simulation in the second application example of the embodiment of the present technology. In order to perform a sphere simulation using the control system in the present embodiment, an image of a sphere to be observed is captured by the image sensor 100, and the amount of movement and direction of movement of the sphere are calculated. Thereby, the trajectory of the sphere can be predicted.

  For example, in the case of simulation golf, the mobile terminal 533 captures an image of the player 531 hitting the ball 532. The portable terminal 533 includes the image sensor 100, and imaging is performed by the imaging unit 110. Target tracking and moment calculation are performed based on the captured image, and the amount and direction of movement of the ball 532 are calculated. Thereby, the trajectory of the sphere is predicted, and the flight distance of the ball 532 is predicted. The flight distance predicted in this way is displayed as the estimated flight distance 534 on the display unit of the portable terminal 533.

  In this example, the example in which the estimated flight distance is displayed has been described. However, for example, the speed and rotation direction of the ball 532 may be displayed. In order to measure the rotation of the ball 532, a marker is attached to the ball 532, and the state of rotation can be observed by tracking the marker.

  FIG. 6 is a diagram illustrating an example of a trajectory of a sphere in the second application example of the embodiment of the present technology.

  For example, when a sphere is thrown at a speed of 150 km / h, when the sphere is observed at 30 fps as in a conventional camera, the distance between the spheres is about 140 cm between adjacent frames, and it is difficult to predict the trajectory of the sphere. is there. On the other hand, in this embodiment, assuming a high frame rate of 1000 fps or more, the distance between the spheres between adjacent frames is about 4.2 cm, and the trajectory of the sphere can be accurately predicted.

  That is, it is possible to predict the trajectory of the sphere by the initial operation by calculating a highly accurate moving speed at the initial stage of pitching. Further, by determining whether or not there is a vertical movement amount in the middle of the trajectory, it is possible to recognize whether or not the vertex has been reached and predict that it will subsequently descend.

  This example of the sphere simulation can be used for, for example, baseball batting measurement, tennis serve measurement, and soccer free kick measurement in addition to the above-described simulation golf. By assuming the high frame rate of the present embodiment, not only the confirmation of the form but also the speed of the hit ball, the estimated flight distance, the rotation direction, etc. can be observed simultaneously from the side of the player.

  FIG. 7 is a flowchart illustrating a processing procedure example of the sphere simulation in the second application example of the embodiment of the present technology.

  First, image data is acquired by imaging a sphere that is a measurement target by the imaging unit 110 (step S831). The acquired image data constitutes a time-series frame. Noise is removed from each acquired frame by the filter processing unit 120 (step S832).

  Then, the binarization processing unit 130 performs binarization based on the color and brightness of the image in the frame (step S833). Here, assuming that a marker is attached to the sphere, the marker is detected. If no marker is attached, the entire sphere is detected.

  An area is set for the detected marker (step S834), and a difference between frames adjacent in time series in the area is generated by the tracking processing unit 140. Thereby, tracking of the marker (target tracking) is performed in the set region (step S835). Further, moment calculation is performed by the moment generator 150 for the marker (step S836).

  Based on the calculated moment, the movement amount of the marker is calculated by the centroid position generation unit 160 (step S837). Based on the calculated moment, the totalization processing unit 210 calculates the movement direction of the marker (step S837). The calculated marker movement amount and direction are displayed on the display device 320 via the interface 230 (step S838).

  In addition, a process is selected in accordance with an instruction from the operation input device 310 (step S839), and when the rotation rate is calculated, the rotation rate of the marker is calculated in the aggregation processing unit 210 (step S841). The calculated rotation rate of the marker is displayed on the display device 320 via the interface 230 (step S842).

  Based on the calculated movement amount and direction of the marker (or the rotation rate when the rotation rate is calculated), the marker trajectory and the estimated flight distance are predicted (step S843).

  These processes are repeated for each frame of image data arranged in time series.

  As described above, in this second application example, assuming a high frame rate of 1000 fps or more as the frame rate, the sphere trajectory is predicted based on the movement amount and the movement direction of the sphere, thereby estimating the sphere. Distance can be predicted.

<4. Third application example>
FIG. 8 is a diagram illustrating an example of an in-field announcement in the third application example of the embodiment of the present technology. In order to make an announcement in the field using the control system in the present embodiment, a camera 541 including the image sensor 100 is installed in each place of the venue (ball field), and a speaker 542 is installed in the vicinity of the camera 541. When it is predicted that a ball that has become a home run or a foul ball will enter the passenger seat by following the whereabouts of the ball by the camera 541, control is performed so as to output an announcement for calling attention from the speaker 542 near the falling point.

  FIG. 9 is a diagram illustrating a mode in which the intrusion of the ball into the passenger seat is predicted in the third application example of the embodiment of the present technology. In the stadium, a fence 543 is provided between the ground and the audience seat. If the position is lower than the fence 543, the ball hits the fence 543 and falls to the ground. However, if the ball exceeds this fence 543, it may fall into the audience seat and hit the audience, causing injury.

  In the present embodiment, the camera 541 follows the ball at a high frame rate, predicts the trajectory of the ball, and quickly catches a ball that is likely to exceed the fence 543 before the fence 543 is exceeded. As a result, it is possible to avoid an inadvertent injury by sending an announcement to alert the audience. Since the speaker 542 that plays the announcement can be limited to the vicinity of the fall point, the range can be narrowed down to perform appropriate alerting. Moreover, this can avoid making the fence 543 unnecessarily high, and can suppress a reduction in the presence of the stadium.

  FIG. 10 is a diagram illustrating a system configuration example in the third application example of the embodiment of the present technology. In this example, cameras 541 including the image sensor 100 are installed at various locations on the fence 543, and information obtained by the image sensor 100 is supplied to the control device 545. When a ball intrusion into the passenger seat is predicted, a control signal is supplied from the control device 545 to the speaker 542 so as to output a warning announcement.

  The camera 541 is desirably provided with a protective cover 544 in order to relax the impact when the ball hits.

  FIG. 11 is a flowchart illustrating an example of a processing procedure of in-place announcement in the third application example of the embodiment of the present technology.

  First, image data is acquired by imaging the ball which is the measurement object by the imaging unit 110 (step S851). The acquired image data constitutes a time-series frame. Noise is removed from each acquired frame by the filter processing unit 120 (step S852).

  Then, the binarization processing unit 130 performs binarization based on the color and brightness of the image in the frame, and detects the target ball (step S853).

  For the detected ball, a difference between adjacent frames in time series is generated by the tracking processing unit 140, and the ball is tracked (target tracking) (step S854). Further, moment calculation is performed on the ball by the moment generator 150 (step S855). As a result, if a motion is detected, the following processing from step S857 is performed, and if no motion is detected, the processing from step S851 is repeated.

  Based on the calculated moment, the total processing unit 210 calculates the distance between the camera 541 and the ball (step S857). Further, the totaling processing unit 210 determines the moving direction and locus of the ball (step S858). Furthermore, according to the position of the ball, the totalization processing unit 210 determines whether it is a foul ball or a home run (step S859). Based on these pieces of information, a control signal is supplied from the control unit 220 via the interface 230 so as to output a warning announcement to the speaker 542, and an automatic announcement is reproduced (step S861).

  These processes are repeated for each frame of image data arranged in time series.

  As described above, in the third application example, by assuming a high frame rate of 1000 fps or more as a frame rate, the ball trajectory is predicted to estimate the ball falling point, and an in-field announcement corresponding to the situation It can be performed.

<5. Fourth application example>
FIG. 12 is a diagram illustrating an example of a volcanic bomb warning in the fourth application example of the embodiment of the present technology. In order to perform a volcanic bomb warning using the control system in the present embodiment, the volcanic crater to be observed is imaged by the image sensor 100 as shown in FIG. And when a volcanic bomb occurs, it follows the whereabouts of the volcanic bomb, predicts the trajectory of the volcanic bomb, and estimates the fall point. A warning such as an evacuation recommendation is given with the estimated fall point as a dangerous area.

  With a normal telephoto camera, it is possible to recognize volcanic smoke from observation images, but it is difficult to predict the reach of volcanic bombs. In the example of the present embodiment, a volcanic bomb can be captured at a high frame rate, the size and moving speed of the volcanic bomb can be measured, and its reach can be estimated.

  FIG. 13 is a flowchart illustrating an example of a volcanic bomb warning processing procedure in the fourth application example of the embodiment of the present technology. When the image sensor 100 is installed, calibration is performed so that appropriate imaging is performed.

  Image data is acquired by imaging the volcano crater to be observed by the imaging unit 110 (step S871). The acquired image data constitutes a time-series frame. Each acquired frame is subjected to correction such as contrast by the filter processing unit 120 (step S872). Thereby, the influence by weather and time can be relieved.

  Then, the binarization processing unit 130 performs binarization based on the color and luminance of the image in the frame, and a difference between frames adjacent in time series is generated by the tracking processing unit 140 to extract a moving object (step) S873). About the extracted moving body, moment calculation is performed by the moment generation part 150, and the vector of the area and movement amount is measured (step S874).

  It is determined whether or not a volcanic bomb has been generated from the measured area of the moving object and the vector of the movement amount (step S875). If it is determined that a volcanic bomb has been generated (step S875: Yes), the processing after step S876 is performed. When it is determined that no volcanic bomb has been generated (step S875: No), the processing after step S871 is repeated.

  Based on the measured area of the moving object, the mass of the volcanic bomb is estimated in the totalization processing unit 210 (step S876). Further, the reach of the volcanic bomb is estimated based on the measured area of the moving object and the vector of the movement amount (step S877). And alert information is produced | generated about this estimated reach (step S878), and the alert information is alert | reported to a related organization (step S879).

  These processes are repeated for each frame of image data arranged in time series.

  As described above, in the fourth application example, by assuming a high frame rate of 1000 fps or more as a frame rate, by predicting the trajectory of the volcanic bomb, the fall point of the volcanic bomb is estimated, and the warning regarding the reach is given. Information can be generated.

<6. Fifth application example>
FIG. 14 is a diagram illustrating an example of tracking imaging in the fifth application example of the embodiment of the present technology. In order to perform tracking imaging using the control system in this embodiment, a lens driving actuator 552 is provided in the lens portion of the camera 550 including the image sensor 100. Further, an automatic tracking button 551 for controlling the lens driving actuator 552 is provided. While the automatic tracking button 551 is pressed, the lens driving actuator 552 is controlled so that the object detected as the target is fixedly captured in the screen. Since the camera 550 is gripped by a photographer, camera shake or the like often occurs and the position of the target is not constant. According to this embodiment, such a target is captured at a fixed position and imaged. Can do.

  FIG. 15 is a diagram illustrating an aspect of tracking imaging in the fifth application example of the embodiment of the present technology. Here, as an example, an example in which a ball is captured as a target during a soccer game is shown.

  When the normal imaging mode is changed to the sports mode, a detection frame 553 is displayed around the ball. This indicates that the image sensor 100 can recognize the ball. When the automatic tracking button 551 is pressed while the detection frame 553 is displayed, the movement of the ball is tracked by the tracking processing unit 140 and the moment calculation is performed by the moment generating unit 150. Then, based on the calculated moment, the movement amount of the ball is calculated by the gravity center position generation unit 160. Also, the ball movement direction 554 is calculated in the totalization processing unit 210 based on the calculated moment. The aggregation processing unit 210 predicts the ball trajectory from the calculated movement amount and movement direction of the ball.

  The control unit 220 supplies control information to the lens driving actuator 552 in accordance with the predicted trajectory of the ball. With this control information, the lens driving actuator 552 is driven to move the lens. At this time, if the camera moves, the position of the center of gravity of the camera shifts, so control information is further fed back from the control unit 220 to the lens driving actuator 552. Thereby, the lens is moved by driving the lens driving actuator 552, and imaging is performed so that the position of the ball becomes a fixed position in the screen. The lens is an example of an optical element described in the claims.

  In this example, it is assumed that the display position of the ball when the automatic tracking button 551 is pressed is maintained. However, when the automatic tracking button 551 is pressed, the display position of the ball is moved to the center, and thereafter The center position may be maintained. Further, a low pass filter may be applied in the filter processing unit 120 so that the trajectory of the ball does not change sharply. Further, when the tracking of the ball is momentarily off, trajectory prediction may be performed for about several frames.

  FIG. 16 is a diagram illustrating another aspect of the tracking imaging in the fifth application example of the embodiment of the present technology. Here, the zoom mode is assumed as the imaging mode. In this zoom mode, the detection of the face is performed together with the detection of the ball. In the figure, on the screen 555, a detection frame 558 is displayed around the ball, and a detection frame 559 is displayed around the face.

  When the detection frames 558 and 559 enter the zoom-in frame 557, an enlargement (zoom-in) operation is performed. On the other hand, when at least one of the detection frames 558 and 559 is removed from the zoom-out frame 556, a reduction (zoom-out) operation is performed. As a result, the zoom-in or zoom-out operation is performed so that both the detection frames 558 and 559 always enter the screen 555.

  FIG. 17 is a flowchart illustrating an example of a tracking imaging process procedure according to the fifth application example of the embodiment of the present technology. In the drawing, the flowchart on the left represents the processing procedure of the camera 550 main body, and the flowchart on the right represents the processing procedure of the image sensor 100.

  When imaging with the camera 550 starts, the image sensor 100 is activated. When the power of the image sensor 100 is off (step S731: Yes), the operation of the image sensor 100 is not performed. When the image sensor 100 is powered on (step S731: No), the process waits until a target is designated by the camera 550 (step S732).

  The camera 550 performs imaging and display in the monitoring mode (step S711). In this monitoring mode, a target to be tracked is designated (step S712). For example, a ball is designated as a target in a soccer game. When target detection starts (step S713: Yes), the coordinates of the target are notified to the image sensor 100 (step S714). When the target tracking fails (step S715: No), the target is detected again (step S713).

  When the target is specified and its coordinates are notified (step S732: Yes), the image sensor 100 detects the target. If the detection is successful (step S733: Yes), the image sensor 100 returns a detection response to the camera 550 (step S734). In addition, even if the detection is successful, if the subsequent target tracking fails (step S735: No), a notification that the tracking is not performed is returned to the camera 550 (step S738), and the target is detected again. (Step S733).

  When the camera 550 receives a notification of target detection from the image sensor 100 or a notification that it is out of tracking (step S716: Yes), the camera 550 waits for the automatic tracking button 551 to be pressed. When the automatic tracking button 551 starts to be pressed (step S717: Yes), the image sensor 100 is instructed to perform tracking control (step S718). When the pressing of the automatic tracking button 551 is completed (step S719: Yes), the image sensor 100 is instructed to control tracking (step S721).

  Upon receiving control related to tracking from the camera 550 (step S736: Yes), the image sensor 100 predicts the trajectory of the ball and controls the lens driving actuator 552 based on the predicted trajectory (step S737). Meanwhile, target tracking is repeated (step S735).

  The camera 550 repeats target tracking (step S715) until the mode is switched (step S722). When the mode is switched (step S722: Yes), the process returns to the target designation unless the power is turned off (step S712).

  As described above, in the fifth application example, assuming the high frame rate of 1000 fps or more as the frame rate, the target locus is predicted, and thus the lens driving actuator 552 is driven based on the predicted locus. Control. Thereby, it is possible to take an image so that the position of the ball becomes a fixed position in the screen.

<7. Sixth application example>
FIG. 18 is a diagram illustrating an example of image superposition in the sixth application example of the embodiment of the present technology. In order to perform image superimposition using the control system in the present embodiment, an original image is captured by the image sensor 100. Then, a part to be superimposed is detected in the original image, and another image is superimposed on the part. Thereby, the superimposed image can be supplied as necessary.

  In FIG. 18 a, as an example, an example in which an advertisement portion described in a player's uniform is replaced with another advertisement during a soccer game is shown. It is assumed that the company name 561 of “P Shoji” is described as an advertisement on the player's uniform on the spot. Since this is physical, it does not change itself. The on-site camera takes a picture of this state as it is.

  The captured images are broadcast to various places through the facilities of the broadcasting station. At that time, the company name or brand name corresponding to the advertising contract is superimposed on the advertisement part of the photographed image. For example, as shown in FIG. 18b, the company name “Q industry” is superimposed on “P trading”, and the superimposed image is broadcast. Thereby, the company name 562 of “Q industry” appears as an advertisement on the player's uniform displayed on the television.

  FIG. 19 is a diagram illustrating an example of image superimposition timing in the sixth application example of the embodiment of the present technology. Here, 30 fps is assumed as the frame rate of the broadcasting camera. On the other hand, a frame rate of a high-speed camera including the image sensor 100 according to the present embodiment is assumed to be 1000 fps.

  In this case, target tracking is performed by using the advertisement portion to be superimposed in the image captured by the high-speed camera as the tracking target. Thereby, the coordinate position and shape of an advertisement part are recognized. Then, a new advertisement texture is generated, and the generated texture is pasted on the advertisement part of the frame of the broadcast camera. Since the frame rate of the high-speed camera is sufficiently faster than the frame rate of the broadcast camera, after performing these processes on the frame of the high-speed camera, the frame can be superimposed on the frame of the broadcast camera.

  FIG. 20 is a flowchart illustrating an example of an image superimposition processing procedure in the sixth application example of the embodiment of the present technology.

  First, image data is acquired by the image sensor 100 (step S881). The acquired image data constitutes a time-series frame. Noise is removed from each acquired frame by the filter processing unit 120 (step S882).

  Then, the binarization processing unit 130 performs binarization based on the color and brightness of the image in the frame, and detects a marker to be superimposed (step S883). An area is set for the detected marker (step S884), and a difference between frames adjacent in time series in the area is generated by the tracking processing unit 140. Thereby, marker tracking (target tracking) is performed in the set region (step S885). In addition, moment calculation is performed on the marker by the moment generator 150 (step S886).

  Further, processing is selected in accordance with an instruction from the operation input device 310 (step S887). In the case of outputting the superimposed video, after the superimposition processing is performed in the aggregation processing unit 210 (step S888), the superimposed video is output through the interface 230 (step S889). On the other hand, when outputting the original video, the original video is output via the interface 230 without performing the superimposing process (step S889).

  These processes are repeated for each frame of image data arranged in time series.

  As described above, in the sixth application example, assuming a high frame rate of 1000 fps or more as the frame rate, the locus of the superimposition target is predicted to superimpose and replace the image at the superimposition target position. Can do.

<8. Seventh application example>
FIG. 21 is a diagram illustrating an example of distance calculation in the seventh application example of the embodiment of the present technology. Conventionally, object tracking used in motion capture or the like has been performed using a plurality of cameras in order to accurately measure a three-dimensional movement. Therefore, the system is large and expensive. Therefore, in order to perform distance calculation using the control system in the present embodiment, an image of the object to be measured is captured by the image sensor 100, and the distance of the object is calculated.

In this example, in order to calculate the distance, it is necessary to perform calibration in advance. That is, the tracking target object 571 is arranged at a predetermined coordinate position (0, 0), and the distance and area at that time are recorded as the reference distance and the reference area. Thereafter, tracking is performed on the object 571, the center of gravity (x, y) and the area s of the object 571 are calculated, and the distance z is calculated from the area and the result of calibration by the following equation.
Distance z = reference distance × (area s / reference area) ^1/2

  FIG. 22 is a diagram illustrating an aspect of object tracking in the seventh application example of the embodiment of the present technology. In this example, since a high frame rate of 1000 fps or more is assumed as the frame rate captured by the image sensor 100, the amount of motion in units of frames is very small. On the premise of this, tracking can be performed with a simple mechanism by localizing the detection range and obtaining the area and the center of gravity in the range.

The vicinity of the center of gravity 573 of the object 572 in the previous frame is set as a detection target frame 574. At this time, one side (width and height) of the detection range is expressed by the following equation. Where α is a search range coefficient.
One side of the detection range = (0th moment of the previous frame) ^1/2 × α

When the tracking processing unit 140 detects the object 575 in the next frame in the detection target frame 574, the binarization processing unit 130 performs binarization processing. As a result, “1” is indicated as the bitmap I (x, y) at the position (x, y) of the object 575, and “0” is indicated at other positions. Based on the bitmap I (x, y), the moment generator 150 performs moment calculation as shown in the following equation.
0th moment (area) = ΣΣI (x, y)
Horizontal first moment = ΣΣx × I (x, y)
Vertical first moment = ΣΣy × I (x, y)
However, ΣΣ in the above equation represents the summation in the horizontal and vertical directions.

Based on the moment generated in this way, the center-of-gravity position generation unit 160 generates the coordinates of the center-of-gravity position by the following expression.
Center of gravity x coordinate = horizontal first moment / zero order moment Center of gravity y coordinate = vertical first moment / zero order moment

  FIG. 23 is a diagram illustrating an aspect of distance calculation in the seventh application example of the embodiment of the present technology. As described above, in this example, the distance can be calculated from the area obtained by the camera 561 by calibrating the relationship between the distance and the area at the reference position in advance.

Here, the area A0 at the reference position is set as the reference area, and the distance D0 at that time is set as the reference distance. When the area A1 is obtained when performing object tracking, the distance D1 is calculated by the following equation.
D1 = D0 × (A1 / A0) ^1/2

Similarly, when the area A2 is obtained when performing object tracking, the distance D2 is calculated by the following equation.
D2 = D0 × (A2 / A0) ^1/2

  FIG. 24 is a flowchart illustrating an example of a processing procedure of distance calculation in the seventh application example of the embodiment of the present technology. It is assumed that calibration is performed as a precondition for the distance calculation.

  First, image data is acquired by the image sensor 100 (step S891). The acquired image data constitutes a time-series frame. Noise is removed from each acquired frame by the filter processing unit 120 (step S892).

  Then, the binarization processing unit 130 performs binarization based on the color and brightness of the image in the frame, and detects a marker to be superimposed (step S893). An area is set for the detected marker (step S 894), and a difference between frames adjacent in time series in the area is generated by the tracking processing unit 140. Thereby, tracking of the marker (target tracking) is performed in the set region (step S895).

  Further, moment calculation is performed by the moment generator 150 for the marker (step S896). Based on the calculated moment, the movement amount of the marker is calculated by the gravity center position generation unit 160 (step S897).

  In addition, a process is selected in accordance with an instruction from the operation input device 310 (step S898), and when the distance is calculated, the distance is calculated in the totalization processing unit 210 (step S899). The calculated distance is displayed on the display device 320 via the interface 230, for example.

  These processes are repeated for each frame of image data arranged in time series.

  Thus, in the seventh application example, it is possible to calculate the distance to the object by predicting the trajectory of the object assuming a high frame rate of 1000 fps or more as the frame rate.

  As described so far, according to the embodiment of the present technology, the object trajectory is predicted by imaging the object at a high frame rate and performing image processing in the image sensor 100, and the prediction result thereof. The control target device 330 can be controlled based on the above.

  The above-described embodiment shows an example for embodying the present technology, and the matters in the embodiment and the invention-specific matters in the claims have a corresponding relationship. Similarly, the invention specific matter in the claims and the matter in the embodiment of the present technology having the same name as this have a corresponding relationship. However, the present technology is not limited to the embodiment, and can be embodied by making various modifications to the embodiment without departing from the gist thereof.

  Further, the processing procedure described in the above embodiment may be regarded as a method having a series of these procedures, and a program for causing a computer to execute these series of procedures or a recording medium storing the program. You may catch it. As this recording medium, for example, a CD (Compact Disc), an MD (MiniDisc), a DVD (Digital Versatile Disc), a memory card, a Blu-ray disc (Blu-ray (registered trademark) Disc), or the like can be used.

  In addition, the effect described in this specification is an illustration to the last, Comprising: It does not limit and there may exist another effect.

In addition, this technique can also take the following structures.
(1) an image sensor that images a target and generates frames of image data arranged in time series;
A binarization processing unit that performs binarization processing on each of the frames to generate a binarized frame;
A tracking processing unit that generates a difference between the binarized frames adjacent to each other in time series and tracks a change in the position of the object included in the binarized frame;
A moment generation unit that calculates a moment of the object included in the binarized frame based on a result of the tracking processing unit;
A center-of-gravity position generation unit that generates a center-of-gravity position of the object included in the binarized frame based on the moment generated by the moment generation unit;
A trajectory generator that predicts and generates a trajectory of the object based on the barycentric position of the object generated by the centroid position generator;
An image sensor comprising: a control signal supply unit configured to supply a control signal to the control target device based on the trajectory of the object generated by the trajectory generation unit.
(2) The trajectory generation unit predicts a trajectory of the distance to the object,
The image sensor according to (1), wherein the control signal supply unit supplies the control signal when the predicted distance satisfies a predetermined condition.
(3) The image sensor according to (2), wherein the control signal is a control signal for controlling the operation of the buffer member.
(4) The image sensor according to (2), wherein the control signal is a control signal for controlling the operation of the warning device.
(5) The trajectory generator predicts a trajectory of the trajectory of the object,
The image sensor according to any one of (1) to (4), wherein the control signal supply unit supplies the control signal for displaying the predicted trajectory.
(6) The trajectory generator predicts the flight distance of the object,
The image sensor according to (5), wherein the control signal supply unit supplies the control signal for displaying the predicted flight distance.
(7) The locus generation unit predicts the arrival position of the object,
The image sensor according to (5), wherein the control signal supply unit supplies the control signal for displaying the predicted arrival position.
(8) The image sensor according to (7), wherein the control signal supply unit supplies the control signal for outputting an audio signal corresponding to the predicted arrival position.
(9) further comprising an optical element that supplies an image including the object to the image sensor;
The image sensor according to any one of (1) to (8), wherein the control signal is a control signal for controlling an operation of the optical element.
(10) The image sensor according to any one of (1) to (9), wherein the control signal is a control signal that is displayed by replacing a part of the frame with a predetermined image.

DESCRIPTION OF SYMBOLS 100 Image sensor 110 Imaging part 120 Filter process part 130 Binarization process part 140 Tracking process part 150 Moment generation part 160 Center of gravity position generation part 210 Aggregation process part 220 Control part 230 Interface 310 Operation input apparatus 320 Display apparatus 330 Control object apparatus 511 515 Protection target 512, 516, 521, 523, 525 Buffer member 513, 517, 522, 524, 526 Image sensor 514, 518 Moving object 531 Player 532 Ball 533 Mobile terminal 534 Estimated flight distance 541, 550, 561 Camera 542 Speaker 543 Fence 544 Protective cover 545 Control device 551 Automatic tracking button 552 Lens drive actuator

Claims (10)

An image sensor that images a target and generates frames of image data arranged in time series; and
A binarization processing unit that performs binarization processing on each of the frames to generate a binarized frame;
A tracking processing unit that generates a difference between the binarized frames adjacent to each other in time series and tracks a change in the position of the object included in the binarized frame;
A moment generation unit that calculates a moment of the object included in the binarized frame based on a result of the tracking processing unit;
A center-of-gravity position generation unit that generates a center-of-gravity position of the object included in the binarized frame based on the moment generated by the moment generation unit;
A trajectory generator that predicts and generates a trajectory of the object based on the barycentric position of the object generated by the centroid position generator;
An image sensor comprising: a control signal supply unit configured to supply a control signal to the control target device based on the trajectory of the object generated by the trajectory generation unit. The trajectory generator predicts a trajectory of the distance to the object;
The image sensor according to claim 1, wherein the control signal supply unit supplies the control signal when the predicted distance satisfies a predetermined condition.   The image sensor according to claim 2, wherein the control signal is a control signal for controlling an operation of the buffer member.   The image sensor according to claim 2, wherein the control signal is a control signal for controlling an operation of the warning device. The trajectory generator predicts the trajectory of the trajectory of the object;
The image sensor according to claim 1, wherein the control signal supply unit supplies the control signal for displaying the predicted trajectory. The trajectory generator predicts the flight distance of the object,
The image sensor according to claim 5, wherein the control signal supply unit supplies the control signal for displaying the predicted flight distance. The locus generation unit predicts the arrival position of the object;
The image sensor according to claim 5, wherein the control signal supply unit supplies the control signal for displaying the predicted arrival position. The image sensor according to claim 7, wherein the control signal supply unit supplies the control signal for outputting an audio signal corresponding to the predicted arrival position. An optical element for supplying an image including the object to the image sensor;
The image sensor according to claim 1, wherein the control signal is a control signal for controlling an operation of the optical element.   The image sensor according to claim 1, wherein the control signal is a control signal that is displayed by replacing a part of the frame with a predetermined image.

Images