JP2024066386A - Information processing device, information processing method, and information processing program - Google Patents

Abstract

To provide an information processing device, an information processing method, and an information processing program that detect a moving object.SOLUTION: In an information processing device 10, a motion processing unit (MoPU) 12 as a first processor extracts a point indicating an existing position of an object from an image of the object; and outputs movement information indicating a movement of the point indicating the existing position of the object along a predetermined coordinate axis, at a frame rate of 1000 frames per second or higher.SELECTED DRAWING: Figure 2

Description

The disclosed technology relates to an information processing device, an information processing method, and an information processing program that detect moving objects.

Patent document 1 describes a vehicle with an autonomous driving function.

JP 2022-035198 A

The information processing device according to the disclosed technology includes a first processor. The first processor extracts a point indicating the location of an object from an image of the object, and outputs motion information indicating the motion of the point indicating the location of the object along a predetermined coordinate axis at a frame rate of 1000 frames/second or more.

The first processor may output, as the motion information, vector information of the motion of the center point or center of gravity of the object along a predetermined coordinate axis. The first processor may output the vector information for at least two diagonal vertices of a rectangle that encloses the contour of the object.

The images may include infrared images. The images may include visible light images and infrared images that are synchronized with each other.

The two first processors may be used to output, as the motion information, vector information of the movement of a point indicating the location of the object along each of three coordinate axes in a three-dimensional orthogonal coordinate system.

The first processor may derive the distance to the object based on the electromagnetic waves irradiated to the object and reflected from the object, and output, as the movement information, vector information of the movement of a point indicating the location of the object along each of three coordinate axes in a three-dimensional orthogonal coordinate system.

The information processing device may further include a second processor that outputs images of the object at a frame rate of less than 1000 frames per second, and a third processor that performs response control for the object based on the movement information and the images output from the second processor.

The information processing device according to the disclosed technology includes a first processor. The first processor extracts a point indicating the location of an object from an image in which the object appears, and outputs the point indicating the location of the object.

The information processing device includes a camera capable of changing the frame rate, and the first processor calculates a score related to the external environment, determines the frame rate of the camera according to the score, outputs a control signal to the camera instructing the camera to capture an image at the determined frame rate, extracts a point indicating the location of the object from the image captured by the camera, and outputs the point indicating the location of the object.

The information processing device is mounted on a vehicle, and the first processor calculates a degree of danger regarding the traveling of the vehicle as a score regarding the external environment, determines a frame rate of the camera according to the degree of danger, outputs a control signal to the camera instructing the camera to capture an image at the determined frame rate, extracts a point indicating the location of the object from the image captured by the camera, and outputs the point indicating the location of the object.

The first processor extracts an object from the image, and if the object is located in a predetermined area, extracts a point indicating the object's location, and outputs the point indicating the object's location.

The first processor extracts objects from the image, calculates a score for each object, extracts points indicating the location of the object where the score is equal to or greater than a predetermined threshold, and outputs the points indicating the location of the object.

The information processing device according to the disclosed technology is an information processing device that includes a camera capable of changing the frame rate and a processor, and the processor detects objects appearing in images captured by the camera at each time, and controls the camera to change the frame rate according to at least one of a time series of the number of the detected objects, a time series of the acceleration of the objects, and a time series of the size of the objects.

When changing the frame rate according to the time series of the number of objects, the processor may control the frame rate to be higher when the number of objects in the image at the current time is greater than the number of objects in the image at the previous time, and control the frame rate to be lower when the number of objects in the image at the current time is less than the number of objects in the image at the previous time.

When changing the frame rate according to the time series of the acceleration of the object, the processor may control the frame rate to be higher if the acceleration of the object in the image at the current time is greater than the acceleration of the object in the image at the previous time, and control the frame rate to be lower if the acceleration of the object in the image at the current time is less than the acceleration of the object in the image at the previous time.

When changing the frame rate in accordance with the time series of the size of the object, the processor may control the frame rate to be higher when the size of the object in the image at the current time is larger than the size of the object in the image at the previous time, and control the frame rate to be lower when the size of the object in the image at the current time is smaller than the size of the object in the image at the previous time.

The processor may calculate a score related to the external environment according to at least one of a time series of the number of the objects, a time series of the acceleration of the objects, and a time series of the size of the objects, and may control the frame rate to be changed according to the score related to the external environment.

The processor may extract a point indicating the location of the object from an image captured by the camera, and output the point indicating the location of the object.

The information processing device may use the two processors to output, as the motion information, vector information of the motion of a point indicating the location of the object along each of three coordinate axes in a three-dimensional orthogonal coordinate system.

The information processing method according to the disclosed technology is an information processing method executed by an information processing device including a camera capable of changing the frame rate and a processor, in which the processor detects objects appearing in images captured by the camera at each time, and controls the camera to change the frame rate according to at least one of a time series of the number of the detected objects, a time series of the acceleration of the objects, and a time series of the size of the objects.

The information processing program related to the disclosed technology is an information processing program to be executed by a processor of an information processing device including a camera capable of changing the frame rate and a processor, in which the processor detects objects appearing in images captured by the camera at each time, and controls the camera to change the frame rate according to at least one of a time series of the number of the detected objects, a time series of the acceleration of the objects, and a time series of the size of the objects.

Note that the above summary of the invention does not list all of the necessary features of the present invention. Also, subcombinations of these features may also be inventions.

FIG. 1 is a schematic diagram showing an example of a vehicle equipped with a Central Brain. 1 is a block diagram showing an example of a configuration of an information processing device according to an embodiment of the disclosed technology. FIG. 11 is a diagram showing an example of a manner of data transfer from a MoPU to a Central Brain according to an embodiment of the disclosed technology. FIG. 11 is a diagram showing an example of a manner of data transfer from a MoPU to a Central Brain according to an embodiment of the disclosed technology. 1 is a diagram showing an example of an aspect of object coordinate detection by MoPU according to an embodiment of the disclosed technology; FIG. 11 is a diagram illustrating an example of MoPU processing according to an embodiment of the disclosed technology. FIG. 11 is a diagram illustrating an example of MoPU processing according to an embodiment of the disclosed technology. 1 is a block diagram showing an example of a configuration of an information processing device according to an embodiment of the disclosed technology. 1 is a block diagram showing an example of a configuration of an information processing device according to an embodiment of the disclosed technology. FIG. 1 is a diagram for explaining an embodiment of the disclosed technology. FIG. 1 is a diagram for explaining an embodiment of the disclosed technology.

The present invention will be described below through embodiments of the invention, but the following embodiments do not limit the invention according to the claims. Furthermore, not all of the combinations of features described in the embodiments are necessarily essential to the solution of the invention.

First Embodiment
The information processing device according to the embodiment of the disclosed technology may be configured to accurately determine an index value required for driving control based on a large amount of information related to vehicle control. Therefore, the information processing device according to the present disclosure may be at least partially mounted on a vehicle to realize vehicle control.

In addition, an information processing device according to an embodiment of the disclosed technology can realize Autonomous Driving in real time based on data obtained from Level 6 AI/multivariate analysis/goal seek/strategy planning/optimal probability solution/optimal speed solution/optimal course management/multiple sensor inputs at the edge, and can provide a driving system that is adjusted based on the delta optimal solution.

Figure 1 is a schematic diagram showing an example of a vehicle equipped with a Central Brain. As shown in Figure 1, the Central Brain may be connected to multiple Gate Ways so that they can communicate with each other. The Central Brain of this embodiment can achieve Level L6 autonomous driving based on multiple pieces of information acquired via the Gate Ways.

"Level 6" is a level that represents autonomous driving, and is equivalent to a level higher than Level 5, which represents fully autonomous driving. Although Level 5 represents fully autonomous driving, it is at the same level as a human driving, and there is still a chance of accidents occurring. Level 6 represents a level higher than Level 5, and is equivalent to a level where the chance of accidents occurring is lower than at Level 5.

The computing power at Level 6 is about 1000 times that of Level 5. Therefore, it is possible to achieve high-performance operation control that was not possible at Level 5.

FIG. 2 is a block diagram showing an example of the configuration of an information processing device 10 according to an embodiment of the disclosed technology. The information processing device 10 includes an IPU (Image Processing Unit) 11, an MoPU (Motion Processing Unit) 12, a Central Brain 15, and a memory 16. The Central Brain 15 includes a GNPU (Graphics Neural network Processing Unit) 13 and a CPU (Central Processing Unit) 14.

IPU 11 may be built into an ultra-high resolution camera (not shown) installed in the vehicle. IPU 11 performs predetermined image processing such as Bayer transformation, demosaicing, noise removal, and sharpening on images of objects present around the vehicle, and outputs the processed images of the objects at a frame rate of, for example, 10 frames per second and a resolution of 12 million pixels. The images output from IPU 11 are supplied to Central Brain 15 and memory 16. IPU 11 is an example of a "second processor" in the disclosed technology.

The MoPU 12 may be built into a camera other than the ultra-high resolution camera installed in the vehicle. The MoPU 12 outputs motion information indicating the motion of the captured object from an image of the object captured at a frame rate of 1000 frames/second or more, for example, at a frame rate of 1000 frames/second or more. That is, the frame rate of the output of the MoPU 12 is 100 times the frame rate of the output of the IPU 11. The MoPU 12 outputs motion information, which is vector information of the motion of a point indicating the location of the object along a predetermined coordinate axis. That is, the motion information output from the MoPU 12 does not include information necessary to identify what the captured object is (e.g., whether it is a person or an obstacle), but only includes information indicating the motion (movement direction and movement speed) on the coordinate axes (x-axis, y-axis, z-axis) of the center point (or center of gravity) of the object. The information output from the MoPU 12 is supplied to the Central Brain 15 and the memory 16. Because the motion information does not include image information, the amount of information transferred to the Central Brain 15 and memory 16 can be dramatically reduced. The MoPU 12 is an example of a "first processor" in the disclosed technology.

Based on the image output from IPU 11 and the motion information output from MoPU 12, Central Brain 15 executes driving control of the vehicle as a response control to the object. For example, Central Brain 15 recognizes objects (people, animals, roads, traffic lights, signs, crosswalks, obstacles, buildings, etc.) around the vehicle based on the image output from IPU 11. Also, Central Brain 15 recognizes the motion of recognized objects around the vehicle based on the motion information output from MoPU 12. Based on the recognized information, Central Brain 15 performs, for example, control of the motor that drives the wheels (speed control), brake control, and steering control. In Central Brain 15, GNPU 13 may be responsible for processing related to image recognition, and CPU 14 may be responsible for processing related to vehicle control. Central Brain 15 is an example of a "third processor" in the disclosed technology.

Ultra-high resolution cameras are generally used for image recognition in autonomous driving. It is possible to recognize what objects are included in an image captured by a high resolution camera. However, this alone is insufficient for autonomous driving in the Level 6 era. In the Level 6 era, it is also necessary to recognize the movement of objects with higher accuracy. By recognizing the movement of objects with higher accuracy using the MoPU 12, for example, a vehicle traveling by autonomous driving can perform evasive actions to avoid obstacles with higher accuracy. However, a high resolution camera can only capture images at about 10 frames per second, and the accuracy of analyzing the movement of objects is lower than that of a camera equipped with a MoPU 12. On the other hand, a camera equipped with a MoPU 12 can output at a high frame rate of, for example, 1000 frames per second.

Therefore, the technology disclosed herein uses two independent processors, IPU11 and MoPU12. The high-resolution camera (IPU11) is given the role of acquiring the image information necessary to recognize what the captured object is, and MoPU12 is given the role of detecting the movement of the object. MoPU12 captures the object as a point, and analyzes in which direction the coordinates of that point move on the x-axis, y-axis, and z-axis, and at what speed. Since it is possible to detect the overall contour of an object and what the object is using images from the high-resolution camera, it is possible to determine how the entire object will behave by simply knowing, for example, how the center point of the object moves using MoPU12.

A method of analyzing only the movement and speed of the center point of an object makes it possible to significantly reduce the amount of information transferred to the Central Brain 15 and the amount of calculations in the Central Brain 15, compared to determining how the entire image of the object moves. For example, when an image of 1000 pixels x 1000 pixels is transmitted to the Central Brain 15 at a frame rate of 1000 frames per second, including color information, 4 billion bits per second of data will be transmitted to the Central Brain 15. By having the MoPU 12 transmit only the motion information indicating the movement of the center point of the object, the amount of data transmitted to the Central Brain 15 can be compressed to 20,000 bits per second. In other words, the amount of data transmitted to the Central Brain 15 is compressed to 1/200,000.

In this way, by combining the low frame rate, high resolution images output from IPU11 with the high frame rate, lightweight motion information output from MoPU12, it is possible to realize object recognition, including object motion, with a small amount of data.

When one MoPU 12 is used, it is possible to obtain vector information of the movement of a point indicating the location of an object along each of two coordinate axes (x-axis and y-axis) in a three-dimensional orthogonal coordinate system. Using the principle of a stereo camera, two MoPUs 12 may be used to output vector information of the movement of a point indicating the location of an object along each of three coordinate axes (x-axis, y-axis, z-axis) in a three-dimensional orthogonal coordinate system. The z-axis is the axis along the depth direction (vehicle travel).

Also, as shown in FIG. 3, an image from a camera mounted on the left side of the vehicle and an image from a camera mounted on the right side of the vehicle may be input to the core 12A of the MoPU 12. Each of these images may be, for example, an image including color information of 1000 pixels by 1000 pixels, and may be input to the core 12A at a frame rate of 1000 frames/second. Based on these images, the core 12A of the MoPU 12 may transfer motion vector information along each of the three coordinate axes (x-axis, y-axis, z-axis) in a three-dimensional orthogonal coordinate system to the Central Brain 15 at a frame rate of 1000 frames/second. Also, as shown in FIG. 4, an image from a camera mounted on the left side of the vehicle and an image from a camera mounted on the right side of the vehicle may be processed using separate cores 12A ₁ and 12A ₂ , respectively.

In the above description, the MoPU 12 outputs motion information indicating the motion of the center point of the object. However, the disclosed technology is not limited to this mode. The MoPU 12 may output motion information for at least two coordinate points that are diagonal to the vertices of a rectangle that surrounds the contour of the object recognized from the image captured by the camera. FIG. 5 illustrates a mode in which the MoPU 12 sets bounding boxes 21, 22, 23, and 24 that surround the contour of each of the four objects included in the image, and outputs motion information for two coordinate points that are diagonal to the vertices of the bounding boxes 21, 22, 23, and 24. In this way, the MoPU 12 may capture the object as an object having a certain size, rather than as a point. When capturing an object as an object having a certain size, it is not necessary to extract only two coordinate points that are diagonal to the vertices of a rectangle that surrounds the contour of the object recognized from the image captured by the camera, and multiple coordinate points including the contour may be extracted.

As shown in FIG. 6, MoPU 12 may output motion information based on at least one of a visible light image and an infrared image. The visible light image is an image captured by a visible light camera, and the infrared image is an image captured by an infrared camera. The visible light image and the infrared image are each input to core 12A at a frame rate of 1000 frames/second or more. It is preferable that the visible light image and the infrared image are synchronized with each other. By using an infrared image in object detection by MoPU 12, object detection becomes possible even when object detection using a visible light image is difficult, such as at night. MoPU 12 may output motion information based only on the infrared image out of the visible light image and the infrared image, or may output motion information based on both the visible light image and the infrared image.

As shown in FIG. 7, MoPU 12 may output motion information based on an image and a radar signal. The radar signal is a signal based on electromagnetic waves irradiated to an object and reflected from the object. MoPU 12 may derive the distance to the object based on the image and the radar signal, and output motion vector information of a point indicating the location of the object along each of three axes in a three-dimensional orthogonal coordinate system as motion information. The image may include at least one of a visible light image and an infrared image. The image and radar signal are input to core 12A at a frame rate of 1000 frames/second or more.

In the above description, the Central Brain 15 performs vehicle driving control based on the image output from the IPU 11 and the motion information output from the MoPU 12, but the disclosed technology is not limited to this aspect. The Central Brain 15 may perform motion control of the robot as response control to the object based on the image output from the IPU 11 and the motion information output from the MoPU 12. The robot may be, for example, a humanoid smart robot that performs work in place of a human. For example, the Central Brain 15 may perform motion control of the robot's arms, palms, fingers, and feet based on the image output from the IPU 11 and the motion information output from the MoPU 12 to grasp, grab, hold, carry, move, carry, throw, kick, avoid, and the like of the object. The IPU 11 and the MoPU 12 may be mounted, for example, at the positions of the right and left eyes of the robot. In other words, the right eye may be equipped with an IPU11 and MoPU12 for the right eye, and the left eye may be equipped with an IPU11 and MoPU12 for the left eye.

Second Embodiment
Next, a second embodiment will be described. The second embodiment differs from the first embodiment in that the frame rate at which an image is captured is variable.

(Information processing device installed in vehicle: Smart Car)
8 is a block diagram of an information processing device 210 according to the second embodiment, which is mounted on a vehicle. As shown in FIG. 8, the information processing device 210 mounted on the vehicle includes a MoPU 12ML corresponding to the left eye, a MoPU 12MR corresponding to the right eye, an IPU 11, a core 12X, and a Central Brain 15.

The MoPU 12ML is equipped with a camera 30L, a radar 32L, and an infrared camera 34L. The MoPU 12MR is equipped with a camera 30R, a radar 32R, and an infrared camera 34R. The radars 32L and 32R detect the above-mentioned radar signal. The infrared cameras 34L and 34R acquire the above-mentioned infrared images.

The IPU 11 is equipped with a high-resolution camera (not shown) as described above, detects objects from high-resolution images captured by the high-resolution camera, and outputs information indicating the type of object (hereinafter simply referred to as "label information").

Note that below, we will only explain the processing of MoPU12ML, which corresponds to the left eye.

The camera 30L of the MoPU12ML captures images at a higher frame rate (120, 240, 480, 960, or 1920 frames/sec) than the high-resolution camera of the IPU11 (for example, capturing images at 10 frames/sec). The camera 30L is a camera that can change the frame rate.

Core 12L of MoPU12ML (e.g., composed of one or more CPUs) extracts feature points for each frame of image captured by 30L and outputs their coordinate values (X, Y). MoPU12ML outputs, for example, the center point (center of gravity) of the object extracted from the image as the feature point. Note that the feature points may be two diagonal vertices of a rectangle that virtually surrounds the object. Furthermore, when capturing an object as an object of a certain size, it is not necessary to extract only at least two diagonal coordinate points of the vertices of a rectangle that surrounds the contour of the object recognized from the image captured by the camera, and multiple coordinate points including the contour may be extracted.

Specifically, MoPU12ML outputs the coordinate values (X, Y) of a feature point extracted from one object. Note that, for example, when multiple objects (e.g., object A, object B, and object C, etc.) appear in one image, MoPU12ML may output the coordinate values (Xn, Yn) of the feature points extracted from each of the multiple objects. The series of feature points in the images captured at each time corresponds to object movement information.

It is also possible that, for example, darkness may prevent MoPU 12ML from identifying an object. In this case, MoPU 12ML may use infrared camera 34L to detect the heat of the object, and output the coordinates (Xn, Yn) of the object based on the infrared image resulting from the detection and the image captured by camera 30L. Also, the capture of the image by camera 30L may be synchronized with the capture of the infrared image by infrared camera 34L. In this case, for example, the number of images captured per second by camera 30L is synchronized with the number of images captured per second by infrared camera 34L (e.g., 1920 frames/second).

MoPU12ML may also acquire the Z-axis coordinate value of the object based on the three-dimensional point cloud data acquired by radar 32L. In this case, the capturing of images by camera 30L and the acquisition of three-dimensional point cloud data by radar 32L may be synchronized. For example, the number of three-dimensional point cloud data acquired per second by radar 32L is synchronized with the number of images captured per second by camera 30L (e.g., 1920 frames/second).

In addition, the number of images taken per second by the camera 30L, the number of images taken per second by the infrared camera 34L, and the number of 3D point cloud data acquired per second by the radar 32L may be set to be the same, and the timing of data acquisition may be synchronized.

Core 12X acquires the coordinates of the feature points output from MoPU 12ML and the label information of the object (information indicating whether the object is a dog, a cat, or a bear) output from IPU 11. Core 12X then outputs the label information in association with the coordinates corresponding to the feature points. This makes it possible to associate information about what the object represented by the feature points represents with movement information of the object represented by the feature points.

The above is the processing of MoPU12ML, which corresponds to the left eye. MoPU12MR, which corresponds to the right eye, performs the same processing as MoPU12ML, which corresponds to the left eye.

In addition, the depth coordinate value Zn of the feature point may be further calculated using the principle of a stereo camera based on the image captured by the camera 30L of the MoPU12ML and the image captured by the camera 30R of the MoPU12MR.

(Information processing device installed in a robot: Smart Robot)
Fig. 9 is a block diagram of an information processing device 310 according to the second embodiment, which is mounted on a robot. As shown in Fig. 9, the information processing device 310 mounted on the robot includes a MoPU 12ML corresponding to a left eye, a MoPU 12MR corresponding to a right eye, an infrared camera 34, a structured light 36, a core 12X, and a central brain 15. The information processing device 310 mounted on the robot has the same functions as the information processing device 210 mounted on the vehicle.

For example, if the MoPU 12ML cannot identify an object due to the effects of darkness, the MoPU 12ML detects the heat of the object using the infrared camera 34, and outputs the coordinates (Xn, Yn) of the object based on the infrared image that is the detection result and the image captured by the camera 30L. Also, the capture of the image by the camera 30L and the capture of the infrared image by the infrared camera 34 may be synchronized. In this case, for example, the number of images captured per second by the camera 30L and the number of images captured per second by the infrared camera 34 are synchronized (e.g., 1920 frames/second).

The core 12X may also use structured light 36 to obtain the depth coordinate Zn of the object. The structured light 36 is disclosed, for example, in the reference document (http://ex-press.jp/wp-content/uploads/2018/10/018_teledyne_3rd.pdf). In this case, the capturing of images by the cameras 30L and 30R and the measurement of three-dimensional data by the structured light 36 may be synchronized. For example, the number of images captured per second by the cameras 30L and 30R may be synchronized with the number of three-dimensional data measured per second by the structured light 36 (for example, 1920 frames/second).

Furthermore, both the infrared images captured by the infrared camera 34 and the three-dimensional data measured by the structured light 36 may be used in combination.

(Change frame rate according to external environment)
The information processing devices 210 and 310 may change the frame rate of the camera depending on the external environment. For example, the information processing devices 210 and 310 calculate a score related to the external environment, and determine the frame rate of the cameras 30L and 30R depending on the score. Then, the information processing devices 210 and 310 output a control signal to the cameras 30L and 30R to instruct them to capture an image at the determined frame rate. The cameras 30L and 30R capture images at the determined frame rate. Then, the information processing devices 210 and 310 extract points indicating the location of an object from the images captured by the cameras 30L and 30R, and output the points indicating the location of the object.

The information processing device 210 mounted on the vehicle is equipped with multiple types of sensors (not shown). One or more processors equipped in the information processing device 210 mounted on the vehicle calculate the degree of danger of the vehicle traveling in a dangerous place in the future as a score regarding the external environment based on sensor information (e.g., weight center of gravity shift, detection of road material, detection of outside air temperature, detection of outside air humidity, detection of up, down, side, diagonal inclination angle of a slope, road freezing condition, detection of moisture content, material of each tire, wear condition, detection of air pressure, road width, presence or absence of overtaking prohibition, oncoming vehicles, vehicle type information of front and rear vehicles, cruising state of those vehicles, or surrounding conditions (birds, animals, soccer balls, wrecked vehicles, earthquakes, housework, wind, typhoon, heavy rain, light rain, blizzard, fog, etc.) etc.) acquired from multiple types of sensors (not shown).

Then, one or more processors included in the information processing device 210 switch the number of images captured per second (frame rate) based on the calculated risk level and a threshold value. The risk level is, for example, a value between 0 and 1.0. In this case, for example, a first threshold, a second threshold, a third threshold, and a fourth threshold are set in advance as threshold values. For example, the first threshold value may be set to 0.2, the second threshold value to 0.4, the third threshold value to 0.6, and the fourth threshold value to 0.8.

For example, when the risk level is lower than the first threshold, one or more processors provided in the information processing device 210 select 120 frames per second and output control signals to the cameras 30L, 30R, the radars 32L, 32R, and the infrared cameras 34L, 34R to capture images, acquire radar signals, or capture infrared images at that frame rate.

Also, for example, when the risk level is equal to or higher than the first threshold and lower than the fourth threshold, one or more processors included in the information processing device 210 select one of 240, 480, or 960 frames/second, and output a control signal to each device to acquire various data at that frame rate. For example, when the risk level is equal to or higher than the first threshold and lower than the second threshold, one or more processors included in the information processing device 210 select a frame rate of 240 frames/second, and output a control signal to each device to acquire various data at that frame rate. Also, when the risk level is equal to or higher than the second threshold and lower than the third threshold, one or more processors included in the information processing device 210 select a frame rate of 480 frames/second, and output a control signal to each device to acquire various data at that frame rate. Also, when the risk level is equal to or higher than the third threshold and lower than the fourth threshold, one or more processors included in the information processing device 210 select a frame rate of 960 frames/second, and output a control signal to each device to acquire various data at that frame rate. Also, for example, when the risk level is equal to or higher than the fourth threshold, one or more processors included in the information processing device 210 select 1920 frames/second and output a control signal to each device to acquire various types of data at that frame rate.

In addition, one or more processors included in the information processing device 210 may use big data related to driving that is known before the vehicle starts driving, such as long-tail incident AI (Artificial Intelligence) DATA (e.g., trip data of a vehicle equipped with a level 5 autonomous driving control system) or map information, as information for predicting the level of danger, to predict the level of danger.

The one or more processors of the information processing device 310 mounted on the robot may calculate a score related to the external environment based on, for example, the speed of an object captured by the cameras 30L and 30R, and change the frame rate according to the score. For example, the score related to the external environment is calculated to be larger as the speed of the object increases. For this reason, when the score related to the external environment is large, the one or more processors of the information processing device 310 mounted on the robot select 1920 frames/second, and output control signals to each device to acquire various data at that frame rate. When the score related to the external environment is small, the one or more processors of the information processing device 310 mounted on the robot select 120 frames/second, and output control signals to each device to acquire various data at that frame rate. Other controls are the same as those of the information processing device 210 mounted on the vehicle described above.

(Output of feature points according to the area where the object is detected)
The information processing device 210, 310 may output a point indicating the location of an object when the location of the object in the image is a predetermined area. In this case, the information processing device 210, 310 determines whether or not to output feature points of the object depending on the area in which the object is detected. For example, the core 12L, 12R of the information processing device 210 mounted on the vehicle does not extract feature points from objects (e.g., objects existing on the sidewalk) different from the object detected in the road area on which the vehicle runs. FIG. 10 shows a diagram for explaining the process in the case where feature points are not extracted from objects existing on the sidewalk, for example.

In FIG. 10(A), objects B1 to B4 are extracted. Normally, coordinates representing feature points for each of objects B1 to B4 would be extracted.

In this case, for example, as shown in FIG. 10(B), the cores 12L, 12R included in the information processing device 210 mounted on the vehicle sequentially detect road boundaries L from an image in front of the vehicle using known technology. Then, the cores 12L, 12R included in the information processing device 210 extract coordinates representing feature points only from objects B1 to B3 located on the road identified by the road boundaries L.

In addition, for example, the cores 12L and 12R provided in the information processing device 210 may extract coordinates representing feature points only from the objects B1 to B3, without extracting the object area itself for the object B4 that is different from the objects B1 to B3 located on the road.

(Output of feature points according to object movement)
The information processing device 210, 310 may calculate a score for each object appearing in the image, and extract a point indicating the location of the object whose score is equal to or greater than a predetermined threshold. In this case, for example, the information processing device 210, 310 may determine whether or not to output feature points of the object according to the movement of the object. For example, the cores 12L, 12R of the information processing device 210 may not extract feature points from objects that do not affect the running of the vehicle. Specifically, the cores 12L, 12R of the information processing device 210 calculate the moving direction or speed of the object appearing in the image by using AI, for example. And, for example, the cores 12L, 12R of the information processing device 210 do not extract feature points from pedestrians moving away from the road. On the other hand, the cores 12L, 12R of the information processing device 210 extract feature points from objects approaching the road (for example, a child about to run out onto the road).

The one or more processors included in the information processing device 210 can also extract feature points from images captured by, for example, an event camera (https://dendenblog.xyz/event-based-camera/). Figure 11 shows a diagram for explaining an image captured by an event camera.

As shown in FIG. 11, in an image captured by an event camera, the difference between the image captured at the current time and the image captured at the previous time is extracted as points. For this reason, when an event camera is used, for example, points of each moving part of the person area shown in FIG. 11(A) are extracted, as shown in FIG. 11(B).

In response to this, one or more processors included in the information processing device 210 extract the person as an object, as shown in FIG. 11(C), and then extract the coordinates of the feature points (for example, only one point) that represent the person area. This makes it possible to reduce the amount of information transferred to the Central Brain 15 and the memory 16. Images captured by the event camera allow the person as an object to be extracted at any frame rate, so the cameras 30L and 30R mounted on the MoPUs 12ML and 12MR capture images at a maximum frame rate of 1920 frames/second, but in the case of the event camera, extraction is not limited to 1920 frames/second and can also be performed at a higher frame rate, making it possible to capture object movement information with greater accuracy.

As described above, the information processing devices 210 and 310 of the second embodiment extract points indicating the location of an object from an image in which the object appears, and output the points indicating the location of the object. This makes it possible to reduce the amount of information transferred to the core 12X, the Central Brain 15, and the memory 16. In addition, by associating the points indicating the location of the object with the label information output from the IPU 11, information regarding what kind of object is moving is obtained. In particular, the cameras 30L and 30R mounted on the MoPUs 12ML and 12MR are capable of capturing images at a maximum frame rate of 1920 frames/second, and therefore can capture object movement information with high accuracy.

In addition, the information processing devices 210, 310 are equipped with cameras 30L, 30R that can change the frame rate, calculate a score related to the external environment, and determine the frame rate of the camera according to the score. The information processing devices 210, 310 then output a control signal to the cameras 30L, 30R to instruct them to capture an image at the determined frame rate, extract a point indicating the location of an object from the image captured by the cameras 30L, 30R, and output the point indicating the location of the object. This allows images to be captured at a frame rate suitable for the external environment.

In addition, the information processing device 210 calculates the degree of danger regarding the vehicle's traveling as a score regarding the external environment, determines the frame rate of the cameras 30L and 30R according to the degree of danger, outputs a control signal to the cameras 30L and 30R to instruct them to capture images at the determined frame rate, and extracts points indicating the location of objects from the images captured by the cameras 30L and 30R. This makes it possible to change the frame rate according to the degree of danger regarding the vehicle's traveling.

In addition, the information processing device 210, 310 extracts an object from an image, and if the object is located in a specified area, extracts a point indicating the object's location and outputs the point indicating the object's location. This allows the cores 12L, 12R of the information processing device 210, 310 to avoid acquiring points in areas that are less important for control processing.

In addition, the information processing device 210, 310 extracts objects from the image, calculates a score for each object, extracts points indicating the location of objects whose scores are equal to or greater than a predetermined threshold, and outputs the points indicating the location of the objects. This allows the cores 12L, 12R of the information processing device 210, 310 to avoid acquiring points for objects that are less important for the control process.

Third Embodiment
Next, the third embodiment will be described. The configuration of the information processing device of the third embodiment is the same as that of the first or second embodiment, so the same reference numerals are used and the description is omitted. The information processing device of the third embodiment is different from the first and second embodiments in that the frame rate of the camera (for example, at least one of the camera 30L, the radar 32L, the infrared camera 34L, the camera 30R, the radar 32R, and the infrared camera 34R) is controlled to change depending on at least one of the time series of the number of objects appearing in the image, the time series of the acceleration of the objects appearing in the image, and the time series of the size of the objects appearing in the image.

As in the first or second embodiment, consider a case where an image is captured by a camera mounted on a vehicle or robot, and the driving of the vehicle or robot is controlled according to the movement of an object in the image. In this case, if the number of objects in the image captured at the current time is greater than the number of objects in the image captured at the previous time, the number of objects in the image has increased, so it is preferable to increase the frame rate of the camera mounted on the vehicle or robot to acquire more images. Similarly, if the acceleration of an object in the image captured at the current time is greater than the acceleration of an object in the image captured at the previous time, it is also preferable to increase the frame rate of the camera to acquire more images. This is because in this case, the object is accelerating, and it is preferable to acquire images at shorter time intervals. Also, if the size of an object in the image captured at the current time is greater than the size of the object in the image captured at the previous time, it represents that the object is approaching the vehicle or robot, so it is also preferable to increase the frame rate of the camera to acquire more images.

Therefore, at least one processor (e.g., at least one of core 12L, core 12R, and core 12X) of the information processing device of the third embodiment detects objects appearing in images captured by the camera at each time. Then, at least one processor of the information processing device controls the camera to change its frame rate according to at least one of the time series of the number of detected objects, the time series of the acceleration of the objects, and the time series of the size of the objects.

For example, at least one processor of the information processing device controls the camera to increase the frame rate when the number of objects captured in the image at the current time is greater than the number of objects captured at the previous time. Also, at least one processor of the information processing device controls the camera to decrease the frame rate when the number of objects captured in the image at the current time is less than the number of objects captured at the previous time.

Specifically, at least one processor of the information processing device calculates the degree of danger as an example of a score related to the external environment, similar to the second embodiment.

In this case, at least one processor of the information processing device calculates the risk level so that the risk level value increases if the number of objects appearing in the image increases between the previous time and the current time. Also, at least one processor of the information processing device calculates the risk level so that the risk level value decreases if the number of objects appearing in the image decreases between the previous time and the current time. Note that, as in the second embodiment, a first threshold, a second threshold, a third threshold, and a fourth threshold are set in advance as thresholds, and may be set, for example, as the first threshold = 0.2, the second threshold = 0.4, the third threshold = 0.6, and the fourth threshold = 0.8.

For example, when the number of objects captured in an image is one and the number of objects has not changed between the previous time and the current time, at least one processor of the information processing device calculates the danger level, which is an example of a score related to the external environment, to be 0.1. The danger level calculated in this case is lower than the first threshold value. Therefore, at least one processor of the information processing device selects 120 frames per second, and outputs control signals to the cameras 30L, 30R, the radars 32L, 32R, and the infrared cameras 34L, 34R so as to capture images, acquire radar signals, or capture infrared images at that frame rate.

For example, when the number of objects in the image increases from 1 to 10 between the previous time and the current time, at least one processor of the information processing device calculates the risk level, which is an example of a score related to the external environment, to be 0.9. In this case, the calculated risk level is higher than the fourth threshold value. Therefore, at least one processor of the information processing device selects 1920 frames/second, and outputs control signals to the cameras 30L, 30R, the radars 32L, 32R, and the infrared cameras 34L, 34R so as to capture images, acquire radar signals, or capture infrared images at that frame rate.

Also, for example, when the risk level is equal to or greater than the first threshold and lower than the second threshold, at least one processor of the information processing device selects a frame rate of 240 frames/second and outputs a control signal to each device to acquire various data at that frame rate. When the risk level is equal to or greater than the second threshold and lower than the third threshold, at least one processor of the information processing device selects a frame rate of 480 frames/second and outputs a control signal to each device to acquire various data at that frame rate. When the risk level is equal to or greater than the third threshold and lower than the fourth threshold, at least one processor of the information processing device selects a frame rate of 960 frames/second and outputs a control signal to each device to acquire various data at that frame rate.

The information processing device of the third embodiment uses two processors, cores 12L and 12R, to output, as motion information, vector information of the movement of a point indicating the location of an object along each of the three coordinate axes in a three-dimensional orthogonal coordinate system.

In this way, at least one processor of the information processing device controls the camera to increase the frame rate if the number of objects captured in the image increases during the time period between the current time and the previous time. Also, at least one processor of the information processing device controls the camera to decrease the frame rate if the number of objects captured in the image decreases during the time period between the current time and the previous time. As a result, when the number of objects captured in the image increases, it becomes possible to accurately capture the movement of each of the objects, and the vehicle or robot can be appropriately controlled according to the object movement. Also, when the number of objects decreases, the frame rate is lowered so that not many images are captured, thereby, for example, saving the amount of electricity required to drive the camera.

For example, at least one processor of the information processing device does not change the camera frame rate when the amount of change representing the difference between the number of objects in the image at the previous time and the number of objects in the image at the current time is less than a threshold value related to the amount of change in the number of objects. On the other hand, at least one processor of the information processing device controls the camera frame rate to be changed when the amount of change representing the difference between the number of objects in the image at the previous time and the number of objects in the image at the current time is equal to or greater than the threshold value related to the amount of change in the number of objects. For example, when the amount of change in the number of objects represents a decrease in the number of objects, at least one processor of the information processing device controls the camera frame rate to be lowered. On the other hand, when the amount of change in the number of objects represents an increase in the number of objects, at least one processor of the information processing device controls the camera frame rate to be higher.

In addition, at least one processor of the information processing device may control the camera to change the frame rate according to the time series of the acceleration of an object captured in an image, as described above. In this case, at least one processor of the information processing device calculates the acceleration of an object captured in an image at each time using a known technique based on an image captured by the camera at each time. Then, at least one processor of the information processing device controls the camera to increase the frame rate when the acceleration of an object captured in an image at the current time is greater than the acceleration of an object captured in an image at the previous time. On the other hand, at least one processor of the information processing device controls the frame rate to decrease when the acceleration of an object captured in an image at the current time is smaller than the acceleration of an object captured in an image at the previous time. Note that, when multiple objects are captured in an image, at least one processor of the information processing device may control the frame rate to change according to the maximum or minimum acceleration of the multiple objects. Alternatively, when multiple objects are captured in an image, at least one processor of the information processing device may control the frame rate to change according to the average value of the accelerations of the multiple objects.

For example, at least one processor of the information processing device does not change the camera frame rate when the amount of change representing the difference between the acceleration of an object captured in an image at a previous time and the acceleration of an object captured in an image at a current time is less than a threshold value related to the amount of change in the acceleration of the object. On the other hand, at least one processor of the information processing device controls the camera frame rate to be changed when the amount of change representing the difference between the acceleration of an object captured in an image at a previous time and the acceleration of an object captured in an image at a current time is equal to or greater than the threshold value related to the amount of change in the acceleration of the object. For example, when the amount of change in the acceleration of the object represents a decrease in the acceleration of the object, at least one processor of the information processing device controls the camera frame rate to be lowered. On the other hand, when the amount of change in the acceleration of the object represents an increase in the acceleration of the object, at least one processor of the information processing device controls the camera frame rate to be higher.

Also, at least one processor of the information processing device may control the camera to change the frame rate according to the time series of the size of an object captured in an image, as described above. In this case, at least one processor of the information processing device controls the camera to increase the frame rate when the size of an object captured in an image at the current time is larger than the size of an object captured in an image at the previous time. On the other hand, at least one processor of the information processing device controls the frame rate to decrease when the size of an object captured in an image at the current time is smaller than the size of an object captured in an image at the previous time. Note that, when multiple objects are captured in an image, at least one processor of the information processing device may control the frame rate to change according to the maximum or minimum size of each of the multiple objects. Alternatively, when multiple objects are captured in an image, at least one processor of the information processing device may control the frame rate to change according to the average value of the sizes of each of the multiple objects.

For example, at least one processor of the information processing device does not change the camera frame rate when the amount of change representing the difference between the size of an object in an image at a previous time and the size of the object in an image at a current time is less than a threshold value related to the amount of change in the size of the object. On the other hand, at least one processor of the information processing device controls the camera frame rate to be changed when the amount of change representing the difference between the size of an object in an image at a previous time and the size of the object in an image at a current time is equal to or greater than the threshold value related to the amount of change in the size of the object. For example, when the amount of change in the size of the object represents a decrease in the size of the object, at least one processor of the information processing device controls the camera frame rate to be lowered. On the other hand, when the amount of change in the acceleration of the object represents an increase in the size of the object, at least one processor of the information processing device controls the camera frame rate to be higher.

As described above, the information processing device of the third embodiment controls to change the frame rate of the camera (for example, at least one of the camera 30L, the radar 32L, the infrared camera 34L, the camera 30R, the radar 32R, and the infrared camera 34R) according to at least one of the time series of the number of objects captured in the image, the time series of the acceleration of the object captured in the image, and the time series of the size of the object captured in the image. Then, the information processing device of the third embodiment extracts a point indicating the location of the object from the image in which the object is captured, and outputs a point indicating the location of the object. This makes it possible to obtain an appropriate number of images at each time according to the time series of the number of objects captured in the image, the time series of the acceleration of the object captured in the image, and the time series of the size of the object captured in the image, and to appropriately control the vehicle or robot. More specifically, when the number of objects captured in the image increases, the information processing device of the third embodiment controls to increase the frame rate of the camera, making it possible to obtain images at shorter time intervals, and to appropriately control the vehicle or robot. Furthermore, when the acceleration of an object in an image increases, the information processing device of the third embodiment controls the camera to increase its frame rate, making it possible to capture images at shorter time intervals and appropriately control the vehicle or robot. Furthermore, when the size of an object in an image increases, the information processing device of the third embodiment controls the camera to increase its frame rate, making it possible to capture images at shorter time intervals and appropriately control the vehicle or robot.

The blocks in the flowcharts and block diagrams of the present embodiment may represent stages of a process in which an operation is performed or "parts" of a device responsible for performing the operation. Particular stages and "parts" may be implemented by dedicated circuitry, programmable circuitry provided with computer-readable instructions stored on a computer-readable storage medium, and/or a processor provided with computer-readable instructions stored on a computer-readable storage medium. Dedicated circuitry may include digital and/or analog hardware circuitry, and may include integrated circuits (ICs) and/or discrete circuits. Programmable circuitry may include reconfigurable hardware circuitry including AND, OR, XOR, NAND, NOR, and other logical operations, flip-flops, registers, and memory elements, such as, for example, field programmable gate arrays (FPGAs), programmable logic arrays (PLAs), and the like.

A computer-readable storage medium may include any tangible device capable of storing instructions that are executed by a suitable device, such that a computer-readable storage medium having instructions stored thereon comprises an article of manufacture that includes instructions that can be executed to create means for performing the operations specified in the flowchart or block diagram. Examples of computer-readable storage media may include electronic storage media, magnetic storage media, optical storage media, electromagnetic storage media, semiconductor storage media, and the like. More specific examples of computer-readable storage media may include floppy disks, diskettes, hard disks, random access memories (RAMs), read-only memories (ROMs), erasable programmable read-only memories (EPROMs or flash memories), electrically erasable programmable read-only memories (EEPROMs), static random access memories (SRAMs), compact disk read-only memories (CD-ROMs), digital versatile disks (DVDs), Blu-ray disks, memory sticks, integrated circuit cards, and the like.

The computer readable instructions may include either assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source or object code written in any combination of one or more programming languages, including object-oriented programming languages such as Smalltalk (registered trademark), JAVA (registered trademark), C++, etc., and conventional procedural programming languages such as the "C" programming language or similar programming languages.

The computer-readable instructions may be provided to a processor of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, or a programmable circuit, either locally or over a local area network (LAN), a wide area network (WAN), such as the Internet, etc., so that the processor of the general-purpose computer, special-purpose computer, or other programmable data processing apparatus, or the programmable circuit, executes the computer-readable instructions to generate means for performing the operations specified in the flowcharts or block diagrams. Examples of processors include computer processors, processing units, microprocessors, digital signal processors, controllers, microcontrollers, etc.

The present invention has been described above using an embodiment, but the technical scope of the present invention is not limited to the scope described in the above embodiment. It is clear to those skilled in the art that various modifications and improvements can be made to the above embodiment. It is clear from the claims that forms incorporating such modifications or improvements can also be included in the technical scope of the present invention.

The order of execution of each process, such as operations, procedures, steps, and stages, in the devices, systems, programs, and methods shown in the claims, specifications, and drawings is not specifically stated as "before" or "prior to," and it should be noted that the processes may be performed in any order, unless the output of a previous process is used in a later process. Even if the operational flow in the claims, specifications, and drawings is explained using "first," "next," etc. for convenience, it does not mean that it is essential to perform the processes in that order.

(Appendix 1)
An information processing device including a first processor,
The first processor extracts points indicating a location of an object from an image of the object, and outputs motion information indicating a movement of the points indicating the location of the object along a predetermined coordinate axis at a frame rate of 1000 frames/second or more.

(Appendix 2)
The information processing device according to claim 1, wherein the first processor outputs, as the motion information, vector information of a motion of a center point or a center of gravity of the object along a predetermined coordinate axis.

(Appendix 3)
The information processing device according to claim 1, wherein the first processor outputs, as the motion information, vector information of motion along a predetermined coordinate axis for at least two diagonal vertices of a rectangle surrounding a contour of the object.

(Appendix 4)
The information processing device according to claim 1, wherein the image includes an infrared image.

(Appendix 5)
The information processing device of claim 1, wherein the images include a visible light image and an infrared image that are synchronized with each other.

(Appendix 6)
The information processing device according to claim 1, wherein the two first processors are used to output, as the motion information, vector information of a motion of a point indicating a position of the object along each of three coordinate axes in a three-dimensional orthogonal coordinate system.

(Appendix 7)
The information processing device described in Appendix 7, wherein the first processor derives a distance to the object based on a wave reflected from the object of an electromagnetic wave irradiated to the object, and outputs, as the movement information, vector information of a movement of a point indicating the position of the object along each of three coordinate axes in a three-dimensional orthogonal coordinate system.

(Appendix 8)
a second processor for outputting images of the object at a frame rate of less than 1000 frames per second;
a third processor that performs response control for the object based on the movement information and the image output from the second processor;
2. The information processing device according to claim 1, further comprising:

(Appendix 9)
A camera capable of changing the frame rate;
A processor;
An information processing device comprising:
The processor,
Detecting an object appearing in an image captured by the camera at each time;
controlling the frame rate of the camera to be changed in response to at least one of a time series of the number of the detected objects, a time series of the acceleration of the objects, and a time series of the size of the objects;
Information processing device.

(Appendix 10)
The processor,
When changing the frame rate in response to the time series of the number of objects,
When the number of the objects captured in the image at the current time is greater than the number of the objects captured in the image at the previous time, the frame rate is controlled to be increased;
When the number of the objects appearing in the image at the current time is smaller than the number of the objects appearing in the image at the previous time, the frame rate is controlled to be lowered.
10. The information processing device according to claim 9.

(Appendix 11)
The processor,
When changing the frame rate in response to a time series of the acceleration of the object,
When the acceleration of the object captured in the image at the current time is greater than the acceleration of the object captured in the image at the previous time, the frame rate is controlled to be increased;
When the acceleration of the object captured in the image at the current time is smaller than the acceleration of the object captured in the image at the previous time, the frame rate is controlled to be lowered.
11. The information processing device according to claim 9 or 10.

(Appendix 12)
The processor,
When changing the frame rate in accordance with the time series of the size of the object,
When a size of the object captured in an image at a current time is larger than a size of the object captured in an image at a previous time, the frame rate is controlled to be increased;
When a size of the object shown in an image at a current time is smaller than a size of the object shown in an image at a previous time, the frame rate is controlled to be lowered.
12. The information processing device according to claim 9.

(Appendix 13)
The processor,
calculating a score related to an external environment in response to at least one of the time series of the number of objects, the time series of the acceleration of the objects, and the time series of the size of the objects;
Controlling the frame rate so as to change the frame rate in accordance with the score regarding the external environment.
13. The information processing device according to claim 9.

(Appendix 14)
The processor,
extracting a point indicating a position of the object from the image captured by the camera, and outputting the point indicating the position of the object;
14. The information processing device according to claim 9.

(Appendix 15)
the information processing device outputs, using the two processors, motion vector information of a point indicating a position of the object along each of three coordinate axes in a three-dimensional orthogonal coordinate system as the motion information;
15. The information processing device according to claim 9.

(Appendix 16)
A camera capable of changing the frame rate;
A processor;
An information processing method executed by an information processing device comprising:
The processor,
Detecting an object appearing in an image captured by the camera at each time;
controlling the frame rate of the camera to be changed in response to at least one of a time series of the number of the detected objects, a time series of the acceleration of the objects, and a time series of the size of the objects;
Information processing methods.

(Appendix 17)
A camera capable of changing the frame rate;
A processor;
An information processing program for causing the processor of an information processing device to execute the information processing program,
The processor,
Detecting an object appearing in an image captured by the camera at each time;
controlling the frame rate of the camera to be changed in response to at least one of a time series of the number of the detected objects, a time series of the acceleration of the objects, and a time series of the size of the objects;
Information processing program.

10 Information processing device 11 IPU
12 MoPU
13. GNPU
14 CPU
15 Central Brein

Claims (9)

A camera capable of changing the frame rate;
A processor;
An information processing device comprising:
The processor,
Detecting an object appearing in an image captured by the camera at each time;
controlling the frame rate of the camera to be changed in response to at least one of a time series of the number of the detected objects, a time series of the acceleration of the objects, and a time series of the size of the objects;
Information processing device. The processor,
When changing the frame rate in response to the time series of the number of objects,
When the number of the objects captured in the image at the current time is greater than the number of the objects captured in the image at the previous time, the frame rate is controlled to be increased;
When the number of the objects appearing in the image at the current time is smaller than the number of the objects appearing in the image at the previous time, the frame rate is controlled to be lowered.
The information processing device according to claim 1 . The processor,
When changing the frame rate in response to a time series of the acceleration of the object,
When the acceleration of the object captured in the image at the current time is greater than the acceleration of the object captured in the image at the previous time, the frame rate is controlled to be increased;
When the acceleration of the object captured in the image at the current time is smaller than the acceleration of the object captured in the image at the previous time, the frame rate is controlled to be lowered.
The information processing device according to claim 1 . The processor,
When changing the frame rate in accordance with the time series of the size of the object,
When a size of the object captured in an image at a current time is larger than a size of the object captured in an image at a previous time, the frame rate is controlled to be increased;
When a size of the object shown in an image at a current time is smaller than a size of the object shown in an image at a previous time, the frame rate is controlled to be lowered.
The information processing device according to claim 1 . The processor,
calculating a score related to an external environment in response to at least one of the time series of the number of objects, the time series of the acceleration of the objects, and the time series of the size of the objects;
Controlling the frame rate so as to change the frame rate in accordance with the score regarding the external environment.
The information processing device according to claim 1 . The processor,
extracting a point indicating a position of the object from the image captured by the camera, and outputting the point indicating the position of the object;
The information processing device according to claim 1 . the information processing device outputs, using the two processors, motion vector information of a point indicating a position of the object along each of three coordinate axes in a three-dimensional orthogonal coordinate system as the motion information;
The information processing device according to claim 1 . A camera capable of changing the frame rate;
A processor;
An information processing method executed by an information processing device comprising:
The processor,
Detecting an object appearing in an image captured by the camera at each time;
controlling the frame rate of the camera to be changed in response to at least one of a time series of the number of the detected objects, a time series of the acceleration of the objects, and a time series of the size of the objects;
Information processing methods. A camera capable of changing the frame rate;
A processor;
An information processing program for causing the processor of an information processing device to execute the information processing program,
The processor,
Detecting an object appearing in an image captured by the camera at each time;
controlling the frame rate of the camera to be changed in response to at least one of a time series of the number of the detected objects, a time series of the acceleration of the objects, and a time series of the size of the objects;
Information processing program.

Images