JP2020042446A - Information processing system, information processing method, and information processing program - Google Patents

Abstract

To provide an information processing system capable of estimating presence of an object that does not appear in a captured image.SOLUTION: An information processing system 100 includes: an optical sensing section 10 for generating optical information within a predetermined range by optical sensing; a reaction object detection section 22 for detecting a reaction object from optical information; a state recognition section 23 for recognizing a state of a reaction object on the basis of optical information; and a presence estimation section 24 for estimating presence of an observation impossible object that is not detected from optical information on the basis of a state of a reaction object.SELECTED DRAWING: Figure 3

Description

  The present invention relates to an information processing system for processing optical information, an information processing method, and an information processing program.

  2. Description of the Related Art Conventionally, there has been known a system that analyzes an image captured by a camera installed on a moving body, detects an object such as another moving body, and performs movement support or automatic driving of the moving body. As such a system, for example, a driving support system that detects a pedestrian or another vehicle ahead of a vehicle by analyzing an image captured by a vehicle-mounted camera and outputs a warning to the driver has been put into practical use ( For example, Patent Document 1).

JP 2014-6700 A

  However, in the conventional system, an object reflected in a captured image of a camera can be detected, but driving assistance or the like is performed based on the presence of an object not reflected in the captured image, such as an object shielded by another object or an object out of the angle of view. No information was available for autonomous driving.

  Therefore, an object of the present invention is to provide an information processing system, an information processing method, and an information processing program that can obtain information on the presence of an object that does not appear in a captured image.

  An information processing system according to one embodiment of the present invention includes an optical sensing unit (10) that generates optical information within a predetermined range by optical sensing, and a reactive object detection unit (22) that detects a reactive object from the optical information. A state recognizing unit (23) for recognizing a state of the reactive object based on the optical information; and a presence estimating unit for estimating the presence of an unobservable object not detected from the optical information based on the state of the reactive object. (24).

  According to the present invention, the presence of an unobservable object in an unobservable region is estimated based on the state of a reaction object reflected in a captured image. Therefore, the presence of an unobservable object that cannot be imaged by a camera is also estimated. be able to.

FIG. 1 is a diagram showing a use situation of the information processing system according to the embodiment of the present invention. FIG. 2 is a diagram illustrating an image captured by the optical sensing device mounted on the vehicle C1 of FIG. FIG. 3 is a block diagram showing a configuration of the information processing system according to the embodiment of the present invention. FIG. 4 is a diagram illustrating an example of a rule table when the reaction object is a pedestrian. FIG. 5 is an operation flowchart of the information processing system according to the present embodiment. FIG. 6 is an operation flowchart of the presence estimation process by the presence estimation unit.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. The embodiment described below shows an example in which the present invention is implemented, and the present invention is not limited to the specific configuration described below. In carrying out the present invention, a specific configuration according to the embodiment may be appropriately adopted.

  FIG. 1 is a diagram showing a use situation of the information processing system according to the embodiment of the present invention. In the example of FIG. 1, the information processing system 100 according to the embodiment of the present invention is applied to a vehicle C1. In the example of FIG. 1, the vehicle C1 to which the information processing system 100 is applied is traveling on a road, and is approaching an intersection. The vehicle C1 is provided with an optical sensing device 10 that constitutes the information processing system 100.

  In the present embodiment, the optical sensing device 10 is a camera that generates a color image by capturing an image. The optical axis of the camera faces the front of the vehicle C1, and can capture an image of the front of the vehicle C1. The camera can capture an image at an angle of view F. However, the hatched area in FIG. 1 that is shielded by the left and right buildings becomes the blind spot of the camera, and the situation cannot be directly grasped by the image captured by the camera. Also, naturally, the situation outside the angle of view F cannot be directly grasped by the image captured by the camera. In the example of FIG. 1, the vehicle C2 that is about to enter the intersection in front of the vehicle C1 from the lateral direction is at the blind spot of the camera, and is not reflected in the image captured by the camera.

  On the other hand, in the example of FIG. 1, a pedestrian as the reactive object R is located on the other side of the intersection, and is facing the vehicle C1. This pedestrian is observed by the camera without being blocked by another object, and is reflected in the captured image.

  FIG. 2 is a diagram illustrating an image captured by the optical sensing device 10 mounted on the vehicle C1 of FIG. The hatched areas on the left and right in the captured image are the blind spot areas O1 and O2 of the captured image. The hatched areas outside the left and right surroundings of the captured image are the out-of-angle regions O3 and O4 of the captured image.

  Although the vehicle C2 exists behind the wall in the blind spot area O1 on the left side, the vehicle C2 is not shown in the captured image. Also, regarding the blind spot area O2 on the right side, even if there is another vehicle behind the wall, the other vehicle is not reflected in the captured image. Further, regarding the out-of-view areas O3 and O4, even if there is another vehicle, the other vehicle is not reflected in the captured image. In the following, a region where there is a possibility that an unobservable object (hereinafter, referred to as an “unobservable object”) such as the blind spot regions O1 and O2 and the out-of-angle regions O3 and O4 is referred to as an “inobservable region”. That.

  On the other hand, a pedestrian as a reaction object R is reflected on the other side of the front intersection. The information processing system 100 according to the present embodiment, when a captured image as illustrated in FIG. 2 is obtained in the situation as illustrated in FIG. 1, uses the captured image to determine whether an unobservable object exists in the unobservable area. Is intended to estimate that the vehicle C2 exists in the blind spot area O1 in the example of FIG.

  FIG. 3 is a block diagram showing a configuration of the information processing system according to the embodiment of the present invention. The information processing system 100 includes an optical sensing device 10 and an information processing device 20. The optical sensing device 10 generates optical information within a predetermined range by using optical sensing means. In the present embodiment, a camera including an optical system, an image sensor, an image processor, and the like is employed as the optical sensing means. This camera generates, as optical information, a two-dimensional color image corresponding to the angle of view of the camera.

  The information processing device 20 inputs a captured image generated by the optical sensing device 10 by imaging, and estimates the presence of an unobservable object that is not detected from the captured image based on the captured image. For this purpose, the information processing device 20 includes an area setting unit 21, a reaction object detection unit 22, a state recognition unit 23, and a presence estimation unit 24. Note that the information processing device 20 is realized by a computer including a processor, a memory, a storage device, and the like executing a predetermined program.

  The information processing device 20 is mounted on a vehicle and estimates the presence of an unobservable object in the vehicle. Note that some or all of the configuration of the information processing device 20 may be provided in another computer. In this case, a vehicle equipped with a part or all of the information processing device 20 and a vehicle equipped with the optical sensing device 10 can communicate with each other. In this case, the optical sensing device 10 transmits the obtained optical information to the information processing device 20.

  The region setting unit 21 sets an unobservable region as a target for estimating the presence of an unobservable object in an image input from the optical sensing device 10. Specifically, the area setting unit 21 detects a blind spot area from the captured image input from the optical sensing device 10, and sets the blind spot area as an unobservable area. Any method can be used as a method for detecting a blind spot area from an image. For example, the blind spot area may be detected from the image input from the optical sensing device 10 by learning the blind spot area of a large number of images of the vehicle-mounted camera by machine learning such as a neural network.

  In addition, the region setting unit 21 sets a region outside the angle of view around a predetermined direction of the image input from the optical sensing device 10 as a non-observable region. In the present embodiment, the region setting unit 21 sets the region outside the angle of view around the left and right of the image as the unobservable region.

  The reactive object detection unit 22 detects a reactive object from the captured image input from the optical sensing device 10. Estimation based on machine learning, such as a neural network, can also be used for this reaction object detection. Specifically, the reaction object detection unit 22 captures images using R-CNN (Regions with Convolutional Neural Networks), Faster-RCNN, SSD (Single Shot Multibox Detector), Yoro (You only lock once), Mask R-CNN, and the like. The reaction object can be detected from the image.

  The reaction object detection unit 22 first detects an object from a captured image, performs an annotation on the detected object, and outputs “pedestrian”, “vehicle”, “motorcycle”, “bicycle”, “crosswalk” as annotation data. ”,“ Signs ”, etc. Of these labels, movable objects are tagged with that effect. When the tag attached to the detected object has a tag indicating that the object is movable, the reactive object detection unit 22 detects the object as a reactive object. In the present embodiment, a tag indicating that the object is a moving object is attached to a label of “pedestrian”, “vehicle”, “motorcycle”, and “bicycle”.

  A reaction object is an object that reacts according to the surrounding environment. That is, pedestrians, vehicles, two-wheeled vehicles, and bicycles react to surrounding obstacles (other vehicles and the like) and stop responding or move in a hurry. In the case of a pedestrian, the robot may react to surrounding obstacles, turn the gaze, and react in a surprising manner.

  The state recognizing unit 23 recognizes the state of the reactive object detected by the reactive object detection unit 22 based on the captured image in order to estimate the presence of the unobservable object. In the present embodiment, the state recognizing unit 23 estimates the behavior of the reactive object such as stopped or moving as the state of a pedestrian, vehicle, two-wheeled vehicle, bicycle, or the like. The state recognition unit 23 can recognize the state of the reaction object by learning a large amount of the state corresponding to the certain reaction object by machine learning such as a neural network.

  The state recognition unit 23 labels the state of the reaction object. The state of the reaction object is defined for each reaction object. For example, for a pedestrian, labels such as stop, walk, fast-walk, rush, left gaze, right gaze, surprise, and raised hand are prepared as state labels. Further, for example, for a vehicle, labels such as stop, run, slow, left blinker, right blinker, hazard, left turn, right turn, and reverse are prepared as state labels.

  The presence estimating unit 24 estimates the presence of an unobservable object that is not detected from the image based on the state of the reaction object recognized by the state recognition unit 23. First, the presence estimating unit 24 associates a blind spot area for estimating the presence of an unobservable object with the reactive object based on the detected position of the reactive object and the state of the reactive object. In the present embodiment, the presence of an unobservable object in the blind spot area is estimated for the blind spot area closest to the detected reaction object based on the state of the reaction article.

  For this purpose, the existence estimating unit 24 obtains the respective centers of gravity of the detected unobservable region and the reaction object. The presence estimating unit 24 calculates the distance between the center of gravity of each reaction object and the center of gravity of each unobservable region, and associates the reaction object with the closest distance between the centers of gravity with each unobservable region. Then, the presence estimating unit 24 estimates the presence of the unobservable object in the unobservable region based on the state of the reaction object associated with each unobservable region.

  The presence estimating unit 24 has a rule table that defines the correspondence between the state of the reaction object recognized by the state recognition unit 23 and the existence probability of the unobservable object in the unobservable region associated with the reaction object.

  FIG. 4 is a diagram illustrating an example of a rule table when the reaction object is a pedestrian. In this example, as the state of the pedestrian, surprise, stop, change direction, hurry, and constant speed are defined, and for each of them, the existence probability of the unobservable object in the unobservable area corresponding to the pedestrian is defined. I have. The presence estimating unit 24 estimates the presence of the unobservable object by obtaining the existence probability of the unobservable object corresponding to the state of the reaction object recognized by the state recognition unit 23 with reference to the rule table. The existence estimating unit 24 outputs the existence probability of the unobservable object as the estimation result.

  The estimation result output from the existence estimation unit 24 can be used for various purposes. For example, when the information processing system 100 is applied to a vehicle, the vehicle may perform vehicle control such as braking based on the estimation result, or may alert the driver.

  FIG. 5 is an operation flowchart of the information processing system according to the present embodiment. First, the optical sensing device 10 performs imaging to generate a captured image (Step S51). Next, the area setting unit 21 detects a blind spot area from the captured image, and sets it as an unobservable area together with the area outside the angle of view (step S52). Specifically, the area setting unit 21 detects a blind spot area if there is one, and if there is no blind spot area, sets only the surrounding area outside the angle of view as an unobservable area.

  The reaction object detection unit 22 detects a reaction object from the captured image (Step S53). If the reaction object has not been detected from the captured image (NO in step S53), the process ends. If there is a reaction object in the captured image, the reaction object detection unit 22 detects the reaction object by labeling it from the captured image (YES in step S53).

  The state recognition unit 23 recognizes the detected state of the reaction object (step S54). Note that the area setting processing by the area setting unit 21 (step S52), the object detection processing by the object detection unit 22 (step S53), and the state recognition processing by the state recognition unit 23 (step S54) are performed in parallel. Good.

  The presence estimating unit 24 estimates the presence of the unobservable object based on the unobservable region set by the region setting unit 21 and the state of the reaction object recognized by the state recognizing unit 23 (step S55). .

  FIG. 6 is an operation flowchart of the presence estimation process by the presence estimation unit 24. First, the presence estimating unit 24 reads all the unobservable regions set by the region setting unit 21 (step S61), and determines a corresponding reaction object for each unobservable region (step S62).

  When a corresponding reactive object is determined for each unobservable region (YES in step S62), the presence of the unobservable object is estimated from the state of the corresponding reactive object and the rule table for the unobservable region. (Step S63). If a corresponding reaction object cannot be determined for each unobservable region (NO in step S62), the existence estimating unit 24 determines that it is unknown whether there is an unobservable object for the unobservable region ( Step S64).

  The existence estimating unit 24 outputs estimation results for all unobservable regions (step S65). Specifically, for the unobservable region where the existence of the unobservable object could be estimated, the existence probability of the unobservable object is output as the estimation result, and the unobservable region where the existence of the unobservable object could not be estimated is unknown. Is output.

  As described above, according to the information processing system 100 of the present embodiment, the presence of the unobservable object in the unobservable region is estimated based on the state of the reaction object reflected in the captured image. Even for unobservable unobservable objects, their existence can be estimated. The above embodiment is merely an embodiment of the present invention, and various modifications are possible. Hereinafter, modified examples will be described.

  The information processing system 100 according to the above-described embodiment has the optical sensing device 10 mounted on a vehicle and estimates the presence of an unobservable object in front of the vehicle. It can also be applied to applications. For example, the information processing system 100 is applicable to an autonomous mobile robot. In this case, the optical sensing device 10 is installed on the robot, and when the robot moves autonomously, it can be estimated that there is a person as an unobservable object at a blind spot at a turning corner ahead in the traveling direction. By estimating the presence of such a person, it is possible to perform autonomous movement control such as making a large round trip based on the estimation result.

  Further, the information processing system 100 is applicable to a forklift. In this case, the optical sensing device 10 is installed on the forklift, and for example, it is estimated that there is an unobservable object at the blind spot at the front corner, and the driver is alerted by a buzzer or vibration based on the estimation result. be able to. Further, the information processing system 100 can be applied to a visually impaired person support system. In this case, a visually impaired person carries or wears the optical sensing device 10, for example, presumes that a person or a vehicle is present as an unobservable object at a blind spot in a forward corner, and outputs a buzzer or vibration based on the estimation result. Can alert the visually impaired. Further, the information processing system 100 is also applicable to a wearable camera and an action camera. In this case, the user can be notified by a buzzer or vibration that there may be a person in the blind spot.

  In the above embodiment, a camera is used as the optical sensing device 10, an image is generated as optical information, and the information processing device 20 processes a single captured image obtained by the optical sensing device 10. Although the presence of the unobservable object has been estimated, the information processing device 20 may estimate the presence of the unobservable object by processing a plurality of continuous images.

  In this case, the camera continuously captures images, generates continuous captured images, and outputs the images to the information processing device 20. The reaction object detection unit 22 of the information processing device 20 can detect a reaction object based on these continuous captured images. Further, the state recognition unit 23 can also recognize the state of the reaction object based on the continuous captured images. By using not only spatial information but also time-series information, the reactive object detection unit 22 and the state recognition unit 23 can use the Two-Stream CNN, C3D, RNN (Recurrent Neural Network), and LSTM (Long Short- Using a technique such as Term Memory), a reaction object can be detected or the state of the reaction object can be recognized.

  Further, in the above-described embodiment, the reaction object detection unit 22 detects the reaction object, and the state recognition unit 23 recognizes the detected state of the reaction object. However, these processes may be performed simultaneously. That is, when detecting the reaction object from the captured image, the state of the reaction object may be recognized at the same time. For example, a reactive object may be detected, such as a stopped pedestrian or a vehicle emitting a left blinker. In particular, such detection can be effectively performed by using time-series information in addition to image information.

  Further, in the above-described embodiment, a camera that generates an image by imaging (that is, a monocular camera) is adopted as the optical sensing device 10, but the optical sensing device 10 uses the optical information within a predetermined range by optical sensing. Other modes may be used as long as they generate. For example, the optical sensing device 10 may be a three-dimensional imaging system that acquires three-dimensional information within the angle of view. The optical sensing device 10 as a three-dimensional imaging system may be a stereo camera, or, in addition to a monocular camera, a range image using an infrared depth sensor, an ultrasonic sensor, a millimeter wave radar, and LiDAR (Light Detection and Ranging). May be generated.

  As described above, the optical sensing device 10 preferably obtains not only planar information but also three-dimensional information such as three-dimensional information or distance information as optical information, so that the area setting unit 21 suitably detects a blind spot area. be able to. The area setting unit 21 can detect a blind spot area based on a depth jump value based on stereoscopic information or distance information.

  Further, in the above embodiment, the presence estimating unit 24 associates each non-observable region with the reaction object closest to the region, and performs observation in the non-observable region based on the state of the associated reaction object. The existence of impossible objects was estimated. In this case, while one reaction object may be associated with a plurality of unobservable regions, there may be unobservable regions that cannot be associated with any of the reaction objects.

  The association between the unobservable region and the reaction object is not limited to the above. The presence estimating unit 24 may associate an unobservable region with the closest distance between the centers of gravity for each reaction object. In this case, one unobservable region may be associated with a plurality of reactive objects, but there may be a reactive object that is not associated with any of the unobservable regions.

  The presence estimating unit 24 obtains the distance between the centers of gravity of all the unobservable regions and all the reaction objects, and associates all the unobservable regions with the distance between the centers of gravity equal to or smaller than a predetermined threshold with the reactive objects. Is also good. Further, the presence estimation unit 24 may associate the closest reaction object whose distance between the centers of gravity is equal to or less than a predetermined value with the unobservable region.

  Further, the presence estimating unit 24 recognizes the gaze direction of the human (pedestrian or driver of the vehicle) by the state recognizing unit 23, and the presence estimating unit 24 determines the correspondence between the unobservable region and the reactive object based on the gaze direction. May be attached. For example, when the pedestrian's line of sight passes through the unobservable region, those pedestrians may be associated with the unobservable region. In addition, when the distance between the pedestrian's line of sight and the center of gravity of the unobservable object is equal to or less than a predetermined value, those pedestrians may be associated with the unobservable region.

  In the above embodiment, the presence estimation unit 24 uses the distance between the centers of gravity to evaluate the distance between the unobservable region and the reaction object. However, the present invention is not limited thereto. Correspondence may be performed based on the distance between the center of gravity and a portion closest to the center of gravity of the reaction object in the unobservable region. Further, the existence estimation unit 24 may measure the distance between the unobservable region and the reaction object using a point other than the center of gravity of the reaction object as a reference point.

  In the above embodiment, the existence estimating unit 24 outputs the probability that an unobservable object exists for each unobservable region as an estimation result. Whether or not to do so may be output as the estimation result. In addition, the presence estimation unit 24 performs threshold processing, and outputs an estimation result that discretely expresses the existence probability as a low, a medium, a high, and the like in a plurality of stages, instead of the binary determination that the presence or absence is present. You may. Furthermore, the presence estimating unit 24 may output a signal to that effect only when the presence probability is equal to or more than a predetermined threshold.

  The area setting unit 21 detects a blind spot area as an unobservable area from the captured image. If the detected reaction object can be associated with the unobservable area, the blind spot area can be accurately positioned from the captured image. And the shape need not be detected, and it is only necessary to specify the approximate position and shape in the image where it exists.

  In addition, a reference point may be set at the edge of the captured image for the area outside the angle of view, and may be associated based on the positional relationship between the reference point and the reference point (for example, the center of gravity) of the reaction object. Note that the area setting unit 21 may set only the area outside the angle of view without detecting the blind area as the unobservable area. Conversely, the area setting unit 21 may detect only the blind area without setting the area outside the angle of view. May be set.

  Further, in the above embodiment, the presence estimating unit 24 refers to the rule table to determine the presence probability corresponding to the recognized state, but the method of estimating the presence of the unobservable object is not limited to this. . The presence estimating unit 24 may, for example, collect data on the presence of an unobservable object corresponding to the state of the reaction object, and estimate the presence of the unobservable object with statistical probability. Further, the existence estimating unit 24 may obtain the existence probability of the unobservable object corresponding to the state of the reaction object by machine learning such as a neural network.

  Furthermore, when a plurality of reaction objects are associated with one unobservable region, the existence estimation unit 24 may output the maximum probability of the existence probability estimated based on each reaction object, The existence probability of the unobservable object may be obtained by taking a weighted sum of the existence probabilities estimated based on each reaction object.

  Further, the existence estimating unit 24 may not only calculate the existence probability of the unobservable object according to the state of the reaction object, but also estimate its content, that is, what the unobservable object is.

  Further, in the above-described embodiment, the information processing system 100 is configured as the optical sensing device 10 and the information processing device 20 are separate devices, but the information processing system 100 is configured by integrating them. Is also good.

  As described above, the present invention estimates the presence of an unobservable object in an unobservable region based on the state of a reaction object reflected in a captured image. Can be estimated, and is useful as an information processing system or the like for processing optical information.

10 optical sensing device, 20 information processing device, 21 area setting unit,
22 Reaction object detection unit, 23 State recognition unit, 24 Presence estimation unit

Claims (13)

An optical sensing unit (10) for generating optical information within a predetermined range by optical sensing;
A reaction object detection unit (22) for detecting a reaction object from the optical information;
A state recognition unit (23) for recognizing a state of the reaction object based on the optical information;
A presence estimating unit (24) for estimating the presence of an unobservable object that is not detected from the optical information based on a state of the reaction object;
An information processing system comprising: An area setting unit (21) for setting an unobservable area to be a target for estimating the presence of the unobservable object;
The information processing system according to claim 1, wherein the presence estimation unit estimates the presence of the unobservable object in the unobservable region.   The information processing system according to claim 2, wherein the area setting unit detects a blind spot area within the predetermined range from the optical information, and sets the blind spot area as the non-observable area.   The information processing system according to claim 1, wherein the area setting unit sets an area outside the predetermined range as the unobservable area.   The information processing system according to claim 1, wherein the optical sensing unit uses a camera to generate an image within an angle of view range as the optical information.   The information processing system according to claim 1, wherein the optical sensing unit uses a three-dimensional imaging system to generate a three-dimensional image within an angle of view range as the optical information. The optical sensing unit continuously generates the optical information,
The information processing system according to claim 1, wherein the state recognition unit recognizes a state of the reaction object based on the plurality of pieces of optical information that are continuously generated. The state recognition unit can recognize at least a state in which the reaction object is stopped and a state in which the reaction object is moving,
The information processing system according to claim 1, wherein the presence estimation unit estimates that the unobservable object exists at least when the reaction object is in a stopped state. The area setting unit sets the unobservable area closest to the reaction object as an unobservable area corresponding to the reaction object,
The information processing system according to claim 2, wherein the presence estimation unit estimates the presence of the unobservable object in the unobservable region corresponding to the reaction object, based on the state of the reaction object. The region setting unit sets the unobservable region within a predetermined distance from the reaction object as an unobservable region corresponding to the reaction object,
The information processing system according to claim 2, wherein the presence estimation unit estimates the presence of the unobservable object in the unobservable region corresponding to the reaction object, based on the state of the reaction object. The state recognition unit recognizes the attention direction of the reaction object,
The area setting unit sets the unobservable area in the attention direction of the reaction object as an unobservable area corresponding to the reaction object,
The information processing system according to claim 2, wherein the presence estimation unit estimates the presence of the unobservable object in the unobservable region corresponding to the reaction object, based on the state of the reaction object. An imaging step of generating optical information within a predetermined range by optical sensing,
A reaction object detection step of detecting a reaction object from the optical information,
A state recognition step of recognizing a state of the reaction object based on the optical information,
Based on the state of the reaction object, presence estimation step of estimating the presence of an unobservable object not detected from the optical information,
An information processing method comprising: A computer that obtains the optical information from an optical sensing device that generates optical information within a predetermined range by optical sensing,
A reaction object detection unit that detects a reaction object from the optical information,
A state recognition unit that recognizes the state of the reaction object based on the optical information, and a presence estimation unit that estimates the presence of an unobservable object that is not detected from the optical information based on the state of the reaction object.
Information processing program to function as

Images