JP2023044054A - Device for estimating surrounding image - Google Patents

Abstract

To estimate a near-future surrounding image of a moving object.SOLUTION: A surrounding image estimating device 1 comprises: an estimating unit 12 for estimating a surrounding image captured by a camera 3 after a predetermined time based on the surrounding image captured by the camera 3 provided in a moving object 2; a calculating unit 13 for calculating a coincident degree between the surrounding image estimated by the estimating unit 12 and a surrounding image actually captured by the camera 3 after a predetermined time; and an output unit 14 that performs output based on the coincident degree calculated by the calculation unit 13. The output unit 14 may calculate a predetermined time when the coincident degree satisfies a predetermined standard based on the coincident degree calculated by the calculating unit 13, and may output a surrounding image after the predetermined time estimated by the estimating unit 12.SELECTED DRAWING: Figure 3

Description

One aspect of the present disclosure relates to a surrounding image estimation device that estimates a surrounding image.

Japanese Unexamined Patent Application Publication No. 2002-200001 discloses an image information processing apparatus that uses a surrounding image obtained by capturing the surroundings of a moving object.

JP 2021-117833 A

Since the image information processing apparatus uses an actually captured image of the surroundings, for example, it is not possible to perform processing based on an estimated image of the surroundings in the near future. Therefore, it is desired to estimate a near-future surrounding image of a moving object.

A surrounding image estimating device according to one aspect of the present disclosure includes an estimating unit that estimates a surrounding image captured by an imaging means after a predetermined time based on a surrounding image captured by an imaging means provided for a moving object; A calculating unit that calculates the degree of matching between the surrounding image estimated by the estimating unit and the surrounding image actually captured by the imaging means after a predetermined time, and an output unit that outputs based on the degree of matching calculated by the calculating unit. And prepare.

In this aspect, an image of the surroundings captured after a predetermined period of time by an imaging means provided on a moving object is estimated. That is, it is possible to estimate a near-future surrounding image of a moving object.

According to one aspect of the present disclosure, a near-future ambient image of a moving object can be estimated.

It is a figure which shows an example of the system configuration|structure of the surrounding image estimation system containing the surrounding image estimation apparatus which concerns on embodiment. 1 is a conceptual diagram of Scenario 1; FIG. It is a figure which shows an example of the functional structure of the surrounding image estimation apparatus which concerns on embodiment. It is a figure which shows an example of a surrounding image. It is a figure which shows an example of the surrounding image of the past, the present, and the future. FIG. 10 is a diagram showing a comparison between a future ambient image and an actual ambient image; 4 is a flowchart showing an example of processing executed by the surrounding image estimation device according to the embodiment; FIG. 11 is a conceptual diagram of Scenario 2; It is a figure which shows an example of the surrounding image of T-junction vicinity. It is a figure which shows an example of the surrounding image at the time of turning right at the T-junction. It is a conceptual diagram at the time of turning right at the T-junction. It is a figure which shows an example of the surrounding image estimated at the time of turning right at the T-junction. FIG. 11 is a conceptual diagram of Scenario 3; FIG. 10 is a diagram showing an example of an estimated surrounding image (including an object) when turning right at a T-junction; 9 is a flowchart showing another example of processing executed by the surrounding image estimation device according to the embodiment; It is a figure which shows an example of the hardware constitutions of the computer used with the surrounding image estimation apparatus which concerns on embodiment.

Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In the description of the drawings, the same elements are denoted by the same reference numerals, and overlapping descriptions are omitted. In addition, the embodiments of the present disclosure in the following description are specific examples of the present invention, and the present invention is not limited to these embodiments unless specifically stated to limit the present invention.

FIG. 1 is a diagram showing an example of a system configuration of a surrounding image estimation system 4 including a surrounding image estimation device 1 according to an embodiment. As shown in FIG. 1, the surrounding image estimation system 4 includes a surrounding image estimation device 1 (surrounding image estimation device), a moving body 2 (moving object), and a camera 3 (imaging means) provided on the moving body 2. Consists of The surrounding image estimation device 1 and the moving object 2 are connected to communicate with each other via a network or the like, and can transmit and receive arbitrary information to and from each other at arbitrary timing.

The surrounding image estimation device 1 is a computer device that estimates a near-future surrounding image of the mobile object 2 . The surrounding image estimation device 1 may be located at a location (remote) away from the mobile object 2 or may be provided in the mobile object 2 . Alternatively, part of the surrounding image estimation device 1 may be located at a location (remote) away from the moving body 2 , and the rest of the surrounding image estimation device 1 may be provided on the moving body 2 . In that case, a part and the remaining part of the surrounding image estimation device 1 are connected to communicate with each other via a wireless network or the like, and the part and the remaining part function together as the surrounding image estimation device 1 . Details of the surrounding image estimation device 1 will be described later.

The moving body 2 is a moving (movable) object. Objects may be inanimate, such as man-made objects, or animate, such as people. Examples of the mobile object 2 include wheelchairs, bicycles, vehicles such as cars and trucks, aircraft such as airplanes, helicopters, rockets and drones, ships (including submarines), people, and animals. The moving body 2 may move using its own movement ability, or may move using another movement ability without having any movement ability itself. The moving body 2 may include a GPS (Global Positioning System) capable of acquiring position information, an acceleration sensor, an angular velocity sensor, a magnetic sensor, and the like. The moving body 2 uses various sensors to periodically (for example, every second) generate movement information including its own current position, acceleration, speed, movement (advancement) direction, and current time stamp, and via a network or the like It may be transmitted to the surrounding image estimation device 1 .

The camera 3 is imaging means. The camera 3 captures an image of the surroundings (periphery, neighborhood, near, surroundings, vicinity, side, perimeter) and outputs a surrounding image, which is an image of the surroundings. The camera 3 may output a surrounding video (surrounding video) that is a temporal series of surrounding images. Although it is assumed that the surrounding image is used in this embodiment, the surrounding image may be used instead. That is, in the present embodiment, the term "image" may be appropriately replaced with "video" or "moving image."

The camera 3 is composed of a camera 3a (first camera) capable of measuring the distance (depth information) to an object in the captured image, and a camera 3b (second camera) capable of capturing the surroundings. may The camera 3a is, for example, a stereo camera or a LiDAR (Light Detection and Ranging or Laser Imaging Detection and Ranging) camera. The camera 3b is, for example, a 360-degree camera or a camera capable of capturing images in multiple directions. The camera 3b may be a general camera instead of a special camera. The camera 3 may be composed of a plurality of (two or more) cameras. In this embodiment, a plurality of surrounding images captured by the camera 3 configured by a plurality of cameras are collectively referred to as a surrounding image captured by the camera 3 as appropriate. When outputting the surrounding image, the camera 3 may output surrounding information, which is arbitrary information related to the surrounding image. The surrounding information may include a time stamp of the timing at which the surrounding image was captured, and information about objects in the surrounding image and distances from the objects.

The surrounding image and surrounding information output by the camera 3 are transmitted to the surrounding image estimation device 1 via the mobile object 2 and a network, etc. 2 may be displayed to the controller controlling the movement of . In addition, the controller transmits control information regarding movement control (forward, right turn, stop, etc.) of the moving body 2 from the surrounding image estimation device 1 to the moving body 2, and the moving body 2 moves according to the control information. you can go That is, the controller may remotely control (remote control, remote control) the movement of the moving body 2 based on the surrounding image and surrounding information transmitted from the moving body 2 .

The camera 3 includes the above-described various sensors included in the mobile body 2, and the camera 3 (not the mobile body 2) periodically (for example, every second) generates the above-described movement information, and estimates the surrounding image via a network or the like. It may be sent to device 1 .

In FIG. 1, a mobile object 2 is a vehicle that runs (moves) on a road. The surrounding image S is an image in the traveling direction taken by the camera 3 provided on the mobile body 2 and directed in the traveling direction (forward direction) of the mobile body 2 . In the surrounding image S, mainly the road on which the moving body 2 is traveling, the sidewalk beside the road, a person walking on the sidewalk, and the scenery such as the mountains spreading farther in the traveling direction are shown.

[Scenario 1]
Hereinafter, an embodiment in Scenario 1 will be described. FIG. 2 is a conceptual diagram of Scenario 1. FIG. In Scenario 1, as shown in FIG. 1, a moving body 2 such as a car runs on a road with good visibility in the direction of travel, and a (moving) object B1 such as a person or an obstacle exists in the direction of travel. Imagine a scene. In Scenario 1, a surrounding image in the near future is estimated from the surrounding image (and surrounding information) that can be captured by the camera 3 (only) provided on the moving object 2 .

FIG. 3 is a diagram showing an example of the functional configuration of the surrounding image estimation device 1 according to the embodiment. As shown in FIG. 3, the surrounding image estimation device 1 includes a storage unit 10 (storage unit), an acquisition unit 11 (acquisition unit), an estimation unit 12 (estimation unit), a calculation unit 13 (calculation unit), and an output 14 (output unit). Note that the functional configuration shown in FIG. 3 and specific contents of the functional configuration described later are not limited to Scenario 1, and are common to Scenarios 2 and 3 described later.

Each functional block of the surrounding image estimation device 1 is assumed to function within the surrounding image estimation device 1, but it is not limited to this. For example, some of the functional blocks of the surrounding image estimation device 1 are computer devices different from the surrounding image estimation device 1, and are connected to the surrounding image estimation device 1 via a network. It may function while transmitting and receiving information as appropriate. Further, some functional blocks of the surrounding image estimation device 1 may be omitted, a plurality of functional blocks may be integrated into one functional block, or one functional block may be decomposed into a plurality of functional blocks. good too.

Each function of the surrounding image estimation device 1 shown in FIG. 3 will be described below.

The storage unit 10 stores arbitrary information used in calculations in the surrounding image estimation device 1, calculation results in the surrounding image estimation device 1, and the like. The information stored by the storage unit 10 may be referred to by each function of the surrounding image estimation device 1 as appropriate.

Acquisition unit 11 acquires arbitrary information that is used for calculation in surrounding image estimation device 1 . More specifically, the acquisition unit 11 acquires information from an external device or the like via a network or the like. The acquisition unit 11 causes the storage unit 10 to store the acquired information, and outputs the information to other functional blocks of the surrounding image estimation device 1 .

The acquisition unit 11 acquires surrounding images, surrounding information, and movement information from the mobile object 2 via a network or the like. The ambient image, ambient information, and movement information acquired by the acquisition unit 11 are stored by the storage unit 10 as history information in chronological order based on the time stamps included in the ambient information and the movement information.

The estimating unit 12 estimates a surrounding image captured by the camera 3 after a predetermined time based on the surrounding image captured by the camera 3 provided on the moving body 2 (or the history information stored by the storage unit 10). . For example, the estimating unit 12, based on a plurality of surrounding images captured by the camera 3 up to the present time, determines the time after a predetermined time "2 seconds" from the reference time (for example, the present time) "13:00:00". A surrounding image captured by the camera 3 at "13:00:02" is estimated. Here, since the mobile object 2 is movable, there is a possibility that the mobile object 2 is in a different position at the time "13:00:02" from the position at the time "13:00:00". be. The estimation unit 12 may periodically (for example, every second) and continuously estimate the surrounding image. A specific estimation process by the estimation unit 12 will be described later.

The estimation unit 12 may estimate surrounding images corresponding to a plurality of different predetermined times. For example, the estimating unit 12 captures the ambient image captured by the camera 3 at the time “13:00:02” after the predetermined time “2 seconds” from the reference time “13:00:00” and the reference time “13:00:00”. A surrounding image captured by the camera 3 at a time "13:00:04" after a predetermined time "4 seconds" from the reference time "13:00:00" and a predetermined time "6 The surrounding image captured by the camera 3 at the time "13:00:06" after "seconds" is estimated.

The estimation unit 12 may estimate based on the distance to the object in the image captured by the camera 3a and the surrounding image captured by the camera 3b. A specific estimation process will be described later.

The estimation unit 12 may cause the storage unit 10 to store the estimated surrounding image, or may output it to another functional block of the surrounding image estimation device 1 .

FIG. 4 is a diagram showing an example of a surrounding image. The situation in FIG. 4 and the like are the same as in FIG. 1, so differences will be mainly described. The surrounding image Sa is a surrounding image stereoscopically captured by the camera 3a (so that the distance to the object can be measured). The surrounding image Sb is a surrounding image captured omnidirectionally by the camera 3b. In the example shown in FIG. 4, the surrounding image Sb is a surrounding image obtained by imaging an area wider than the surrounding image Sa, and the area of the surrounding image Sb includes (includes) the area of the surrounding image Sa. The area indicated by the surrounding image Sa may be an attention area that the controller pays attention to. In that case, of the area shown by the surrounding image Sb, the area excluding the area shown by the surrounding image Sa is outside the attention area that the controller does not pay attention to. In the surrounding image Sa and the surrounding image Sb, a person B2 (object) walking on the sidewalk on the left side from the back side to the front side (in the direction opposite to the traveling direction of the moving body 2) is shown. . In the surrounding image Sb, a person B3 (object) walking on the sidewalk on the right side from the front side toward the back side (in the traveling direction of the moving body 2) is shown.

FIG. 5 is a diagram showing an example of past/present/future surrounding images.

The past surrounding images shown in the left diagram in FIG. 5 are the surrounding images Sa and Sb captured by the cameras 3a and 3b in the past (for example, two seconds before the current time). Of the past surrounding images, the surrounding image Sa, which is the region of interest, is called a past frame. The distance "20 m" to the person B2 shown in the past frame indicates the distance to the person B2 measured by the camera 3a. In the present embodiment, the information about the distance to the object measured by the camera 3a is displayed in the surrounding image Sa in FIG. 5 for the sake of convenience. and

The current surrounding images shown in the middle diagram in FIG. 5 are the surrounding image Sa and the surrounding image Sb currently (currently) captured by the cameras 3a and 3b, respectively. Among the current surrounding images, the surrounding image Sa, which is the attention area, is called a current frame. The distance to the person B2 shown in the current frame is "15 m", and is closer to the moving object 2 than in the past frame. On the other hand, the person B3 shown in the current surrounding image is farther away from the moving object 2 than in the past surrounding image. The current frame moves upward in the region of the surrounding image Sb compared to the past frames.

The future surrounding images shown in the diagram on the right side of FIG. is. Of the future surrounding images, the surrounding image Sa, which is the region of interest, is called a future frame. The distance to the person B2 shown in the future frame is "10 m", and the person is closer to the moving object 2 than in the current frame. On the other hand, the person B3 shown in the future ambient image (and future frame) is further away from the vehicle 2 than in the current ambient image. The future frame moves upward within the region of the surrounding image Sb compared to the current frame.

Specific estimation processing of the future frame by the estimation unit 12 will be described with reference to the future surrounding image of FIG. 5 . First, for the surrounding images Sa excluding the person B2 and the person B3, the past surrounding images Sa and their time stamps, the current surrounding images Sa and Sb and their time stamps, and t seconds is estimated (by the estimation unit 12). For example, the difference between the time stamps of the surrounding image Sa of the past surrounding image and the surrounding image Sa of the current surrounding image, and the position of the surrounding image Sa of the past surrounding image and the position of the surrounding image Sa of the current surrounding image. From the difference (upward movement amount) of , the upward movement amount in t seconds is calculated (assuming that it moves linearly upward), and the surrounding image Sa of the current surrounding image is converted to the calculated upward movement amount A surrounding image moved by the movement amount is estimated as a future frame (excluding the person B2 and the person B3). The image of the region R1 (upper side of the future frame) shown in FIG. 5, which requires new estimation when moving upward, may be combined based on the surrounding image Sb. For example, the image of the area R1 may be generated using the image of the corresponding portion in the surrounding image Sb of the current surrounding image. In addition, in the above calculation, by using information about the position, acceleration, speed, and movement direction of the moving body 2 included in the movement information of the history information stored by the storage unit 10, accurate movement of the moving body 2 can be performed. The estimation accuracy of the surrounding image that is estimated and finally estimated may be increased. The surrounding image Sb of the future surrounding image may also be estimated in the same manner as the future frame.

Next, the person B2 in the future frame is estimated (by the estimation unit 12) based on the surrounding information associated with the past frame and the current frame. For example, based on the distance from the person B2 included in the surrounding information associated with the past frame and the current frame, the movement vector of the person B2 in the x, y, and z axis directions is calculated from the time difference between the past frame and the current frame, Based on the calculated movement vector and t seconds, the person B2 in the future frame is estimated in the same manner as the estimation of the surrounding image Sa described above.

Finally, the person B3 in the future frame is estimated (by the estimation unit 12) based on the surrounding image Sb acquired by the camera 3b in the past surrounding image and the current surrounding image. For example, based on the surrounding image Sb acquired by the camera 3b in the past surrounding image and the current surrounding image, the movement vector of the person B3 in the x- and y-axis directions is obtained from the time difference between the past surrounding image and the current surrounding image. Based on the calculated movement vector and t seconds, the person B3 in the future frame is estimated in the same manner as the estimation of the surrounding image Sa described above. That is, it is estimated that the person B3 will be reflected in the attention area from the movement vector calculated based on the surrounding image Sb, and superimposed on the future frame.

As described above, the estimating unit 12 estimates the future frame (surrounding image) captured by the camera 3 after t seconds, based on the surrounding image captured by the camera 3 provided on the moving object 2 .

The estimation unit 12 may output the estimation accuracy for each object when estimating the surrounding image to another functional block in association with the surrounding image. For example, the estimation unit 12 may output low estimation accuracy for fast moving objects and non-straight moving objects, and output high estimation accuracy for stationary objects and slow straight moving objects.

The calculation unit 13 calculates the degree of matching between the surrounding image estimated by the estimation unit 12 and the surrounding image actually captured by the camera 3 after a predetermined time. For example, when the estimation unit 12 estimates the surrounding image captured by the camera 3 at the time "13:00:02" after the predetermined time "2 seconds" from the reference time "13:00:00" , the calculation unit 13 actually captures the estimated surrounding image with the camera 3 at time ``13:00:02'' after a predetermined time ``2 seconds'' from the reference time ``13:00:00''. The degree of matching with the imaged surrounding image is calculated.

The surrounding image estimated by the estimating unit 12, which is one of the objects for which the calculating unit 13 calculates the degree of matching, may be output by the estimating unit 12 or stored by the storage unit 10. There may be. The other surrounding image for which the calculation unit 13 calculates the degree of matching, which is actually captured by the camera 3 after a predetermined period of time, is received by the surrounding image estimation device 1 from the moving body 2 via a network or the like, and stored. stored by the unit 10;

Match is the degree to which two images match. The degree of matching may be the degree of similarity between images, which is an existing technology. The degree of matching is represented by, for example, a real number from "0" to "1". good too.

The calculation unit 13 may calculate the matching degree based on the position of the object in the surrounding image. For example, assume that an object at position A (a, b) is at position B (c, d) after a predetermined period of time. It is also assumed that the estimation unit 12 estimates that the object will be at the position X(x, y) after a predetermined time. In that case, for example, the calculation unit 13 may calculate the matching degree by calculating the degree of similarity between the surrounding image of the position B and the surrounding image of the position X.

The calculation unit 13 may calculate the matching degree for each of the plurality of surrounding images estimated by the estimation unit 12 . For example, the estimating unit 12 generates a surrounding image at a time “13:00:02” after a predetermined time “2 seconds” from the reference time “13:00:00” and a reference time “13:00:00”. Surrounding image at time ``13:00:04'' after a predetermined time ``4 seconds'' from the reference time ``13:00:00'' , and the surrounding image of . In this case, the calculation unit 13 calculates the degree of matching between the estimated surrounding image at time “13:00:02” and the surrounding image actually captured at that time, ” and the actually captured image at that time are calculated, and the estimated ambient image at time “13:00:06” and the actually captured image at that time are calculated. Calculate the degree of matching with the surrounding image.

The calculation unit 13 may cause the storage unit 10 to store the calculated matching degree, or may output it to another functional block of the surrounding image estimation device 1 .

FIG. 6 is a diagram showing a comparison between a future ambient image and an actual ambient image. The future surrounding image shown in the upper diagram in FIG. 6 is the same as the future surrounding image in FIG. That is, the future frame in the future surrounding image is the surrounding image estimated by the estimation unit 12 . The actual surrounding images shown in the lower diagram in FIG. 6 are the surrounding image Sa and the surrounding image actually captured by the cameras 3a and 3b, respectively, after t seconds (predetermined time) from the time when the current frame in FIG. 5 was captured. Sb. Of the actual surrounding images, the surrounding image Sa, which is the attention area, is called an actual frame. The calculator 13 calculates (compares) the degree of matching between the future frame and the actual frame shown in FIG. Note that when the future frame and the actual frame are compared, parameter tuning of the estimation performed by the estimation unit 12 may be performed based on the comparison result to improve the accuracy of the estimation. By performing parameter tuning, for example, it is possible to reflect the characteristics of the speed and traveling direction of the moving body 2 due to the habits of the controller.

The output unit 14 outputs based on the degree of matching calculated by the calculation unit 13 . For example, the output unit 14 may display the degree of matching calculated by the calculation unit 13 to the controller via the output device 1006 (described later) or the like. Further, for example, the output unit 14 outputs the degree of matching calculated by the calculating unit 13 and the surrounding image estimated by the estimating unit 12 on which the degree of matching was calculated, via the output device 1006 (described later). May be displayed to the controller. The output from the output unit 14 is not limited to display, and may be, for example, output to another functional block of the surrounding image estimation device 1 or transmission to another device via a network or the like.

Based on the degree of matching calculated by the calculating unit 13, the output unit 14 calculates a predetermined time when the degree of matching satisfies a predetermined standard, and outputs the surrounding image after the predetermined time estimated by the estimating unit 12. good too. For example, when the degree of matching is represented by a real number between “0” and “1” as described above, the output unit 14 outputs a predetermined A time (for example, “2 seconds”) is calculated, and the surrounding image after the predetermined time (from the reference time) estimated by the estimation unit 12 is output.

The output unit 14 selects a degree of matching that satisfies a predetermined criterion from among the plurality of degrees of matching calculated by the calculating unit 13, and outputs a predetermined time based on the selected degree of matching. It may be calculated as a predetermined time that satisfies a predetermined criterion. For example, the matching degree for the surrounding image at the time ``13:00:02'' after the predetermined time ``2 seconds'' from the reference time ``13:00:00'' is ``0.9''. The matching degree for the surrounding image at the time "13:00:04" after the predetermined time "4 seconds" from the reference time "13:00:00" is "0.7". When the matching degree for the surrounding image at the time "13:00:06" after the time "6 seconds" is "0.5" and the predetermined criterion is "0.8" or higher, the output unit 14: A matching degree of “0.9” that satisfies a predetermined criterion of “0.8” or higher is selected, and a predetermined time period of “2 seconds” that is the basis of the selected matching degree is set to a predetermined Calculate as hours. Then, the output unit 14 outputs the surrounding image after the predetermined time “2 seconds” estimated by the estimation unit 12 (from the reference time).

The output unit 14 may perform output further based on the transmission delay time until the surrounding image captured by the camera 3 is displayed to the controller who controls the movement of the moving body 2 . The transmission delay time is, for example, the delay time associated with image transmission from the moving object 2 to a remote controller (remote location). The transmission delay time is specified (predicted) by the difference between the time stamp included in the surrounding information associated with the surrounding image and the time stamp obtained by the time function provided in advance in the surrounding image estimation device 1 (at the timing of displaying to the controller). ) may be It is assumed that the surrounding image estimation device 1 and the camera 3 are synchronized in time. For example, based on the degree of matching calculated by the calculating unit 13, the output unit 14 determines that the degree of matching satisfies a predetermined standard and the difference from the transmission delay time is a predetermined standard (for example, "0.5 seconds"). The following predetermined time may be calculated, and the surrounding image after the predetermined time estimated by the estimation unit 12 may be output.

Based on the degree of matching calculated by the calculating unit 13, the output unit 14 calculates a predetermined time period when the degree of matching satisfies a predetermined criterion, and when the transmission delay time is within the predetermined time period, the transmission delay time is estimated by the estimating unit 12. Alternatively, the surrounding image after the transmission delay time may be output. For example, in the same way as in the above example, the predetermined time "2 seconds" satisfying the predetermined criterion "0.8" or higher is calculated, and if the transmission delay time is "1.5 seconds", the transmission delay time " 1.5 seconds” is within the predetermined time of “2 seconds”, the output unit 14 outputs the surrounding image after the transmission delay time “1.5 seconds” estimated by the estimation unit 12 .

The output unit 14 may output further based on the estimation accuracy of the estimation by the estimation unit 12 . For example, the output unit 14 may output (display) the value of the estimation accuracy for each object in the surrounding image output by the estimation unit 12 on the object, or may color-code the object according to the estimation accuracy. may be used to output the estimated accuracy to the controller.

Next, an example of processing executed by the surrounding image estimation device 1 will be described with reference to FIG. FIG. 7 is a flowchart showing an example of processing executed by the surrounding image estimation device 1 according to the embodiment.

First, the estimation unit 12 estimates the surrounding image S captured by the camera 3 after a predetermined time t seconds (step S1, estimation step ). Next, the calculation unit 13 calculates the degree of matching between the surrounding image S estimated in S1 and the surrounding image S actually captured by the camera 3 after a predetermined time t seconds (step S2, calculation step). Next, the output unit 14 outputs based on the degree of matching calculated in S2 (step S3, output step).

The above is the description of the embodiment in Scenario 1.

[Scenario 2]
An embodiment in Scenario 2 will be described below. FIG. 8 is a conceptual diagram of Scenario 2. FIG. Scenario 2 assumes a scene in which a moving object 2 such as a wheelchair is traveling along a corridor in a building and turning right at a T-junction (turning corner). FIG. 9 is a diagram showing an example of a surrounding image around a T-junction. Similar to Scenario 1, Mobile 2 is remotely controllable by the controller. Before the moving body 2 makes a right turn at the T-junction, the camera 3 of the moving body 2 cannot capture the right turn ahead of the T-junction. . FIG. 10 is a diagram showing an example of a surrounding image when turning right at a T-junction. The surrounding image shown in FIG. 10 was actually captured by the camera 3 of the moving body 2 after the moving body 2 turned right at the T-shaped road. I can't even guess.

The estimating unit 12 may further perform the estimation based on map information around the predicted position of the moving body 2 after a predetermined time. FIG. 11 is a conceptual diagram when turning right at a T-junction. The estimating unit 12 estimates a surrounding image of an imaging range that can be captured from a point after a predetermined time t seconds based on map information (map data) around the point. The map data can be obtained from the cloud via a network or the like, and the map data may include image data (aerial photographs, textures, landscape photographs, etc.) associated with the map data. The estimating unit 12 may predict the point after a predetermined time of t seconds from the speed of the moving body 2 and the current position. FIG. 12 is a diagram showing an example of an estimated surrounding image when turning right at a T-junction. A region R2 in the surrounding image shown in FIG. 12 is a region estimated by the estimation unit 12 based on the map information. The above is the description of the embodiment in Scenario 2.

[Scenario 3]
Hereinafter, an embodiment in Scenario 3 will be described. FIG. 13 is a conceptual diagram of Scenario 3. FIG. In Scenario 3, in addition to the scene of Scenario 2, a scene is assumed in which a surveillance camera C, a person B4, and another moving object 2a different from the moving object 2 are present in the corridor at the back of the T-junction. In Scenario 3, the person B4 (object, obstacle, dynamic object) who turns right at the T-junction is also estimated.

The estimating unit 12 may further perform the estimation based on a surrounding image taken by another camera 3 (surveillance camera C or a camera included in the moving object 2a) existing near the predicted position of the moving object 2 after a predetermined time. . The estimating unit 12 further estimates based on map information near the predicted position of the moving object 2 after a predetermined time and objects included in surrounding images taken by another camera 3 and present near the predicted position. good too.

That is, when the moving body 2a or the monitoring camera C exists at the right turn ahead of the T-junction, the image captured by the moving body 2a or the monitoring camera C is used to estimate the person B4. The moving body 2a has the same functional configuration as the moving body 2, and transmits the imaged surrounding image to the surrounding image estimation device 1 together with the positional information of the moving body 2a. The moving object 2a or the monitoring camera C estimates the position of the person B4 after a predetermined time of t seconds by adopting a method equivalent to scenario 1. FIG. The estimating unit 12 of the moving object 2 estimates the surrounding image after a predetermined time t seconds by using the same method as in Scenario 2, and then estimates the person B4 based on the position coordinates (the person B4 can be reproduced). . FIG. 14 is a diagram showing an example of an estimated surrounding image (including an object) when turning right at a T-junction. A person B4 estimated by the estimation unit 12 is included in a region R2 of the surrounding image shown in FIG. The above is the description of the embodiment in Scenario 3.

Next, an example of processing of scenarios 1 to 3 executed by the surrounding image estimation device 1 will be described with reference to FIG. FIG. 15 is a flowchart showing another example of processing executed by the surrounding image estimation device 1 according to the embodiment.

First, the surrounding image estimation device 1 predicts the transmission delay time d (step S10). Next, the surrounding image estimation device 1 predicts the position (self-position) of the moving body 2 after d seconds (step S11). Next, the surrounding image estimating device 1 can estimate the surrounding image after d seconds only from the surrounding image captured by the moving body 2 (the surrounding image captured from the self position after d seconds has no corners, and the moving body 2 It is determined whether or not it is possible to estimate only from the surrounding image captured in (step S12). If it is determined in S12 that estimation is possible (S12: YES), the surrounding image estimation device 1 estimates the surrounding image using the pattern of Scenario 1 (Step S13). Next, the surrounding image estimation device 1 performs parameter tuning by comparing with the surrounding image actually captured after d seconds (step S14), and the process ends. If it is determined in S12 that estimation is not possible (S12: NO), the surrounding image estimation device 1 estimates the surrounding image in the pattern of Scenario 2 (Step S15). Following S15, it is determined whether or not a camera present near the predicted position of the moving body 2 after d seconds has detected an object (step S16). If it is determined in S16 that no detection has been made (step S16: NO), the process continues to S14. If it is determined that the object is detected in S16 (step S16: YES), the surrounding image estimation device 1 estimates the object according to the pattern of Scenario 3 (step S17), and continues to S14.

Next, effects of the surrounding image estimation device 1 according to the embodiment will be described.

According to the surrounding image estimating device 1, the estimating unit 12 estimates the surrounding image captured by the camera 3 after a predetermined time based on the surrounding image captured by the camera 3 provided on the moving body 2, and the calculating unit 13 calculates the matching degree between the surrounding image estimated by the estimating unit 12 and the surrounding image actually captured by the camera 3 after a predetermined time, and the output unit 14 outputs the matching degree calculated by the calculating unit 13. based output. With this configuration, the surrounding image captured by the camera 3 provided on the moving body 2 after a predetermined time is estimated. That is, a near-future surrounding image of the moving object 2 can be estimated. In addition, since the output is performed based on the degree of matching between the estimated surrounding image and the actually captured surrounding image, for example, the estimation accuracy of the estimated surrounding image can be easily and effectively utilized.

Further, according to the surrounding image estimation device 1, the output unit 14 calculates a predetermined time period in which the matching degree satisfies a predetermined standard based on the matching degree calculated by the calculating unit 13, and the estimation unit 12 estimates A surrounding image after the predetermined time may be output. With this configuration, it is possible to output an estimated surrounding image with a certain degree of estimation accuracy.

Further, according to the surrounding image estimation device 1, the estimation unit 12 estimates the surrounding images corresponding to each of a plurality of different predetermined times, and the calculation unit 13 estimates each of the plurality of surrounding images estimated by the estimation unit 12 The output unit 14 selects a degree of matching that satisfies a predetermined criterion from among the plurality of degrees of matching calculated by the calculating unit 13, and selects a predetermined time on which the selected degree of matching is based, It may be calculated as a predetermined time when the degree of matching satisfies a predetermined criterion. With this configuration, it is possible to more easily and reliably output an estimated surrounding image with a certain degree of estimation accuracy.

According to the surrounding image estimation device 1, the camera 3 is composed of the camera 3a capable of measuring the distance to an object in the captured image and the camera 3b capable of capturing the surroundings. , the distance to the object in the image captured by the camera 3a and the surrounding image captured by the camera 3b. With this configuration, it is possible to perform estimation with consideration given to the distance to the object, and it is possible to perform estimation based on images captured by different cameras, so that it is possible to perform estimation with higher accuracy. In addition, by acquiring information in the z-axis (depth) direction with the camera 3a, it is possible to estimate movement with a large vector in the z-axis direction.

Further, according to the surrounding image estimation device 1, the output unit 14 outputs an output further based on the transmission delay time until the surrounding image captured by the camera 3 is displayed to the controller controlling the movement of the moving body 2. you can go With this configuration, it is possible to perform output in consideration of transmission delay time. For example, by outputting a near-future surrounding image for the transmission delay time to the controller, it is possible for the controller to output a real-time surrounding image of the mobile object 2 .

Further, according to the surrounding image estimation device 1, the output unit 14 calculates a predetermined time period in which the degree of matching satisfies a predetermined standard based on the degree of matching calculated by the calculating unit 13, and the transmission delay time is In the case of less than, the surrounding image after the transmission delay time estimated by the estimation unit 12 may be output. With this configuration, for example, it is possible to output a real-time surrounding image of the moving object 2 with a certain degree of estimation accuracy.

Further, according to the surrounding image estimation device 1 , the output unit 14 may output based on the estimation accuracy of the estimation by the estimation unit 12 . With this configuration, estimation accuracy can be easily and effectively utilized.

Further, according to the surrounding image estimation device 1, the estimation unit 12 may further perform estimation based on map information around the predicted position of the moving object 2 after a predetermined time. With this configuration, parts that cannot be estimated by the moving body 2 itself can be compensated for by the map information.

Further, according to the surrounding image estimation device 1, the estimation unit 12 may further perform estimation based on a surrounding image captured by another camera 3 that will be present in the vicinity of the predicted position of the moving body 2 after a predetermined time. With this configuration, the part that cannot be estimated by the moving body 2 itself can be compensated for by the surrounding image captured by another camera 3 .

Further, according to the surrounding image estimation device 1, the estimation unit 12 combines the map information near the predicted position of the moving body 2 after a predetermined time with the surrounding image captured by another camera 3 existing near the predicted position. It may be estimated further based on the included objects. With this configuration, parts that cannot be estimated by the moving body 2 itself can be compensated for by objects included in the surrounding image captured by the map information and another camera 3 .

Here, conventional problems will be described. In order to operate a moving object from a remote location, it is necessary for a remote operator to grasp the local situation in real time, and currently the mainstream method is to grasp the local situation by video. Due to processing such as video transmission, encoding, and decoding, a delay of several tens to hundreds of milliseconds is generated in order to deliver local images to the remote operator. end up

The surrounding image estimation device 1 of this embodiment considers the delay that occurs when transmitting an image (video) to the controller (remote operator), and takes several tens to several hundreds of milliseconds according to the transmission delay time. Realize a remote control system that does not make you feel image transmission delays by virtually generating future local images. The surrounding image estimating device 1 can generate an image of the controller's attention area portion in the future by the transmission delay time related to image transmission from the direction and speed of the moving body 2 and the vectors of the surrounding objects. When creating a future image, the surrounding image estimation device 1 generates a future image corresponding to the image transmission delay at that time. However, if the match rate is less than the predetermined match rate, a slightly delayed future image is generated so as to achieve the match rate. For example, assuming that there is a delay of 5 seconds, if a 5-second future image is generated, the matching rate will decrease. According to the surrounding image estimating device 1, it is possible to prevent a response delay in remote control and to realize remote control without discomfort. The method using the surrounding image estimation device 1 is a near-future video generation method for real-time remote control.

It should be noted that the block diagrams used in the description of the above embodiments show blocks in units of functions. These functional blocks (components) are realized by any combination of at least one of hardware and software. Also, the method of implementing each functional block is not particularly limited. That is, each functional block may be implemented using one device that is physically or logically coupled, or directly or indirectly using two or more devices that are physically or logically separated (e.g. , wired, wireless, etc.) and may be implemented using these multiple devices. A functional block may be implemented by combining software in the one device or the plurality of devices.

Functions include judging, determining, determining, calculating, calculating, processing, deriving, investigating, searching, checking, receiving, transmitting, outputting, accessing, resolving, selecting, choosing, establishing, comparing, assuming, expecting, assuming, Broadcasting, notifying, communicating, forwarding, configuring, reconfiguring, allocating, mapping, assigning, etc. can't For example, a functional block (component) that makes transmission work is called a transmitting unit or a transmitter. In either case, as described above, the implementation method is not particularly limited.

For example, the surrounding image estimation device 1 according to the embodiment of the present disclosure may function as a computer that performs the processing of the surrounding image estimation method of the present disclosure. FIG. 16 is a diagram illustrating an example of a hardware configuration of the surrounding image estimation device 1 according to an embodiment of the present disclosure. The ambient image estimation device 1 described above may be physically configured as a computer device including a processor 1001, a memory 1002, a storage 1003, a communication device 1004, an input device 1005, an output device 1006, a bus 1007, and the like.

Note that in the following description, the term "apparatus" can be read as a circuit, device, unit, or the like. The hardware configuration of the surrounding image estimation device 1 may be configured to include one or more of each device shown in the figure, or may be configured without some devices.

Each function in the surrounding image estimation device 1 is performed by the processor 1001 performing calculations by loading predetermined software (programs) onto hardware such as the processor 1001 and the memory 1002, controlling communication by the communication device 1004, It is realized by controlling at least one of data reading and writing in the memory 1002 and the storage 1003 .

The processor 1001, for example, operates an operating system to control the entire computer. The processor 1001 may be configured by a central processing unit (CPU) including an interface with peripheral devices, a control device, an arithmetic device, registers, and the like. For example, the acquisition unit 11 , the estimation unit 12 , the calculation unit 13 , the output unit 14 and the like described above may be implemented by the processor 1001 .

The processor 1001 also reads programs (program codes), software modules, data, etc. from at least one of the storage 1003 and the communication device 1004 to the memory 1002, and executes various processes according to these. As the program, a program that causes a computer to execute at least part of the operations described in the above embodiments is used. For example, the acquiring unit 11, the estimating unit 12, the calculating unit 13, and the output unit 14 may be stored in the memory 1002 and implemented by a control program that operates on the processor 1001, and other functional blocks may be implemented in the same way. good too. Although it has been explained that the above-described various processes are executed by one processor 1001, they may be executed simultaneously or sequentially by two or more processors 1001. FIG. Processor 1001 may be implemented by one or more chips. Note that the program may be transmitted from a network via an electric communication line.

The memory 1002 is a computer-readable recording medium, and is composed of at least one of, for example, ROM (Read Only Memory), EPROM (Erasable Programmable ROM), EEPROM (Electrically Erasable Programmable ROM), and RAM (Random Access Memory). may be The memory 1002 may also be called a register, cache, main memory (main storage device), or the like. The memory 1002 can store executable programs (program code), software modules, etc. for implementing a wireless communication method according to an embodiment of the present disclosure.

The storage 1003 is a computer-readable recording medium, for example, an optical disk such as a CD-ROM (Compact Disc ROM), a hard disk drive, a flexible disk, a magneto-optical disk (for example, a compact disk, a digital versatile disk, a Blu-ray disk), smart card, flash memory (eg, card, stick, key drive), floppy disk, magnetic strip, and/or the like. Storage 1003 may also be called an auxiliary storage device. The storage medium described above may be, for example, a database, server, or other suitable medium including at least one of memory 1002 and storage 1003 .

The communication device 1004 is hardware (transmitting/receiving device) for communicating between computers via at least one of a wired network and a wireless network, and is also called a network device, a network controller, a network card, a communication module, or the like. The communication device 1004 includes a high-frequency switch, a duplexer, a filter, a frequency synthesizer, and the like, for example, in order to realize at least one of frequency division duplex (FDD) and time division duplex (TDD). may consist of For example, the acquisition unit 11 , the estimation unit 12 , the calculation unit 13 , the output unit 14 and the like described above may be implemented by the communication device 1004 .

The input device 1005 is an input device (for example, keyboard, mouse, microphone, switch, button, sensor, etc.) that receives input from the outside. The output device 1006 is an output device (for example, display, speaker, LED lamp, etc.) that outputs to the outside. Note that the input device 1005 and the output device 1006 may be integrated (for example, a touch panel).

Devices such as the processor 1001 and the memory 1002 are connected by a bus 1007 for communicating information. The bus 1007 may be configured using a single bus, or may be configured using different buses between devices.

In addition, the surrounding image estimation device 1 includes hardware such as a microprocessor, a digital signal processor (DSP), an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), and an FPGA (Field Programmable Gate Array). part or all of each functional block may be implemented by the hardware. For example, processor 1001 may be implemented using at least one of these pieces of hardware.

Notification of information is not limited to the aspects/embodiments described in this disclosure, and may be performed using other methods.

Each aspect/embodiment described in the present disclosure includes LTE (Long Term Evolution), LTE-A (LTE-Advanced), SUPER 3G, IMT-Advanced, 4G (4th generation mobile communication system), 5G (5th generation mobile communication system), FRA (Future Radio Access), NR (new Radio), W-CDMA (registered trademark), GSM (registered trademark), CDMA2000, UMB (Ultra Mobile Broadband), IEEE 802.11 (Wi-Fi (registered trademark) )), IEEE 802.16 (WiMAX®), IEEE 802.20, UWB (Ultra-WideBand), Bluetooth®, and other suitable systems and extended It may be applied to at least one of the next generation systems. Also, a plurality of systems may be applied in combination (for example, a combination of at least one of LTE and LTE-A and 5G, etc.).

The processing procedures, sequences, flowcharts, etc. of each aspect/embodiment described in this disclosure may be rearranged as long as there is no contradiction. For example, the methods described in this disclosure present elements of the various steps using a sample order, and are not limited to the specific order presented.

Input/output information and the like may be stored in a specific location (for example, memory), or may be managed using a management table. Input/output information and the like can be overwritten, updated, or appended. The output information and the like may be deleted. The entered information and the like may be transmitted to another device.

The determination may be made by a value represented by one bit (0 or 1), by a true/false value (Boolean: true or false), or by numerical comparison (for example, a predetermined value).

Each aspect/embodiment described in the present disclosure may be used alone, may be used in combination, or may be used by switching according to execution. In addition, the notification of predetermined information (for example, notification of “being X”) is not limited to being performed explicitly, but may be performed implicitly (for example, not notifying the predetermined information). good too.

Although the present disclosure has been described in detail above, it should be apparent to those skilled in the art that the present disclosure is not limited to the embodiments described in this disclosure. The present disclosure can be practiced with modifications and variations without departing from the spirit and scope of the present disclosure as defined by the claims. Accordingly, the description of the present disclosure is for illustrative purposes and is not meant to be limiting in any way.

Software, whether referred to as software, firmware, middleware, microcode, hardware description language or otherwise, includes instructions, instruction sets, code, code segments, program code, programs, subprograms, and software modules. , applications, software applications, software packages, routines, subroutines, objects, executables, threads of execution, procedures, functions, and the like.

Software, instructions, information, etc. may also be sent and received over a transmission medium. For example, the software uses wired technology (coaxial cable, fiber optic cable, twisted pair, Digital Subscriber Line (DSL), etc.) and/or wireless technology (infrared, microwave, etc.) to create websites, Wired and/or wireless technologies are included within the definition of transmission medium when sent from a server or other remote source.

Information, signals, etc. described in this disclosure may be represented using any of a variety of different technologies. For example, data, instructions, commands, information, signals, bits, symbols, chips, etc. that may be referred to throughout the above description may refer to voltages, currents, electromagnetic waves, magnetic fields or magnetic particles, light fields or photons, or any of these. may be represented by a combination of

The terms explained in this disclosure and the terms necessary for understanding the present disclosure may be replaced with terms having the same or similar meanings.

As used in this disclosure, the terms "system" and "network" are used interchangeably.

In addition, the information, parameters, etc. described in the present disclosure may be expressed using absolute values, may be expressed using relative values from a predetermined value, or may be expressed using other corresponding information. may be represented.

The names used for the parameters described above are not limiting names in any way. Further, the formulas, etc., using these parameters may differ from those expressly disclosed in this disclosure.

As used in this disclosure, the terms "determining" and "determining" may encompass a wide variety of actions. "Judgement", "determining" are, for example, judging, calculating, computing, processing, deriving, investigating, looking up, searching, inquiring (eg, lookup in a table, database, or other data structure), ascertaining as "judged" or "determined", and the like. Also, "judgment" and "decision" are used for receiving (e.g., receiving information), transmitting (e.g., transmitting information), input, output, access (accessing) (for example, accessing data in memory) may include deeming that something has been "determined" or "decided". In addition, "judgment" and "decision" are considered to be "judgment" and "decision" by resolving, selecting, choosing, establishing, comparing, etc. can contain. In other words, "judgment" and "decision" may include considering that some action is "judgment" and "decision". Also, "judgment (decision)" may be read as "assuming", "expecting", "considering", or the like.

The terms "connected," "coupled," or any variation thereof mean any direct or indirect connection or connection between two or more elements, It can include the presence of one or more intermediate elements between two elements being "connected" or "coupled." Couplings or connections between elements may be physical, logical, or a combination thereof. For example, "connection" may be read as "access". As used in this disclosure, two elements are defined using at least one of one or more wires, cables, and printed electrical connections and, as some non-limiting and non-exhaustive examples, in the radio frequency domain. , electromagnetic energy having wavelengths in the microwave and optical (both visible and invisible) regions, and the like.

As used in this disclosure, the phrase "based on" does not mean "based only on," unless expressly specified otherwise. In other words, the phrase "based on" means both "based only on" and "based at least on."

Any reference to elements using the "first," "second," etc. designations used in this disclosure does not generally limit the quantity or order of those elements. These designations may be used in this disclosure as a convenient method of distinguishing between two or more elements. Thus, reference to a first and second element does not imply that only two elements can be employed or that the first element must precede the second element in any way.

The “means” in the configuration of each device described above may be replaced with “unit”, “circuit”, “device”, or the like.

Where "include," "including," and variations thereof are used in this disclosure, these terms are inclusive, as is the term "comprising." is intended. Furthermore, the term "or" as used in this disclosure is not intended to be an exclusive OR.

In this disclosure, where articles have been added by translation, such as a, an, and the in English, the disclosure may include the plural nouns following these articles.

In the present disclosure, the term "A and B are different" may mean "A and B are different from each other." The term may also mean that "A and B are different from C". Terms such as "separate," "coupled," etc. may also be interpreted in the same manner as "different."

DESCRIPTION OF SYMBOLS 1... Surrounding image estimation apparatus 2*2a... Moving body 3*3a*3b... Camera 4... Surrounding image estimation system 10... Storage part 11... Acquisition part 12... Estimation part 13... Calculation part 14 1001: Processor 1002: Memory 1003: Storage 1004: Communication device 1005: Input device 1006: Output device 1007: Bus B1: Object B2/B3/B4: People C: Monitor Cameras, R1 and R2...areas, S, Sa, and Sb...surrounding images.

Claims (10)

an estimating unit for estimating a surrounding image captured by the imaging means after a predetermined time based on the surrounding image captured by the imaging means provided for the moving object;
a calculating unit for calculating a degree of matching between the surrounding image estimated by the estimating unit and the surrounding image actually captured by the imaging means after the predetermined time;
an output unit that outputs based on the degree of matching calculated by the calculation unit;
An ambient image estimation device comprising: The output unit calculates the predetermined time when the degree of matching satisfies a predetermined standard based on the degree of matching calculated by the calculating unit, and outputs the surrounding image after the predetermined time estimated by the estimating unit. Output,
The ambient image estimation device according to claim 1. The estimating unit estimates surrounding images corresponding to each of the plurality of different predetermined times,
The calculation unit calculates the matching degree for each of the plurality of surrounding images estimated by the estimation unit;
The output unit selects the degree of matching that satisfies a predetermined criterion from among the plurality of degrees of matching calculated by the calculating unit, and outputs the predetermined time on which the selected degree of matching is based on the degree of matching. Calculate as the predetermined time that satisfies a predetermined criterion,
The ambient image estimation device according to claim 2. The imaging means comprises a first camera capable of measuring the distance to an object in the image being imaged and a second camera capable of imaging the surroundings,
The estimation unit estimates based on a distance to an object in an image captured by the first camera and a surrounding image captured by the second camera.
A surrounding image estimation device according to any one of claims 1 to 3. The output unit performs output further based on a transmission delay time until the surrounding image captured by the imaging means is displayed to a controller controlling movement of the object.
The ambient image estimation device according to any one of claims 1 to 4. The output unit calculates the predetermined time when the degree of matching satisfies a predetermined standard based on the degree of matching calculated by the calculating unit, and calculates the predetermined time when the transmission delay time is within the predetermined time. outputting a surrounding image after the transmission delay time estimated by the unit;
The ambient image estimation device according to claim 5. The output unit performs output further based on the estimation accuracy of the estimation by the estimation unit.
The ambient image estimation device according to any one of claims 1 to 6. The estimation unit further estimates based on map information near the predicted position of the object after the predetermined time.
The surrounding image estimation device according to any one of claims 1 to 7. The estimating unit further estimates based on a surrounding image captured by another imaging means existing near the predicted position of the object after the predetermined time.
A surrounding image estimation device according to any one of claims 1 to 8. The estimating unit further estimates based on map information in the vicinity of the predicted position of the object after the predetermined time and an object included in a surrounding image captured by another imaging means existing in the vicinity of the predicted position,
A surrounding image estimation device according to any one of claims 1 to 9.

Images