WO2023100415A1

WO2023100415A1 - Information processing device, movable body, information processing method and program

Info

Publication number: WO2023100415A1
Application number: PCT/JP2022/028507
Authority: WO
Inventors: 則政岸; 雅司高田; 秀雄廣重
Original assignee: 株式会社光庭インフォ
Priority date: 2021-11-30
Filing date: 2022-07-22
Publication date: 2023-06-08
Also published as: CN118511505A; JPWO2023100415A1

Abstract

According to an aspect of the present invention, an information processing device is provided. This information processing device comprises a control unit. The control unit generates a single image on the basis of both a bird's eye view image related to the traveling direction of a movable body and a top view image including both the movable body and the periphery of the movable body. The single image includes a first area for displaying the bird's eye view image related to the traveling direction and a second area for displaying the top view image. The control unit performs a control such that the single image is displayed on a display unit that can be seen by the operator of the movable body.

Description

Information processing device, mobile object, information processing method and program

The present invention relates to an information processing device, a mobile object, an information processing method, and a program.

Display systems exist for mobile objects.
Japanese Patent Laid-Open No. 2002-200001 discloses a technique aimed at displaying a display that makes it easy for a passenger to understand the traveling direction of a moving object and to easily concentrate on monitoring the surroundings.

JP 2021-094939 A

　The surrounding information of the moving object is presented in an easy-to-understand manner.

According to one aspect of the present invention, an information processing device is provided. This information processing device has a control unit. The control unit generates one image based on a bird's-eye view image regarding the traveling direction of the moving object and a top view image including the moving object and its surroundings. One image includes a first area displaying a bird's-eye view image related to the traveling direction and a second area displaying a top view image. Control is performed so that one image is displayed on a display unit visible to the operator of the mobile object.

FIG. 1 is a diagram showing an example of a system configuration of an information processing system 1000. As shown in FIG. FIG. 2 is a diagram showing an example of the hardware configuration of the information processing device 110. As shown in FIG. FIG. 3 is a diagram showing an example of the functional configuration of the information processing device 110. As shown in FIG. FIG. 4 is an activity diagram showing an example of information processing by the information processing apparatus 110. As shown in FIG. FIG. 5 is a diagram (part 1) showing an example of a bird's-eye view image. FIG. 6 is a diagram (part 1) showing an example of a top view image. FIG. 7 is a diagram (part 1) showing an example of a synthesized image. FIG. 8 is a diagram (part 1) for explaining adjustment of the aspect ratio. FIG. 9 is a diagram (part 2) for explaining adjustment of the aspect ratio. FIG. 10 is a diagram (part 3) for explaining adjustment of the aspect ratio. FIG. 11 is a diagram (part 2) showing an example of a bird's-eye view image. FIG. 12 is a diagram (part 2) showing an example of a top view image. FIG. 13 is a diagram (part 2) showing an example of a synthesized image. FIG. 14 is a diagram showing an example of a displacement curve when per is decreasing away from an object. FIG. 15 is a diagram showing an example of a displacement curve when per approaches an object and increases. FIG. 16 is a diagram showing an example of a top view image and a bird's eye view image when the vehicle is in reverse. FIG. 17 is a diagram showing an example of a synthesized image. FIG. 18 is a diagram illustrating an example of image processing for a top view image and a bird's eye view image. FIG. 19 is a diagram showing an example in which projective transformation is performed so that a rectangle becomes an inverted trapezoid in a bird's-eye view image. 20A and 20B are diagrams illustrating an example of image processing for a top view image and a bird's eye view image in Modification 3. FIG. FIG. 21 is a diagram showing an example in which a distance sensor is provided in automobile 100. As shown in FIG. FIG. 22 is a diagram illustrating an example of a display mode of a bird's-eye view image.

Embodiments of the present invention will be described below with reference to the drawings. Various features shown in the embodiments shown below can be combined with each other.

In this specification, the term "unit" may include, for example, a combination of hardware resources implemented by circuits in a broad sense and software information processing that can be specifically realized by these hardware resources. In addition, various information is handled in the present embodiment, and these information are, for example, physical values of signal values representing voltage and current, and signal values as binary bit aggregates composed of 0 or 1. It is represented by high and low, or quantum superposition (so-called quantum bit), and communication and operation can be performed on a circuit in a broad sense.

A circuit in a broad sense is a circuit realized by at least appropriately combining circuits, circuits, processors, memories, and the like. That is, Application Specific Integrated Circuit (ASIC), programmable logic device (for example, Simple Programmable Logic Device (SPLD), Complex Programmable Logic Device (CPLD), and field It includes a programmable gate array (Field Programmable Gate Array: FPGA)).

<Embodiment 1>
1. System Configuration FIG. 1 is a diagram showing an example of the system configuration of an information processing system 1000. As shown in FIG. As shown in FIG. 1, an information processing system 1000 includes an automobile 100 as a system configuration. A car is an example of a mobile object. Automobile 100 includes information processing device 110 , display 120 , and multiple cameras 130 . The information processing device 110 is a device that executes the processing of this embodiment. In the present embodiment, the information processing device 110 is described as being included in the vehicle 100, but may not be included in the vehicle 100 as long as it can communicate with the vehicle 100 or the like. The display 120 is a display device that displays a 3D synthesized image and the like, which will be described later, under the control of the information processing device 110 . Camera 130 is a camera that captures an image of the surroundings of automobile 100 . More specifically, the camera 130 is an RGB camera or the like that adds and describes color information as an image. Camera 130 ₁ is provided behind automobile 100 . Camera 130 ₂ is provided in front of automobile 100 . Although not shown in FIG. 1, in addition to the cameras _130-1 and _130-2 , cameras 130 are provided on the left and right sides of the automobile 100, respectively. These cameras are hereinafter simply referred to as camera 130 . In addition, although not shown in FIG. 1 for simplification of explanation, the vehicle 100 has a plurality of depth sensors. The depth sensor measures the shape of objects around the automobile 100 and the distance from the automobile 100 to the objects using laser light or the like. However, the depth sensor is not limited to laser light, and may measure distance using ultrasonic waves, or may measure distance using a camera with a depth sensor.

2. Hardware Configuration FIG. 2 is a diagram illustrating an example of the hardware configuration of the information processing apparatus 110. As illustrated in FIG. The information processing apparatus 110 includes a control unit 201, a storage unit 202, and a communication unit 203 as a hardware configuration. The control unit 201 is a CPU (Central Processing Unit) or the like, and controls the entire information processing apparatus 110 . The storage unit 202 is any one of HDD (Hard Disk Drive), ROM (Read Only Memory), RAM (Random Access Memory), SSD (Solid State Drive), etc., or any combination thereof, and stores programs and controls. It stores data and the like used when the unit 201 executes processing based on a program. The control unit 201 executes processing based on a program stored in the storage unit 202 to configure the functional configuration of the information processing apparatus 110 as shown in FIG. Activity diagram processing and the like are realized. In this embodiment, it is assumed that data used when the control unit 201 executes processing based on a program is stored in the storage unit 202, but the data is stored in another device with which the information processing device 110 can communicate. You may make it memorize|store in a memory|storage part. The communication unit 203 is a NIC (Network Interface Card) or the like, connects the information processing apparatus 110 to a network, and controls communication with other apparatuses (eg, display 120, camera 130, etc.).

3. Functional Configuration FIG. 3 is a diagram illustrating an example of the functional configuration of the information processing device 110 . The information processing apparatus 110 includes a peripheral information receiving unit 301, an object recognition unit 302, a behavior information receiving unit 303, and a display control unit 304 as functional configurations.

(Peripheral information receiving unit 301)
The peripheral information receiving unit 301 receives information on objects around the vehicle 100, information on colors around the vehicle 100, and the like from the camera 130 as peripheral information. Further, the peripheral information receiving unit 301 receives the shapes of objects around the vehicle 100 and the distance from the vehicle 100 to the objects as peripheral information from the plurality of depth sensors.

(Object recognition unit 302)
The object recognition unit 302 recognizes objects around the automobile 100 based on the surrounding information received by the surrounding information receiving unit 301 . Surrounding objects include, for example, buildings, roads, other vehicles, parking lots, and the like.

(Behavior information receiving unit 303)
The behavior information receiving unit 303 receives behavior information of the moving body. More specifically, the behavior information receiving unit 303 converts information detected by a wheel speed sensor, a steering angle sensor, an inertial measurement unit (IMU), etc. included in the vehicle 100 into the behavior information of the vehicle 100. to get as An inertial measurement device is a ^device that detects three-dimensional inertial motion (translational motion and rotational motion in orthogonal three-axis directions). Rotational motion is detected by [deg/sec]. Also, the behavior information receiving unit 303 may receive information about the behavior of the automobile 100 from SLAM (Simultaneous Localization and Mapping) included in the automobile 100, LiDAR, a stereo camera, a gyro sensor, etc. included in the automobile 100.
Here, the behavior information includes, for example, position information and orientation information of the vehicle 100 .

(Display control unit 304)
The display control unit 304 converts a surrounding object into a 3D model based on information on the object recognized by the object recognition unit 302, information from the SLAM, and the like. The display control unit 304 also generates a 3D model of the automobile 100 based on information from the camera 130, information from SLAM, behavior information of the automobile 100 received by the behavior information receiving unit 303, and the like. The display control unit 304 also generates a top view image including the automobile 100 and its surroundings based on the 3D model of the surrounding objects, the 3D model of the automobile 100, information from the camera 130, and the like. Here, the top view image is an image that has undergone perspective transformation so that the viewpoint can be viewed from above the road surface, and is an image that includes the entire automobile 100 or at least half or more of the automobile 100 . In the present embodiment, an image of the ground taken from directly above is used as a bottom image, and an image synthesized and morphed with an image of the automobile 100 or the like is used as a top image. The display control unit 304 generates one composite image based on the bird's-eye view image regarding the traveling direction of the automobile 100 and the top view image including the automobile 100 and its surroundings. Here, the bird's-eye view image regarding the traveling direction of the vehicle 100 is, for example, a bird's-eye view image in front of the vehicle 100 when the vehicle 100 is moving forward, and is, for example, a bird's-eye view image in front of the vehicle 100 when the vehicle 100 is moving backward. 2 is a bird's-eye view image of the rear of the automobile 100. FIG. The forward bird's eye view image is an image obtained by morphing the image captured by the camera ₁₃₀₂ based on the peripheral information of the automobile 100 acquired by the peripheral information receiving unit 301 . Similarly, the rear bird's-eye view image is obtained by morphing the image captured by the camera ₁₃₀₁ based on the peripheral information of the automobile 100 acquired by the peripheral information receiving unit 301 . Here, the synthesized image includes a first area displaying a bird's-eye view image related to the traveling direction and a second area displaying a top view image. The composite image is shown in FIG. 7 and the like, which will be described later. The display control unit 304 controls to display the generated composite image on the display 120 of the automobile 100 or the like. The display 120 of the automobile 100 is an example of a display visible to the operator of the automobile 100 .

Note that part or all of the functional configuration of FIG. 3 may be implemented in the information processing device 110 or the automobile 100 as a hardware configuration. Also, part or all of the functional configuration of the information processing device 110 may be implemented in the vehicle 100 .

4. Information Processing FIG. 4 is an activity diagram showing an example of information processing of the information processing apparatus 110 .
At A401, the peripheral information receiving unit 301 receives information on objects around the vehicle 100, information on colors around the vehicle 100, etc. from the camera 130 as peripheral information. Further, the peripheral information receiving unit 301 receives the shapes of objects around the vehicle 100 and the distance from the vehicle 100 to the objects as peripheral information from the plurality of depth sensors.
At A<b>402 , the object recognition unit 302 recognizes objects around the automobile 100 based on the surrounding information received by the surrounding information receiving unit 301 .

At A403, the behavior information receiving unit 303 receives information about the behavior of the automobile 100.

At A404, the display control unit 304 converts the surrounding objects into a 3D model based on the information on the object recognized by the object recognition unit 302, the information from the SLAM, and the like. The display control unit 304 also generates a 3D model of the automobile 100 based on information from the camera 130, information from SLAM, behavior information of the automobile 100 received by the behavior information receiving unit 303, and the like. The display control unit 304 also generates a top view image including the automobile 100 and its surroundings based on the 3D model of the surrounding objects, the 3D model of the automobile 100, information from the camera 130, and the like. The display control unit 304 also generates a bird's-eye view image regarding the traveling direction of the automobile 100 based on information from the camera 130 and the like.

In A405, the display control unit 304 generates one composite image based on the bird's eye view image regarding the traveling direction of the automobile 100 and the top view image including the automobile 100 and its surroundings.
In A406, the display control unit 304 controls to display the generated composite image on the display 120 of the automobile 100 or the like.
FIG. 5 is a diagram (Part 1) showing an example of a bird's-eye view image regarding the traveling direction of the automobile 100. FIG. An image 501 is an example of a bird's-eye view image of the traveling direction of the automobile 100 . FIG. 5 is a diagram of a scene in which the vehicle 100 is about to be parked in a parking space in reverse. The display control unit 304 identifies the driving scene of the automobile 100 based on the distance to objects around the automobile 100 obtained from a depth sensor or the like. For simplicity of explanation, in the present embodiment, a bird's-eye view image (back-view bird's-eye view image) of the camera ₁₃₀₁ when the automobile 100 is moving backward will be described as an example. However, the same applies to a bird's-eye view image (front-view bird's-eye view image) of the camera ₁₃₀₂ when the automobile 100 is moving forward.

(Change of display ratio - step by step)
The display control unit 304 changes the ratio of the first area and the ratio of the second area in the synthesized image based on the distance between the automobile 100 and the object existing in the traveling direction. The first display area and the second display area will be described later with reference to FIG. 7 and the like.
First, the display control unit 304 changes the proportion of the first area and the proportion of the second area in the composite image in stages based on the distance between the automobile 100 and the object existing in the traveling direction. An example to change is explained. An example of an object existing in the direction of travel is an obstacle such as a car stop. However, this does not limit the present embodiment, and other examples of objects existing in the direction of travel include parking lot walls, other cars parked in the parking lot, and parking spaces in the parking lot. It may be a mark or the like. Here, the display control unit 304 may recognize and extract an object present in the direction of travel by image processing from a bird's-eye view image related to the direction of travel, or an object around the automobile 100 recognized by the object recognition unit 302 may be extracted. can be recognized by A car stop, a mark indicating a parking place, or the like is an example of a car parking position.

For the sake of simplicity of explanation, an example of display control when the distance L between the vehicle 100 and the object behind the vehicle 100 is greater than or equal to a predetermined distance and when it is less than the predetermined distance will be described below. However, this does not limit the present embodiment, and the display control unit 304 does not have two stages, but three stages (for example, 5 m or more, less than 5 m, 3 m or more, less than 3 m, etc.), four stages (for example, 5 m or more, less than 5 m, 4 m or more, less than 4 m, 3 m or more, less than 3 m, etc.). In addition, as an example of the predetermined distance, 3 m will be described below as an example. In addition, 3m is an example.

When the vertical and horizontal lengths of the screen presented on the display 120 by the display control unit 304 are H and V, the vertical and horizontal lengths of the top view image subjected to image processing such as cropping to generate the composite image are Hf and Vf, and the composite image is generated. Let Hd and Vd be bird's-eye view images of the traveling direction of the automobile 100 that have undergone image processing, such as cropping, for the purpose of obtaining the image.
When the distance L between the automobile 100 and the object behind the automobile 100 is 3 m or more, the display control unit 304 controls Hf and Hd to be one to one.
More specifically, the display control unit 304 cuts out a portion of the frame 502 from the image 501 in FIG. That is, when the distance L between the automobile 100 and an object behind the automobile 100 is 3 m or more, the display control unit 304 presents an image far from the automobile 100 in the bird's-eye view image regarding the traveling direction of the automobile 100. Delete the image near 100. The display control unit 304 always keeps the aspect ratio of the cropped image constant.

Also, the display control unit 304 cuts out a portion of a frame 602 from the top view image 601 in FIG. FIG. 6 is a diagram (part 1) showing an example of a top view image. The display control unit 304 combines the image 503 and the image 604 to generate the composite image 701 shown in FIG. FIG. 7 is a diagram (part 1) showing an example of a synthesized image. In FIG. 7, Hd and Hf are one-to-one. In FIG. 7, 702 is the first area. 703 is the second area. As shown in FIG. 7, the first region is placed on top of the composite image 701 . The second area is placed at the bottom of the composite image 701 . The first area is an area where a bird's-eye view image regarding the traveling direction of the automobile 100 is displayed. The second area is an area in which a top view image including the automobile 100 and its surroundings is displayed. In the synthesized image, the first area and the second area match the flow of surrounding information with respect to the movement of the car. Therefore, the operator of the vehicle 100 can intuitively understand the surrounding conditions and the like.

In addition, the display control unit 304 adjusts the aspect ratio of each area so that the display distances of the respective display areas are substantially the same at the locations where the first area and the second area in the synthesized image are in contact with each other. . The adjustment of the aspect ratio will be described below with reference to FIGS. 8 to 10. FIG.
FIG. 8 is a diagram (part 1) for explaining adjustment of the aspect ratio. In FIG. 8, 704 represents the automobile 100 . The lines in the first area 702 and the second area 703 are attached for explanation. represents a line.
When there is no object in the traveling direction of the automobile 100, the display control unit 304 performs image processing to compress the portion corresponding to the automobile 100 in the top view image so that the left and right sides of the automobile 100 can be seen, as shown in FIG. Then, the top view image and the bird's eye view image are synthesized. FIG. 9 is a diagram (part 2) for explaining adjustment of the aspect ratio.
On the other hand, if there is an object in the traveling direction of the automobile 100, the display control unit 304 performs projective transformation so that the rectangle becomes a trapezoid in the top view image as shown in FIG. A top view image and a bird's eye view image are synthesized. FIG. 10 is a diagram (part 3) for explaining adjustment of the aspect ratio. By performing image processing in this manner, the lines of the top view image and the bird's eye view image can be made as if they were straight lines. FIG. 10 is a diagram (part 3) for explaining adjustment of the aspect ratio.

When the distance L between the car 100 and the object behind the car 100 is less than 3 m, the display control unit 304 controls Hf and Hd to be 1:2. More specifically, the display control unit 304 cuts out a frame 802 from the image 801 in FIG. 11 to obtain an image 803 . That is, when the distance L between the automobile 100 and the object behind the automobile 100 is less than 3 m, the display control unit 304 controls the image near the automobile 100 because distant images are not important in the bird's-eye view image regarding the traveling direction of the automobile 100 . Cut out the image of FIG. 11 is a diagram (No. 2) showing an example of a bird's-eye view image regarding the traveling direction of the automobile 100. As shown in FIG. The display control unit 304 changes the scale of the aspect ratio of the cropped image 803 to obtain an image 804 .

Further, the display control unit 304 cuts out the portion of the frame 902 from the top view image 901 of FIG. FIG. 12 is a diagram (part 2) showing an example of a top view image. The display control unit 304 changes the scale of the horizontal ratio of the image 904 to obtain an image 905 . The display control unit 304 combines the image 804 and the image 905 to generate the composite image 1001 in FIG. 13 . FIG. 13 is a diagram (part 2) showing an example of a synthesized image. In FIG. 13, Hd and Hf are 1:2.

(change of display ratio - continuous)
Next, an example in which the display control unit 304 continuously changes the proportion of the first area and the proportion of the second area in the synthesized image will be described. More specifically, the display control unit 304 performs control such that the closer the vehicle 100 is to the object, the smaller the proportion of the first area in the composite image than the proportion of the second area.
The display control unit 304 uses the area change rate between the image displayed in the first area and the image displayed in the second area, and the displacement curves shown in FIGS. 14 and 15 to be described later, Change the area of the image displayed in each area.

The display control unit 304 obtains per indicating the distance ratio.
In the case of retreating, the display control unit 304 measures the distance to the object behind and divides it by the maximum distance (5 m) to obtain per. In addition, 5m is an example of a predetermined distance, and is not limited to this.
per = 1 - distance to object behind/maximum distance (5m)
In the case of moving forward, the display control unit 304 measures the distance to the object ahead and divides it by the maximum distance (5 m) to obtain per.
per = 1 - distance to object in front/maximum distance (5m)
If the object is not closer than the maximum distance, the subsequent processing is not performed.
per(0<=per<=1)

The display control unit 304 obtains the display area division (dif) from the displacement curve using the obtained per.
Here, when the automobile 100 is parked, there is a case where the vehicle is turned back. Also, the vehicle 100 may be separated from the object due to the steering angle. In such a case, in order to prevent a large change in the image (video) displayed on the display 120, the displacement curve is set as shown in FIG. (curveInc) The automobile 100 is configured to have two of FIG. Also, curveInc is always located on the right side of curveDec. FIG. 14 is a diagram showing an example of a displacement curve when per is decreasing away from an object. FIG. 15 is a diagram showing an example of a displacement curve when per approaches an object and increases. In both FIGS. 14 and 15, the horizontal axis is per and the vertical axis is dif.

(change of viewpoint)
In addition, the display control unit 304 changes the depression angle with respect to the viewpoint of the top view image based on the distance between the automobile 100 and an object existing in the traveling direction. More specifically, the display control unit 304 performs control such that the closer the vehicle 100 is to the object, the greater the angle of depression with respect to the viewpoint of the top view image. At this time, the display control unit 304 uses the displacement curve described above to perform control such that the closer the vehicle 100 is to the object, the greater the angle of depression with respect to the viewpoint of the top view image. The display control unit 304 may change the rate of change in the angle of depression depending on whether the vehicle 100 is moving closer to the object or when the vehicle 100 is moving away from the object.
The display control unit 304 may change the display ratio described above when the automobile 100 is moving backward, and may change the viewpoint when the automobile 100 is moving forward.

According to the present embodiment, it is possible for the operator to easily understand the traveling direction of the mobile object, and to present the peripheral information of the mobile object in an easy-to-understand manner. More specifically, the information processing apparatus 110 presents the composite image on one screen, and changes the viewpoint and/or the presentation area of the perspective transformation according to the surrounding conditions, thereby obtaining a bird's-eye view of the direction of travel. The image and the top view image are represented continuously. As a result, intuitive surrounding information can be presented to the operator. In particular, the information processing device 110 considers the distance to objects such as obstacles around the automobile 100, and changes the viewpoint and/or the presentation area, thereby realizing a display method that makes the obstacles more recognizable. As a result, the operator can more intuitively understand the forward and backward movements of the vehicle when the vehicle is backing up, and can drive the automobile 100 safely.

(Modification 1)
A modification of the first embodiment will be described.
In the first embodiment, the automobile 100 has been described as an example of a moving object. However, the mobile object may be, for example, a so-called drone such as an unmanned aerial vehicle. When the moving object is a drone, the information processing device 110 may be included in the drone or may be included in the controller of the operator who operates the drone. Further, when the moving body is a drone, a display section that can be visually observed by the operator is provided in the controller. Further, when the moving object is a drone, examples of the object include a mark indicating the departure and arrival location of the drone.
According to the modified example, it is also possible for the operator to easily understand the moving direction of the moving object, and to present the peripheral information of the moving object in an easy-to-understand manner.

(Modification 2)
A modification of the first embodiment will be described.
FIG. 16 is a diagram showing an example of a top view image and a bird's eye view image when the vehicle is in reverse. Arrow 1601 indicates the traveling direction of automobile 100 . Image 1602 is a top view image. An image 1603 is a bird's-eye view image.
FIG. 17 is a diagram showing an example of a synthesized image. A synthesized image 1701 is an image created by synthesizing the top view image 1602 and the overhead view image 1603 shown in FIG.
FIG. 18 is a diagram illustrating an example of image processing for a top view image and a bird's eye view image.
In S1801, the display control unit 304 turns the top view image upside down so that the vehicle faces downward.
In S1802, the display control unit 304 performs rectangular clipping on the vertically inverted top view image so as to have a predetermined area Sa. That is, the display control unit 304 does not draw a portion other than the predetermined area Sa of the top view image that is vertically inverted.
In S1803, the display control unit 304 projectively transforms the rectangle-clipped top view image so that the rectangle becomes a trapezoid.
On the other hand, in S1804, the display control unit 304 performs rectangular clipping so that the bird's-eye view image has a predetermined area Sb. That is, the display control unit 304 does not draw a portion other than the predetermined area Sb of the predetermined bird's-eye view image.
Here, the display control unit 340 changes the ratio of Sa to Sb described above according to the distance between the automobile 100 and the obstacle.
In S1805, the display control unit 304 performs projective transformation so as to correct the distortion of the bird's-eye view image subjected to rectangular clipping.
In S1806, the display control unit 304 synthesizes the projectively transformed top view image and the projectively transformed bird's eye view image to generate one synthesized image.
Note that while the automobile 100 is moving, the processing shown in FIG. 18 is repeatedly executed.

(Modification 3)
A modification of the first embodiment will be described.
FIG. 19 is a diagram showing an example in which projective transformation is performed so that a rectangle becomes an inverted trapezoid in a bird's-eye view image.
When an object exists in the traveling direction of the automobile 100, as shown in FIG. Projective transformation is performed so that the rectangle becomes an inverted trapezoid, and one image is generated by synthesizing the top view image subjected to the projective transformation and the overhead view image subjected to the projective transformation. By performing projective transformation so that the rectangle in the top view image becomes a trapezoid and performing projective transformation so that the rectangle in the bird's eye view image becomes an inverse trapezoid, the top view image and the bird's eye view image are transformed as shown in FIG. , it is possible to connect straight lines smoothly.

20A and 20B are diagrams illustrating an example of image processing for a top view image and a bird's eye view image in Modification 3. FIG.
In S2001, the display control unit 304 turns the top view image upside down so that the vehicle faces downward.
In S2002, the display control unit 304 sets an ROI (Region Of Interest). At this time, the display control unit 304 sets the ROI so that the upper side of the top view image and the lower side of the bird's eye view image match in three-dimensional position.
In S2003, the display control unit 304 performs projective transformation so that the rectangle of the top view image becomes a trapezoid.
On the other hand, in S2004, the display control unit 304 sets an ROI (Region Of Interest). At this time, the display control unit 304 sets the ROI so that the lower side of the bird's-eye view image and the upper side of the top view image match at three-dimensional positions. Note that when the automobile 100 approaches an obstacle, the display control unit 304 sets the ROI so that the near area is enlarged in the vertical direction and the display area in front is enlarged in the horizontal direction.
In S2005, the display control unit 304 performs projective transformation so that the rectangle of the overhead view image becomes an inverted trapezoid.
In S2006, the display control unit 304 combines the top view image that has undergone the projective transformation and the overhead view image that has undergone the projective transformation.
In S<b>2007 , the display control unit 304 enlarges or reduces the combined image to fit the display 120 . Also, the display control unit 304 clips the combined image.
Note that while the automobile 100 is moving, the processing shown in FIG. 20 is repeatedly executed.

(Modification 4)
A modification of the first embodiment will be described.
FIG. 21 is a diagram showing an example in which a distance sensor is provided in automobile 100. As shown in FIG. FIG. 21 shows an example in which distance sensors are provided on the rear left and right sides of the automobile 100 . However, the number of distance sensors is not limited to two. Also, the distance sensor may measure the distance to an object existing on the side or front of the automobile 100 . As shown in FIG. 21, for example, when an object exists in the right rear of the automobile 100, the display control unit 304 of Modification 4 displays the left portion of the bird's-eye view image as shown in FIG. Projective transformation is performed so that the rectangle becomes an inverted trapezoid. Note that in FIG. 22, projective transformation is not performed on the right side of the bird's-eye view image so that the rectangle becomes an inverted trapezoid. Similarly, for example, when an object exists on the left rear of the automobile 100, the display control unit 304 performs projective transformation so that the rectangle becomes an inverted trapezoid on the right side of the bird's-eye view image. In this case as well, the left portion of the bird's-eye view image is not subjected to projective transformation such that the rectangle becomes an inverted trapezoid. That is, the display control unit 304 performs control so that the display form of the bird's-eye view image is changed depending on whether the object is on the left or right of the moving object.
FIG. 22 is a diagram illustrating an example of a display mode of a bird's-eye view image. In the example of FIG. 22, projective transformation is performed on the left side of the bird's eye view image so that the rectangle becomes an inverted trapezoid, and projective transformation is not performed on the right side of the bird's eye view image.

<Appendix>
The invention may be provided in each of the aspects described below.

(1) An information processing apparatus having a control unit, wherein the control unit generates a bird's-eye view image related to a moving direction of a moving body and a top view image including the moving body and the periphery of the moving body. based on, one image is generated, the one image includes a first area for displaying a bird's eye view image related to the traveling direction and a second area for displaying the top view image, An information processing device that controls to display the one image on a display unit that is visible to an operator of the mobile object.

(2) In the information processing device according to (1) above, the first area is arranged above the one image, and the second area is arranged below the one image. Information processing equipment.

(3) In the information processing apparatus according to (1) or (2) above, the control unit, based on the distance between the moving object and the object existing in the traveling direction, in the one image, An information processing device that changes a ratio of the first area and a ratio of the second area.

(4) In the information processing apparatus according to (3) above, the control unit controls, based on the distance, the ratio of the first area and the ratio of the second area in the one image, An information processing device that continuously changes the

(5) In the information processing apparatus according to (3) above, the control unit controls, based on the distance, the ratio of the first area and the ratio of the second area in the one image, Information processing device that changes step by step.

(6) In the information processing apparatus according to any one of (3) to (5) above, the control unit controls the area of the first area in the one image as the moving object approaches the object. An information processing device that performs control so that the ratio is smaller than the ratio of the second region.

(7) In the information processing apparatus according to any one of (3) to (6) above, when an object exists in the direction of travel, the control unit causes the rectangle to become a trapezoid in the top view image. An information processing device that generates the one image by performing a projective transformation and synthesizing it with the bird's-eye view image.

(8) In the information processing apparatus described in (7) above, when an object exists in the traveling direction, the control unit performs projective transformation so that a rectangle becomes a trapezoid in the top view image, and An information processing device that performs projective transformation so that a rectangle in an image becomes an inverted trapezoid, and generates the one image by synthesizing the projectively transformed top view image and the projectively transformed overhead view image.

(9) In the information processing device according to (8) above, the control unit may change the display form of the bird's-eye view image depending on whether the object is on the left or right of the moving object. Information processing device to control.

(10) In the information processing apparatus according to any one of (3) to (9) above, the control unit adjusts the overhead view image based on the distance between the moving object and the object. An information processing device that changes the depression angle related to the viewpoint.

(11) In the information processing apparatus according to (10) above, the control unit performs control such that the depression angle with respect to the viewpoint of the bird's-eye view image increases as the moving object approaches the object.

(12) In the information processing device described in (10) or (11) above, the control unit changes the bird's-eye view image when the moving body approaches the object and when the moving body moves away from the object. An information processing device that changes the rate of change in depression angle with respect to a viewpoint.

(13) In the information processing apparatus according to any one of (3) to (12) above, the moving object is an automobile, and the object is related to a parking position of the automobile. Device.

(14) A mobile body, comprising the information processing device according to any one of (1) to (13) above.

(15) The mobile object according to (14) above, wherein the mobile object is an automobile.

(16) An information processing method executed by an information processing apparatus, wherein one An image is generated, wherein the one image includes a first area for displaying a bird's-eye view image related to the traveling direction and a second area for displaying the top view image, and the one image is generated. An information processing method for controlling display on a display unit of the moving body.

(17) A program for causing a computer to function as a control unit of the information processing apparatus according to any one of (1) to (13) above.
Of course, this is not the only case.

For example, it may be provided as a computer-readable non-temporary storage medium that stores the above program.
Moreover, you may combine a modification arbitrarily.
Also, in the above-described embodiments and the like, the back view has been mainly explained as an example, but the above-described effects can be obtained by executing the same processing for the front view as well.

Finally, although various embodiments of the present invention have been described, these are presented as examples and are not intended to limit the scope of the invention. The novel embodiments can be embodied in various other forms, and various omissions, replacements, and modifications can be made without departing from the scope of the invention. Embodiments and modifications thereof are included in the scope and gist of the invention, and are included in the scope of the invention described in the claims and equivalents thereof.

100: Car 110: Information processing device 120: Display 130: Camera 201: Control unit 202: Storage unit 203: Communication unit 301: Peripheral information reception unit 302: Object recognition unit 303: Behavior information reception unit 304: Display control unit 340: Display control unit 501 : Image 502 : Frame 503 : Image 601 : Top view image 602 : Frame 603 : Image 604 : Image 701 : Synthetic image 702 : First area 703 : Second area 801 : Image 802 : Frame 803 : Image 804 : Image 901 : Top view image 902 : Frame 903 : Image 904 : Image 905 : Image 1000 : Information processing system 1001 : Synthetic image 1301 : Camera 1302 : Camera 1601 : Arrow 1701 : Synthetic image

Claims

An information processing device,
having a control unit,
The control unit
Generating one image based on a bird's-eye view image of the moving direction of the moving body and a top view image including the moving body and the periphery of the moving body,
The one image includes a first area that displays a bird's eye view image related to the traveling direction and a second area that displays the top view image,
controlling to display the one image on a display unit visible to an operator of the mobile body;
Information processing equipment.
In the information processing device according to claim 1,
The first area is arranged at the top of the one image,
The second area is arranged at the bottom of the one image,
Information processing equipment.
In the information processing apparatus according to claim 1 or claim 2,
The control unit
changing the ratio of the first area and the ratio of the second area in the one image based on the distance between the moving object and the object existing in the traveling direction;
Information processing equipment.
In the information processing device according to claim 3,
The control unit
continuously changing the ratio of the first area and the ratio of the second area in the one image based on the distance;
Information processing equipment.
In the information processing device according to claim 3,
The control unit
Based on the distance, stepwise changing the ratio of the first area and the ratio of the second area in the one image.
Information processing equipment.
In the information processing apparatus according to any one of claims 3 to 5,
The control unit
controlling so that the ratio of the first area in the one image becomes smaller than the ratio of the second area as the moving object approaches the object;
Information processing equipment.
In the information processing apparatus according to any one of claims 3 to 6,
The control unit
When an object exists in the traveling direction, projective transformation is performed so that the rectangle becomes a trapezoid in the top view image, and the one image is generated by synthesizing it with the bird's eye view image.
Information processing equipment.
In the information processing device according to claim 7,
The control unit
When an object exists in the traveling direction, projective transformation is performed so that the rectangle becomes a trapezoid in the top view image, and projective transformation is performed so that the rectangle becomes an inverted trapezoid in the bird's eye view image. generating the one image by synthesizing the diagram image and the bird's-eye view image that has undergone the projective transformation;
Information processing equipment.
In the information processing device according to claim 8,
The control unit
controlling to change the display form of the bird's-eye view image according to whether the object is on the left or right of the moving object;
Information processing equipment.
In the information processing apparatus according to any one of claims 3 to 9,
The control unit
changing a depression angle with respect to the viewpoint of the bird's-eye view image based on the distance between the moving body and the object;
Information processing equipment.
In the information processing device according to claim 10,
The control unit
controlling such that the closer the moving body approaches the object, the greater the angle of depression with respect to the viewpoint of the bird's-eye view image;
Information processing equipment.
In the information processing device according to claim 10 or claim 11,
The control unit
changing the rate of change in the depression angle with respect to the viewpoint of the bird's-eye view image between when the moving body approaches the object and when the moving body moves away from the object;
Information processing equipment.
In the information processing apparatus according to any one of claims 3 to 12,
The moving object is an automobile,
the object relates to a parking position of the car;
Information processing equipment.
being mobile,
Having the information processing device according to any one of claims 1 to 13,
Mobile.
In the moving body according to claim 14,
the moving object is an automobile,
Mobile.
An information processing method executed by an information processing device,
Generating one image based on a bird's-eye view image of the moving direction of the moving body and a top view image including the moving body and the periphery of the moving body,
The one image includes a first area that displays a bird's eye view image related to the traveling direction and a second area that displays the top view image,
controlling to display the one image on the display unit of the moving body;
Information processing methods.
a program,
the computer,
A program for functioning as a control unit of the information processing apparatus according to any one of claims 1 to 13.