US12482170B2 - Image processing device, image processing method, and image processing program - Google Patents

Image processing device, image processing method, and image processing program

Info

Publication number
US12482170B2
US12482170B2 US18/516,334 US202318516334A US12482170B2 US 12482170 B2 US12482170 B2 US 12482170B2 US 202318516334 A US202318516334 A US 202318516334A US 12482170 B2 US12482170 B2 US 12482170B2
Authority
US
United States
Prior art keywords
bird
dimensional
eye view
area
view image
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active, expires
Application number
US18/516,334
Other versions
US20240096006A1 (en
Inventor
Katsuyuki OKONOGI
Takayuki Kato
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Socionext Inc
Original Assignee
Socionext Inc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Socionext Inc filed Critical Socionext Inc
Publication of US20240096006A1 publication Critical patent/US20240096006A1/en
Application granted granted Critical
Publication of US12482170B2 publication Critical patent/US12482170B2/en
Active legal-status Critical Current
Adjusted expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60R—VEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR
    • B60R1/00—Optical viewing arrangements; Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles
    • B60R1/20—Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles
    • B60R1/22—Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles for viewing an area outside the vehicle, e.g. the exterior of the vehicle
    • B60R1/23—Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles for viewing an area outside the vehicle, e.g. the exterior of the vehicle with a predetermined field of view
    • B60R1/27—Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles for viewing an area outside the vehicle, e.g. the exterior of the vehicle with a predetermined field of view providing all-round vision, e.g. using omnidirectional cameras
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T15/00—Three-dimensional [3D] image rendering
    • G06T15/10—Geometric effects
    • G06T15/20—Perspective computation
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/50—Image enhancement or restoration using two or more images, e.g. averaging or subtraction
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/50—Context or environment of the image
    • G06V20/56—Context or environment of the image exterior to a vehicle by using sensors mounted on the vehicle
    • G06V20/58—Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
    • G06V20/586—Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads of parking space
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N5/00—Details of television systems
    • H04N5/74—Projection arrangements for image reproduction, e.g. using eidophor
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N7/00—Television systems
    • H04N7/18—Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20212—Image combination
    • G06T2207/20221—Image fusion; Image merging

Definitions

  • the present disclosure relates to an image processing device, an image processing method, and an image processing program.
  • a three-dimensional bird's eye view image generation system that photographs the surroundings of a mobile entity such as a vehicle by using multiple imaging devices, and that generates a three-dimensional bird's eye view image viewing the whole surroundings of the mobile entity from above by projecting multiple photographed images onto a three-dimensional projection surface by using projection information.
  • the three-dimensional bird's eye view image generation system for example, when a driver is parking his/her vehicle, it is possible to assist the driver's driving operation by providing the driver with a three-dimensional bird's eye view image of the vehicle.
  • At least one aspect of the present disclosure therefore aims to display an object naturally when generating a three-dimensional bird's eye view image from multiple images.
  • an image processing device is configured to generate a three-dimensional bird's eye view image by projecting captured images acquired from a plurality of imaging devices onto a three-dimensional projection surface
  • this image processing device includes: an acquiring part configured to acquire position information of a first area, in which an object is to be placed, in a three-dimensional space; a conversion part configured to convert the position information of the first area into position information in accordance with the three-dimensional projection surface; and an output part configured to output the three-dimensional bird's eye view image, on which the object is to be placed on a second area in the three-dimensional space, the second area being specified by the converted position information.
  • FIG. 1 is a diagram that illustrates an example system structure of a three-dimensional bird's eye view image generation system and an example functional structure of an image processing device;
  • FIG. 2 is a diagram that illustrates an example arrangement of devices in a three-dimensional bird's eye view image generation system
  • FIG. 3 is a diagram that illustrates example outer structures of an operating device and a display device
  • FIG. 4 is a diagram that illustrates an example hardware structure of an image processing device
  • FIG. 5 is a first diagram that illustrates a specific example of a process in a first generation part
  • FIG. 6 is a second diagram that illustrates a specific example of the process in the first generation part
  • FIG. 7 is a first diagram that illustrates a specific example of a process in a second generation part
  • FIG. 8 is a second diagram that illustrates a specific example of the process in the second generation part
  • FIG. 9 is a first flowchart that shows a flow of a display image generation process
  • FIG. 10 is a diagram that illustrates an example of a scene in which the three-dimensional bird's eye view image generation system is used;
  • FIG. 11 is a diagram that illustrates a specific example of a display image
  • FIG. 12 is a third diagram that illustrates a specific example of the process in the second generation part
  • FIG. 13 is a second flowchart that shows a flow of a display image generation process.
  • FIG. 14 is a fourth diagram that illustrates a specific example of the process in the second generation part.
  • FIG. 1 is a diagram that illustrates an example system structure of the three-dimensional bird's eye view image generation system and an example functional structure of the image processing device.
  • the example system structure of FIG. 1 shows a system structure in which a three-dimensional bird's eye view image generation system 100 is mounted on a vehicle 10 .
  • the three-dimensional bird's eye view image generation system 100 mounted on the vehicle 10 includes imaging devices 111 to 114 , an image processing device 120 , a controller area network (CAN) 170 , an operating device 181 , and a display device 182 .
  • CAN controller area network
  • the imaging devices 111 to 114 generate captured images by photographing the front, left, right, and rear sides of the vehicle 10 , respectively, and send the thus-generated captured images to the image processing device 120 .
  • An image processing program is installed in the image processing device 120 , and, when this program is executed, the image processing device 120 functions as a first generation part 130 , a second generation part 140 , and an output part 150 .
  • the captured image acquiring part 131 When the captured image acquiring part 131 receives, from the operating device 181 , command information including a command for generating a display image, the captured image acquiring part 131 acquires the captured images transmitted from the imaging devices 111 to 114 .
  • the first projection part 132 projects each captured image acquired by the captured image acquiring part 131 onto a three-dimensional projection surface, based on projection information stored in a projection information storage part 160 .
  • the first bird's eye view image generation part 133 generates a first bird's eye view image in a predetermined view position and direction based on each captured image projected on the three-dimensional projection surface.
  • the second generation part 140 generates a second bird's eye view image, which is a three-dimensional bird's eye view image in which an object is to be placed.
  • the second generation part 140 includes a position information acquiring part 141 , a second projection part 142 , and a second bird's eye view image generation part 143 .
  • the object included in the second bird's eye view image may be one that is not included in the captured images transmitted from the imaging devices 111 to 114 .
  • the position information acquiring part 141 acquires position information of the vehicle 10 in a three-dimensional space from the CAN 170 . Also, the position information acquiring part 141 acquires position information of the site (first area), on which the object is to be placed, in the three-dimensional space.
  • the second projection part 142 is an example of a conversion part, and projects the site, on which the object is to be placed, onto a three-dimensional projection surface based on projection information stored in the projection information storage part 160 . Also, the second projection part 142 projects the site that is projected on the three-dimensional projection surface, onto the bottom plane of the three-dimensional space, according to the view position and direction for generating the first bird's eye view image.
  • the second bird's eye view image part generation 143 places the object on the site (second area) projected on the bottom plane of the three-dimensional space, and generates a second bird's eye view image, which is a three-dimensional bird's eye view image based on the above view position and direction.
  • the second bird's eye view image may be an image of the object based on the above view position and direction, not including the captured images transmitted from the imaging devices 111 to 114 .
  • the output part 150 generates a display image by superimposing the second bird's eye view image on the first bird's eye view image.
  • the output part 150 for example, prioritizes the second bird's eye view image and superimposes it on the first bird's eye view image.
  • the output part 150 can transmit a three-dimensional bird's eye view image, in which an object is placed, as a display image to the display device 182 , and display it on the display device 182 .
  • FIG. 2 is a diagram that illustrates an example arrangement of devices in a three-dimensional bird's eye view image generation system.
  • 200 a in FIG. 2 is a top view that shows an example arrangement of imaging devices in the vehicle 10 .
  • the imaging device 111 is placed in the center of the front of the vehicle 10 , and photographs an area in front of the vehicle 10 (the range indicated by a dotted line 201 ).
  • the imaging device 112 is placed in the center of the left side of the vehicle 10 and photographs an area in the left side of the vehicle 10 (the range indicated by a dotted line 202 ).
  • the imaging device 113 is placed in the center of the right side of the vehicle 10 and photographs an area in the right side of the vehicle 10 (the range indicated by a dotted line 203 ).
  • the imaging device 114 is placed in the center of the rear of the vehicle 10 and photographs an area in the rear side of the vehicle 10 (the range indicated by a dotted line 204 ). Note that the number and arrangement of imaging devices is not limited to what is shown in FIG. 2 . The number of imaging devices may be two or more, for example.
  • 200 b in FIG. 2 is a side view that shows an example arrangement of the image processing device 120 , the operating device 181 , and the display device 182 in the vehicle 10 .
  • the image processing device 120 is placed in the back of the center console in the vehicle 10 .
  • the operating device 181 and the display device 182 are placed on the center console of the vehicle 10 .
  • FIG. 3 is a diagram that illustrates example outer structures of the operating device and the display device.
  • the present embodiment assumes that the operating device 181 and the display device 182 are integrated together, configured as what is referred to as a “touch panel” 300 .
  • the touch panel 300 is, for example, embedded and installed in a center console 301 near the center position of the center console 301 .
  • FIG. 4 is a diagram that illustrates an example hardware structure of the image processing device.
  • the image processing device 120 includes a central processing unit (CPU) 401 , a read only memory (ROM) 402 , and a random access memory (RAM) 403 .
  • the CPU 401 , ROM 402 , and RAM 403 form what is referred to as a computer.
  • the image processing device 120 has a secondary memory device 404 , a connecting device 405 , a communicating device 406 , and a drive device 407 . Note that each piece of hardware in the image processing device 120 is connected with each other via a bus 408 .
  • the CPU 401 executes various programs (for example, an image processing program, etc.) installed in the secondary memory device 404 .
  • the ROM 402 is a non-volatile memory.
  • the ROM 402 functions as a primary memory device that stores various programs, data, and so forth, that the CPU 401 needs when executing various programs installed in the secondary memory device 404 .
  • the ROM 402 stores boot programs such as a basic input/output system (BIOS), an extensible firmware interface (EFI), and so forth.
  • BIOS basic input/output system
  • EFI extensible firmware interface
  • the RAM 403 is a volatile memory such as a dynamic random access memory (DRAM) or a static random access memory (SRAM).
  • the RAM 403 functions as a primary memory device that provides a work area in which various programs installed in the secondary memory device 404 are expanded when executed by the CPU 401 .
  • the secondary memory device 404 stores various installed programs and various information used when executing the programs.
  • the connecting device 405 is for transmitting and receiving signals between external devices (for example, the imaging devices 111 to 114 , the operating device 181 , and the display device 182 ) and the image processing device 120 .
  • the communicating device 406 allows communication with external devices via the CAN 170 .
  • the drive device 407 is a device for setting a computer-readable recording medium 410 .
  • the recording medium 410 here includes a medium for recording information optically, electrically, or magnetically, such as a CD-ROM, a flexible disk, or a magneto-optical disk.
  • the recording medium 410 may include, for example, a semiconductor memory that records information electrically, such as a ROM or a flash memory.
  • various programs may be installed in the secondary memory device 404 by, for example, setting a distributed recording medium 410 in the drive device 407 , and reading various programs recorded in the recording medium 410 by the drive device 407 .
  • various programs may be downloaded from the network via the communicating device 406 , and installed in the secondary memory device 404 .
  • FIG. 5 is a first diagram that illustrates a specific example of the process in the first generation part.
  • projection information 500 is an example of projection information stored in a projection information storage part 160 .
  • the projection information 500 includes “Imaging device” and “Three-dimensional projection surface coordinates” as information items.
  • Captured pixel coordinates stores the coordinates of each pixel of captured images 511 to 514 , photographed by the imaging device 111 to imaging device 114 and acquired by the captured image acquiring part 131 .
  • the captured images 511 to 514 all have n ⁇ m pixels.
  • “Captured pixel coordinates” for the imaging device 111 stores n ⁇ m coordinates of the captured image 511 , from (s 11 , t 11 ) to (s 1n , t 1m ).
  • “Captured pixel coordinates” for the imaging device 112 stores n ⁇ m coordinates of the captured image 512 , from (s 21 , t 21 ) to (s 2n , t 2m ).
  • “Captured pixel coordinates” for the imaging device 113 (Type 3) stores n ⁇ m coordinates of the captured image 513 , from (s 31 , t 31 ) to (s 3n , t 3m ).
  • “Captured pixel coordinates” of the imaging device 114 (type 4) stores n ⁇ m coordinates of the captured image 514 , from (s 41 , t 41 ) to (s 4n , t 4m ).
  • the first projection part 132 projects the captured pixel coordinates of the captured images 511 to 514 onto the three-dimensional projection surface 520 , and “Three-dimensional projection surface coordinates” stores these coordinates as projected on the three-dimensional projection surface 520 . Therefore, each coordinate stored in “Three-dimensional projection surface coordinates” is associated with a corresponding coordinate stored in “Captured pixel coordinates.”
  • the captured pixel coordinates (s 11 , t 11 ) of the captured image 511 are associated with the three-dimensional projection surface coordinates (x 11 , y 12 , z 11 ) on the three-dimensional projection surface 520 .
  • the captured pixel coordinates (s 21 , t 21 ) of the captured image 512 are associated with the three-dimensional projection surface coordinates (x 21 , y 22 , z 21 ) on the three-dimensional projection surface 520 .
  • the captured pixel coordinates (s 31 , t 31 ) of the captured image 513 are associated with the three-dimensional projection surface coordinates (x 31 , y 32 , z 31 ) on the three-dimensional projection surface 520 .
  • the captured pixel coordinates (s 41 , t 41 ) of the captured image 514 are associated with the three-dimensional projection surface coordinates (x 41 , y 42 , z 41 ) on the three-dimensional projection surface 520 .
  • the captured images 511 to 514 photographed by the imaging devices 111 to 114 , are acquired by the captured image acquiring part 131 of the image processing device 120 , and projected on the three-dimensional projection surface 520 by the first projection part 132 , based on the projection information 500 .
  • the captured images 511 to 514 projected on the three-dimensional projection surface 520 are combined in the first bird's eye view image generation part 133 , and the three-dimensional image 530 of the vehicle is incorporated therein, thereby generating the first bird's eye view image.
  • FIG. 6 is a second diagram that illustrates a specific example of the process in the first generation part.
  • 600 a in FIG. 6 is a top view of the three-dimensional projection surface 520 , and shows specific examples of the view position and direction that are used when the first bird's eye view image generation part 133 generates the first bird's eye view image, in the top view of the three-dimensional projection surface 520 .
  • an arrow 601 is the view position and direction used when generating the first bird's eye view image with a view from the front side of the vehicle.
  • an arrow 602 is the view position and direction used when generating the first bird's eye view image with a view from the left rear of the vehicle.
  • an arrow 603 is the view position and direction used when generating the first bird's eye view image with a view from the right rear of the vehicle.
  • 600 b in FIG. 6 shows an example of a first bird's eye view image 610 generated by the first bird's eye view image generation part 133 based on the view position and direction indicated by the arrow 602 .
  • the first bird's eye view image generation part 133 can generate a first bird's eye view image viewed diagonally from the rear left of the vehicle, which, for example, shows a three-dimensional image 530 of the vehicle on an area 611 , which shows the current position of the vehicle.
  • FIG. 7 and FIG. 8 are first and second diagrams that show specific examples of the process in the second generation part.
  • 7 a to 7 c and 8 a to 8 b each show coordinates in the three-dimensional space in which the vehicle 10 moves.
  • a site 700 (first area) is an area on which an object is to be placed.
  • the example of 7 a illustrates a situation in which the position information acquiring part 141 acquires, from the CAN 170 , (X a1 , Y a1 ), (X a1 , Y a2 ), (X a2 , Y a1 ), and (X a2 , Y a2 ) as position information of the site 700 on the bottom plane of the three-dimensional space.
  • position information 710 shows the position of the vehicle 10 in the three-dimensional space.
  • the example of 7 b illustrates a situation in which the position information acquiring part 141 acquires (X v , Y v ), from the CAN 170 , as position information of the vehicle 10 in the three-dimensional space.
  • a three-dimensional projection surface 720 is a three-dimensional projection surface that is generated at the position of the vehicle 10 .
  • the position information of the vehicle 10 may be used as the origin in the three-dimensional space, in which case (X v , Y v ) is defined as (0, 0).
  • an area 730 is an area of the three-dimensional projection surface 720 where the site 700 is projected.
  • the example of 7 c shows a situation in which the second projection part 142 projects the site 700 onto the three-dimensional projection surface 720 , and (x a1 , y a1 , z a1 ), (x a2 , y a2 , z a2 ), (x a3 , y a3 , z a3 ), and (x a4 , y a4 , z a4 ) are calculated as position information of the projected area 730 .
  • 8 a of FIG. 8 shows a site 810 (second area), in which the projected area 730 , projected on the three-dimensional projection surface 720 , is projected on the bottom plane of the three-dimensional space based on a predetermined view position and direction 820 .
  • the example of 8 a shows a situation in which the second projection part 142 projects the area 730 onto the bottom plane of the three-dimensional space, and (X′ a1 , Y′ a1 ), (X′ a1 , Y′ a2 ), (X′ a2 , Y′ a1 ), and (X′ a2 , Y′ a2 ) are calculated as position information of the projected site 810 .
  • an object 830 indicates an object that is placed on the projected site 810 .
  • the example of 8 b shows a situation in which the second bird's eye view image generation part 143 places a rectangular parallelepiped object 830 on the projected site 810 .
  • the second bird's eye view image generation part 143 can generate a second bird's eye view image (a bird's eye view image on which the object 830 is placed) that does not look unnatural with respect to the first bird's eye view image generated by the first bird's eye view image generation part 133 .
  • FIG. 9 is a first flowchart that shows a flow of the display image generation process, which is executed upon receipt of a command for generating a display image from the operating device 181 .
  • step S 901 the first generation part 130 acquires a three-dimensional image of the vehicle, which is used when generating the first bird's eye view image.
  • step S 902 the first generation part 130 acquires captured images photographed by the imaging devices 111 to 114 .
  • step S 903 the first generation part 130 projects the captured images onto a three-dimensional projection surface.
  • step S 904 the first generation part 130 generates the first bird's eye view image, in which a three-dimensional image of the vehicle is incorporated, based on a predetermined view position and direction.
  • step S 905 the second generation part 140 acquires, from the CAN 170 , position information of the site on which the object is to be placed, on the bottom plane of the three-dimensional space.
  • step S 906 the second generation part 140 acquires position information of the vehicle in the three-dimensional space, from the CAN 170 . Note that when, for example, the position of the vehicle is defined as the origin, it is possible to skip acquiring the position information of the vehicle.
  • step S 907 the second generation part 140 projects the site, on which the object is to be placed, onto a three-dimensional projection surface, at a position in accordance with the position information of the vehicle.
  • step S 908 the second generation part 140 projects the site, which is projected on the three-dimensional projection surface, onto the bottom plane of the three-dimensional space, based on the predetermined view position and direction.
  • step S 909 the second generation part 140 places the object on the site projected on the bottom plane of the three-dimensional space.
  • step S 910 the second generation part 140 generates a second bird's eye view image, in which the three-dimensional space where the object is placed is captured based on the predetermined view position and direction.
  • step S 911 the output part 150 generates a display image, which is a three-dimensional bird's eye view image, by superimposing the second bird's eye view image on the first bird's eye view image. Also, the output part 150 displays the display image generated thus, on the display device 182 .
  • FIG. 10 is a diagram that illustrates an example scene in which the three-dimensional bird's eye view image generation system is used, and shows a parking lot 1000 from above.
  • the solid-line quadrilaterals indicate vehicles that are parked, and the dotted-line quadrilaterals indicate parking areas where no vehicle is parked.
  • a dotted-line quadrilateral 1001 indicates a parking area where the vehicle 10 can be parked.
  • the three-dimensional bird's eye view image generation system 100 assumes that the parking area indicated by the dotted-line quadrilateral 1001 is the site where the object is to be placed, and acquires its position information in the three-dimensional space. Also, the three-dimensional bird's eye view image generation system 100 places a three-dimensional image of the vehicle, as the object, on the parking area indicated by the dotted-line quadrilateral 1001 . By this means, prior to parking, the driver of the vehicle 10 can visually check a three-dimensional bird's eye view image, in which his/her vehicle is parked in a parking area that is available, as a display image.
  • FIG. 11 is a diagram that illustrates a specific example of a display image.
  • a site 1110 corresponding to the parking area indicated by the dotted-line quadrilateral 1001 , can be displayed naturally in the display image 1100 .
  • a three-dimensional image 1120 of the vehicle can be displayed naturally in the display image 1100 .
  • the image processing device 120 according to the first embodiment:
  • FIG. 12 is a third diagram that illustrates a specific example of the process in the second generation part.
  • the difference here from FIG. 8 is that, in 12 a , a site 1210 , which is an area where the area 730 projected on the three-dimensional projection surface 720 is projected on the bottom plane of the three-dimensional space based on a predetermined view position and direction 820 , has a shape other than a rectangle.
  • the second bird's eye view image generation part 143 identifies an inscribed rectangle 1220 in the site 1210 , and places an object 1230 , included in the specified inscribed rectangle 1220 , within the inscribed rectangle 1220 in the site 1210 .
  • FIG. 12 shows a situation in which the object 1230 , included within the inscribed rectangle 1220 , is placed within the inscribed rectangle 1220 in the site 1210 .
  • the image processing device 120 may be configured to generate a display image every time captured images are acquired at predetermined intervals, and to display, sequentially, the display images generated thus.
  • a third embodiment will be described below, focusing on differences from the first embodiment described above.
  • FIG. 13 is a second flowchart that shows a flow of the display image generation process.
  • step S 1301 the first generation part 130 determines whether or not to end the display image generation process. If it is determined in step S 1301 not to end the process (NO in step S 1301 ), the process returns to step S 902 . By this means, the process from step S 902 to step S 912 is repeated, and a display image is generated every time captured images are acquired at predetermined intervals.
  • step S 1301 if it is determined to end the process in step S 1301 (YES in step S 1301 ), the display image generation process ends.
  • FIG. 14 is a fourth diagram that illustrates a specific example of the process in the second generation part. However, FIG. 14 also shows the positions of the vehicle 10 in the parking lot 1000 at different times.
  • FIG. 14 a in FIG. 14 shows a case in which the vehicle 10 is located far from the site 700 .
  • the three-dimensional projection surface 720 is generated at a position far from the site 700 in the three-dimensional space.
  • FIG. 14 a in FIG. 14 shows a case in which the vehicle 10 is located close to the site 700 .
  • the three-dimensional projection surface 720 is generated at a position close to the site 700 in the three-dimensional space.
  • the image processing device 120 can display a three-dimensional image of the vehicle (object), in the display image, even if the vehicle 10 moves.
  • the above embodiments have all been described assuming that the image processing device 120 is installed in the vehicle 10 .
  • the entity in which the image processing device 120 is installed is by no means limited to the vehicle 10 , and mobile entities other than the vehicle 10 may be applied as well.
  • the object to be placed is by no means limited to a three-dimensional image of a vehicle, and any three-dimensional image may be used.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Multimedia (AREA)
  • General Physics & Mathematics (AREA)
  • Signal Processing (AREA)
  • Computing Systems (AREA)
  • Geometry (AREA)
  • Computer Graphics (AREA)
  • Mechanical Engineering (AREA)
  • Closed-Circuit Television Systems (AREA)
  • Image Processing (AREA)

Abstract

An image processing device is configured to generate a three-dimensional bird's eye view image by projecting captured images acquired from a plurality of imaging devices onto a three-dimensional projection surface, and includes: an acquiring part configured to acquire position information of a first area, on which an object is to be placed, in a three-dimensional space; a conversion part configured to convert the position information of the first area into position information in accordance with the three-dimensional projection surface; and an output part configured to output the three-dimensional bird's eye view image, on which the object is to be placed on a second area in the three-dimensional space, the second area being specified by the converted position information. By this means, when the three-dimensional bird's eye view image is generated from multiple images, it is possible to display the object naturally in the three-dimensional bird's eye view image.

Description

CROSS-REFERENCE TO RELATED APPLICATION
The present application is a continuation application filed under 35 U.S.C. 111(a) claiming the benefit under 35 U.S.C. 120 and 365(c) of PCT International Application No. PCT/JP 2021/019968, filed on May 26, 2021, and designating the U.S. The entire contents of PCT International Application No. PCT/JP2021/019968 are incorporated herein by reference.
BACKGROUND OF THE INVENTION 1. Field of the Invention
The present disclosure relates to an image processing device, an image processing method, and an image processing program.
2. Description of the Related Art
There is a three-dimensional bird's eye view image generation system that photographs the surroundings of a mobile entity such as a vehicle by using multiple imaging devices, and that generates a three-dimensional bird's eye view image viewing the whole surroundings of the mobile entity from above by projecting multiple photographed images onto a three-dimensional projection surface by using projection information.
According to the three-dimensional bird's eye view image generation system, for example, when a driver is parking his/her vehicle, it is possible to assist the driver's driving operation by providing the driver with a three-dimensional bird's eye view image of the vehicle.
CITATION LIST Patent Document
    • Patent Document 1: Unexamined Japanese Patent Application Publication No. 2009-96347
    • Patent Document 2: Unexamined Japanese Patent Application Publication No. 2009-232310
    • Patent Document 3: Unexamined Japanese Patent Application Publication No. 2009-265870
    • Patent Document 4: Unexamined Japanese Patent Application Publication No. 2010-268369
    • Patent Document 5: Unexamined Japanese Patent Application Publication No. 2021-27366
SUMMARY OF INVENTION Technical Problem
Here, in the event the above system is used, while it is possible to display an object that is included in the multiple photographed images in the three-dimensional bird's eye view image, it is still difficult to display a new object in the three-dimensional bird's eye view image naturally.
At least one aspect of the present disclosure therefore aims to display an object naturally when generating a three-dimensional bird's eye view image from multiple images.
Solution to Problem
According to at least one aspect of the present disclosure, an image processing device is configured to generate a three-dimensional bird's eye view image by projecting captured images acquired from a plurality of imaging devices onto a three-dimensional projection surface, and this image processing device includes: an acquiring part configured to acquire position information of a first area, in which an object is to be placed, in a three-dimensional space; a conversion part configured to convert the position information of the first area into position information in accordance with the three-dimensional projection surface; and an output part configured to output the three-dimensional bird's eye view image, on which the object is to be placed on a second area in the three-dimensional space, the second area being specified by the converted position information.
Advantageous Effects of Invention
According to at least one aspect of the present disclosure, it is possible to display an object naturally when generating a three-dimensional bird's eye view image from multiple images.
BRIEF DESCRIPTION OF DRAWINGS
FIG. 1 is a diagram that illustrates an example system structure of a three-dimensional bird's eye view image generation system and an example functional structure of an image processing device;
FIG. 2 is a diagram that illustrates an example arrangement of devices in a three-dimensional bird's eye view image generation system;
FIG. 3 is a diagram that illustrates example outer structures of an operating device and a display device;
FIG. 4 is a diagram that illustrates an example hardware structure of an image processing device;
FIG. 5 is a first diagram that illustrates a specific example of a process in a first generation part;
FIG. 6 is a second diagram that illustrates a specific example of the process in the first generation part;
FIG. 7 is a first diagram that illustrates a specific example of a process in a second generation part;
FIG. 8 is a second diagram that illustrates a specific example of the process in the second generation part;
FIG. 9 is a first flowchart that shows a flow of a display image generation process;
FIG. 10 is a diagram that illustrates an example of a scene in which the three-dimensional bird's eye view image generation system is used;
FIG. 11 is a diagram that illustrates a specific example of a display image;
FIG. 12 is a third diagram that illustrates a specific example of the process in the second generation part;
FIG. 13 is a second flowchart that shows a flow of a display image generation process; and
FIG. 14 is a fourth diagram that illustrates a specific example of the process in the second generation part.
 DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
Now, embodiments of the present disclosure will be described below with reference to the accompanying drawings. Note that, in this specification and the accompanying drawings, components having substantially the same functional structure are designated by the same reference codes and redundant explanation will be omitted.
First Embodiment
<Three-Dimensional Bird's Eye View Image Generation System and Image Processing Device>
First, a system structure of a three-dimensional bird's eye view image generation system installed in a mobile entity such as a vehicle, and a functional structure of an image processing device included in the three-dimensional bird's eye view image generation system will be described. FIG. 1 is a diagram that illustrates an example system structure of the three-dimensional bird's eye view image generation system and an example functional structure of the image processing device.
The example system structure of FIG. 1 shows a system structure in which a three-dimensional bird's eye view image generation system 100 is mounted on a vehicle 10. As shown in FIG. 1 , the three-dimensional bird's eye view image generation system 100 mounted on the vehicle 10 includes imaging devices 111 to 114, an image processing device 120, a controller area network (CAN) 170, an operating device 181, and a display device 182.
The imaging devices 111 to 114 generate captured images by photographing the front, left, right, and rear sides of the vehicle 10, respectively, and send the thus-generated captured images to the image processing device 120.
An image processing program is installed in the image processing device 120, and, when this program is executed, the image processing device 120 functions as a first generation part 130, a second generation part 140, and an output part 150.
The first generation part 130 generates a first bird's eye view image, which is a three-dimensional bird's eye view image around the vehicle 10. To implement this function, the first generation part 130 includes a captured image acquiring part 131, a first projection part 132, and a first bird's eye view image generation part 133.
When the captured image acquiring part 131 receives, from the operating device 181, command information including a command for generating a display image, the captured image acquiring part 131 acquires the captured images transmitted from the imaging devices 111 to 114.
The first projection part 132 projects each captured image acquired by the captured image acquiring part 131 onto a three-dimensional projection surface, based on projection information stored in a projection information storage part 160.
The first bird's eye view image generation part 133 generates a first bird's eye view image in a predetermined view position and direction based on each captured image projected on the three-dimensional projection surface.
The second generation part 140 generates a second bird's eye view image, which is a three-dimensional bird's eye view image in which an object is to be placed. To implement this function, the second generation part 140 includes a position information acquiring part 141, a second projection part 142, and a second bird's eye view image generation part 143. Note that the object included in the second bird's eye view image may be one that is not included in the captured images transmitted from the imaging devices 111 to 114.
The position information acquiring part 141 acquires position information of the vehicle 10 in a three-dimensional space from the CAN 170. Also, the position information acquiring part 141 acquires position information of the site (first area), on which the object is to be placed, in the three-dimensional space.
The second projection part 142 is an example of a conversion part, and projects the site, on which the object is to be placed, onto a three-dimensional projection surface based on projection information stored in the projection information storage part 160. Also, the second projection part 142 projects the site that is projected on the three-dimensional projection surface, onto the bottom plane of the three-dimensional space, according to the view position and direction for generating the first bird's eye view image.
The second bird's eye view image part generation 143 places the object on the site (second area) projected on the bottom plane of the three-dimensional space, and generates a second bird's eye view image, which is a three-dimensional bird's eye view image based on the above view position and direction. Note that the second bird's eye view image may be an image of the object based on the above view position and direction, not including the captured images transmitted from the imaging devices 111 to 114.
The output part 150 generates a display image by superimposing the second bird's eye view image on the first bird's eye view image. When generating the display image, the output part 150, for example, prioritizes the second bird's eye view image and superimposes it on the first bird's eye view image. By this means, the output part 150 can transmit a three-dimensional bird's eye view image, in which an object is placed, as a display image to the display device 182, and display it on the display device 182.
Hereinafter, each device in the three-dimensional bird's eye view image generation system 100, and the hardware structure and each functional part of the image processing device 120 will be described in detail in order.
<Details of Each Device in the Three-Dimensional Bird's Eye View Image Generation System>
First, an example arrangement and outer structures of devices in the three-dimensional bird's eye view image generation system 100 will be described.
(1) Example of Arrangement
FIG. 2 is a diagram that illustrates an example arrangement of devices in a three-dimensional bird's eye view image generation system. In particular, 200 a in FIG. 2 is a top view that shows an example arrangement of imaging devices in the vehicle 10.
As shown in 200 a in FIG. 2 , the imaging device 111 is placed in the center of the front of the vehicle 10, and photographs an area in front of the vehicle 10 (the range indicated by a dotted line 201). The imaging device 112 is placed in the center of the left side of the vehicle 10 and photographs an area in the left side of the vehicle 10 (the range indicated by a dotted line 202). The imaging device 113 is placed in the center of the right side of the vehicle 10 and photographs an area in the right side of the vehicle 10 (the range indicated by a dotted line 203). The imaging device 114 is placed in the center of the rear of the vehicle 10 and photographs an area in the rear side of the vehicle 10 (the range indicated by a dotted line 204). Note that the number and arrangement of imaging devices is not limited to what is shown in FIG. 2 . The number of imaging devices may be two or more, for example.
200 b in FIG. 2 is a side view that shows an example arrangement of the image processing device 120, the operating device 181, and the display device 182 in the vehicle 10. As shown in 200 b of FIG. 2 , the image processing device 120 is placed in the back of the center console in the vehicle 10. Also, the operating device 181 and the display device 182 are placed on the center console of the vehicle 10.
(2) Outer Structures of the Operating Device and Display Device
Next, outer structures of the operating device 181 and the display device 182 placed on the center console of the vehicle 10 will be described. FIG. 3 is a diagram that illustrates example outer structures of the operating device and the display device. The present embodiment assumes that the operating device 181 and the display device 182 are integrated together, configured as what is referred to as a “touch panel” 300. As shown in FIG. 3 , the touch panel 300 is, for example, embedded and installed in a center console 301 near the center position of the center console 301.
<Hardware Structure of Image Processing Device>
Next, the hardware structure of the image processing device 120 will be described. FIG. 4 is a diagram that illustrates an example hardware structure of the image processing device.
As shown in FIG. 4 , the image processing device 120 includes a central processing unit (CPU) 401, a read only memory (ROM) 402, and a random access memory (RAM) 403. The CPU 401, ROM 402, and RAM 403 form what is referred to as a computer. Also, the image processing device 120 has a secondary memory device 404, a connecting device 405, a communicating device 406, and a drive device 407. Note that each piece of hardware in the image processing device 120 is connected with each other via a bus 408.
The CPU 401 executes various programs (for example, an image processing program, etc.) installed in the secondary memory device 404.
The ROM 402 is a non-volatile memory. The ROM 402 functions as a primary memory device that stores various programs, data, and so forth, that the CPU 401 needs when executing various programs installed in the secondary memory device 404. To be more specific, the ROM 402 stores boot programs such as a basic input/output system (BIOS), an extensible firmware interface (EFI), and so forth.
The RAM 403 is a volatile memory such as a dynamic random access memory (DRAM) or a static random access memory (SRAM). The RAM 403 functions as a primary memory device that provides a work area in which various programs installed in the secondary memory device 404 are expanded when executed by the CPU 401.
The secondary memory device 404 stores various installed programs and various information used when executing the programs.
The connecting device 405 is for transmitting and receiving signals between external devices (for example, the imaging devices 111 to 114, the operating device 181, and the display device 182) and the image processing device 120.
The communicating device 406 allows communication with external devices via the CAN 170.
The drive device 407 is a device for setting a computer-readable recording medium 410. The recording medium 410 here includes a medium for recording information optically, electrically, or magnetically, such as a CD-ROM, a flexible disk, or a magneto-optical disk. Alternatively, the recording medium 410 may include, for example, a semiconductor memory that records information electrically, such as a ROM or a flash memory.
Note that various programs may be installed in the secondary memory device 404 by, for example, setting a distributed recording medium 410 in the drive device 407, and reading various programs recorded in the recording medium 410 by the drive device 407. Alternatively, various programs may be downloaded from the network via the communicating device 406, and installed in the secondary memory device 404.
<Details of Each Functional Part of the Image Processing Device>
Next, each functional part (here, the first generation part 130 and the second generation part 140) of the image processing device 120 will be described based on specific examples.
(1) Specific Example of the Process in the First Generation Part
First, a specific example of the process in the first generation part 130 will be described with reference to FIG. 5 and FIG. 6 . FIG. 5 is a first diagram that illustrates a specific example of the process in the first generation part. In FIG. 5 , projection information 500 is an example of projection information stored in a projection information storage part 160. As shown in FIG. 5 , the projection information 500 includes “Imaging device” and “Three-dimensional projection surface coordinates” as information items.
“Imaging device” further includes “Type” and “Captured pixel coordinates.” “Type” stores information for identifying between the imaging devices 111 to 114. That is, in the projection information 500, type=1 is the imaging device 111, type=2 is the imaging device 112, type=3 is the imaging device 113, and type=4 is the imaging device 114.
“Captured pixel coordinates” stores the coordinates of each pixel of captured images 511 to 514, photographed by the imaging device 111 to imaging device 114 and acquired by the captured image acquiring part 131. In the present embodiment, the captured images 511 to 514 all have n×m pixels.
Therefore, in the projection information 500, for example, “Captured pixel coordinates” for the imaging device 111 (Type 1) stores n×m coordinates of the captured image 511, from (s11, t11) to (s1n, t1m). Similarly, “Captured pixel coordinates” for the imaging device 112 (Type 2) stores n×m coordinates of the captured image 512, from (s21, t21) to (s2n, t2m). Also, “Captured pixel coordinates” for the imaging device 113 (Type 3) stores n×m coordinates of the captured image 513, from (s31, t31) to (s3n, t3m). Furthermore, “Captured pixel coordinates” of the imaging device 114 (type 4) stores n×m coordinates of the captured image 514, from (s41, t41) to (s4n, t4m).
The first projection part 132 projects the captured pixel coordinates of the captured images 511 to 514 onto the three-dimensional projection surface 520, and “Three-dimensional projection surface coordinates” stores these coordinates as projected on the three-dimensional projection surface 520. Therefore, each coordinate stored in “Three-dimensional projection surface coordinates” is associated with a corresponding coordinate stored in “Captured pixel coordinates.”
For example, in the case of the projection information 500, the captured pixel coordinates (s11, t11) of the captured image 511 are associated with the three-dimensional projection surface coordinates (x11, y12, z11) on the three-dimensional projection surface 520. Similarly, the captured pixel coordinates (s21, t21) of the captured image 512 are associated with the three-dimensional projection surface coordinates (x21, y22, z21) on the three-dimensional projection surface 520. Similarly, the captured pixel coordinates (s31, t31) of the captured image 513 are associated with the three-dimensional projection surface coordinates (x31, y32, z31) on the three-dimensional projection surface 520. Similarly, the captured pixel coordinates (s41, t41) of the captured image 514 are associated with the three-dimensional projection surface coordinates (x41, y42, z41) on the three-dimensional projection surface 520.
The captured images 511 to 514, photographed by the imaging devices 111 to 114, are acquired by the captured image acquiring part 131 of the image processing device 120, and projected on the three-dimensional projection surface 520 by the first projection part 132, based on the projection information 500.
The captured images 511 to 514 projected on the three-dimensional projection surface 520 are combined in the first bird's eye view image generation part 133, and the three-dimensional image 530 of the vehicle is incorporated therein, thereby generating the first bird's eye view image.
FIG. 6 is a second diagram that illustrates a specific example of the process in the first generation part. In particular, 600 a in FIG. 6 is a top view of the three-dimensional projection surface 520, and shows specific examples of the view position and direction that are used when the first bird's eye view image generation part 133 generates the first bird's eye view image, in the top view of the three-dimensional projection surface 520.
In 600 a, for example, an arrow 601 is the view position and direction used when generating the first bird's eye view image with a view from the front side of the vehicle. Also, an arrow 602 is the view position and direction used when generating the first bird's eye view image with a view from the left rear of the vehicle. Furthermore, an arrow 603 is the view position and direction used when generating the first bird's eye view image with a view from the right rear of the vehicle.
600 b in FIG. 6 shows an example of a first bird's eye view image 610 generated by the first bird's eye view image generation part 133 based on the view position and direction indicated by the arrow 602. As shown in 600 b of FIG. 6 , the first bird's eye view image generation part 133 can generate a first bird's eye view image viewed diagonally from the rear left of the vehicle, which, for example, shows a three-dimensional image 530 of the vehicle on an area 611, which shows the current position of the vehicle.
(2) Specific Example of the Process in the Second Generation Part
Next, a specific example of the process in the second generation part 140 will be described. FIG. 7 and FIG. 8 are first and second diagrams that show specific examples of the process in the second generation part. In FIG. 7 and FIGS. 8, 7 a to 7 c and 8 a to 8 b each show coordinates in the three-dimensional space in which the vehicle 10 moves.
In 7 a of FIG. 7 , a site 700 (first area) is an area on which an object is to be placed. The example of 7 a illustrates a situation in which the position information acquiring part 141 acquires, from the CAN 170, (Xa1, Ya1), (Xa1, Ya2), (Xa2, Ya1), and (Xa2, Ya2) as position information of the site 700 on the bottom plane of the three-dimensional space.
In 7 b of FIG. 7 , position information 710 shows the position of the vehicle 10 in the three-dimensional space. The example of 7 b illustrates a situation in which the position information acquiring part 141 acquires (Xv, Yv), from the CAN 170, as position information of the vehicle 10 in the three-dimensional space. Note that, in 7 b, a three-dimensional projection surface 720 is a three-dimensional projection surface that is generated at the position of the vehicle 10. In the example of 7 b, the position information of the vehicle 10 may be used as the origin in the three-dimensional space, in which case (Xv, Yv) is defined as (0, 0).
In 7 c of FIG. 7 , an area 730 is an area of the three-dimensional projection surface 720 where the site 700 is projected. The example of 7 c shows a situation in which the second projection part 142 projects the site 700 onto the three-dimensional projection surface 720, and (xa1, ya1, za1), (xa2, ya2, za2), (xa3, ya3, za3), and (xa4, ya4, za4) are calculated as position information of the projected area 730.
Next, 8 a of FIG. 8 shows a site 810 (second area), in which the projected area 730, projected on the three-dimensional projection surface 720, is projected on the bottom plane of the three-dimensional space based on a predetermined view position and direction 820. The example of 8 a shows a situation in which the second projection part 142 projects the area 730 onto the bottom plane of the three-dimensional space, and (X′a1, Y′a1), (X′a1, Y′a2), (X′a2, Y′a1), and (X′a2, Y′a2) are calculated as position information of the projected site 810.
In 8 b of FIG. 8 , an object 830 indicates an object that is placed on the projected site 810. The example of 8 b shows a situation in which the second bird's eye view image generation part 143 places a rectangular parallelepiped object 830 on the projected site 810.
By this means, the second bird's eye view image generation part 143 can generate a second bird's eye view image (a bird's eye view image on which the object 830 is placed) that does not look unnatural with respect to the first bird's eye view image generated by the first bird's eye view image generation part 133.
<Flow of the Display Image Generation Process>
Next, the flow of the display image generation process in the image processing device 120 will be described. FIG. 9 is a first flowchart that shows a flow of the display image generation process, which is executed upon receipt of a command for generating a display image from the operating device 181.
In step S901, the first generation part 130 acquires a three-dimensional image of the vehicle, which is used when generating the first bird's eye view image.
In step S902, the first generation part 130 acquires captured images photographed by the imaging devices 111 to 114.
In step S903, the first generation part 130 projects the captured images onto a three-dimensional projection surface.
In step S904, the first generation part 130 generates the first bird's eye view image, in which a three-dimensional image of the vehicle is incorporated, based on a predetermined view position and direction.
In step S905, the second generation part 140 acquires, from the CAN 170, position information of the site on which the object is to be placed, on the bottom plane of the three-dimensional space.
In step S906, the second generation part 140 acquires position information of the vehicle in the three-dimensional space, from the CAN 170. Note that when, for example, the position of the vehicle is defined as the origin, it is possible to skip acquiring the position information of the vehicle.
In step S907, the second generation part 140 projects the site, on which the object is to be placed, onto a three-dimensional projection surface, at a position in accordance with the position information of the vehicle.
In step S908, the second generation part 140 projects the site, which is projected on the three-dimensional projection surface, onto the bottom plane of the three-dimensional space, based on the predetermined view position and direction.
In step S909, the second generation part 140 places the object on the site projected on the bottom plane of the three-dimensional space.
In step S910, the second generation part 140 generates a second bird's eye view image, in which the three-dimensional space where the object is placed is captured based on the predetermined view position and direction.
In step S911, the output part 150 generates a display image, which is a three-dimensional bird's eye view image, by superimposing the second bird's eye view image on the first bird's eye view image. Also, the output part 150 displays the display image generated thus, on the display device 182.
<Scene in which Three-Dimensional Bird's Eye View Image Generation System is Used>
Next, a scene in which the three-dimensional bird's eye view image generation system 100 is used will be described. FIG. 10 is a diagram that illustrates an example scene in which the three-dimensional bird's eye view image generation system is used, and shows a parking lot 1000 from above.
In FIG. 10 , the solid-line quadrilaterals indicate vehicles that are parked, and the dotted-line quadrilaterals indicate parking areas where no vehicle is parked. Among these, a dotted-line quadrilateral 1001 indicates a parking area where the vehicle 10 can be parked.
In such a scene of use, the three-dimensional bird's eye view image generation system 100 assumes that the parking area indicated by the dotted-line quadrilateral 1001 is the site where the object is to be placed, and acquires its position information in the three-dimensional space. Also, the three-dimensional bird's eye view image generation system 100 places a three-dimensional image of the vehicle, as the object, on the parking area indicated by the dotted-line quadrilateral 1001. By this means, prior to parking, the driver of the vehicle 10 can visually check a three-dimensional bird's eye view image, in which his/her vehicle is parked in a parking area that is available, as a display image.
FIG. 11 is a diagram that illustrates a specific example of a display image. As shown in FIG. 11 , according to the three-dimensional bird's eye view image generation system 100, a site 1110, corresponding to the parking area indicated by the dotted-line quadrilateral 1001, can be displayed naturally in the display image 1100. Also, as shown in FIG. 11 , according to the three-dimensional bird's eye view image generation system 100, a three-dimensional image 1120 of the vehicle can be displayed naturally in the display image 1100.
Note that in the display image 1100, only the area 611, which indicates the current position of the vehicle, is shown in the first bird's eye view image looking down on the vehicle from the side, and the three-dimensional image 530 of the vehicle on this area 611 is omitted.
SUMMARY
As is clear from the above description, the image processing device 120 according to the first embodiment:
    • generates a first bird's eye view image by projecting captured images acquired from multiple imaging devices onto a three-dimensional projection surface;
    • acquires position information of a site, on which an object to be placed, in the three-dimensional space;
    • converts the position information of the site into position information in accordance with the three-dimensional projection surface; and
    • generates and outputs a display image by superimposing a second bird's eye view image, on which the object is to be placed on the site in the three-dimensional space specified by the converted position information, on the first bird's eye view image.
By this means, according to the image processing device 120 of the first embodiment, when a three-dimensional bird's eye view image is generated from multiple captured images, an object can be displayed naturally.
Second Embodiment
A case has been described with the above first embodiment where the second projection part 142 projects a site of a rectangular shape onto the bottom plane of a three-dimensional space. However, the shape of the site that is projected need not be always rectangular. So, a case will be described below with a second embodiment where the site that is projected has a shape other than a rectangular shape.
FIG. 12 is a third diagram that illustrates a specific example of the process in the second generation part. The difference here from FIG. 8 is that, in 12 a, a site 1210, which is an area where the area 730 projected on the three-dimensional projection surface 720 is projected on the bottom plane of the three-dimensional space based on a predetermined view position and direction 820, has a shape other than a rectangle.
In this case, the second bird's eye view image generation part 143 identifies an inscribed rectangle 1220 in the site 1210, and places an object 1230, included in the specified inscribed rectangle 1220, within the inscribed rectangle 1220 in the site 1210.
12 b in FIG. 12 shows a situation in which the object 1230, included within the inscribed rectangle 1220, is placed within the inscribed rectangle 1220 in the site 1210. By this means, with the second bird's eye view image generation part 143, it is possible to avoid a situation in which the object looks distorted in the second bird's eye view image.
Third Embodiment
A case has been described with the above first embodiment where the display image 1100 is generated once. However, the image processing device 120 may be configured to generate a display image every time captured images are acquired at predetermined intervals, and to display, sequentially, the display images generated thus. A third embodiment will be described below, focusing on differences from the first embodiment described above.
<Flow of the Display Image Generation Process>
First, the flow of the display image generation process in the image processing device 120 will be described. FIG. 13 is a second flowchart that shows a flow of the display image generation process. The difference from the first flowchart described earlier with reference to FIG. 9 lies in step S1301.
In step S1301, the first generation part 130 determines whether or not to end the display image generation process. If it is determined in step S1301 not to end the process (NO in step S1301), the process returns to step S902. By this means, the process from step S902 to step S912 is repeated, and a display image is generated every time captured images are acquired at predetermined intervals.
Note that if it is determined to end the process in step S1301 (YES in step S1301), the display image generation process ends.
Specific Example of the Process in the Second Generation Part
Next, a specific example of the process in the second generation part 140 will be described. FIG. 14 is a fourth diagram that illustrates a specific example of the process in the second generation part. However, FIG. 14 also shows the positions of the vehicle 10 in the parking lot 1000 at different times.
14 a in FIG. 14 shows a case in which the vehicle 10 is located far from the site 700. In this case, the three-dimensional projection surface 720 is generated at a position far from the site 700 in the three-dimensional space.
14 a in FIG. 14 shows a case in which the vehicle 10 is located close to the site 700. In this case, the three-dimensional projection surface 720 is generated at a position close to the site 700 in the three-dimensional space.
In this way, as the vehicle 10 moves, the relationship between the relative positions of the site 700 and the three-dimensional projection surface 720 changes; accordingly, the position information of the site projected on the three-dimensional projection surface also changes. As a result of this, the site projected in the three-dimensional space also changes.
In other words, by generating a display image every time captured images are acquired, the image processing device 120 can display a three-dimensional image of the vehicle (object), in the display image, even if the vehicle 10 moves.
Other Embodiments
The above embodiments have all been described assuming that the image processing device 120 is installed in the vehicle 10. However, the entity in which the image processing device 120 is installed is by no means limited to the vehicle 10, and mobile entities other than the vehicle 10 may be applied as well.
Also, although the above embodiments have all been described with reference to cases in which a three-dimensional image of a vehicle is placed as an object, the object to be placed is by no means limited to a three-dimensional image of a vehicle, and any three-dimensional image may be used.
Note that the present invention is by no means limited to the structures or configurations described with the above embodiments or combinations thereof with other elements. The present invention therefore can be modified within a range not departing from the spirit of the present invention, and their mode of use/application can be determined as appropriate.

Claims (7)

The invention claimed is:
1. An image processing device configured to generate a three-dimensional bird's eye view image by projecting captured images acquired from a plurality of imaging devices onto a three-dimensional projection surface, the image processing device comprising:
an acquirer configured to acquire position information of a first area, on which an object is to be placed, in a three-dimensional space;
a converter configured to:
project the first area onto the three-dimensional projection surface, and
convert the position information of the first area into the converted position information in accordance with the three-dimensional projection surface by projecting the first area projected on the three-dimensional projection surface onto a second area in the three-dimensional space, based on a view position for generating the three-dimensional bird's eye view image, the second area being specified by the converted position information;
a first bird's eye view image generator configured to generate a first bird's eye view image based on the view position, by projecting the captured images acquired from the plurality of imaging devices onto the three-dimensional projection surface;
a second bird's eye view image generator configured to place the object within the second area in the three-dimensional space, and generate a second bird's eye view image based on the view position; and
a display configured to output the three-dimensional bird's eye view image, on which the object is placed within the second area in the three-dimensional space by prioritizing the second bird's eye view image and superimposing the second bird's eye view image on the first bird's eye view image,
wherein the imaging devices are mounted on a vehicle, and
the object is located around a three-dimensional image of the vehicle.
2. The image processing device according to claim 1, wherein the object is placed within a quadrilateral inscribed in the second area.
3. The image processing device according to claim 1, wherein the first bird's eye view image of the vehicle is generated by projecting the captured images acquired from the plurality of imaging devices, which are mounted on the vehicle and photograph areas in front, rear, and sides of the vehicle, onto the three-dimensional projection surface.
4. The image processing device according to claim 3, wherein the first area is a parking area where the vehicle can be parked.
5. The image processing device according to claim 3,
wherein, every time captured images are acquired from the plurality of imaging devices at predetermined intervals, the first bird's eye view image generator generates the first bird's eye view image by projecting the captured images onto the three-dimensional projection surface in accordance with position information of the vehicle, and
wherein, every time the first area projected on the three-dimensional projection surface in accordance with the position information of the vehicle is projected on the second area in the three-dimensional space, the second bird's eye view image generator places the object on the second area and generates the second bird's eye view image.
6. An image processing method that can be executed on an image processing device configured to generate a three-dimensional bird's eye view image by projecting captured images acquired from a plurality of imaging devices onto a three-dimensional projection surface, the image processing method comprising:
acquiring position information of a first area, on which an object is to be placed, in a three-dimensional space;
projecting the first area onto the three-dimensional projection surface;
converting the position information of the first area into the converted position information in accordance with the three-dimensional projection surface by projecting the first area projected on the three-dimensional projection surface onto a second area in the three-dimensional space, based on a view position for generating the three-dimensional bird's eye view image, the second area being specified by the converted position information;
generating a first bird's eye view image based on the view position, by projecting the captured images acquired from the plurality of imaging devices onto the three-dimensional projection surface;
generating a second bird's eye view image based on the view position by placing the object within the second area in the three-dimensional space; and
outputting the three-dimensional bird's eye view image, on which the object is placed within the second area in the three-dimensional space by prioritizing the second bird's eye view image and superimposing the second bird's eye view image on the first bird's eye view image,
wherein the imaging devices are mounted on a vehicle, and
the object is located around a three-dimensional image of the vehicle.
7. A non-transitory computer-readable recording medium storing an image processing program that, when executed on a computer of an image processing device configured to generate a three-dimensional bird's eye view image by projecting captured images acquired from a plurality of imaging devices onto a three-dimensional projection surface, causes the computer to:
acquire position information of a first area, on which an object is to be placed, in a three-dimensional space;
project the first area onto the three-dimensional projection surface;
convert the position information of the first area into the converted position information in accordance with the three-dimensional projection surface by projecting the first area projected on the three-dimensional projection surface onto a second area in the three-dimensional space, based on a view position for generating the three-dimensional bird's eye view image, the second area being specified by the converted position information;
generate a first bird's eye view image based on the view position, by projecting the captured images acquired from the plurality of imaging devices onto the three-dimensional projection surface;
generate a second bird's eye view image based on the view position by placing the object within the second area in the three-dimensional space; and
output the three-dimensional bird's eye view image, on which the object is placed within the second area in the three-dimensional space by prioritizing the second bird's eye view image and superimposing the second bird's eye view image on the first bird's eye view image,
wherein the imaging devices are mounted on a vehicle, and
the object is located around a three-dimensional image of the vehicle.
US18/516,334 2021-05-26 2023-11-21 Image processing device, image processing method, and image processing program Active 2041-11-25 US12482170B2 (en)

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/JP2021/019968 WO2022249318A1 (en) 2021-05-26 2021-05-26 Image processing device, image processing method, and image processing program

Related Parent Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2021/019968 Continuation WO2022249318A1 (en) 2021-05-26 2021-05-26 Image processing device, image processing method, and image processing program

Publications (2)

Publication Number Publication Date
US20240096006A1 US20240096006A1 (en) 2024-03-21
US12482170B2 true US12482170B2 (en) 2025-11-25

Family

ID=84228640

Family Applications (1)

Application Number Title Priority Date Filing Date
US18/516,334 Active 2041-11-25 US12482170B2 (en) 2021-05-26 2023-11-21 Image processing device, image processing method, and image processing program

Country Status (4)

Country Link
US (1) US12482170B2 (en)
JP (1) JP7677412B2 (en)
CN (1) CN117356087A (en)
WO (1) WO2022249318A1 (en)

Citations (20)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2009096347A (en) 2007-10-17 2009-05-07 Denso Corp Parking assistance device
JP2009232310A (en) 2008-03-25 2009-10-08 Fujitsu Ltd Image processor for vehicle, image processing method for vehicle, image processing program for vehicle
JP2009265870A (en) 2008-04-24 2009-11-12 Alpine Electronics Inc Driving support device and its display method
US20090309970A1 (en) * 2008-06-04 2009-12-17 Sanyo Electric Co., Ltd. Vehicle Operation System And Vehicle Operation Method
US20100245573A1 (en) * 2009-03-25 2010-09-30 Fujitsu Limited Image processing method and image processing apparatus
JP2010268369A (en) 2009-05-18 2010-11-25 Konica Minolta Opto Inc Peripheral display device
US20120069153A1 (en) * 2009-05-25 2012-03-22 Panasonic Corporation Device for monitoring area around vehicle
JP2015015527A (en) 2013-07-03 2015-01-22 クラリオン株式会社 Video display system, video synthesis device and video synthesis method
WO2019038901A1 (en) * 2017-08-25 2019-02-28 株式会社ソシオネクスト Correction device, correction program, and recording medium
US20190215465A1 (en) * 2017-02-28 2019-07-11 JVC Kenwood Corporation Bird's-eye view image generating device, bird's-eye view image generating system, bird's-eye view image generating method, and medium
JP2019160118A (en) 2018-03-16 2019-09-19 株式会社デンソーテン Information provider and method for providing information
US20200031283A1 (en) * 2016-09-30 2020-01-30 Aisin Seiki Kabushiki Kaisha Periphery monitoring apparatus
US20200139888A1 (en) * 2018-09-06 2020-05-07 Aisin Seiki Kabushiki Kaisha Vehicle periphery display device
JP6730619B2 (en) * 2017-05-17 2020-07-29 株式会社Ｊｖｃケンウッド Overhead video generation device, overhead video display device, overhead video generation method and program
JP2020127171A (en) 2019-02-06 2020-08-20 アイシン精機株式会社 Perimeter monitoring device
US20200404191A1 (en) * 2018-03-05 2020-12-24 Denso Corporation Image creation device and image creation method
JP2021027366A (en) 2019-07-31 2021-02-22 株式会社デンソーテン Image generating apparatus
US20210107465A1 (en) * 2019-10-11 2021-04-15 Toyota Jidosha Kabushiki Kaisha Parking assist apparatus
US20210107511A1 (en) * 2019-10-11 2021-04-15 Toyota Jidosha Kabushiki Kaisha Parking assist apparatus
US20210188289A1 (en) * 2018-09-13 2021-06-24 Sony Semiconductor Solutions Corporation Information processing device, moving apparatus, method, and program

Patent Citations (21)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2009096347A (en) 2007-10-17 2009-05-07 Denso Corp Parking assistance device
JP2009232310A (en) 2008-03-25 2009-10-08 Fujitsu Ltd Image processor for vehicle, image processing method for vehicle, image processing program for vehicle
JP2009265870A (en) 2008-04-24 2009-11-12 Alpine Electronics Inc Driving support device and its display method
US20090309970A1 (en) * 2008-06-04 2009-12-17 Sanyo Electric Co., Ltd. Vehicle Operation System And Vehicle Operation Method
US20100245573A1 (en) * 2009-03-25 2010-09-30 Fujitsu Limited Image processing method and image processing apparatus
JP2010268369A (en) 2009-05-18 2010-11-25 Konica Minolta Opto Inc Peripheral display device
US20120069153A1 (en) * 2009-05-25 2012-03-22 Panasonic Corporation Device for monitoring area around vehicle
JP2015015527A (en) 2013-07-03 2015-01-22 クラリオン株式会社 Video display system, video synthesis device and video synthesis method
US20200031283A1 (en) * 2016-09-30 2020-01-30 Aisin Seiki Kabushiki Kaisha Periphery monitoring apparatus
US20190215465A1 (en) * 2017-02-28 2019-07-11 JVC Kenwood Corporation Bird's-eye view image generating device, bird's-eye view image generating system, bird's-eye view image generating method, and medium
JP6730619B2 (en) * 2017-05-17 2020-07-29 株式会社Ｊｖｃケンウッド Overhead video generation device, overhead video display device, overhead video generation method and program
WO2019038901A1 (en) * 2017-08-25 2019-02-28 株式会社ソシオネクスト Correction device, correction program, and recording medium
US20200186788A1 (en) 2017-08-25 2020-06-11 Socionext Inc. Information processing apparatus and recording medium
US20200404191A1 (en) * 2018-03-05 2020-12-24 Denso Corporation Image creation device and image creation method
JP2019160118A (en) 2018-03-16 2019-09-19 株式会社デンソーテン Information provider and method for providing information
US20200139888A1 (en) * 2018-09-06 2020-05-07 Aisin Seiki Kabushiki Kaisha Vehicle periphery display device
US20210188289A1 (en) * 2018-09-13 2021-06-24 Sony Semiconductor Solutions Corporation Information processing device, moving apparatus, method, and program
JP2020127171A (en) 2019-02-06 2020-08-20 アイシン精機株式会社 Perimeter monitoring device
JP2021027366A (en) 2019-07-31 2021-02-22 株式会社デンソーテン Image generating apparatus
US20210107465A1 (en) * 2019-10-11 2021-04-15 Toyota Jidosha Kabushiki Kaisha Parking assist apparatus
US20210107511A1 (en) * 2019-10-11 2021-04-15 Toyota Jidosha Kabushiki Kaisha Parking assist apparatus

Non-Patent Citations (4)

* Cited by examiner, † Cited by third party
Title
International Search Report dated Aug. 10, 2021 issued in International Patent Application No. PCT/JP2021/019968, with English translation.
Written Opinion of the International Searching Authority dated Aug. 10, 2021 issued in International Patent Application No. PCT/JP2021/019968, with English translation.
International Search Report dated Aug. 10, 2021 issued in International Patent Application No. PCT/JP2021/019968, with English translation.
Written Opinion of the International Searching Authority dated Aug. 10, 2021 issued in International Patent Application No. PCT/JP2021/019968, with English translation.

Also Published As

Publication number Publication date
CN117356087A (en) 2024-01-05
WO2022249318A1 (en) 2022-12-01
JP7677412B2 (en) 2025-05-15
JPWO2022249318A1 (en) 2022-12-01
US20240096006A1 (en) 2024-03-21

Similar Documents

Publication Publication Date Title
US11102405B2 (en) Rear-stitched view panorama for rear-view visualization
US9754179B2 (en) Image generating device, electronic device, image generating method and recording medium
US20210326608A1 (en) Object detection apparatus, object detection method, and computer readable recording medium
US20090185054A1 (en) System and method for displaying an image indicating a positional relation between partially overlapping images
US10764561B1 (en) Passive stereo depth sensing
JP7038729B2 (en) Image compositing device and image compositing method
JP2008271308A (en) Image processor and method, and vehicle
JP2009017020A (en) Image processing apparatus and display image generation method
JP2008187564A (en) Camera calibration apparatus and method, and vehicle
JP6762570B2 (en) Image processing equipment, image processing method, and image processing program
US11132817B2 (en) Image processing apparatus, image processing system, and image processing method for generating a vehicle see-through rear view image
CN111402404B (en) Panorama complementing method and device, computer readable storage medium and electronic equipment
CN111815752A (en) Image processing method, device and electronic device
JP2019185520A (en) Image processor and method
JP2019080174A (en) Image processing apparatus, imaging system, communication system, image processing method, and program
US12482170B2 (en) Image processing device, image processing method, and image processing program
KR102418852B1 (en) Electronic device and method for controlling an image display
US11715218B2 (en) Information processing apparatus and information processing method
US12579818B2 (en) Information processing device, and control program
KR101762726B1 (en) Around view monitor system for generating top view image and method thereof
JP3154676B2 (en) In-vehicle camera image synthesis fluoroscope
JP2009077022A (en) Driving support system and vehicle
JP6734136B2 (en) Image processing device
JP2001086492A (en) In-vehicle camera image synthesis fluoroscope
JP6855254B2 (en) Image processing device, image processing system, and image processing method

Legal Events

Date Code Title Description
AS Assignment

Owner name: SOCIONEXT INC., JAPAN

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:OKONOGI, KATSUYUKI;KATO, TAKAYUKI;REEL/FRAME:065638/0471

Effective date: 20231110

FEPP Fee payment procedure

Free format text: ENTITY STATUS SET TO UNDISCOUNTED (ORIGINAL EVENT CODE: BIG.); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY

STPP Information on status: patent application and granting procedure in general

Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION

STPP Information on status: patent application and granting procedure in general

Free format text: NON FINAL ACTION MAILED

STPP Information on status: patent application and granting procedure in general

Free format text: RESPONSE TO NON-FINAL OFFICE ACTION ENTERED AND FORWARDED TO EXAMINER

STPP Information on status: patent application and granting procedure in general

Free format text: FINAL REJECTION COUNTED, NOT YET MAILED

STPP Information on status: patent application and granting procedure in general

Free format text: FINAL REJECTION MAILED

STPP Information on status: patent application and granting procedure in general

Free format text: RESPONSE AFTER FINAL ACTION FORWARDED TO EXAMINER

STPP Information on status: patent application and granting procedure in general

Free format text: ALLOWED -- NOTICE OF ALLOWANCE NOT YET MAILED

STPP Information on status: patent application and granting procedure in general

Free format text: NOTICE OF ALLOWANCE MAILED -- APPLICATION RECEIVED IN OFFICE OF PUBLICATIONS

STPP Information on status: patent application and granting procedure in general

Free format text: PUBLICATIONS -- ISSUE FEE PAYMENT RECEIVED

STPP Information on status: patent application and granting procedure in general

Free format text: PUBLICATIONS -- ISSUE FEE PAYMENT VERIFIED

STCF Information on status: patent grant

Free format text: PATENTED CASE