KR20210156360A - Appratus and method for generating bird viewpoint image - Google Patents

Abstract

In order to generate a bird's-eye-view viewpoint image, a method for generating the bird's-eye-view viewpoint image generates a plurality of images for a current view using a plurality of cameras installed in a vehicle, generates an initial bird's-eye view viewpoint image for the current view having a hole area based on the plurality of images, determines a target area for where the vehicle is located at the current view on a previous bird's-eye-view viewpoint image for a previous view, and generates the bird's-eye-view viewpoint image of the current view, by filling at least one part of the hole area of the initial bird's-eye-view viewpoint image, based on the target area.

Description

Method and apparatus for generating a bird's-eye view image {APPRATUS AND METHOD FOR GENERATING BIRD VIEWPOINT IMAGE}

The following embodiments relate to a technology for generating a bird's-eye view image while a vehicle is being driven.

With the development of image capturing and processing technology, a camera is mounted on a recently produced vehicle, and a bird's eye view system for assisting the driving of the vehicle is provided to the user or the vehicle. The bird's eye view system may generate a bird's eye view image or a top view image using images captured by a plurality of cameras. The bird's eye view image can provide the driver with a screen that looks down on the vehicle from the sky, thereby completely eliminating blind spots on the front, rear, left, and right sides of the vehicle.

An embodiment may provide a method and apparatus for generating a bird's-eye view image.

Another embodiment may provide a method and apparatus for generating a bird's-eye view image in which a floor of an area where a vehicle is located appears.

According to an embodiment, a method for generating a bird's eye view image, performed by an electronic device, includes generating a plurality of images for a current time using a plurality of cameras installed in a vehicle, and based on the plurality of images, the generating an initial bird's-eye view image for the current time, wherein the initial bird's-eye view image has a hall area; generating movement information that the vehicle has moved from a previous time to the current time, based on the movement information determining a target area in which the vehicle is located at the current time on the previous bird's-eye view image for the previous time, and filling at least a part of the hall area of the initial bird's-eye view image based on the target area. and generating a bird's-eye view image of the sight.

The hall area of the initial bird's eye view image may include an area in which the vehicle is located at the current time.

The hole area of the initial bird's eye view image may correspond to an area deviating from a viewing angle of the plurality of cameras.

The generating of the movement information may include receiving a steering angle of a steering wheel of the vehicle and a speed of the vehicle, and based on the steering angle and the speed, a rotation angle of a target wheel of the vehicle and an actual movement of the target wheel Calculating a distance, and calculating a target movement distance by converting the actual movement distance into a scale of an image plane of a bird's eye view, wherein the movement information may include the rotation angle and the target movement distance .

The determining of the target area based on the movement information may include: a distance moved by the horizontal axis, a distance moved by a vertical axis, and an angle rotated with respect to the centripetal point on the image plane of the bird's eye view of the vehicle based on the movement information. calculating, and based on the calculated distance moved on the horizontal axis, the distance moved on the vertical axis, and the rotated angle with respect to the centripetal point, at the current time on the previous bird's eye view image. The method may include determining an area in which the vehicle is located as the target area.

Calculating the distance moved by the horizontal axis, the distance moved by the vertical axis, and the rotation angle with respect to the centripetal point on the image plane of the bird's eye view of the vehicle based on the movement information is calculated according to the Ackerman-Jantoud type (Ackerman-Jantoud type) ) based on the movement information, calculating the distance the vehicle moves on the horizontal axis on the image plane of the bird's eye view, the distance moved on the vertical axis, and the angle rotated with respect to the centripetal point.

Calculating the distance the vehicle moves on the horizontal axis, the distance moved on the vertical axis, and the rotation angle with respect to the centripetal point on the image plane at the bird's eye view point based on the movement information includes the rotation angle of the target wheel of the vehicle. calculating the rotation angle of a preset virtual center wheel based on the step, determining the centripetal point based on the track length of the vehicle, the wheelbase length of the vehicle, and the rotation angle of the center wheel step, calculating the actual moving distance of the center wheel based on the actual moving distance of the target wheel, and based on the centripetal point and the actual moving distance of the center wheel, the vehicle moves horizontally on the image plane at the bird's eye view It may include calculating the distance moved along the axis, the distance moved along the vertical axis, and the rotation angle with respect to the centripetal point.

The generating of the bird's eye view image for the current time by filling at least a part of the hole area of the initial bird's eye view image based on the target area may include: accumulating a partial target area excluding the area where the vehicle is located in the vehicle area image, determining the area in which the vehicle is located at the current time as a copy area on the vehicle area image, and setting the duplicate area as the initial bird's eye view The method may include generating the bird's-eye view image by copying it to at least a part of the hole area of the viewpoint image.

The step of accumulating a partial target area excluding the area where the vehicle is located at the previous time on the previous bird's eye view image of the target area in the vehicle area image includes: the cumulative rotation angle of the vehicle from a reference time to the previous time calculating, generating a partial target area excluding the area where the vehicle is located at the previous time on the previous bird's eye view image among the target areas, rotating the partial target area to correspond to the cumulative rotation angle; and accumulating the partial target area on the vehicle area image by moving the rotated partial target area to the center of the vehicle area image.

The electronic device may be mounted in an autonomous driving vehicle or a vehicle supporting Advanced Driver Assistance Systems (ADAS).

According to another aspect, an electronic device for generating a bird's-eye view image includes a processor for executing a program for generating a bird's-eye view image, and a memory for storing the program, the program comprising: a plurality of cameras installed in a vehicle; generating a plurality of images for the current time using the generating movement information in which the vehicle has moved from generating a bird's eye view image for the current time by filling at least a part of the hole area of the initial bird's eye view image based on the target area.

The electronic device may further include a plurality of cameras that generate the plurality of images.

The generating of the movement information may include receiving a steering angle of a steering wheel of the vehicle and a speed of the vehicle, and based on the steering angle and the speed, a rotation angle of a target wheel of the vehicle and an actual movement of the target wheel Calculating a distance, and calculating a target movement distance by converting the actual movement distance into a scale of an image plane of a bird's eye view, wherein the movement information may include the rotation angle and the target movement distance .

The determining of the target area based on the movement information may include: a distance moved by the horizontal axis, a distance moved by a vertical axis, and an angle rotated with respect to the centripetal point on the image plane of the bird's eye view of the vehicle based on the movement information. calculating, and based on the calculated distance moved on the horizontal axis, the distance moved on the vertical axis, and the rotated angle with respect to the centripetal point, at the current time on the previous bird's eye view image. The method may include determining an area in which the vehicle is located as the target area.

Calculating the distance the vehicle moves on the horizontal axis, the distance moved on the vertical axis, and the rotation angle with respect to the centripetal point on the image plane at the bird's eye view point based on the movement information includes the rotation angle of the target wheel of the vehicle. calculating the rotation angle of a preset virtual center wheel based on the step, determining the centripetal point based on the track length of the vehicle, the wheelbase length of the vehicle, and the rotation angle of the center wheel step, calculating the actual moving distance of the center wheel based on the actual moving distance of the target wheel, and based on the centripetal point and the actual moving distance of the center wheel, the vehicle moves horizontally on the image plane at the bird's eye view It may include calculating the distance moved along the axis, the distance moved along the vertical axis, and the rotation angle with respect to the centripetal point.

The electronic device may be mounted in an autonomous driving vehicle or a vehicle supporting Advanced Driver Assistance Systems (ADAS).

A method and apparatus for generating a bird's-eye view image may be provided.

A method and apparatus for generating a bird's-eye view image in which a floor of an area where a vehicle is located appears may be provided.

1 illustrates a bird's-eye view of a moving vehicle according to an example.
2 is an initial bird's-eye view image including a hole area according to an example.
3 is a block diagram of an electronic device according to an exemplary embodiment.
4 is a flowchart of a method of generating a bird's-eye view image according to an example.
5 is an initial bird's-eye view image of a previous time and an initial bird's-eye view image of a current time, according to an example.
6 is a flowchart of a method of generating movement information in which a vehicle moves from a previous time to a current time according to an example.
7 illustrates a target area in which a vehicle is located at a current time determined on a previous bird's-eye view image according to an example.
8 is a flowchart of a method of determining a target area in which a vehicle is located at a current time on a previous bird's-eye view image based on movement information according to an example.
9 is a flowchart of a method of calculating a distance that a vehicle moves in a horizontal axis, a distance moved in a vertical axis, and a rotation angle with respect to a centripetal point on an image plane at a bird's eye view based on movement information according to an example.
10 illustrates a method of determining a centripetal point based on movement information according to an example.
11 illustrates a method of calculating a rotation angle of a vehicle with respect to a centripetal point using a virtual center wheel according to an example.
12 illustrates a method of calculating a distance moved by a horizontal axis and a distance moved by a vertical axis of the vehicle on an image plane at a bird's eye view based on an angle at which the vehicle rotates with respect to a centripetal point, according to an example.
13 is a flowchart of a method of generating a bird's eye view image for a current time by filling at least a part of a hole area of an initial bird's eye view image based on a target area according to an example.
14 is a flowchart of a method of accumulating a partial target area excluding an area in which a vehicle is located at a previous time on a previous bird's eye view image among target areas in a vehicle area image, according to an example.
15 illustrates a method of rotating a partial target area to correspond to an accumulated rotation angle according to an example.
16 illustrates a method of moving a rotated partial target area to the center of a vehicle area image according to an example.
17 illustrates a method of generating a bird's-eye view image for a current time based on a radiation area determined on a vehicle area image according to an example.
18 is a vehicle area image according to an example.
19 illustrates a bird's-eye view image generated using a radiation area determined on a vehicle area image and the radiation area according to an example.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. However, since various changes may be made to the embodiments, the scope of the patent application is not limited or limited by these embodiments. It should be understood that all modifications, equivalents and substitutes for the embodiments are included in the scope of the rights.

The terms used in the examples are used for the purpose of description only, and should not be construed as limiting. The singular expression includes the plural expression unless the context clearly dictates otherwise. In this specification, terms such as "comprise" or "have" are intended to designate that a feature, number, step, operation, component, part, or a combination thereof described in the specification exists, but one or more other features It should be understood that this does not preclude the existence or addition of numbers, steps, operations, components, parts, or combinations thereof.

Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which the embodiment belongs. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with the meaning in the context of the related art, and should not be interpreted in an ideal or excessively formal meaning unless explicitly defined in the present application. does not

In addition, in the description with reference to the accompanying drawings, the same components are given the same reference numerals regardless of the reference numerals, and the overlapping description thereof will be omitted. In describing the embodiment, if it is determined that a detailed description of a related known technology may unnecessarily obscure the gist of the embodiment, the detailed description thereof will be omitted.

In addition, in describing the components of the embodiment, terms such as first, second, A, B, (a), (b), etc. may be used. These terms are only for distinguishing the elements from other elements, and the essence, order, or order of the elements are not limited by the terms. When it is described that a component is "connected", "coupled" or "connected" to another component, the component may be directly connected or connected to the other component, but another component is between each component. It will be understood that may also be "connected", "coupled" or "connected".

Components included in one embodiment and components having a common function will be described using the same names in other embodiments. Unless otherwise stated, descriptions described in one embodiment may be applied to other embodiments as well, and detailed descriptions within the overlapping range will be omitted.

1 illustrates a bird's-eye view of a moving vehicle according to an example, and FIG. 2 is an initial bird's-eye view image including a hall area according to an example.

If the moving vehicle 110 can be observed from a bird's eye view, a lot of information about the surrounding situation of the vehicle 110 can be obtained. For example, the positional relationship between vehicles located in the vicinity of the vehicle 110 , the speed of the vehicles, obstacles, and the like may be obtained as information.

In order to observe the vehicle 110 from a bird's eye view, a camera for generating an image is located above the vehicle 110 , but this method is difficult to use in reality. Accordingly, a bird's eye view image of the vehicle 110 is generated by photographing the side surfaces of the vehicle using the cameras installed in the vehicle 110 , converting the captured images into a bird's eye view, and synthesizing the images converted to the bird's eye view. method can be considered. A system that provides a plurality of images or a bird's-eye view image to a user of the vehicle 110 may be an Around View Monitoring (AVM) system.

However, the initial bird's-eye view image 200 generated based on images of the side surfaces of the vehicle 110 is an area of the vehicle 110 that is out of the viewing angle of the cameras as shown in FIG. 2 below. A hole area 210 is indicated for . The hole area 210 is an area in which there is no pixel value information on a corresponding pixel in the image.

The unnatural disconnection occurring in the bird's eye view image by the hall area 210 makes the user feel a sense of distance from the actual driving environment. According to one aspect, as a method of filling the hole area 210 , a method of determining an area corresponding to the hole area 210 from a previous bird's eye view image and filling the hole area 210 based on the determined area may be considered. . A method of filling the hole area 210 may be based on a Transparent Vehicle Chassis (TVC) technology. In the TVC technology, when the hall area 210 is reconstructed using an image of a previous time, this sense of disconnection can be resolved.

In addition, when a bird's-eye view image is generated by applying TVC technology, the floor surface of the road that is obscured by the vehicle appears in the image. The user can additionally grasp road surface information such as speed limit and child protection zone display through the bird's eye view image. In particular, since the user can grasp the entire parking line in real time while parking, parking may be possible. Specifically, since the user can grasp the floor surface of the vehicle, there is an effect of accurately grasping information in the parking space, such as a parking line and a parking mark.

Hereinafter, a method of generating a bird's eye view image by filling in the hole region of the initial bird's eye view image will be described in detail with reference to FIGS. 3 to 19 .

3 is a block diagram of an electronic device according to an exemplary embodiment.

According to one aspect, the electronic device 300 includes a communication unit 310 , a processor 320 , a memory 330 , and a camera 340 . The electronic device 300 may be included in a vehicle. For example, the electronic device 300 may be a device such as an electronic control unit (ECU). As another example, the electronic device 300 may be an independent device connected to the ECU.

The communication unit 310 is connected to the processor 320 , the memory 330 , and the camera 340 to transmit and receive data. The communication unit 310 may be connected to another external device to transmit/receive data. Hereinafter, the expression "transmitting and receiving "A" may indicate transmitting and receiving "information or data representing A".

The communication unit 310 may be implemented as circuitry in the electronic device 300 . For example, the communication unit 310 may include an internal bus and an external bus. As another example, the communication unit 310 may be an element that connects the electronic device 300 and an external device. The communication unit 310 may be an interface. The communication unit 310 may receive data from an external device and transmit the data to the processor 320 and the memory 330 .

The processor 320 processes data received by the communication unit 310 and data stored in the memory 330 . A “processor” may be a data processing device implemented in hardware having circuitry having a physical structure for performing desired operations. For example, desired operations may include code or instructions included in a program. For example, a data processing device implemented as hardware includes a microprocessor, a central processing unit, a processor core, a multi-core processor, and a multiprocessor. , an Application-Specific Integrated Circuit (ASIC), and a Field Programmable Gate Array (FPGA).

Processor 320 executes computer readable code (eg, software) stored in memory (eg, memory 330 ) and instructions issued by processor 320 .

The memory 330 stores data received by the communication unit 310 and data processed by the processor 320 . For example, the memory 330 may store a program (or an application, software). The stored program may be a set of syntaxes coded to generate a bird's-eye view image and executable by the processor 320 .

According to one aspect, memory 330 may include one or more of volatile memory, non-volatile memory and random access memory (RAM), flash memory, hard disk drive, and optical disk drive.

The memory 330 stores an instruction set (eg, software) for operating the electronic device 300 . The instruction set for operating the electronic device 300 is executed by the processor 320 .

The camera 340 creates an image by photographing a scene. The camera 340 may include a plurality of cameras. For example, the plurality of cameras may be respectively disposed on the front, rear, left and right sides of the vehicle.

The communication unit 310 , the processor 320 , the memory 330 , and the camera 340 will be described in detail below with reference to FIGS. 4 to 19 .

4 is a flowchart of a method for generating a bird's-eye view image according to an example.

The following steps 410 to 450 are performed by the electronic device 300 described above with reference to FIG. 3 .

In operation 410 , the electronic device 300 generates a plurality of images for the current time using a plurality of cameras installed in the vehicle. Each of the plurality of cameras may continuously generate an image. For example, an image most recently generated by the first camera may be an image for the current time, and an image generated before that may be an image for a previous time. A plurality of images may be generated at the same time by synchronizing the photographing period of the plurality of cameras.

In operation 420 , the electronic device 300 generates an initial bird's-eye view image of the current time based on the plurality of images. Since the plurality of images do not include image information about the vehicle itself, the initial bird's-eye view image has a hall area corresponding to the vehicle area. That is, the hall area includes an area in which the vehicle is located at the current time. The hole area corresponds to an area deviating from the viewing angles of the plurality of cameras. Since the vehicle area in the initial bird's eye view image is constant for all views, the hole area in the initial bird's eye view image may be set as a region of interest (ROI).

In order to generate a bird's-eye view image, a plurality of images captured by a plurality of cameras must be merged. In order to merge a plurality of images, the coordinate systems of the images must match. In other words, partial bird's-eye view images may be generated by converting a plurality of images into a common coordinate system preset for the vehicle, and an initial bird's eye view image may be generated by merging the partial bird's-eye view images. For example, a partial bird's eye view image may be generated by warping each image.

According to one aspect, in order to generate an accurate bird's eye view image, a plurality of cameras may need to be calibrated periodically or at every impact. A method for calibrating a plurality of cameras is not limited, and various methods may be used.

Referring to FIG. 5 , a bird's-eye view image 510 generated at a previous time and an initial bird's-eye view image 522 generated at a current time according to an example are shown. The previous time is the time at which the most recently generated bird's eye view image 510 was generated (hereinafter, the most recently generated bird's eye view image is referred to as a "previous bird's eye view image") or the previous bird's eye view image 510. It refers to the time at which a plurality of basic images are generated. The illustrated previous bird's-eye view image 510 includes an area 512 in which the vehicle is located at the previous time. Based on the current time, the area 512 may be an area filled with information. The initial bird's eye view image 520 includes an area 522 in which the vehicle is located at the current time. Based on the current time, the area 522 may be a hole area in which information is not filled.

According to one aspect, when implementing the TVC technology in hardware, the main consideration is the efficient use of memory. A bird's eye view image uses a relatively large amount of memory space. In the case of storing the entire previous bird's-eye view image and the current bird's-eye view image, the efficiency of memory usage decreases and the operating speed of the program becomes slow. Accordingly, when the memory area for storing the previous bird's-eye view image and the memory area for storing the current bird's-eye view image are shared, the operating efficiency of the program may increase.

In operation 430 , the electronic device 300 generates movement information of the vehicle moving from the previous time to the current time. According to an aspect, a steering angle of a steering wheel of a vehicle and a speed of the vehicle may be obtained from a previous time to a current time, and movement information of the vehicle moving on the scale of the image plane at the bird's eye view may be generated based on this. A method of generating movement information of a vehicle will be described in detail below with reference to FIG. 6 .

According to another aspect, the rotation angle of the wheel of the vehicle and the actual moving distance of the wheel from the previous time to the present time may be provided as movement information. For example, the rotation angle and the actual moving distance of the wheel may be directly measured based on at least one sensor attached to the wheel.

In operation 440 , the electronic device 300 determines a target area in which the vehicle is located at the current time on the previous bird's-eye view image based on the movement information. Referring to FIG. 7 , a target area 710 in which the vehicle is located at the current time on the previous bird's eye view image 510 is determined based on movement information. For example, on the previous bird's eye view image 510 , an area moved from the area 512 where the vehicle is located at the previous time by the amount of movement corresponding to the movement information may be determined as the target area 710 .

A method of determining the target area will be described in detail below with reference to FIGS. 8 to 12 .

In operation 450 , the electronic device 300 generates a bird's eye view image for the current time by filling at least a part of the hole area of the initial bird's eye view image based on the target area. A method of generating a bird's eye view image for the current time will be described in detail below with reference to FIGS. 13 to 19 .

6 is a flowchart of a method of generating movement information in which a vehicle moves from a previous time to a current time according to an example.

According to one aspect, the step 430 described above with reference to FIG. 4 may include the following steps 610 to 630 .

In step 610 , the electronic device 300 receives the steering angle of the steering wheel of the vehicle and the speed of the vehicle. For example, a controller area network (CAN) is installed in a vehicle, and a steering angle of a steering wheel of the vehicle and a speed of the vehicle may be generated through the CAN device. The electronic device 300 may receive the steering angle of the steering wheel of the vehicle and the speed of the vehicle from the CAN device.

In operation 620 , the electronic device 300 calculates a rotation angle of the target wheel of the vehicle and an actual moving distance of the target wheel based on the steering angle of the steering wheel and the speed of the vehicle.

For example, when the wheels rotated by the handle are the front wheels, rotation angles for each of the front wheels that are the target wheels may be calculated by multiplying the steering angle of the handle by a preset coefficient.

For example, the actual distance traveled by the target wheel or vehicle during the target time may be calculated based on the speed and the difference between the previous time and the current time (hereinafter, the difference between the previous time and the current time is "target time"). have.

In operation 630, the electronic device 300 calculates the target movement distance by converting the actual movement distance of the target wheel into the scale of the image plane of the bird's eye view. For example, the target movement distance may be calculated by multiplying the actual movement distance by a preset value. For example, when 1 m corresponds to the length of 100 pixels on the image plane of the bird's eye view, and the actual movement distance is 50 cm, the calculated target movement distance may correspond to the length of 50 pixels.

The movement information includes the generated rotation angle of the target wheel and the target movement distance. In other words, the movement information is information that the vehicle has moved on the image plane of the bird's eye view from the previous time to the present time.

8 is a flowchart of a method of determining a target area in which a vehicle is located at a current time on a previous bird's-eye view image based on movement information according to an example.

According to one aspect, step 440 described above with reference to FIG. 4 may include steps 810 and 820 below.

In operation 810 , the electronic device 300 calculates a distance the vehicle moves on the horizontal axis on the image plane of a bird's eye view, a distance it moves on a vertical axis, and a rotation angle with respect to the centripetal point based on the movement information.

According to one aspect, in order to calculate the distance the vehicle moved on the horizontal axis, the distance moved on the vertical axis, and the rotation angle with respect to the centripetal point on the image plane at the bird's eye view, based on the movement information, the Ackerman- Jantoud type) can be used.

A method of calculating the distance the vehicle moves on the horizontal axis, the distance moved on the vertical axis, and the rotation angle with respect to the centripetal point on the image plane of the bird's eye view will be described in detail below with reference to FIGS. 9 to 12 .

In step 820, the electronic device 300 based on the distance moved along the horizontal axis of the vehicle (the first distance), the distance moved along the new axis (the second distance), and the angle rotated with respect to the centripetal point (target angle) Thus, an area in which the vehicle is located at the current time on the previous bird's-eye view image is determined as the target area.

In the example with reference to FIG. 7 , the area 710 where the vehicle is located at the current time on the previous bird's eye view image 510 based on the distance moved along the horizontal axis, the distance moved to the new axis, and the angle rotated with respect to the centripetal point. ) as the target area. In the area 512 where the vehicle is located at the previous time, the area in which the center of the vehicle moves the first distance on the horizontal axis, moves the second distance on the vertical axis, and rotates at a target angle with respect to the centripetal point is the vehicle at the current time. This may be an area 710 in which it is located.

9 is a flowchart of a method of calculating a distance that a vehicle moves in a horizontal axis, a distance moved in a vertical axis, and a rotation angle with respect to a centripetal point on an image plane at a bird's eye view based on movement information according to an example.

According to one aspect, the step 810 described above with reference to FIG. 8 may include the following steps 910 to 940 .

In operation 910 , the electronic device 300 calculates a preset rotation angle of the virtual center wheel based on the rotation angle of the target wheel of the vehicle.

For example, the center wheel may be preset to be located on an axis connecting the front wheels in a straight line. Specifically, the center wheel may be located at the center of the shaft, but is not limited to the described embodiment. As another example, the center wheel may be preset as a left front wheel or a right front wheel.

When the center wheel is preset to be positioned at the center between the front wheels, the average of the respective rotation angles of the front wheels may be calculated as the rotation angle of the center wheel.

In operation 920 , the electronic device 300 determines a centripetal point based on a track length of the vehicle, a wheelbase length, and a rotation angle of a center wheel.

In the example with reference to FIG. 10 , the centripetal point 1030 is determined based on the track length 1004 of the vehicle 1000 , the wheelbase length 1005 , and the rotation angle θ _{M of the center wheel 1003 .} The center wheel 1003 may be preset based on the target wheels 1001 and 1002 . For example, the rotation angle θ _M of the center wheel may be an angle rotated in a state in which the center wheel is aligned. The intersection of the virtual straight line 1020 extending the central axis of the rotated center wheel 1003 and the virtual straight line 1010 extending the axis of the rear wheels is determined as the centripetal point 1030 . A distance between the center wheel 1003 and the point where the rear axles of the rear wheels 1001 and 1002 meet (eg, the center of the rear axle) and the centripetal point 1030 may be a reference length 1040 .

In operation 930 , the electronic device 300 calculates the actual moving distance of the center wheel based on the actual moving distance of the target wheel.

When the center wheel is preset to be located at the center between the front wheels, an average of the actual moving distances of each of the front wheels may be calculated as the actual moving distance of the center wheel.

In step 940 , the electronic device 300 determines the distance the vehicle moves on the horizontal axis on the image plane of the bird's eye view based on the actual movement distance of the centripetal point and the center wheel, the distance moved on the vertical axis, and the rotation about the centripetal point. Calculate the angle.

In the example shown with reference to FIG. 11, the center wheel 1103 of the vehicle 1110 at the current time by moving the actual moving distance around the centripetal point 1030 for the center wheel 1003 of the vehicle 1000 at the previous time. location can be determined. _{A movement angle d θ} of the center wheel 1103 may be determined based on the position of the center wheel 1103 .

The positional relationship between the vehicle 1000 at the previous time and the vehicle 1110 at the current time illustrated with reference to FIG. 11 is expressed with reference to FIG. 12 based on the center of the vehicle. A center position 1210 of the vehicle 1000 and a center position 1220 of the vehicle 1110 are determined. _{The angle θ C} between the rear axle and the central position 1210 _{is determined around the centripetal point 1030, and the angle d θ} between the central position 1210 and the central position 1220 around the centripetal point 1030 is determined. do. _{The angle d θ} between the central position 1210 and the central position 1220 is the same as the movement angle d _{θ of FIG. 11 .} A horizontal axis distance 1230 and a longitudinal axis distance 1240 between the central position 1210 and the central position 1220 are determined.

Based on the distance 1230 moved along the horizontal axis, the distance 1240 moved along the vertical axis, and the angle d _θ rotated with respect to the centripetal point 1030 of the vehicle 1110, the current on the previous bird's-eye view image 510 An area 710 in which the vehicle is located at the time is determined as the target area.

13 is a flowchart of a method of generating a bird's eye view image for a current time by filling at least a part of a hole area of an initial bird's eye view image based on a target area according to an example.

According to one aspect, the step 450 described above with reference to FIG. 4 may include the following steps 1310 to 1330 .

In operation 1310 , the electronic device 300 accumulates a partial target area excluding the area where the vehicle is located at the previous time on the previous bird's-eye view image among the target areas in the vehicle area image. The vehicle area image may be an image in which only areas corresponding to the vehicle area at each time are accumulated. The center of the vehicle area image may change whenever the time of day changes. For example, the center of the vehicle region image for the previous time may move based on a distance 1230 moved along the horizontal axis and a distance 1240 moved along the vertical axis. Accordingly, the vehicle area at the current time may be located at the center of the vehicle area image.

According to one aspect, an area for storing the vehicle area image may be preset on the memory 330 .

A method of accumulating the partial target area on the vehicle area image will be described in detail below with reference to FIGS. 14 to 16 .

In operation 1320 , the electronic device 300 determines an area in which the vehicle is located at the current time as a copy area on the vehicle area image. In the example with reference to FIG. 17 , the area in which the vehicle is located at the current time on the vehicle area image 1600 is determined as the radiation area 1710 .

In operation 1330, the electronic device 300 generates a bird's-eye view image by copying the copy area to at least a part of the hole area of the initial bird's-eye view image. In the example referring to FIG. 17 , a bird's eye view image for the current time is generated by copying the copy area 1710 to at least a part of the hole area 522 of the initial bird's eye view image 520 .

14 is a flowchart of a method of accumulating a partial target area excluding an area in which a vehicle is located at a previous time on a previous bird's eye view image among target areas in a vehicle area image, according to an example.

According to one aspect, the step 1310 described above with reference to FIG. 13 may include the following steps 1410 to 1440 .

In operation 1410 , the electronic device 300 calculates the cumulative rotation angle of the vehicle from the reference time to the previous time. For example, the reference time may be an initial time at which the vehicle region image is generated. A first rotation angle of the vehicle between the first time and the second time is calculated, a second rotation angle of the vehicle between the second time and the third time is calculated, the reference time is the first time, and the previous time is the second time In the case of 3 time, the sum of the first rotation angle and the second rotation angle may be calculated as the cumulative rotation angle.

In operation 1420 , the electronic device 300 generates a partial target area from among the target areas except for the area where the vehicle is located at the previous time on the previous bird's-eye view image.

In the example with reference to FIG. 15 , a partial target area 1520 is generated in the target area 710 except for the vehicle area 1510 in which the vehicle is located at the previous time on the previous bird's eye view image 510 . Although it is illustrated as if there is no information in the vehicle area 1510 , the area 1510 of the previous bird's-eye view image 510 may be pre-filled based on the vehicle area image at the previous time. Accordingly, since the vehicle area image at the current time may already include information on the area where the vehicle is located at the previous time, in order to reduce the amount of processed data, the vehicle at the previous time on the previous bird's-eye view image of the target area is reduced. You can exclude the area where it is located.

In operation 1430 , the electronic device 300 rotates the partial target area to correspond to the cumulative rotation angle of the vehicle.

In the example with reference to FIG. 15 , the partial target area 1520 is rotated to correspond to the _{cumulative rotation angle θ acc} based on the center of the vehicle area image 1530 for the previous time.

In operation 1440 , the electronic device 300 accumulates the rotated partial target area on the vehicle area image. The center of the vehicle area image in which the partial target area is accumulated may be moved to the center of the partial target image. The vehicle area image in which the partial target area for the current time is accumulated may be an image generated for the current time.

In the example with reference to FIG. 16 , the preliminary vehicle area image is generated by accumulating the rotated partial target area 1520 on the vehicle area image 1530 for the previous time. By moving the center of the preliminary vehicle area image to the center of the partial target area 1520 , the vehicle area image 1600 for the current time is generated.

After step 1440 is performed, steps 1320 and 1330 described above with reference to FIG. 13 may be performed.

In the example with reference to FIG. 17 , in step 1320 , the area in which the vehicle is located at the current time is determined as the radiation area 1710 on the vehicle area image 1600 . The radiation area 1710 may include a partial target area 1520 . An area not included in the partial target area 1520 among the copy area 1710 may be an area copied from previous times of the current time.

In operation 1330 , the copy area 1710 is copied to at least a part of the hole area 522 of the initial bird's eye view image 520 to generate a bird's-eye view image.

18 is a vehicle area image according to an example.

The illustrated vehicle region image 1800 may be an image in which partial target regions continuously generated from an initial time to a current time are accumulated. In other words, the vehicle area image 1800 may be an image of the floor area to which the vehicle has moved.

19 illustrates a bird's-eye view image generated using a radiation area determined on a vehicle area image and the radiation area according to an example.

The area in which the vehicle is located at the current time is determined as the radiation area 1910 on the vehicle area image 1800 . A bird's eye view image is generated by copying the copy area 1910 to at least a part of the hole area 210 of the initial bird's eye view image 200 .

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the embodiment, or may be known and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floppy disks. - includes magneto-optical media, and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

Software may comprise a computer program, code, instructions, or a combination of one or more thereof, which configures a processing device to operate as desired or is independently or collectively processed You can command the device. The software and/or data may be any kind of machine, component, physical device, virtual equipment, computer storage medium or apparatus, to be interpreted by or to provide instructions or data to the processing device. , or may be permanently or temporarily embody in a transmitted signal wave. The software may be distributed over networked computer systems and stored or executed in a distributed manner. Software and data may be stored in one or more computer-readable recording media.

As described above, although the embodiments have been described with reference to the limited drawings, those skilled in the art may apply various technical modifications and variations based on the above. For example, the described techniques are performed in an order different from the described method, and/or the described components of the system, structure, apparatus, circuit, etc. are combined or combined in a different form than the described method, or other components Or substituted or substituted by equivalents may achieve an appropriate result.

Therefore, other implementations, other embodiments, and equivalents to the claims are also within the scope of the following claims.

300: electronic device
310: communication unit
320: processor
330: memory
340: camera

Claims (17)

A bird's-eye view image generating method performed by an electronic device,
generating a plurality of images for a current time using a plurality of cameras installed in a vehicle;
generating an initial bird's eye view image for the current time based on the plurality of images, wherein the initial bird's eye view image has a hole area;
generating movement information in which the vehicle has moved from a previous time to the present time;
determining a target area in which the vehicle is located at the current time on a previous bird's-eye view image for the previous time based on the movement information; and
generating a bird's eye view image for the current time by filling at least a part of the hole area of the initial bird's eye view image based on the target area
containing,
How to create a bird's-eye view image.
According to claim 1,
The hall area of the initial bird's eye view image includes an area in which the vehicle is located at the current time,
How to create a bird's-eye view image.
According to claim 1,
The hole area of the initial bird's eye view image corresponds to an area deviating from the viewing angles of the plurality of cameras,
How to create a bird's-eye view image.
According to claim 1,
The step of generating the movement information comprises:
receiving a steering angle of a steering wheel of the vehicle and a speed of the vehicle;
calculating a rotation angle of a target wheel of the vehicle and an actual moving distance of the target wheel based on the steering angle and the speed; and
Calculating the target movement distance by converting the actual movement distance into a scale of the image plane of the bird's eye view
including,
The movement information includes the rotation angle and the target movement distance,
How to create a bird's-eye view image.
5. The method of claim 4,
Determining the target area based on the movement information comprises:
calculating a distance moved by the horizontal axis, a distance moved in a vertical axis, and a rotation angle with respect to the centripetal point of the vehicle on the image plane at a bird's eye view point based on the movement information; and
Based on the calculated distance moved along the horizontal axis, the distance moved along the vertical axis, and the rotated angle with respect to the centripetal point, the area in which the vehicle is located at the current time on the previous bird's eye view image Determining the target area
containing,
How to create a bird's-eye view image.
6. The method of claim 5,
Calculating the distance moved by the horizontal axis, the distance moved by the vertical axis, and the rotation angle with respect to the centripetal point of the vehicle on the image plane of the bird's eye view point based on the movement information,
Based on the movement information through the Ackerman-Jantoud type, the distance that the vehicle moves on the horizontal axis on the image plane at the bird's eye view, the distance moved on the vertical axis, and the rotation angle with respect to the centripetal point are calculated. step
containing,
How to create a bird's-eye view image.
6. The method of claim 5,
Calculating the distance moved by the horizontal axis, the distance moved by the vertical axis, and the rotation angle with respect to the centripetal point of the vehicle on the image plane of the bird's eye view point based on the movement information,
calculating a rotation angle of a preset virtual center wheel based on the rotation angle of the target wheel of the vehicle;
determining the centripetal point based on a track length of the vehicle, a wheelbase length of the vehicle, and the rotation angle of the center wheel;
calculating an actual moving distance of the center wheel based on the actual moving distance of the target wheel; and
Calculating the distance the vehicle moves along the horizontal axis on the image plane of the bird's eye view, the distance moved along the vertical axis, and the angle rotated with respect to the centripetal point based on the actual movement distance of the centripetal point and the center wheel
containing,
How to create a bird's-eye view image.
According to claim 1,
generating a bird's eye view image for the current time by filling at least a part of the hole area of the initial bird's eye view image based on the target area,
accumulating a partial target area excluding the area where the vehicle is located at the previous time on the previous bird's eye view image of the target area in the vehicle area image;
determining an area in which the vehicle is located at the current time as a copy area on the vehicle area image; and
generating the bird's-eye view image by copying the copy area to at least a part of the hole area of the initial bird's-eye view image
containing,
How to create a bird's-eye view image.
9. The method of claim 8,
The step of accumulating a partial target area excluding the area where the vehicle is located at the previous time on the previous bird's eye view image among the target areas in the vehicle area image,
calculating a cumulative rotation angle of the vehicle from a reference time to the previous time;
generating a partial target area excluding an area in which the vehicle is located at the previous time on the previous bird's eye view image among the target areas;
rotating the partial target area to correspond to the cumulative rotation angle; and
accumulating the rotated partial target area on the vehicle area image
containing,
How to create a bird's-eye view image.
According to claim 1,
The electronic device is mounted on an autonomous driving vehicle or a vehicle supporting Advanced Driver Assistance Systems (ADAS),
How to create a bird's-eye view image.
A computer-readable recording medium containing a program for performing the method of any one of claims 1 to 10.
An electronic device for generating a bird's-eye view image,
a processor executing a program for generating a bird's-eye view image; and
memory to store the program
including,
The program is
generating a plurality of images for a current time using a plurality of cameras installed in a vehicle;
generating an initial bird's eye view image for the current time based on the plurality of images, wherein the initial bird's eye view image has a hole area;
generating movement information in which the vehicle has moved from a previous time to the present time;
determining a target area in which the vehicle is located at the current time on a previous bird's-eye view image for the previous time based on the movement information; and
generating a bird's eye view image for the current time by filling at least a part of the hole area of the initial bird's eye view image based on the target area
to do,
electronic device.
13. The method of claim 12,
A plurality of cameras generating the plurality of images
further comprising,
electronic device.
13. The method of claim 12,
The step of generating the movement information comprises:
receiving a steering angle of a steering wheel of the vehicle and a speed of the vehicle;
calculating a rotation angle of a target wheel of the vehicle and an actual moving distance of the target wheel based on the steering angle and the speed; and
Calculating the target movement distance by converting the actual movement distance into a scale of the image plane of the bird's eye view
including,
The movement information includes the rotation angle and the target movement distance,
electronic device.
15. The method of claim 14,
Determining the target area based on the movement information comprises:
calculating a distance moved by the horizontal axis, a distance moved in a vertical axis, and a rotation angle with respect to the centripetal point of the vehicle on the image plane at a bird's eye view point based on the movement information; and
Based on the calculated distance moved along the horizontal axis, the distance moved along the vertical axis, and the rotated angle with respect to the centripetal point, the area in which the vehicle is located at the current time on the previous bird's eye view image Determining the target area
containing,
electronic device.
16. The method of claim 15,
Calculating the distance moved by the horizontal axis, the distance moved by the vertical axis, and the rotation angle with respect to the centripetal point of the vehicle on the image plane of the bird's eye view point based on the movement information,
calculating a rotation angle of a preset virtual center wheel based on the rotation angle of the target wheel of the vehicle;
determining the centripetal point based on a track length of the vehicle, a wheelbase length of the vehicle, and the rotation angle of the center wheel;
calculating an actual moving distance of the center wheel based on the actual moving distance of the target wheel; and
Calculating the distance the vehicle moves along the horizontal axis on the image plane of the bird's eye view, the distance moved along the vertical axis, and the angle rotated with respect to the centripetal point based on the actual movement distance of the centripetal point and the center wheel
containing,
electronic device.
13. The method of claim 12,
The electronic device is mounted on an autonomous driving vehicle or a vehicle supporting Advanced Driver Assistance Systems (ADAS),
electronic device.

Images