KR102529835B1 - Method and apparatus for outputting graphic objects related to vehicel driving information - Google Patents

Abstract

In order to output a graphic object for vehicle driving information, a first image for a first time is generated using a camera installed on one side of the vehicle, and a second image for a second time is generated using the camera. , Create a depth image for the first image as a reference image among the first image and the second image, determine a target area related to the position of the vehicle based on the depth image on the reference image, and create a graphic object representing the target area. output through the display.

Description

Graphic object output method and apparatus related to vehicle driving information

The following embodiments relate to a technology for outputting a graphic object, and specifically, to a technology for outputting a graphic object related to driving information of a vehicle.

Blind spots may appear as areas that are difficult for a driver to directly observe while driving a vehicle. For example, the blind spot may be an area that is covered by the body of the vehicle or not displayed even by a side mirror or the like. Although the viewing zone can be changed by changing the installation angle of the vehicle's side mirrors, the viewing zone cannot be completely eliminated. To reduce blind spots, side mirrors can be replaced with mirrors with a wide angle of view, or ultrasonic sensors can be installed at the rear of the vehicle. However, there are disadvantages in that replacing the mirror requires additional cost and the driver may lose a sense of an actual distance. In addition, the ultrasonic sensor has a short distance for sensing an object and cannot discriminate the size of the object.

Korean Patent Publication No. 10-2016-0147530 (published on December 23, 2016) discloses a vehicle side mirror blind spot detection device and a control method thereof. The disclosed invention is provided with a cylindrical side mirror, a rear camera, and a monitor, so that air resistance can be reduced while driving, and the rear can be monitored without a blind spot. can

An embodiment may provide a method and apparatus for outputting a graphic object related to driving information of a vehicle.

One embodiment may provide a method and apparatus for controlling a vehicle.

However, the technical challenges are not limited to the above-described technical challenges, and other technical challenges may exist.

According to one aspect, a method of outputting a graphic object, performed by an electronic device, includes generating a first image for a first time using a camera installed on one side of a vehicle, and generating a first image for a second time using the camera. generating a second image for the first image, generating a depth image for the first image as a reference image among the first image and the second image, and determining a target area associated with the position of the vehicle based on the depth image. The method includes determining on a reference image, and outputting a graphic object representing the target region through a display.

The second image may be a next image generated by the camera after generating the first image.

The generating of the depth image may include calculating a transformation relationship between the first image and the second image, and creating a plurality of virtual planes having different depth values based on the first image based on the transformation relationship. setting, generating a plurality of virtual images corresponding to the plurality of virtual planes based on the first image and a homography of each of the plurality of virtual planes with respect to the first image; calculating matching costs for matching pixels in each of the plurality of virtual images corresponding to a target pixel in a first image, determining a final matching pixel based on the matching costs, including the final matching pixel The method may include determining a depth value of a target virtual image among the plurality of virtual images as a target depth value of the target pixel, and generating the depth image based on the target depth value.

Calculating the transformation relationship between the first image and the second image may include calculating the transformation relationship between the first image and the second image using a scale invariant feature transform (SIFT) algorithm, or a PoseNet model Calculating the transformation relationship between the first image and the second image using

A first homography of a first virtual plane among the plurality of virtual planes may be determined based on an external parameter of the first image, an external parameter of the second image, and a first depth value of the first virtual plane.

Calculating matching costs for matching pixels in each of the plurality of virtual images corresponding to a target pixel in the reference image may include sum of absolute difference (SAD) or zero mean normalized cross-correlation (ZNCC) based on Calculating the matching costs may be included.

The target area may be a straight line or an area in the form of a straight line corresponding to a portion of a floor area in a scene represented by the first image, and the graphic object may be an object representing the straight line or the area in the form of a straight line.

The target area may be a 2D area corresponding to a part of a floor area in a scene represented by the first image, and the graphic object may be an object representing a boundary of the 2D area.

The target area may be a 3D area including a part of a floor area in a scene represented by the first image, and the graphic object may be an object representing a boundary of the 3D area.

The graphic object output method may further include determining whether an object exists in the target area, and performing a preset operation if the object exists in the target area.

The step of determining whether an object exists within the target area may include determining a current driving situation of the vehicle, and determining whether an object exists within the target area when the current driving situation corresponds to a preset driving situation. It may include a decision-making step.

Determining whether an object exists in the target area may include determining an object sensing value in the target area at the first time, a previous object sensing value in a previous target area determined at a time before the first time, and The method may include calculating a change amount between object sensing values, and determining whether an object exists when the change amount is greater than or equal to a preset threshold value.

When an object exists in the target area, performing a preset operation may include outputting an alarm to a user of the vehicle as the preset operation.

When an object exists in the target area, performing a preset operation may include controlling at least one of a driving direction or a driving speed of the vehicle as the preset operation.

The display may be included in a side mirror of the vehicle, and the graphic object may be overlaid on a scene displayed on the side mirror.

The outputting of the graphic object representing the target region through the display may include outputting the reference image overlaid with the graphic object through the display.

According to another aspect, an electronic device includes a memory in which a program outputting a graphic object is recorded, and a processor that executes the program, wherein the program is stored at a first time using a camera installed on one side of a vehicle. generating a first image for a second view using the camera; generating a depth image for the first image as a reference image among the first and second images; The steps of determining a target area related to the location of the vehicle on the reference image based on the depth image, and outputting a graphic object representing the target area through a display.

According to another aspect, a vehicle control method, performed by an electronic device, includes generating a first image for a first time point using a camera installed on one side of a vehicle, and a second time point using the camera. Generating a second image for , Generating a depth image for the first image as a reference image among the first and second images, Based on the depth image, a target area associated with the position of the vehicle is determined. Determining on the reference image, determining whether an object exists in the target area, and controlling the vehicle by performing a preset operation when the object exists in the target area.

The step of determining whether an object exists within the target area may include determining a current driving situation of the vehicle, and determining whether an object exists within the target area when the current driving situation corresponds to a preset driving situation. It may include a decision-making step.

Determining whether an object exists in the target area may include determining an object sensing value in the target area at the first time, a previous object sensing value in a previous target area determined at a time before the first time, and The method may include calculating a change amount between object sensing values, and determining whether an object exists when the change amount is greater than or equal to a preset threshold value.

According to another aspect, an electronic device includes a memory in which a program for controlling a vehicle is recorded, and a processor that executes the program, wherein the program is stored at a first time using a camera installed on one side of the vehicle. generating a first image for a second view using the camera; generating a depth image for the first image as a reference image among the first and second images; Determining a target area associated with the position of the vehicle on the reference image based on the depth image, determining whether an object exists in the target area, and if an object exists in the target area, The step of controlling the vehicle by performing a preset operation is performed.

A method and apparatus for outputting a graphic object related to driving information of a vehicle may be provided.

A method and apparatus for controlling a vehicle may be provided.

1 illustrates a blind spot that appears while driving a vehicle according to an example.
2 is a configuration diagram of an electronic device according to an embodiment.
3 is a flowchart of a method of outputting a graphic object according to an exemplary embodiment.
4 is a flowchart of a method of generating a depth image for a reference image according to an example.
5 is a flowchart of a method of calculating a transformation relationship between a first image and a second image according to an example.
6 illustrates different images represented by differences between coordinate systems of cameras according to an example.
7 illustrates a plurality of virtual planes set based on a first image and a second image according to an example.
8 illustrates a method of determining a target depth value for a target pixel of a reference image according to an example.
9A illustrates a target area having a linear shape and a graphic object corresponding to the target area according to an example.
9B illustrates a target area being a 2D area and a graphic object corresponding to the target area according to an example.
9C illustrates a target area being a 3D area and a graphic object corresponding to the target area according to an example.
10 is a flowchart of a method of performing a preset operation based on whether an object exists in a target area according to an example.
11 is a flowchart of a method of determining whether an object exists in a target area based on a current driving situation of a vehicle according to an example.
12 is a flowchart of a method of determining whether an object exists in a target area according to an example.
13 illustrates a target area for determining whether an object exists according to an example.
14 is a flowchart of a method of performing a preset operation according to an example.
15 illustrates an alarm output through a HUD of a vehicle according to an example.
16 is a flowchart of a method for controlling a vehicle according to an exemplary embodiment.

Specific structural or functional descriptions of the embodiments are disclosed for illustrative purposes only, and may be changed and implemented in various forms. Therefore, the form actually implemented is not limited only to the specific embodiments disclosed, and the scope of the present specification includes changes, equivalents, or substitutes included in the technical idea described in the embodiments.

Although terms such as first or second may be used to describe various components, such terms should only be construed for the purpose of distinguishing one component from another. For example, a first element may be termed a second element, and similarly, a second element may be termed a first element.

It should be understood that when an element is referred to as being “connected” to another element, it may be directly connected or connected to the other element, but other elements may exist in the middle.

Singular expressions include plural expressions unless the context clearly dictates otherwise. In this specification, terms such as "comprise" or "have" are intended to designate that the described feature, number, step, operation, component, part, or combination thereof exists, but one or more other features or numbers, It should be understood that the presence or addition of steps, operations, components, parts, or combinations thereof is not precluded.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with the meaning in the context of the related art, and unless explicitly defined in this specification, it should not be interpreted in an ideal or excessively formal meaning. don't

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. In the description with reference to the accompanying drawings, the same reference numerals are given to the same components regardless of reference numerals, and overlapping descriptions thereof will be omitted.

1 illustrates a blind spot that appears while driving a vehicle according to an example.

The driver of the vehicle 110 may change lanes while driving the vehicle 110 . At this time, the user needs to check whether there is a vehicle 115 operating in the right lane in order to change from the current driving lane to the right lane. When the vehicle 115 is close to the vehicle 110, the driver must pay attention to driving. However, when the vehicle 115 is located in the driver's blind spot 120, the driver may not be able to recognize the vehicle 115, and in a state where the driver does not recognize the vehicle 115, the driving lane of the vehicle 110 If you change it, an accident may occur.

According to an embodiment, an image of the surroundings of the vehicle 110 may be generated using a camera of the vehicle 110 to provide a driver with a view of the blind spot 120 . Information on the surrounding area of the vehicle 110 may be sensed based on the generated image, and the sensed information may be provided to the user through a graphic object. The user may accurately and in detail recognize the surrounding area of the vehicle 110 through the graphic object.

Below, a method of outputting a graphic object is described in detail with reference to FIGS. 2 to 16 .

2 is a configuration diagram of an electronic device according to an embodiment.

According to an embodiment, the electronic device 200 includes a communication unit 210, a processor 220, and a memory 230. Additionally, the electronic device 200 may further include a camera (not shown). For example, the camera may be included in a closed circuit television (CCTV) or included in a black box system, and is not limited to the described embodiment.

The communication unit 210 is connected to the processor 220 and the memory 230 to transmit and receive data. The communication unit 210 may transmit/receive data by being connected to another external device. Hereinafter, the expression “transmitting and receiving “A” may indicate transmitting and receiving “information or data indicating A”.

The communication unit 210 may be implemented as circuitry within the electronic device 200 . For example, the communication unit 210 may include an internal bus and an external bus. As another example, the communication unit 210 may be an element that connects the electronic device 200 and an external device. The communication unit 210 may be an interface. The communication unit 210 may receive data from an external device and transmit the data to the processor 220 and the memory 230 .

The processor 220 processes data received by the communication unit 210 and data stored in the memory 230 . The processor 220 may be an Image Signal Processor (ISP).

A “processor” may be a data processing device implemented in hardware having circuitry having a physical structure for executing desired operations. For example, desired operations may include codes or instructions included in a program. For example, a data processing unit implemented in hardware includes a microprocessor, a central processing unit, a processor core, a multi-core processor, and a multiprocessor. , Application-Specific Integrated Circuit (ASIC), and Field Programmable Gate Array (FPGA).

Processor 220 executes computer readable code (eg, software) stored in memory (eg, memory 230 ) and instructions invoked by processor 220 .

The memory 230 stores data received by the communication unit 210 and data processed by the processor 220 . For example, the memory 230 may store a program (or application or software). The stored program may be a set of syntaxes coded to output graphic objects and executable by the processor 220 .

According to one aspect, the memory 230 may include one or more of volatile memory, non-volatile memory and random access memory (RAM), flash memory, a hard disk drive, and an optical disk drive.

The memory 230 stores a command set (eg, software) for operating the electronic device 200 . A set of instructions for operating the electronic device 200 is executed by the processor 220 .

The communication unit 210, the processor 220, and the memory 230 will be described in detail with reference to FIGS. 3 to 15 below.

3 is a flowchart of a method of outputting a graphic object according to an exemplary embodiment.

The steps below are performed by the electronic device 200 described above with reference to FIG. 2 .

In step 310, the electronic device 200 generates a first image for the first view. For example, the processor 220 may generate a first image for a first view using a camera installed on one side of the vehicle (eg, a side mirror or a position corresponding to the side mirror). For example, the first image may include a part (eg, a side part or a rear part) of the vehicle and a rear scene.

In step 320, the electronic device 200 generates a second image for the second view. For example, the processor 220 may generate a second image for a second viewpoint using a camera installed on one side of the vehicle. That is, the camera generating the first image and the second image may be the same camera. The second image may be a next image generated by the camera after generating the first image.

According to one embodiment, the camera may generate images at regular intervals. For example, a camera can produce 60 images in one second (ie, 60 fps). The first image and the second image may be any two temporally consecutive images among continuously generated images.

In step 330, the electronic device 200 generates a depth image for the first image as the reference image among the first image and the second image. Hereinafter, the first image is described as a reference image, but in another embodiment, the second image may be the reference image, and in this case, the description of the first image may be similarly applied to the description of the second image.

According to an embodiment, since the first image and the second image are images generated by photographing almost the same scene, homography between the images may be calculated. For example, in order to calculate the homography, a conversion relationship between a camera that captures a first image at a first point of view and a camera that captures a second image at a second point of view must be calculated. A method of generating a depth image for a reference image (eg, a first image) based on a transformation relationship between cameras and a homography between images will be described in detail with reference to FIGS. 4 to 8 below.

In step 340, the electronic device 200 determines a target area related to the location of the vehicle on the reference image based on the depth image. For example, the target area may include at least a portion of a blind spot of the vehicle. For example, the target area may be an area around the rear of the vehicle.

According to an embodiment, a spatial coordinate system may be set around the vehicle, and a target area may be preset on the spatial coordinate system. By generating a depth image for the reference image, a coordinate system and a spatial coordinate system of the reference image may be matched with each other. A target area may be determined on the reference image based on matching between the coordinate systems.

For example, the target area may be a straight or linear area corresponding to a part of a floor area in a scene indicated by the reference image. As another example, the target area may be a 2D area (eg, a rectangle) corresponding to a part of a floor area in a scene indicated by the reference image. As another example, the target area may be a 3D area (eg, a box-shaped area) including a part of a floor area in a scene indicated by the reference image. The 2D area and the 3D area may be set based on straight line or straight line type areas.

In step 350, the electronic device 200 outputs a graphic object representing the target area through a display. The graphics object may be a computer graphics object. For example, the graphic object may be an object representing a straight line or a straight line-shaped area. As another example, the graphic object may be an object representing a boundary of a 2D area. As another example, the graphic object may be an object representing a boundary of a 3D area.

According to one embodiment, the display may be a display included in a side mirror of a vehicle. The electronic device 200 may output the graphic object to be overlaid on the scene appearing on the side mirror through the display.

According to an embodiment, the display may be a display that outputs images (eg, a first image and a second image) generated using a camera located in a side mirror of a vehicle. For example, the electronic device 200 may synthesize a graphic object with an output image and output the synthesized image of the graphic object through a display. As another example, the electronic device 200 may output the graphic object through the display so that the graphic object is overlaid on the output image.

A graphic object output through a display will be described in detail below with reference to FIG. 9 .

The driver can visually check the blind spots of the vehicle through graphic objects.

4 is a flowchart of a method of generating a depth image for a reference image according to an example.

According to one embodiment, step 330 described above with reference to FIG. 3 may include steps 410 to 470 below.

In operation 410, the electronic device 200 may calculate a conversion relationship between the first image and the second image. For example, a difference between an extrinsic parameter of a camera at a first view of a first image and an extrinsic parameter of a camera at a second view of a second image may be calculated as a transformation relationship. External parameters may include camera rotation information and camera translation information. The transformation relationship may include a difference between rotation information and a difference between position movement information.

In the following, with reference to Fig. 5, a method for calculating the transformation relation is described in detail.

In step 420, the electronic device 200 may set a plurality of virtual planes having different depth values based on the first image based on the transformation relationship. The difference between the number of the plurality of virtual planes and depth values between the virtual planes may be set in advance. For example, the number of virtual planes may be set to 100, and a difference between depth values between virtual planes may be set to 1 meter. In the above example, the first virtual plane having the smallest depth value based on the first image may have a depth value of 1 meter, and the 100th virtual plane having the largest depth value may have a depth value of 100 meters. . The number of virtual planes and a difference between depth values between the virtual planes may vary according to settings.

According to an embodiment, a coordinate system of each of the plurality of virtual planes may be calculated based on a transformation relationship between the first image and the second image. For example, the coordinate system of each virtual plane may be expressed as a homography with respect to the coordinate system of the first image (reference image). A method of calculating the coordinate system of each of the plurality of virtual planes will be described in detail below with reference to FIGS. 6 and 7 .

In step 430, the electronic device 200 generates a plurality of virtual images corresponding to the plurality of virtual planes based on at least one of the first image and the second image and the homography of each of the plurality of virtual planes. can For example, the virtual image may correspond to an image captured by an actual camera having a depth value adjusted in a coordinate system of a corresponding virtual plane. Each of the plurality of virtual images may include a scene corresponding to at least a part of the scene shown in the first image.

In step 440, the electronic device 200 may calculate matching costs for matching pixels in each of a plurality of virtual images corresponding to the target pixel in the first image. For example, matching pixels for the target pixel may be pixels in a plurality of virtual images having the same coordinates as the coordinates in the image of the target pixel.

According to an embodiment, the electronic device 200 may set all pixels in the first image as target pixels and calculate matching costs for each pixel.

According to an embodiment, matching costs for matching pixels corresponding to a target pixel may be calculated based on sum of absolute difference (SAD) or zero mean normalized cross-correlation (ZNCC), but are not limited to the described embodiment. don't

Matching costs calculated for pixels in the first image may be defined as a cost volume. The cost volume is explained in detail with reference to FIG. 8 below.

In operation 450, the electronic device 200 may determine a final matching pixel for the target pixel based on the calculated matching costs. For example, a matching pixel having the lowest matching cost among matching pixels may be determined as a final matching pixel.

In operation 460, the electronic device 200 may determine a depth value of a target virtual image among a plurality of virtual images including a final matching pixel as a target depth value of a target pixel. For example, when the determined final matching pixel is included in the fourth virtual image, a depth value of 4 meters of the fourth virtual image may be determined as a depth value of the target pixel.

In operation 470, the electronic device 200 may generate a depth image based on the target depth value. For example, depth values may be determined for all pixels in the first image, and a depth image having the determined depth values may be generated.

5 is a flowchart of a method of calculating a transformation relationship between a first image and a second image according to an example.

According to one embodiment, step 410 described above with reference to FIG. 4 may include steps 510 to 520 below.

In operation 510, the electronic device 200 may calculate the transformation relationship between the first image and the second image using a scale invariant feature transform (SIFT) algorithm.

In step 520, the electronic device 200 may calculate the transformation relationship between the first image and the second image using the PoseNet model. The PoseNet model may be a model based on a neural network. For example, the PoseNet model may be a model trained using deep-learning methods.

According to an embodiment, the electronic device 200 may calculate a conversion relationship between the first image and the second image through step 510 or step 520, but is not limited to the described embodiment.

6 illustrates different images represented by differences between coordinate systems of cameras according to an example.

Although the first image 614 generated in the coordinate system P _ref of the first camera and the second image 624 generated in the coordinate system P _k of the second camera capture the same scene 610, the coordinate system may be different due to the difference between the image planes 612 and 622 due to the difference between them. For example, the object X in the scene 610 may appear at a position u in the first image 614 and at a position u' in the second image 624 .

The location of the object X in the scene 610 in the image may be changed not only by the difference between coordinate systems of the cameras, but also by the distance difference between the camera and the scene or object X.

A depth value of the object X in the scene 610 may be calculated using different images generated by the difference between the aforementioned image planes 612 and 622 .

Referring to FIGS. 7 and 8 below, a process for calculating a depth value for a pixel in a reference image will be described in detail.

7 illustrates a plurality of virtual planes set based on a first image and a second image according to an example.

According to an embodiment, a plurality of virtual planes having different depth values may be set based on the first image based on a transformation relationship between the first image and the second image. For example, a homography of a first virtual plane among a plurality of virtual planes may be determined based on an external parameter of the first image, an external parameter of the second image, and a first depth value of the first virtual plane.

According to an embodiment, the homography of the coordinate system of the virtual plane may be calculated using [Equation 1] below.

[Equation 1]

In [Equation 1], k represents a camera at the second viewpoint, and P _k represents the coordinate system of the camera at the second viewpoint. m is the number of a plurality of virtual planes, and for example, the value of m is a natural number from 1 to m. n _m is a difference between depth values between adjacent virtual planes, and n _m may have the same value even if m changes. For example, if m is set to 100 and the difference between depth values between adjacent virtual planes is set to 1 meter, then n ₁ , n ₂ , n ₃ , ..., n ₁₀₀ are all 1 meter . d _m represents the depth value of the mth virtual plane, and in the example of the above set values, d ₁ represents 1 meter as the depth value of the first virtual plane, and d ₂ represents 2 as the depth value of the second virtual plane. Indicates a meter, and d ₁₀₀ is a depth value of the 100th virtual plane and may indicate 100 meters. K _k may represent an internal parameter of the camera at the second time, and K _ref may represent an internal parameter of the camera at the first time. For example, internal parameters of the camera may include a focal length, a principal point, and an asymmetry coefficient of the camera.

R _rel represents a difference between rotation information among transformation relationships between the first image and the second image, and may be calculated by [Equation 2] below.

[Equation 2]

In [Equation 2], R _k may represent rotation information of the second image, and R _ref may represent rotation information of the first image.

t _rel may indicate a difference between positional movement information among transformation relationships between the first image and the second image, and may be calculated by [Equation 3] below.

[Equation 3]

In [Equation 3], t _ref may represent location movement information of the first image, and t _k may represent location movement information of the second image.

로 정의될 수 있다. 여기서, d_m이 음수인 이유는, 장면의 시점과 카메라의 시점이 서로 반대이기 때문이다.In [Equation 1],

can be defined as Here, the reason d _m is a negative number is that the viewpoint of the scene and the viewpoint of the camera are opposite to each other.

는 기준 이미지(제1 이미지)의 좌표계에 대한 제m 가상 평면의 호모그래피를 나타낸다. 예를 들어, m이 100인 경우, 총 100개의 가상 평면들에 대한 호모그래피들이 계산될 수 있다.In [Equation 1],

represents the homography of the mth virtual plane with respect to the coordinate system of the reference image (first image). For example, when m is 100, homographies for a total of 100 virtual planes may be calculated.

According to one embodiment, a first homography is calculated for a first virtual plane (711), a second homography is calculated for a second virtual plane (712), and a second homography is calculated for a third virtual plane (713). 3 homography may be calculated, and a fourth homography for the fourth virtual plane 714 may be calculated.

According to an embodiment, the electronic device 200 may generate a virtual image based on the reference image (first image) and the homography of the virtual plane. For example, a first virtual image of a first virtual plane 711 is created, a second virtual image of a second virtual plane 712 is created, and a third virtual image of a third virtual plane 713 is created. An image may be created, and a fourth virtual image of the fourth virtual plane 714 may be created.

8 illustrates a method of determining a target depth value for a target pixel of a reference image according to an example.

For example, a process for determining a target depth value for a target pixel 801 in a reference image 800 is described below. The reference image 800 may include a x b number of pixels.

A cost volume 810 for all pixels in the reference image 800 may be computed. When the number of virtual images generated based on the reference image 800 is c, the cost volume 810 may include a x b x c costs.

A matching pixel having the lowest matching cost (eg, 15) among matching costs 811 of matching pixels for the target pixel 801 may be determined as a final matching pixel. A plane number of a virtual image (eg, a fourth virtual image) including a final matching pixel may be determined as a target depth value for the target pixel 801 . A plane number may correspond to a depth value of a corresponding virtual image. A depth image 820 may be created to include pixels 821 having target depth values for target pixels 801 .

9A illustrates a target area having a linear shape and a graphic object corresponding to the target area according to an example.

According to one embodiment, the target area 911 associated with the location of the vehicle may be a straight line or straight line-shaped area having a preset length from a preset point at the rear of the vehicle. For example, a preset point of the rear of the vehicle may be determined based on the position of the camera 910 . For example, a preset length of the target area 911 may correspond to a typical vehicle width of a vehicle.

A spatial coordinate system based on the position of the camera 900 or the vehicle may be set in advance, and the target area 911 may be determined within the spatial coordinate system. δ _w , δ _h , and δ _l for defining the target region 911 may be predefined. δ _w , δ _h , and δ _l may represent the width, width, and length of the target area 911 .

According to one embodiment, the target area 911 may change based on the steering angle of the vehicle. For example, when the vehicle is going straight, the target area 911 may face the same direction as the vehicle's traveling direction. As another example, even if the vehicle turns right, the target area 911 may be changed to face the direction of the road.

According to one embodiment, the location of the target area 911 may be determined on the reference image 920 . For example, a coordinate system of the reference image 920 may be matched to a spatial coordinate system based on the depth image. When the coordinate system of the reference image 920 and the spatial coordinate system match, the target region 911 may be determined on the reference image.

The position of the target region 911 on the reference image may be determined by δ _x , δ _y , and δ _z . δ _x and δ _y may mean pixel coordinates in the reference image, and δ _z may mean a depth value. For example, δ _x , δ _y , and δ _z can be calculated based on δ _w , δ _h , and δ _l . When the target area 911 appears as a vertex or edge, pixels of the reference image corresponding to the vertex and edge are determined, and the x coordinate, y coordinate of the corresponding pixel and the depth value of the corresponding pixel are δ _x , δ _y , and δ _z can be determined by

A graphic object 921 representing the target area 911 may be output through a display. For example, the graphic object 921 may be output together with the reference image 920 . As another example, the graphic object 921 may be output so as to be overlaid on a scene appearing on the mirror.

9B illustrates a target area being a 2D area and a graphic object corresponding to the target area according to an example.

According to one embodiment, the target area 912 associated with the location of the vehicle may be determined based on the target area 911 described above with reference to FIG. 9A . For example, the target area 912 may be a two-dimensional area including the target area 911 . For example, the 2D area may be set to correspond to a part of a floor area in a scene represented by a reference image.

According to one embodiment, the target area 912 may change based on the steering angle of the vehicle. For example, when the vehicle is going straight, the target area 912 may be directed in the same direction as the vehicle's traveling direction. As another example, even if the vehicle turns right, the target area 912 may be changed to face the direction of the road.

When the coordinate system of the reference image 920 and the spatial coordinate system match, the target area 912 may be determined on the reference image. A graphic object 922 representing the target area 912 may be output through a display. For example, the graphic object 922 may be output together with the reference image 920 . As another example, the graphic object 922 may be output so as to be overlaid on a scene appearing on the mirror.

9C illustrates a target area being a 3D area and a graphic object corresponding to the target area according to an example.

According to one embodiment, the target area 913 associated with the location of the vehicle may be determined based on the target area 911 described above with reference to FIG. 9A and the target area 912 described above with reference to FIG. 9B. For example, the target area 913 may be a 3D area including the target area 911 and the target area 912 . For example, the 3D area may be set to correspond to a space (eg, a box shape) including a part of a floor area in a scene represented by a reference image.

According to one embodiment, the target area 913 may change based on the steering angle of the vehicle. For example, when the vehicle is traveling straight, the target area 913 may face the same direction as the vehicle's traveling direction. As another example, even if the vehicle turns right, the target area 913 may be changed to face the direction of the road.

When the coordinate system of the reference image 920 and the spatial coordinate system match, the target region 913 may be determined on the reference image. A graphic object 923 indicating the target area 913 may be output through a display. For example, the graphic object 923 may be output together with the reference image 920 . As another example, the graphic object 923 may be output so as to be overlaid on a scene appearing on the mirror.

10 is a flowchart of a method of performing a preset operation based on whether an object exists in a target area according to an example.

According to one embodiment, after step 340 described above with reference to FIG. 3 is performed, steps 1010 and 1020 below may be further performed.

In operation 1010, the electronic device 200 may determine whether an object exists within the target area. For example, the target area may be the target area 911 , 912 , and 913 described above with reference to FIG. 9 . A method of determining whether an object exists in the target area will be described in detail with reference to FIGS. 11 to 13 below.

In operation 1020, the electronic device 200 may perform a preset operation when an object exists within the target area. For example, the preset operation may be an operation of outputting an alarm to the user. As another example, the preset operation may be a vehicle control operation. A method of performing a preset operation will be described in detail with reference to FIG. 14 below.

11 is a flowchart of a method of determining whether an object exists in a target area based on a current driving situation of a vehicle according to an example.

According to one embodiment, step 1010 described above with reference to FIG. 10 may include steps 1110 to 1130 below.

In step 1110, the electronic device 200 may determine the current driving situation of the vehicle. For example, the current driving situation may be determined based on whether the vehicle is steered, whether a turn signal is used, whether a brake is used, and the like.

In step 1120, the electronic device 200 may determine whether the current driving situation corresponds to a preset driving situation. For example, the preset driving situation may include a situation in which the vehicle user rotates the vehicle and a situation in which direction indicators are used.

In operation 1130, the electronic device 200 may determine whether an object exists in the target area when the current driving situation corresponds to a preset driving situation. When the current driving condition does not correspond to the preset driving condition, step 1130 may not be performed.

12 is a flowchart of a method of determining whether an object exists in a target area according to an example.

According to one embodiment, step 1010 described above with reference to FIG. 10 may include steps 1210 to 1230 below.

In operation 1210, the electronic device 200 may determine an object sensing value within the target area at a first point in time (ie, a point in time when the first image or reference image is generated).

According to an embodiment, an object sensing value in the target area may be calculated based on pixel values and depth values of pixels constituting the target area in the reference image. For example, a target area in the reference image may include a plurality of sub-target areas, and an average value of pixel values of pixels for each of the plurality of sub-target areas may be determined as the object sensing value. A plurality of sub target regions and an object sensing value corresponding thereto will be described in detail below with reference to FIG. 13 .

In operation 1220, the electronic device 200 may calculate a change amount between a previous object sensing value in the previous target area determined at a previous time point of the first time point and an object sensing value determined at the first time point point in time. For example, when a vehicle that did not exist in the previous target area at the previous time enters the target area at the first time, the amount of change between object sensing values may increase. For example, the previous object sensing value based on the floor of the road is calculated in the previous target area at the previous time, but when a vehicle appears in the target area at the first time, the object sensing value based on the vehicle is calculated, so the amount of change may increase. .

In operation 1230, the electronic device 200 may determine that an object appears or exists when a difference between a previous object sensing value and an object sensing value is greater than or equal to a preset threshold value.

13 illustrates a target area for determining whether an object exists according to an example.

According to an embodiment, the target area 1310 of the vehicle 1300 may include a plurality of sub target areas 1311 , 1312 , and 1313 . For example, each of the plurality of sub target regions 1311, 1312, and 1313 may have a different depth value. For example, since the depth image for the reference image is generated, pixels of interest corresponding to the position of the target region 1310 among pixels in the reference image may be determined. Among the pixels of interest, pixels having the same depth values may constitute the same sub-target region.

According to an embodiment, since the reference image is a projection of a scene onto an image plane, information about a space in which an object does not exist may not appear in the reference image. Accordingly, pixels of interest in an image corresponding to a specific sub-target region (eg, the first sub-target region 1311) may vary according to the generation time of the reference image. For example, when there is no object in the target area 1310, there may be no interest pixels for the first sub-target area 1311. As another example, when the vehicle 1301 exists in the target area 1310 as shown in the illustrated example, there may be many interest pixels for the first sub-target area 1311 .

For example, an average value of pixel values of pixels of interest for each of a plurality of sub target regions may be determined as the object sensing value. As another example, the number of interest pixels for each of the plurality of sub target regions may be determined as an object sensing value.

14 is a flowchart of a method of performing a preset operation according to an example.

According to one embodiment, step 1020 described above with reference to FIG. 10 may include steps 1410 and 1420 below.

In operation 1410, when an object exists in the target area, the electronic device 200 may output an alarm to the user of the vehicle as a preset operation. For example, an alarm may be output through one of sound and video.

According to an embodiment, the electronic device 200 may output an alarm through a display of an instrument panel inside the vehicle. For example, the electronic device 200 may output an alarm through a head up display (HUD) of a vehicle. An alarm output through the HUD will be described in detail below with reference to FIG. 15 .

In step 1420, when an object exists in the target area, the electronic device 200 may control at least one of a driving direction and a driving speed of the vehicle as a preset operation.

According to one aspect, when it is determined that an object exists in the target area while the vehicle is changing lanes, the driving direction of the vehicle may be controlled so that the vehicle returns to the original lane.

15 illustrates an alarm output through a HUD of a vehicle according to an example.

According to an embodiment, the electronic device 200 may output a graphic or animation image 1500 as an alarm through the HUD to call the user's attention. For example, when another vehicle 1520 exists in the target area 1511 in a situation where the user's attention to the target area 1511 of the vehicle 1510 is required, an image 1500 is displayed as an alarm through the HUD. can be output.

16 is a flowchart of a method for controlling a vehicle according to an exemplary embodiment.

The steps below are performed by the electronic device 200 described above with reference to FIG. 2 .

In operation 1610, the electronic device 200 generates a first image for a first view. A description of step 1610 may be replaced with a description of step 310 described above with reference to FIG. 3 .

In step 1620, the electronic device 200 generates a second image for the second view. A description of step 1620 may be replaced with a description of step 320 described above with reference to FIG. 3 .

In operation 1630, the electronic device 200 generates a depth image for the first image as the reference image among the first image and the second image. A description of step 1630 may be replaced with a description of step 330 described above with reference to FIG. 3 .

In step 1640, the electronic device 200 determines a target area related to the location of the vehicle on the reference image based on the depth image. A description of step 1640 may be replaced with a description of step 340 described above with reference to FIG. 3 .

In step 1650, the electronic device 200 determines whether an object exists within the target area. A description of step 1650 may be replaced with a description of step 1010 described above with reference to FIG. 10 .

In operation 1660, the electronic device 200 controls the vehicle by performing a preset operation when an object exists within the target area. A description of step 1660 may be replaced with a description of step 1020 described above with reference to FIG. 10 .

The embodiments described above may be implemented as hardware components, software components, and/or a combination of hardware components and software components. For example, the devices, methods and components described in the embodiments may include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate (FPGA). array), programmable logic units (PLUs), microprocessors, or any other device capable of executing and responding to instructions. The processing device may execute an operating system (OS) and software applications running on the operating system. A processing device may also access, store, manipulate, process, and generate data in response to execution of software. For convenience of understanding, there are cases in which one processing device is used, but those skilled in the art will understand that the processing device includes a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that it can include. For example, a processing device may include a plurality of processors or a processor and a controller. Other processing configurations are also possible, such as parallel processors.

Software may include a computer program, code, instructions, or a combination of one or more of the foregoing, which configures a processing device to operate as desired or processes independently or collectively. The device can be commanded. Software and/or data may be any tangible machine, component, physical device, virtual equipment, computer storage medium or device, intended to be interpreted by or provide instructions or data to a processing device. , or may be permanently or temporarily embodied in a transmitted signal wave. Software may be distributed on networked computer systems and stored or executed in a distributed manner. Software and data may be stored on computer readable media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer readable medium. The computer readable medium may include program instructions, data files, data structures, etc. alone or in combination, and the program instructions recorded on the medium may be specially designed and configured for the embodiment or may be known and usable to those skilled in the art of computer software. may be Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. - includes hardware devices specially configured to store and execute program instructions, such as magneto-optical media, and ROM, RAM, flash memory, and the like. Examples of program instructions include high-level language codes that can be executed by a computer using an interpreter, as well as machine language codes such as those produced by a compiler.

The hardware device described above may be configured to operate as one or a plurality of software modules to perform the operations of the embodiments, and vice versa.

As described above, although the embodiments have been described with limited drawings, those skilled in the art can apply various technical modifications and variations based on this. For example, the described techniques may be performed in an order different from the method described, and/or components of the described system, structure, device, circuit, etc. may be combined or combined in a different form than the method described, or other components may be used. Or even if it is replaced or substituted by equivalents, appropriate results can be achieved.

Therefore, other implementations, other embodiments, and equivalents of the claims are within the scope of the following claims.

200: electronic device
210: communication department
220: processor
230: memory

Claims (23)

The graphic object output method, performed by the electronic device,
generating a first image for a first point of view using a camera installed on one side of the vehicle;
generating a second image for a second viewpoint using the camera;
generating a depth image for the first image as a reference image among the first image and the second image;
determining a target area associated with a position of the vehicle on the reference image based on the depth image; and
outputting a graphic object representing the target area through a display;
including,
Graphic object output method.
According to claim 1,
The second image is a next image generated after the camera generates the first image,
Graphic object output method.
According to claim 1,
The step of generating the depth image,
calculating a transformation relationship between the first image and the second image;
setting a plurality of virtual planes having different depth values based on the first image based on the transformation relationship;
generating a plurality of virtual images corresponding to the plurality of virtual planes based on the first image and a homography of each of the plurality of virtual planes with respect to the first image;
calculating matching costs for matching pixels in each of the plurality of virtual images corresponding to a target pixel in the first image;
determining a final matching pixel based on the matching costs;
determining a depth value of a target virtual image among the plurality of virtual images including the final matching pixel as a target depth value of the target pixel; and
generating the depth image based on the target depth value;
including,
Graphic object output method.
According to claim 3,
Calculating the transformation relationship between the first image and the second image,
calculating the transformation relationship between the first image and the second image using a scale invariant feature transform (SIFT) algorithm; or
Calculating the transformation relationship between the first image and the second image using a PoseNet model
including,
Graphic object output method.
According to claim 3,
A first homography of a first virtual plane among the plurality of virtual planes is determined based on an external parameter of the first image, an external parameter of the second image, and a first depth value of the first virtual plane.
Graphic object output method.
According to claim 3,
Calculating matching costs for matching pixels in each of the plurality of virtual images corresponding to a target pixel in the reference image,
Calculating the matching costs based on sum of absolute difference (SAD) or zero mean normalized cross-correlation (ZNCC)
including,
Graphic object output method.
According to claim 1,
The target area is a straight or linear area corresponding to a part of a floor area in a scene represented by the first image,
The graphic object is an object representing the straight line or the straight line-shaped area,
Graphic object output method.
According to claim 1,
The target area is a two-dimensional area corresponding to a part of a floor area in a scene represented by the first image,
The graphic object is an object representing the boundary of the two-dimensional area,
Graphic object output method.
According to claim 1,
The target area is a three-dimensional area including a part of a floor area in a scene represented by the first image,
The graphic object is an object representing the boundary of the 3D area,
Graphic object output method.
According to claim 9,
determining whether an object exists within the target area; and
performing a preset operation when an object exists within the target area;
Including more,
Graphic object output method.
According to claim 10,
Determining whether an object exists in the target area includes:
determining a current driving condition of the vehicle; and
determining whether an object exists in the target area when the current driving situation corresponds to a preset driving situation;
including,
Graphic object output method.
According to claim 10,
Determining whether an object exists in the target area includes:
determining an object sensing value within the target area at the first time point;
calculating a change amount between a previous object sensing value in a previous target area determined at a time before the first time and the object sensing value; and
Determining that an object exists when the amount of change is equal to or greater than a preset threshold value
including,
Graphic object output method.
According to claim 10,
When an object exists in the target area, performing a preset operation includes:
outputting an alarm to a user of the vehicle as the preset operation;
including,
Graphic object output method.
According to claim 10,
When an object exists in the target area, performing a preset operation includes:
Controlling at least one of a driving direction or a driving speed of the vehicle as the preset operation
including,
Graphic object output method.
According to claim 1,
The display is included in the side mirror of the vehicle,
The graphic object is overlaid on the scene appearing on the side mirror,
Graphic object output method.
According to claim 1,
The step of outputting a graphic object representing the target area through a display,
outputting the reference image overlaid with the graphic object through the display;
including,
Graphic object output method.
A computer program stored in a computer readable recording medium to be combined with hardware to execute the method of any one of claims 1 to 16.
electronic devices,
a memory in which a program outputting graphic objects is recorded; and
Processor that executes the above program
including,
said program,
generating a first image for a first point of view using a camera installed on one side of the vehicle;
generating a second image for a second viewpoint using the camera;
generating a depth image for the first image as a reference image among the first image and the second image;
determining a target area associated with a position of the vehicle on the reference image based on the depth image; and
outputting a graphic object representing the target area through a display;
to do,
electronic device.
A vehicle control method, performed by an electronic device,
generating a first image for a first point of view using a camera installed on one side of the vehicle;
generating a second image for a second viewpoint using the camera;
generating a depth image for the first image as a reference image among the first image and the second image;
determining a target area associated with a position of the vehicle on the reference image based on the depth image;
determining whether an object exists within the target area; and
Controlling the vehicle by performing a preset operation when an object exists within the target area.
including,
vehicle control method.
According to claim 19,
Determining whether an object exists in the target area includes:
determining a current driving condition of the vehicle; and
determining whether an object exists in the target area when the current driving situation corresponds to a preset driving situation;
including,
vehicle control method.
According to claim 19,
Determining whether an object exists in the target area includes:
determining an object sensing value within the target area at the first time point;
calculating a change amount between a previous object sensing value in a previous target area determined at a time before the first time and the object sensing value; and
Determining that an object exists when the amount of change is equal to or greater than a preset threshold value
including,
vehicle control method.
A computer program stored in a computer readable recording medium in order to execute the method of any one of claims 19 to 21 in combination with hardware.
electronic devices,
a memory in which a program for controlling a vehicle is recorded; and
Processor that executes the above program
including,
said program,
generating a first image for a first point of view using a camera installed on one side of the vehicle;
generating a second image for a second viewpoint using the camera;
generating a depth image for the first image as a reference image among the first image and the second image;
determining a target area associated with a position of the vehicle on the reference image based on the depth image;
determining whether an object exists within the target area; and
Controlling the vehicle by performing a preset operation when an object exists within the target area.
to do,
electronic device.

Images