KR20210027974A - Vehicle and controlling method of vehicle - Google Patents

Abstract

The present disclosure provides a vehicle and a vehicle control method for learning the environment around the vehicle as a semantic unit through an artificial neural network, dividing a captured image into the learned semantic unit, and reconstructing it. According to an aspect of the present disclosure, a vehicle comprises: a display unit displaying a captured image; an image capturing unit for capturing an image of the surrounding area; a first artificial neural network for learning a semantic unit of an object existing in the captured image by deep learning; and a control unit which divides the captured image based on the learning result, determines whether a detection target exists in the divided image, simplifies least one of a pattern or a shape of the detection target based on the determination result to emphasize the detection target, restores the divided image into one image, and controls the display unit to generate a virtual image based on the restored image.

Description

Vehicle and vehicle control method {VEHICLE AND CONTROLLING METHOD OF VEHICLE}

The vehicle according to the disclosed aspect relates to autonomous driving of the vehicle. Specifically, the vehicle according to the disclosed aspect relates to capturing and correcting an image of a surrounding environment in the vehicle autonomously driving.

Conventionally, there has been a technology for capturing an environment around a vehicle as an image for safe autonomous driving of a vehicle.

Specifically, in the prior art, in order to make it easier for a user to recognize a captured image, a specific captured image is processed.

However, the above method has a problem of deriving only general feature points through simple image processing irrespective of the purpose for which the user wants to extract feature points from a captured image.

Accordingly, in autonomous driving of a vehicle, there is an urgent need for a technology to extract feature points necessary for the user from the screen itself captured by the vehicle while the vehicle is currently driving.

In order to solve the above problems, one aspect disclosed is a vehicle and a vehicle control method that learns the environment around the vehicle as a semantic unit through an artificial neural network, divides the captured image into the learned semantic unit, and restores it again. Provides.

A vehicle according to the disclosed aspect includes a display unit for displaying a photographed image; An image photographing unit for photographing surrounding images; A first artificial neural network that learns a semantic unit of an object present in the captured image by deep learning; Dividing the captured image based on the learning result, determining whether a detection target exists in the divided image, and highlighting the detection target by simplifying at least one of a pattern or shape of the detection target based on the determination result And a control unit for restoring the divided image into one image and controlling the display unit to generate a virtual image based on the restored image.

The vehicle according to another disclosed aspect further includes a second artificial neural network that learns to combine the divided images, and the controller may generate a virtual image based on the learning result.

The control unit may correct a boundary of an object appearing in the divided image and may remove noise from the divided image.

The controller may assume that the vehicle is driving on a daytime and a sunny day, and may generate a virtual image having a constant illuminance based on the assumed situation.

The controller may differently set color information of the target based on the semantic unit and edge information of the target, and may improve the image quality of the divided image based on the set color.

The control unit may improve the quality of the restored image by using a Hall filter.

The control unit may simplify the shape of the object into a preset shape based on the meaning unit of the object.

The controller may compare a preset verification image with the restored image, learn to determine that the restored image is the same as the verification image as a result of the comparison, and generate a virtual image based on the learning result.

The control unit may remove noise from the divided image using a smoothing filter.

The controller may determine a driving condition of the vehicle and generate the virtual image based on a result of the determination.

A method for controlling a vehicle according to another aspect of the disclosure includes photographing a surrounding image; Display the captured image; Learning the semantic unit of an object present in the captured image by deep learning, segmenting the captured image based on the learning result, determining whether a detection target exists in the segmented image, and determining whether a detection target exists in the segmented image, and based on the determination result Thus, at least one of the pattern or shape of the detection target is simplified to emphasize the detection target, restore the divided image to one image, and generate a virtual image based on the restored image.

The method of controlling a vehicle according to another aspect of the disclosure may further include learning to combine the divided images, and the controlling may generate a virtual image based on the learning result.

The controlling may correct a boundary of an object appearing in the divided image and remove noise from the divided image.

In the controlling, it is assumed that the vehicle is driving during the daytime and on a clear day, and a virtual image having a constant illuminance may be generated based on the assumed situation.

The controlling may differently set color information of the target based on the semantic unit and edge information of the target, and improve the image quality of the divided image based on the set color.

The controlling may improve the quality of the restored image by using a Hall filter.

The controlling may simplify the shape of the object into a preset shape based on the meaning unit of the object.

The controlling may include comparing a preset verification image and the restored image, learning to determine that the restored image is the same as the verification image as a result of the comparison, and generating a virtual image based on the learning result. .

In the controlling, noise of the divided image may be removed by using a smoothing filter.

The controlling may determine the driving condition of the vehicle and generate the virtual image based on the determination result.

The vehicle and vehicle control method according to the disclosed aspect is by learning the environment around the road through deep learning, dividing the image into semantic units based on the learning result, and highlighting the detection target in the divided image, thereby enabling autonomous driving of the vehicle. It is possible to improve the quality of the image required for the application.

In addition, the vehicle and the vehicle control method according to the disclosed aspect may generate a high-quality and stable vehicle surrounding image and generate an image by assuming a certain situation regardless of changes in the external environment of the vehicle.

1 shows a vehicle according to an embodiment.
2 is a control block diagram of a vehicle according to an exemplary embodiment.
3 is a diagram illustrating a process of generating a virtual image by a controller according to an exemplary embodiment.
4 is a diagram illustrating a process of improving the quality of a divided image by a control unit according to an exemplary embodiment.
5 illustrates that a first artificial neural network learns a semantic unit of an object according to an embodiment.
6 is a diagram illustrating a process of learning to combine a segmented image by a second artificial neural network and reconstruct it into a single image according to an exemplary embodiment.
7 is a diagram illustrating a process of generating a virtual image by a controller according to an exemplary embodiment.

The same reference numerals refer to the same elements throughout the specification. This specification does not describe all elements of the embodiments, and general content in the technical field to which the present invention belongs or overlapping content between the embodiments will be omitted. The term'unit, module, member, block' used in the specification may be implemented in software or hardware, and according to embodiments, a plurality of'units, modules, members, blocks' may be implemented as one component, It is also possible for one'unit, module, member, block' to include a plurality of components.

Throughout the specification, when a part is said to be "connected" with another part, this includes not only the case of being directly connected, but also the case of indirect connection, and the indirect connection includes connection through a wireless communication network. do.

In addition, when a part "includes" a certain component, it means that other components may be further included rather than excluding other components unless specifically stated to the contrary.

Terms such as first and second are used to distinguish one component from other components, and the component is not limited by the above-described terms.

Singular expressions include plural expressions, unless the context clearly makes exceptions.

In each step, the identification code is used for convenience of explanation, and the identification code does not describe the order of each step, and each step may be implemented differently from the specified order unless a specific sequence is clearly stated in the context. have.

Hereinafter, the operating principle and embodiments of the present invention will be described with reference to the accompanying drawings.

1 and 2 show a vehicle 1 according to the disclosed embodiment.

1 and 2, the vehicle 1 according to the disclosed embodiment includes an image capturing unit 100, a control unit 200, and a display unit 300.

The image photographing unit 100 according to the disclosed embodiment photographs the surrounding environment of the vehicle 1. Specifically, the image capture unit 100 according to the disclosed embodiment may include a camera. The image capturing unit 100 may capture a front, side, or rear image of the vehicle 1 required for autonomous driving of the vehicle 1. The environment surrounding the vehicle 1 photographed by the image photographing unit 100 may be a fixed object such as a road on which the vehicle 1 runs, a traffic sign existing on the road, a traffic light, a pedestrian road, a street tree, a crosswalk, or a lane. . In addition, the environment surrounding the vehicle 1 photographed by the image photographing unit 100 may be a moving target such as a pedestrian or another vehicle.

However, the environment surrounding the vehicle captured by the image capturing unit 100 is not limited thereto, and may include other factors affecting the autonomous driving of the vehicle 1. In addition, the image capturing unit 100 is not limited to a camera, and may include a component that detects an object around the vehicle 1 such as a radar, a sensor, or a lidar.

The control unit 200 according to the disclosed embodiment includes a first artificial neural network and a second artificial neural network.

The first artificial neural network learns the semantic unit of the object present in the captured image by deep learning, and the second artificial neural network learns to combine the divided image into a screen that is easy for the user to recognize.

The control unit 200 according to the disclosed embodiment divides the image captured by the image capturing unit 100 based on the learning result of the first artificial neural network, and determines whether a detection target exists in the divided image.

In addition, the control unit 200 according to the disclosed embodiment determines the detection target, and highlights the detection target by simplifying at least one of the pattern or shape of the detection target based on the determination result, and converts the divided image into one image. To restore.

In addition, the controller 200 may divide the captured image as will be described later in FIG. 4, obtain edge information of an object present in the divided image, and filter noise of the divided image.

The control unit includes a memory (not shown) that stores data about an algorithm for controlling the operation of components in the vehicle 1 or a program that reproduces the algorithm, and a processor that performs the above-described operation using data stored in the memory ( Not shown). In this case, the memory and the processor may be implemented as separate chips, respectively. Alternatively, the memory and processor may be implemented as a single chip.

The display unit 300 according to the disclosed embodiment displays a virtual image generated by the control unit 200. The virtual image is an image generated by dividing a captured image, simplifying the pattern or shape of the detection target in the divided image, highlighting it, and combining the divided images.

The display unit 300 includes a cathode ray tube (CRT), a digital light processing (DLP) panel, a plasma display panel, a liquid crystal display (LCD) panel, and an electroluminescent ( Electro Luminescence: EL panel, Electrophoretic Display (EPD) panel, Electrochromic Display (ECD) panel, Light Emitting Diode (LED) panel, or Organic Light Emitting Diode: OLED ) It may be provided as a panel or the like, but is not limited thereto.

In addition, the disclosed image capturing unit 100, the control unit 200, and the display unit 300 may include a communication module capable of communicating with respective components. For example, it may include at least one of a short-range communication module, a wired communication module, and a wireless communication module.

The short-range communication module is a Bluetooth module, an infrared communication module, a radio frequency identification (RFID) communication module, a wireless local access network (WLAN) communication module, an NFC communication module, and a Zigbee communication module. It may include various short-range communication modules that transmit and receive.

Wired communication modules include various wired communication modules such as Controller Area Network (CAN) communication modules, Local Area Network (LAN) modules, Wide Area Network (WAN) modules, or Value Added Network (VAN) modules. In addition to communication modules, various cable communication such as USB (Universal Serial Bus), HDMI (High Definition Multimedia Interface), DVI (Digital Visual Interface), RS-232 (recommended standard232), power line communication, or POTS (plain old telephone service) May contain modules.

The wireless communication module includes a Wi-Fi module, a wireless broadband module, a global system for mobile communication (GSM), a code division multiple access (CDMA), a wideband code division multiple access (WCDMA), and a universal mobile telecommunications system (UMTS). ), TDMA (Time Division Multiple Access), LTE (Long Term Evolution), etc. may include a wireless communication module supporting various wireless communication methods.

The wireless communication module may include a wireless communication interface including an antenna for transmitting signals and a transmitter. In addition, the wireless communication module may further include a signal conversion module that modulates a digital control signal output from the control unit 200 into an analog type wireless signal through a wireless communication interface under control of the control unit.

The wireless communication module may include a wireless communication interface including an antenna and a receiver for receiving signals. In addition, the wireless communication module may further include a signal conversion module for demodulating an analog wireless signal received through the wireless communication interface into a digital control signal.

At least one component may be added or deleted corresponding to the performance of the components of the vehicle 1 shown in FIG. 1. In addition, it will be readily understood by those of ordinary skill in the art that the mutual positions of the components may be changed corresponding to the performance or structure of the system.

Meanwhile, each of the components shown in FIGS. 1 and 2 refers to software and/or hardware components such as a Field Programmable Gate Array (FPGA) and an Application Specific Integrated Circuit (ASIC).

Hereinafter, a process in which the controller 200 generates a virtual image and a process in which the first artificial neural network and the second artificial neural network learns will be described.

3 is a diagram illustrating a process in which the controller 200 generates a virtual image.

Referring to FIG. 3, the first artificial neural network learns a semantic unit of an object existing on a road by deep learning (2101). The learning process of the first artificial neural network will be described in detail later in FIG. 5.

The controller 200 divides the captured image based on the learning result of the first artificial neural network (2102).

Specifically, the controller 200 divides the captured image based on the semantic unit of the object existing on the road learned by the first artificial neural network. According to the disclosed embodiment, the first artificial neural network may learn semantic units of objects existing on the road as traffic lights, lanes, pedestrians, and crosswalks. In this case, the controller 200 may be based on the learning result of the first artificial neural network. ) May generate 4 divided images having 4 semantic units. However, the semantic unit used by the controller 200 to divide the image is not limited thereto, and may include other elements necessary for driving of the vehicle 1.

The controller 200 determines whether a detection target exists in the divided image (2103).

Specifically, the detection target refers to an object that the user should intensively recognize when the vehicle 1 is autonomously driving.

According to the disclosed embodiment, when the vehicle 1 is driving during a rainy night, there is a high possibility that the user may not recognize a lane around the vehicle 1. Therefore, in this case, the controller 200 may target a lane existing on the road.

According to another disclosed embodiment, when the vehicle 1 is driving in a child protection zone, the control unit 200 may target a pedestrian existing on a road to be detected.

However, the detection target determined by the control unit 200 is not limited thereto, and may vary according to a road condition or a driving condition of the vehicle.

When it is determined that the detection target exists in the divided image, the controller 200 highlights the detection target (2104, 2105). However, if it is determined that the detection target does not exist in the divided image, the controller 200 continuously determines whether the detection target exists in the divided image (2104).

The method for the control unit 200 to emphasize the detection object may include simplifying and emphasizing the pattern or shape of the detection object.

Simplifying the pattern or shape of the object to be detected means simplifying the shape of the object to the general shape of the object when it is determined that the shape of the object existing on the road is not a typical shape or a state that is difficult for the user to detect. In addition, emphasizing the pattern or shape of the detection target means extracting edge information of the detection target and highlighting the outline and color of the target based on the extracted edge information.

According to the disclosed embodiment, when the control unit 200 determines a lane existing on the road as a detection target, the control unit 200 determines the shape of the erased part of the general lane even when a part of the lane existing on the road is erased. The shape can be corrected, and the color and outline of the lane can also be emphasized.

However, the control unit 200 is not limited to simplifying and emphasizing the pattern or shape of the detection target, and other simplification methods or enhancement methods may be used. In addition, the target determined by the controller 200 as a detection target may include other factors that affect the driving of the vehicle 1.

The controller 200 restores the divided image (2106).

Specifically, the controller 200 combines the divided images and restores them like an actual driving image. The controller 200 may combine the divided images based on the learning result of the second artificial neural network. The second artificial neural network may learn to set the color information of the object differently based on the semantic unit and edge information of the object existing on the road, and to improve the image quality of the divided image based on the set color.

In addition, the second artificial neural network may learn to compare the preset verification image and the reconstructed image, and determine that the reconstructed image as a result of the comparison is the same as the verification image. Here, the preset verification image may be an image running in a daytime or sunny weather environment in order to make it easier for the user to recognize the environment surrounding the vehicle 1 regardless of the driving environment of the vehicle 1. However, the verification image is not limited thereto, and may vary according to a user setting or a driving environment of the vehicle 1.

The controller 200 generates a virtual image based on the reconstructed image (2107).

Specifically, the controller 200 may determine the driving condition of the vehicle 1 and generate the virtual image based on the determination result.

4 is a diagram illustrating a process of improving the image quality of a divided image by the control unit 200 according to the disclosed embodiment.

According to the disclosed embodiment, the image photographing unit 100 photographs the surrounding environment of the vehicle 1 (2201), and the control unit 200 divides the image based on the semantic unit of the photographed object (2202).

The control unit 200 acquires edge information of the object photographed by the divided image (2203). Here, the edge information may be contour information of an object to be detected.

The controller 200 acquires edge information of an object and performs noise filling of the divided image (2204).

Specifically, the control unit 200 may perform noise filtering including a process of removing small noise by applying a smoothing filter such as Bilateral Filtering. In addition, the control unit 200 may perform noise filtering, including a process of filling a hole generated during image segmentation while simply making an outline of an object, and removing unnecessary segmented images that are not a recognition target.

However, the noise filtering process performed by the control unit 200 is not limited thereto, and may include another filtering process capable of improving the image quality of the divided image.

The controller 200 may improve the image quality of the divided image through the above-described process (2205).

5 shows that the first artificial neural network learns an object around a vehicle by deep learning based on the semantic unit of the object.

Referring to FIG. 5, the disclosed first artificial neural network learns a semantic unit of an object through processes f1 to f5. The meaning unit of the object means the elements around the vehicle 1 that can have an influence on the vehicle 1 autonomously driving. As described above, for example, traffic lights, lanes, pedestrians, and crosswalks of the semantic unit of the object may be the semantic units of the object, respectively.

The first artificial neural network according to the disclosed aspect may learn processes f1 to f2 by acquiring an image around the vehicle 1. In addition, the first artificial neural network can convert the learned image into big data and use it as a pin.

The first artificial neural network according to the disclosed aspect may classify the semantic unit of the object based on the semantic unit of the learned object (P2). The first artificial neural network may extract a semantic unit existing in an image captured from a semantic unit of the classified object as Pout. The semantic unit of the extracted object may be used to divide the image captured by the controller 200.

In addition, the first artificial neural network according to the disclosed embodiment may include a convolution neural network (CNN). Here, CNN inputs an image (or an image frame of an image) as an input value, and uses a feature learning to generate a feature map and a generated feature map as an input value. It can be classified into classification, which determines the type of image.

First, the classification may consist of a multi-layer perceptron. Perceptron refers to the basic structure of machine learning that performs the same functions as neurons, which are the basic structures of the human body's neural network. That is, the perceptron is composed of an input value multiplied by a weight, added with a bias, and then a modified input value as a transfer function or activation function. Only the transformed input value that satisfies the threshold value of the transfer function is output as an output value. The perceptron is a structure in which a desired result value is inferred from a repeated input value by self-adjusting a weight and a bias so that an output value is a desired result value.

In order to determine the type of image, the neural network may input each pixel value included in an image frame of an image captured on a multi-layer perceptron into an input-layer Pin. In addition, the CNN, which is an example of the disclosed neural network, may generate a feature image using a convolution technique in feature learning.

However, the first artificial neural network is not limited thereto, and may include another neural network capable of learning a captured image or image by deep learning.

6 is a diagram illustrating a process of learning to combine divided images by a second artificial neural network according to the disclosed embodiment and reconstruct them into a single image.

Referring to FIG. 6, the second artificial neural network plays a role opposite to the above-described first artificial neural network.

Specifically, the second artificial neural network extracts a semantic unit that the user must recognize from the semantic unit of the object classified through the N process. The extracted semantic unit may mean a detection object that needs to be emphasized.

The controller 200 may restore the divided image into one image based on the semantic information of the detection target extracted from the second artificial neural network ⓒ.

7 shows a process in which the controller 200 generates a virtual image.

Referring to FIG. 7, the controller 200 divides a captured image based on a learning result of a first artificial neural network, and restores the segmented image based on a learning result of a second artificial neural network (2301). Here, the reconstructed image is an image in which an object that the user needs to recognize is highlighted.

The controller 200 determines whether the restored image is the same as the preset verification image (2301). Here, the verification image may be an image that satisfies a criterion preset by the user.

The verification image according to the disclosed embodiment may be a driving image assuming a sunny day in the daytime. By comparing the verification image and the reconstructed image as described above, the controller 200 may generate an image having a certain quality.

If it is determined that the restored image is the same as the verification image, the controller 200 generates a virtual image based on the restored image (2303). Here, the virtual image may be an image in which an object that needs to be recognized by the user is emphasized and noise filtering has been completed. However, if it is determined that the restored image is not the same as the verification image, the controller 200 continues to restore the image.

Meanwhile, the disclosed embodiments may be implemented in the form of a recording medium storing instructions executable by a computer. The instruction may be stored in the form of a program code, and when executed by a processor, a program module may be generated to perform the operation of the disclosed embodiments. The recording medium may be implemented as a computer-readable recording medium.

Computer-readable recording media include all types of recording media in which instructions that can be read by a computer are stored. For example, there may be read only memory (ROM), random access memory (RAM), magnetic tape, magnetic disk, flash memory, optical data storage device, and the like.

As described above, the disclosed embodiments have been described with reference to the accompanying drawings. Those of ordinary skill in the art to which the present invention pertains will understand that the present invention may be practiced in a form different from the disclosed embodiments without changing the technical spirit or essential features of the present invention. The disclosed embodiments are illustrative and should not be construed as limiting.

1: vehicle
100: video recording unit
200: control unit
300: display

Claims (20)

A display unit that displays the captured image;
An image photographing unit for photographing surrounding images;
A first artificial neural network that learns a semantic unit of an object present in the captured image by deep learning;
Dividing the captured image based on the learning result, determining whether a detection target exists in the divided image, and highlighting the detection target by simplifying at least one of a pattern or shape of the detection target based on the determination result And a control unit for restoring the divided image into one image and controlling the display unit to generate a virtual image based on the restored image. The method of claim 1,
Further comprising; a second artificial neural network for learning to combine the divided images,
The control unit,
A vehicle that generates a virtual image based on the learning result. The method of claim 1,
The control unit,
A vehicle that corrects a boundary of an object appearing in the divided image and removes noise from the divided image. The method of claim 2,
The control unit,
A vehicle for generating a virtual image having a constant illuminance based on the assumed situation, assuming that the vehicle is driving during the day and on a clear day. The method of claim 1,
The control unit
A vehicle configured to differently set color information of the object based on the semantic unit and edge information of the object, and to improve the image quality of the divided image based on the set color. The method of claim 1,
The control unit,
A vehicle that improves the quality of the restored image by using a Hall filter. The method of claim 1,
The control unit,
A vehicle that simplifies the shape of the object into a preset shape based on the meaning unit of the object. The method of claim 2,
The control unit,
A vehicle that compares a preset verification image and the restored image, learns to determine that the restored image is the same as the verification image as a result of the comparison, and generates a virtual image based on the learning result. The method of claim 3,
The control unit,
A vehicle that removes noise from the segmented image using a smoothing filter. The method of claim 1,
The control unit,
A vehicle that determines the driving condition of the vehicle and generates the virtual image based on the determination result. Photographing surrounding images;
Display the captured image;
Learning the semantic unit of the object present in the captured image by deep learning, segmenting the captured image based on the learning result, determining whether a detection object exists in the segmented image, and determining whether the detection target exists Thus, at least one of the pattern or shape of the detection target is simplified to highlight the detection target, restore the divided image to one image, and generate a virtual image based on the restored image. The method of claim 11,
Learning to combine the divided images; further comprising,
To control the above,
A vehicle control method for generating a virtual image based on the learning result. The method of claim 11,
To control the above,
A vehicle control method for correcting a boundary of an object appearing in the divided image and removing noise from the divided image. The method of claim 12,
To control the above,
A vehicle control method for generating a virtual image having a constant illuminance based on the assumed situation, assuming that the vehicle is driving during the day and on a clear day. The method of claim 11,
To control the above,
A vehicle control method for differently setting color information of the object based on the semantic unit and edge information of the object, and improving the image quality of the divided image based on the set color. The method of claim 11,
To control the above,
A vehicle control method for improving the quality of the restored image by using a hole filter. The method of claim 11,
To control the above,
A vehicle control method for simplifying the shape of the object into a preset shape based on the meaning unit of the object. The method of claim 12,
To control the above,
A vehicle control method for comparing a preset verification image and the restored image, learning to determine that the restored image is the same as the verification image as a result of the comparison, and generating a virtual image based on the learning result. The method of claim 13,
To control the above,
A vehicle control method for removing noise from the divided image using a smoothing filter. The method of claim 11,
To control the above,
A vehicle control method for determining the driving condition of the vehicle and generating the virtual image based on the determination result.

Images