KR102313804B1 - Electronic device for distinguishing front or rear of vehicle and method thereof - Google Patents

Abstract

The present invention relates to an electronic device for distinguishing the front or rear side of a vehicle, which provides a data processing rate, and a method thereof. According to various embodiments of the present invention, the electronic device comprises: a camera module and a processor operatively connected to the camera module. The processor performs the following operations of: controlling a camera module to capture a video of a vehicle to acquire a first image including a vehicle image; using a license plate detection model stored in a memory to detect a first area including at least one license plate image in the first image; acquiring a second image corresponding to a second area extended from the first area by a predetermined area within the first image; using a discrimination model to determine whether the second image includes at least one feature object and determine whether the second image is a front image or a rear image of the vehicle on the basis of a determination result; and checking a driving direction of the vehicle included in the first image on the basis of whether the second image is the front image or the rear image.

Description

ELECTRONIC DEVICE FOR DISTINGUISHING FRONT OR REAR OF VEHICLE AND METHOD THEREOF

Various embodiments disclosed in this document relate to an electronic device, for example, to an electronic device and method for photographing a vehicle and determining a driving or parking direction of the vehicle by using the photographed image.

CCTV (closed-circuit television) for vehicles is used in various fields that require vehicle recognition, such as vehicles violating traffic laws, criminal tracking, and unmanned parking systems. Recently, as the demand for autonomous driving or intelligent transportation systems increases, the use of CCTVs for vehicles and devices for vehicle recognition using the same is increasing.

Reversing the vehicle has a very high risk of causing a traffic accident, and in the event of an accident, it is highly likely to lead to a major accident. However, compared to the risk, crackdown is not carried out significantly. In the case of the prior art, based on the order in which two or more sensors detect the vehicle using two or more loop sensors and piezoelectric sensors embedded in the road surface at regular intervals while shooting a road using a vehicle CCTV, it is determined whether the vehicle is running in reverse. It is determined whether the vehicle is running in reverse or not by measuring the speed using an infrared sensor. In this case, a device other than a vehicle CCTV is required, and additionally, the cost of burying or burying may occur. In addition, the prior art determines whether to run in reverse based on the movement of the same object by shooting a moving image. The above prior art techniques can determine whether the vehicle is traveling in the reverse direction only when the vehicle is running, that is, only when the vehicle is moving. In addition, since it is necessary to analyze images of at least two frames or more, the data processing rate may be lowered compared to determining whether to run in reverse with a single, still image. In addition, when limiting a parking direction (eg, a front parking area), it may be difficult to determine whether a vehicle in a stationary state is in a forward or reverse direction according to the prior art.

delete

Republic of Korea Patent Registration 10-0661902 Korean Patent Publication No. 10-2011-0086638

In order to solve the above-described problem, there is a need for a technology capable of confirming whether the vehicle is driving in the reverse direction or parking the vehicle in the reverse direction using only a single still image. In the case of a section with high speed, a high data processing rate is required to confirm reverse driving within a limited field of view range. In order to guarantee the throughput, a classification technique that effectively searches the area to be analyzed in the still image image is required. In addition, a license plate recognition function is required for notification and traffic regulation violations after reverse driving enforcement. The above technical requirements should be able to be implemented in the embedded environment.

An electronic device according to various embodiments disclosed in this document includes a camera module and a processor operatively connected to the camera module, wherein the processor controls the camera module to capture a moving picture to capture a vehicle image The operation of obtaining a first image including the operation of detecting a first area including at least one license plate image in the first image using the license plate detection model stored in the memory, the first image in the An operation of acquiring a second image corresponding to a second area extended by a certain range from the first area, determining whether at least one feature object is included in the second image using a discrimination model, and the determination An operation of determining whether the second image is a front image or a rear image of a vehicle based on can do.

According to various embodiments, a high data processing rate can be expected by checking whether reverse driving or reverse parking using only a single still image. According to various embodiments, the direction of the vehicle may be checked regardless of whether the vehicle is traveling, and whether the vehicle is traveling in the reverse direction may be determined.

In connection with the description of the drawings, the same or similar reference numerals may be used for the same or similar components.
1 is a diagram illustrating that an electronic device checks a driving direction by photographing a vehicle according to various embodiments of the present disclosure;
2 is a block diagram of an electronic device according to various embodiments of the present disclosure;
3 illustrates a first image and a second image according to various embodiments.
4 is a flowchart illustrating an operation in which an electronic device generates reverse driving information according to various embodiments of the present disclosure;
5 is an example of vehicle front image data for training a discrimination model.
6 is an example of vehicle rear image data for training a discrimination model.
7 illustrates setting of the second area based on the first area according to various embodiments of the present disclosure.

1 is a diagram illustrating that an electronic device checks a driving direction by photographing a vehicle according to various embodiments of the present disclosure;

Referring to FIG. 1 , the electronic device 101 may photograph at least one vehicle (eg, a first vehicle 201 and a second vehicle 202 ). According to an embodiment, the electronic device 101 may be installed in a space where a vehicle can move, park, and/or stop, such as a road surface or a parking lot. According to an embodiment, the electronic device 101 may be installed in a space in which a vehicle can move and/or park, and may capture an image (eg, a first image) in which one or more vehicles are photographed. The image captured by the vehicle may include a still image image and/or a moving image image in which a plurality of still image images are successively captured. According to an embodiment, the first image may be a single still image image. According to an embodiment, the electronic device 101 may be a closed circuit television (CCTV) for a vehicle. The electronic device 101 may continuously and/or temporarily capture an image (eg, a first image) of a surrounding environment in which the electronic device 101 is installed by using the camera module.

According to various embodiments, the electronic device 101 may determine whether the vehicle is traveling in the reverse direction or in the reverse direction by using the first image. According to an embodiment, the electronic device 101 may check whether at least one photographed vehicle (eg, the first vehicle 201 or the second vehicle 202) is traveling in the reverse direction or is parked in the reverse direction. According to an embodiment, the electronic device 101 determines whether the image of the vehicle (eg, the first vehicle 201 or the second vehicle 202 ) included in the first image is a front image or a rear image of the vehicle. can According to an embodiment, the electronic device 101 may determine whether the image of the vehicle included in the first image is the front or the rear by using a discrimination model that determines whether it is a front image or a rear image of the vehicle. According to an embodiment, the electronic device 101 may determine whether the vehicle is the front or the rear using an image (eg, a second image) corresponding to at least a partial region (eg, a second region) of the first image. . The discrimination model may be a model generated using an artificial intelligence algorithm. The discrimination model may be a model learned using a deep learning-based algorithm. For example, the discriminant model may be a model trained to determine whether the vehicle is front or rear based on a convolutional neural network (CNN) algorithm. According to an embodiment, the discrimination model may be a model learned by receiving image data including a vehicle front image and a vehicle rear image and data labeled with information on whether the front or rear side corresponding to each image data is input. According to an embodiment, the electronic device 101 may store the discriminant model in the memory. The electronic device 101 may determine whether an image corresponding to at least a partial region of the first image is a front image or a rear image by using the discrimination model, and generate reverse driving information regarding whether or not driving in the reverse direction based on the confirmation result. . The reverse driving information may be data regarding whether the vehicle is driving in the reverse direction. For example, the reverse running information may include information on whether the vehicle is running in the reverse direction, and may include at least one of text data and a first image for recognizing text on a license plate of the vehicle. The electronic device 101 may generate reverse driving information and temporarily store the generated reverse driving information in a memory.

According to various embodiments, the electronic device 101 may detect the first region from the first image. The first region may be at least one region including a license plate image among regions included in the first image. According to an embodiment, the electronic device 101 may detect the first area in the first image by using the license plate detection model stored in the memory. According to one embodiment, the license plate detection model may be a model learned using a deep learning-based algorithm. For example, the license plate detection model may be a model trained to detect a license plate region within an input image based on a convolutional neural network (CNN) algorithm. According to one embodiment, the license plate detection model may be a model learned using the license plate image data. According to an embodiment, the electronic device 101 may generate data on the size and/or position of the detected license plate area, and temporarily store the generated data in a memory.

According to various embodiments, the electronic device 101 may select the second area and acquire a second image. The second image may be image data to be input to the discrimination model. According to an embodiment, the second image may be an image corresponding to at least a partial region (eg, the second region) of the first image. The first image may be an image including a plurality of vehicles, and since the surrounding environment of the vehicle may be an image taken at the same time, it is checked whether the front or rear of the vehicle is by using at least a partial area of the first image rather than the entire first image. This may be more efficient in terms of processing speed and accuracy. The second area may be an area occupied by the second image in the first image. According to an embodiment, the electronic device 101 may select the second area and obtain a second image corresponding to the second area. According to an embodiment, the second area may be an area including the first area. According to an embodiment, the second area may be an area in which the first area is expanded by a predetermined range based on the size of the first area. According to an embodiment, the second area may be an area generated based on the size and/or location of the first area. According to an embodiment, the second area may include a plurality of areas selected for each vehicle (eg, the first vehicle 201 and the second vehicle 202) included in the first image.

According to various embodiments, the electronic device 101 may recognize the license plate characters included in the first image. According to an embodiment, the electronic device 101 may process a text image included in the first area and recognize it as text data. According to an embodiment, the electronic device 101 may detect the first area and recognize a character in the detected first area. For example, the electronic device 101 may recognize a character included in the first region by performing optical character recognition (OCR). According to an embodiment, the electronic device 101 may pre-process the first image or an image corresponding to a first region included in the first image in order to recognize a character. The pre-processing may include, for example, gray scale transformation, image filtering, and binarization processing. According to an embodiment, the electronic device 101 may store an algorithm and/or instructions for preprocessing in a memory. According to an embodiment, the electronic device 101 may recognize a character using the OCR model. The OCR model may be a model trained to output text data corresponding to an input image based on a convolution neural network (CNN) algorithm. According to an embodiment, the electronic device 101 may store the OCR model in a memory.

The electronic device 101 may be communicatively connected to the external device 102 (eg, a server). The electronic device 101 may be connected to the external device 102 through the wired communication network 103 . According to an embodiment, the electronic device 101 may receive a command and/or data from the external device 102 or transmit a command and/or data to the external device 102 . For example, the external device 102 transmits image data (eg, a first image) captured by the electronic device 101 , text data recognized by the electronic device 101 , and/or information on whether the vehicle is traveling in the reverse direction to the electronic device 101 . ) can be received from The external device 102 may at least partially control the electronic device 101 . According to an embodiment, the electronic device 101 may control the camera module to capture a specific area (eg, a license plate area) including at least one object (eg, a vehicle), and encode the captured image to Streaming can be sent to device 102 . The external device 102 may store data received from the electronic device 101 in a storage (eg, storage) included in the external device 102 .

2 is a block diagram of an electronic device according to various embodiments of the present disclosure;

Referring to FIG. 2 , the electronic device 101 may include a camera module 210 , a memory 220 , and/or a processor 230 . The processor 230 may be electrically or operatively connected to the camera module 210 and/or the memory 220 .

According to various embodiments, the camera module 210 may capture a still image and/or a video. According to an embodiment, the camera module 210 may include one or more lenses, image sensors, and image signal processors. According to an embodiment, the camera module 210 converts light incident from the surrounding environment of the electronic device 101 into an electrical signal, for example, on a road where the electronic device 101 is installed, in a parking lot, etc. (eg, the first image) can be created.

According to various embodiments, the memory 220 may store various data used by at least one component (eg, the processor 230 ) of the electronic device 101 . The memory 220 is for temporarily or permanently storing digital data and may include a volatile memory or a non-volatile memory. The memory 220 may store various instructions that may be executed by the processor 230 . Such instructions may include control commands such as logical operations and data input/output that may be recognized and executed by the processor 230 . According to one embodiment, the memory 220 may store a program as software, and the program may include a model implemented with one or more algorithms. There will be no limitation on the type and/or amount of data that the memory 220 can store, but in this document, the configuration and function of the memory related to the operation of the processor 230 for recognizing a vehicle according to various embodiments are described. only to be explained.

According to one embodiment, the memory 220 may store the license plate detection model. The license plate detection model may be a model learned using a deep learning-based algorithm. For example, the license plate detection model may be a model trained to detect a license plate region within an input image based on a convolutional neural network (CNN) algorithm. According to one embodiment, the license plate detection model may be a model learned using the license plate image data. The license plate detection model may include an instruction stored in the memory 220 to cause the processor 230 of the electronic device 101 to perform a predetermined operation.

According to an embodiment, the memory 220 may store the discriminant model. The discrimination model may be a model generated using an artificial intelligence algorithm. The discrimination model may be a model learned using a deep learning-based algorithm. For example, the discriminant model may be a model trained to determine whether the vehicle is front or rear based on a convolutional neural network (CNN) algorithm. According to an embodiment, the discrimination model may be a model learned by receiving image data including a vehicle front image and a vehicle rear image and data labeled with information on whether the front or rear side corresponding to each image data is input. The discrimination model may include instructions stored in the memory 220 to cause the processor 230 of the electronic device 101 to perform a predetermined operation. The discriminant model is a model learned based on the CNN algorithm and may include a plurality of convolution layers. Each layer includes a plurality of convolution filters, and each filter may assign a weight to a characteristic value of an image. According to an embodiment, the discriminant model may be a model trained by using a predefined loss function.

According to an embodiment, the memory 220 may store the OCR model. The OCR model may be a model trained to output text data corresponding to an input image based on a convolution neural network (CNN) algorithm. The OCR model may include an instruction stored in the memory 220 to cause the processor 230 of the electronic device 101 to perform a predetermined operation.

The processor 230 may control at least one other component (eg, the camera module 210) of the electronic device 101 by executing a program stored in the memory 220, and perform various data processing or operations. can According to an embodiment, as at least part of data processing or operation, the processor 230 stores data received from another component (eg, the camera module 210) in a volatile memory (eg, the memory 220) and , the data stored in the volatile memory may be processed, and the resulting data may be stored in the non-volatile memory. According to an embodiment, the processor 230 may include a hardware structure specialized for processing an artificial intelligence model. Artificial intelligence models (e.g. license plate detection models, discriminant models, OCR models) can be generated through machine learning (e.g. deep learning based on CNN algorithms). There will be no limitation on the type and/or amount of operations, calculations, and data processing that the processor 230 can perform, but in this document, the front of the vehicle is obtained from images captured by the electronic device 101 according to various embodiments. Alternatively, only the configuration and function of the processor 230 related to the operation of recognizing the back side will be described.

According to various embodiments, the processor 230 may acquire the first image by controlling the camera module 210 . The processor 230 may photograph at least one vehicle. According to an embodiment, the electronic device 101 may be installed in a space where a vehicle can move, park, and/or stop, such as a road surface or a parking lot. According to an embodiment, the electronic device 101 may be installed in a space in which a vehicle can move and/or park, and may capture an image (eg, a first image) in which one or more vehicles are photographed. The image captured by the vehicle may include a still image image and/or a moving image image in which a plurality of still image images are successively captured. According to an embodiment, the first image may be a single still image image. According to an embodiment, the processor 230 may be a closed circuit television (CCTV) for a vehicle. The processor 230 may continuously and/or temporarily capture an image (eg, a first image) of a surrounding environment in which the electronic device 101 is installed by using the camera module 210 .

According to various embodiments, the processor 230 may detect the first region. According to an embodiment, the processor 230 may detect the first region from the first image. The first region may be at least one region including a license plate image among regions included in the first image. According to an embodiment, the processor 230 may detect the first area in the first image using the license plate detection model stored in the memory 220 . According to one embodiment, the license plate detection model may be a model learned using a deep learning-based algorithm. For example, the license plate detection model may be a model trained to detect a license plate region (eg, a first region) in an input image based on a convolutional neural network (CNN) algorithm. According to one embodiment, the license plate detection model may be a model learned using the license plate image data. According to an embodiment, the processor 230 may generate data on the detected size and/or position of the first region, and temporarily store the generated data in the memory 220 .

According to various embodiments, the processor 230 may select the second region and acquire the second image. The second image may be image data to be input to the discrimination model. According to an embodiment, the second image may be an image corresponding to at least a partial region (eg, the second region) of the first image. The first image may be an image including a plurality of vehicles, and since the surrounding environment of the vehicle may be an image taken at the same time, it is checked whether the front or rear of the vehicle is by using at least a partial area of the first image rather than the entire first image. This may be more efficient in terms of processing speed and accuracy. The second area may be an area occupied by the second image in the first image. According to an embodiment, the processor 230 may select the second region and acquire a second image corresponding to the second region. According to an embodiment, the second area may be an area including the first area. According to an embodiment, the second area may be an area in which the first area is expanded by a predetermined range based on the size of the first area. According to an embodiment, the second area may be an area generated based on the size and/or location of the first area. According to an embodiment, the second region may include one or more regions selected for each of one or more vehicles included in the first image.

According to various embodiments, the processor 230 may determine whether the front or rear image of the vehicle included in the first image is the front or rear using the determination model for determining whether the image is the front or rear of the vehicle. According to an embodiment, the processor 230 may check whether the vehicle is front or rear using the second image. The discrimination model may be a model generated using an artificial intelligence algorithm. The discrimination model may be a model learned using a deep learning-based algorithm. For example, the discriminant model may be a model trained to determine whether the vehicle is front or rear based on a convolutional neural network (CNN) algorithm. According to an embodiment, the discrimination model may be a model learned by receiving image data including a vehicle front image and a vehicle rear image and data labeled with information on whether the front or rear side corresponding to each image data is input. According to an embodiment, the processor 230 determines whether an image (eg, a second image) corresponding to at least a partial region of the first image is a front image or a rear image using the discrimination model stored in the memory 220 . can The processor 230 may check whether the second image is a vehicle front image or a vehicle rear image by inputting the second image to the discrimination model.

According to various embodiments, the processor 230 may determine whether the vehicle is traveling in the reverse direction or in the reverse direction. According to an embodiment, the processor 230 may determine whether at least one vehicle included in the first image is traveling in the reverse direction or is parked in the reverse direction. The processor 230 may determine whether an image (eg, a second image) corresponding to at least a partial region of the first image is a front image or a rear image using the determination model, and determine whether to run in reverse based on the result of the confirmation. can The processor 230 may determine whether to run in the reverse direction based on a result of whether the second image 230 is a front or a rear side. According to an embodiment, the processor 230 may determine whether to drive in the reverse direction based on the traveling direction of the road. For example, depending on the installation location and/or direction of the electronic device 101 , the front direction of the vehicle may be photographed in the forward direction, or the rear direction of the vehicle may be photographed in the forward direction. The memory 220 may store data on the driving direction of the road in advance, and the processor 230 determines whether the vehicle It can be determined whether the vehicle is running in reverse. For example, the electronic device 101 may be installed to photograph a vehicle traveling in a direction opposite to the photographing direction of the electronic device 101 , and the processor 230 may determine the second image as a vehicle rear image. It can be judged that it is running in reverse.

According to various embodiments, the processor 230 may generate reverse driving information. The reverse driving information may be data regarding whether the vehicle is driving in the reverse direction. For example, the reverse running information may include information on whether the vehicle is running in the reverse direction, and may include at least one of text data and a first image for recognizing text on a license plate of the vehicle. The processor 230 may generate reverse driving information and temporarily store the generated reverse driving information in a memory. According to an embodiment, the reverse driving information may include at least a portion of information on whether or not the reverse driving is performed, a photographing time of the first image corresponding to the reverse driving, character recognition data of the license plate of the vehicle, and the first image. According to an embodiment, the processor 230 may recognize the license plate characters included in the first image. According to an embodiment, the processor 230 may process a text image included in the first area and recognize it as text data. According to an embodiment, the processor 230 may detect the first area and recognize a character in the detected first area. For example, the processor 230 may recognize a character included in the first region by performing optical character recognition (OCR). According to an embodiment, the processor 230 may pre-process the first image or an image corresponding to the first region included in the first image in order to recognize the character. The pre-processing may include, for example, gray scale transformation, image filtering, and binarization processing. According to an embodiment, the processor 230 may store an algorithm and/or instructions for preprocessing in a memory. According to an embodiment, the processor 230 may recognize a character using an OCR model. The OCR model may be a model trained to output text data corresponding to an input image based on a convolution neural network (CNN) algorithm. According to an embodiment, the processor 230 may store the OCR model in a memory. According to an embodiment, the processor 230 may recognize the characters included in the license plate area (eg, the first area) when it is determined that the reverse driving corresponds to the reverse driving after determining whether the reverse driving is performed. Alternatively, the processor 230 may perform character recognition on the continuously detected first area regardless of whether the vehicle is traveling backwards or not. According to one embodiment, the processor 230 generates reverse driving information including at least a portion of information on whether or not reverse driving, a shooting time of a first image corresponding to reverse driving, license plate character recognition data of the vehicle, and a first image, It may be stored in the memory 220 According to an embodiment, the processor 230 may generate reverse driving information and transmit it to an external device (eg, the external device 102 of FIG. 1 ).

3 illustrates a first image and a second image according to various embodiments.

Referring to FIG. 3 , an electronic device (eg, the electronic device 101 of FIG. 1 ) may capture a first image 310 . The first image 310 may be an image of one or more vehicles. According to one embodiment, the first image 310 may be a single still image image. At least one vehicle photographed in the first image 310 may have a license plate 301 attached thereto. An image area (eg, the first area 320 ) corresponding to the license plate 301 may be included in the first image 310 . According to an embodiment, the electronic device 101 may detect at least one license plate 301 in the first image 310 . The electronic device 101 may detect the area in which the license plate 301 is detected (eg, the first area 320 ), and confirm the size and location of the first area 320 . The first image 310 may include a second region 330 . The second region 330 may correspond to at least a partial region of the first image 310 . The second area 330 may be an area selected by the electronic device 101 based on the size and/or location of the first area 320 . According to an embodiment, the second region 330 may be formed over a region including at least the first region 320 . According to an embodiment, the second area 330 may be an area in which the first area 320 is expanded. For example, the second region 330 may have the same center as the first region 320 , and may be a region that is extended by a predetermined multiple in a predetermined direction with respect to the area occupied by the first region 320 . The electronic device 101 may select the second area 330 in the first image 310 based on the first area 320 . Referring to FIG. 3 , the electronic device 101 may acquire a second image 340 . The second image 340 may be an image corresponding to the selected second region 330 . According to an embodiment, the second image 340 may include a first area 320 corresponding to the license plate 301 .

Referring to FIG. 3 , the electronic device 101 may determine whether the image of the vehicle included in the first image is the front or the rear using a discrimination model that determines whether the image is the front or rear of the vehicle. According to an embodiment, the electronic device 101 may use the second image 340 to determine whether the vehicle is front or rear. The discrimination model may be a model generated using an artificial intelligence algorithm. The discrimination model may be a model learned using a deep learning-based algorithm. For example, the discriminant model may be a model trained to determine whether the vehicle is front or rear based on a convolutional neural network (CNN) algorithm. According to an embodiment, the discrimination model may be a model learned by receiving image data including a vehicle front image and a vehicle rear image and data labeled with information on whether the front or rear side corresponding to each image data is input. According to an embodiment, the electronic device 101 may store the discrimination model in a memory (eg, the memory 220 of FIG. 2 ). The electronic device 101 may determine whether the image (eg, the second image 340 ) corresponding to at least a partial region of the first image is a front image or a rear image by using the discrimination model. The electronic device 101 may check whether the second image 340 is a vehicle front image or a vehicle rear image by inputting the second image 340 to the discrimination model.

4 is a flowchart illustrating an operation in which an electronic device generates reverse driving information according to various embodiments of the present disclosure;

Referring to FIG. 4 , the operation of the electronic device (eg, the electronic device 101 of FIG. 1 ) generating reverse driving information is each operation of the processor of the electronic device 101 (eg, the processor 230 of FIG. 2 ). can be explained as For each operation in FIG. 4 , the order and/or part of the operation may be changed, and other operations may be added.

Referring to operation 410 , the processor 230 may acquire a first image. According to various embodiments, the processor 230 may acquire the first image by controlling the camera module (eg, the camera module 210 of FIG. 2 ). The processor 230 may photograph at least one vehicle. According to an embodiment, the electronic device 101 may be installed in a space where a vehicle can move, park, and/or stop, such as a road surface or a parking lot. According to an embodiment, the electronic device 101 may be installed in a space in which a vehicle can move and/or park, and may capture an image (eg, a first image) in which one or more vehicles are photographed. The image captured by the vehicle may include a still image image and/or a moving image image in which a plurality of still image images are successively captured. According to an embodiment, the first image may be a single still image image. According to an embodiment, the processor 230 may be a closed circuit television (CCTV) for a vehicle. The processor 230 may continuously and/or temporarily capture an image (eg, a first image) of a surrounding environment in which the electronic device 101 is installed by using a camera module (eg, the camera module 210 of FIG. 2 ). have.

Referring to operation 420 , the processor 230 may detect the first region. According to an embodiment, the processor 230 may detect the first region from the first image. The first region may be at least one region including a license plate image among regions included in the first image. According to an embodiment, the processor 230 may detect the first area in the first image using the license plate detection model stored in the memory (eg, the memory 220 of FIG. 2 ). According to one embodiment, the license plate detection model may be a model learned using a deep learning-based algorithm. For example, the license plate detection model may be a model trained to detect a license plate region (eg, a first region) in an input image based on a convolutional neural network (CNN) algorithm. According to one embodiment, the license plate detection model may be a model learned using the license plate image data. According to an embodiment, the processor 230 may generate data on the detected size and/or position of the first region, and temporarily store the generated data in a memory (eg, the memory 220 of FIG. 2 ). can be saved

Referring to operation 430 , the processor 230 may select the second region and acquire a second image. The second image may be image data to be input to the discrimination model. According to an embodiment, the second image may be an image corresponding to at least a partial region (eg, the second region) of the first image. The first image may be an image including a plurality of vehicles, and since the surrounding environment of the vehicle may be an image taken at the same time, it is checked whether the front or rear of the vehicle is by using at least a partial area of the first image rather than the entire first image. This may be more efficient in terms of processing speed and accuracy. The second area may be an area occupied by the second image in the first image. According to an embodiment, the processor 230 may select the second region and acquire a second image corresponding to the second region. According to an embodiment, the second area may be an area including the first area. According to an embodiment, the second area may be an area in which the first area is expanded by a predetermined range based on the size of the first area. According to an embodiment, the second area may be an area generated based on the size and/or location of the first area. According to an embodiment, the second region may include one or more regions selected for each of one or more vehicles included in the first image.

Referring to operation 440 , the processor 230 may determine whether the driving vehicle is front or rear. According to various embodiments, the processor 230 may determine whether the front or rear image of the vehicle included in the first image is the front or rear using the determination model for determining whether the image is the front or rear of the vehicle. According to an embodiment, the processor 230 may check whether the vehicle is front or rear using the second image. The discrimination model may be a model generated using an artificial intelligence algorithm. The discrimination model may be a model learned using a deep learning-based algorithm. For example, the discriminant model may be a model trained to determine whether the vehicle is front or rear based on a convolutional neural network (CNN) algorithm. According to an embodiment, the discrimination model may be a model learned by receiving image data including a vehicle front image and a vehicle rear image and data labeled with information on whether the front or rear side corresponding to each image data is input. According to an embodiment, the processor 230 uses the discrimination model stored in the memory (eg, the memory 220 of FIG. 2 ) to display an image (eg, the second image) corresponding to at least a partial region of the first image. You can check whether it is an image or a back image. The processor 230 may check whether the second image is a vehicle front image or a vehicle rear image by inputting the second image to the discrimination model. The second image may be an image corresponding to at least a partial region of the first image. The size of the second image may affect the processing speed of the discriminant model. For example, when the size of the second image is too large, the processing time of the discriminant model may be long. Also, when the size of the second image is too small, the accuracy of determining whether the discrimination model is the front or rear of the vehicle may be lowered. According to an embodiment, the second image may be an image obtained by re-adjusting an image of a partial region included in the first image to a predetermined size. According to an embodiment, the discrimination model may determine whether the second image is a vehicle front image or a vehicle rear image based on whether a feature object is included in the second image. The feature object may be a visual feature that can distinguish the front from the rear of the vehicle. For example, the feature object may include a front grille and/or air intake included only in the front of the vehicle. In addition, the feature object may include an intaglio and/or lighting area around the license plate included only at the rear of the vehicle. The processor 230 may determine whether the second image 340 includes a front or rear surface based on whether the feature object is included. According to an embodiment, the second image may include at least a part of the feature object. According to an embodiment, the size of the second image may be such that at least a partial region of the feature object is included in the second image even if it is applied to various vehicle types, and information about this size is stored in the memory 220 in advance. can be saved. According to an embodiment, the size of the second image may have a range extended by a certain range from an area corresponding to the license plate (eg, the first area). For example, the second image may be an image corresponding to an area extended by three times the height of the license plate up and down with respect to the first area corresponding to the license plate, but this is an example of the expanded multiple, and the expansion multiple is 3 not limited to ships.

Referring to operation 450 , the processor 230 may determine whether the vehicle is traveling in a reverse direction or in a reverse direction. According to an embodiment, the processor 230 may determine whether at least one vehicle included in the first image is traveling in the reverse direction or is parked in the reverse direction. The processor 230 may determine whether an image (eg, a second image) corresponding to at least a partial region of the first image is a front image or a rear image using the determination model, and determine whether to run in reverse based on the result of the confirmation. can The processor 230 may determine whether to run in the reverse direction based on a result of whether the second image 230 is a front or a rear side. According to an embodiment, the processor 230 may determine whether to drive in the reverse direction based on the traveling direction of the road. For example, depending on the installation location and/or direction of the electronic device 101 , the front direction of the vehicle may be photographed in the forward direction, or the rear direction of the vehicle may be photographed in the forward direction. The memory (eg, the memory 220 of FIG. 2 ) may store data on the driving direction of the road in advance, and the processor 230 determines whether the data and the second image 230 are the front or the rear of the road. It may be determined whether the vehicle is traveling in the reverse direction based on the information on the . For example, the electronic device 101 may be installed to photograph a vehicle traveling in a direction opposite to the photographing direction of the electronic device 101 , and the processor 230 may determine the second image as a vehicle rear image. It can be judged that it is running in reverse.

Referring to operation 460 , the processor 230 may generate reverse driving information. The reverse driving information may be data regarding whether the vehicle is driving in the reverse direction. For example, the reverse running information may include information on whether the vehicle is running in the reverse direction, and may include at least one of text data and a first image for recognizing text on a license plate of the vehicle. The processor 230 may generate reverse driving information and temporarily store the generated reverse driving information in a memory. According to an embodiment, the reverse driving information may include at least a portion of information on whether or not the reverse driving is performed, a photographing time of the first image corresponding to the reverse driving, character recognition data of the license plate of the vehicle, and the first image. According to an embodiment, the processor 230 may recognize the license plate characters included in the first image. According to an embodiment, the processor 230 may process a text image included in the first area and recognize it as text data. According to an embodiment, the processor 230 may detect the first area and recognize a character in the detected first area. For example, the processor 230 may recognize a character included in the first region by performing optical character recognition (OCR). According to an embodiment, the processor 230 may pre-process the first image or an image corresponding to the first region included in the first image in order to recognize the character. The pre-processing may include, for example, gray scale transformation, image filtering, and binarization processing. According to an embodiment, the processor 230 may store an algorithm and/or instructions for preprocessing in a memory. According to an embodiment, the processor 230 may recognize a character using an OCR model. The OCR model may be a model trained to output text data corresponding to an input image based on a convolution neural network (CNN) algorithm. According to an embodiment, the processor 230 may store the OCR model in a memory. According to an embodiment, the processor 230 may recognize the characters included in the license plate area (eg, the first area) when it is determined that the reverse driving corresponds to the reverse driving after determining whether the reverse driving is performed. Alternatively, the processor 230 may perform character recognition on the continuously detected first area regardless of whether the vehicle is traveling backwards or not. According to one embodiment, the processor 230 generates reverse driving information including at least a portion of information on whether or not reverse driving, a shooting time of a first image corresponding to reverse driving, license plate character recognition data of the vehicle, and a first image, may be stored in a memory (eg, the memory 220 of FIG. 2 ). According to an embodiment, the processor 230 may generate reverse driving information and transmit it to an external device (eg, the external device 102 of FIG. 1 ). .

5 is an example of vehicle front image data for training a discrimination model.

6 is an example of vehicle rear image data for training a discrimination model.

7 illustrates setting of the second area based on the first area according to various embodiments of the present disclosure.

5 and 6 , the first image (eg, the first image 310 of FIG. 3 or the first image 310 of FIG. 7 ) is the front image 201-1 of the vehicle and/or the vehicle The back image 201 - 2 may be included.

According to an embodiment, the electronic device (eg, the electronic device 101 of FIG. 1 ) uses a discrimination model for determining whether the vehicle is a front image or a rear image of the vehicle, and the image of the vehicle included in the first image (eg, the second image) It may be checked whether the image 340 is the front or rear of the vehicle. According to an embodiment, the electronic device 101 determines whether the vehicle is the front or the rear using an image (eg, the second image 340 ) corresponding to at least a partial region (eg, the second region) of the first image. can be checked The discrimination model may be a model generated using an artificial intelligence algorithm. The discrimination model may be a model learned using a deep learning-based algorithm. For example, the discriminant model may be a model trained to determine whether the vehicle is front or rear based on a convolutional neural network (CNN) algorithm. According to an embodiment, the discrimination model may be a model learned by receiving image data including a vehicle front image and a vehicle rear image and data labeled with information on whether the front or rear side corresponding to each image data is input.

Referring to FIG. 5 , the discrimination model may be trained to recognize the front or rear side of the vehicle using the second image 340 . The second image 340 may include an image of a part of the front surface 201-1 of the vehicle or an image of a part of the rear surface 201-2 of the vehicle. According to an embodiment, the discrimination model may be trained to recognize the front or rear side of the vehicle by using the second image 340 as training data. The vehicle front image 201-1 of FIG. 5, the vehicle rear image 201-2 and the second image 340 of FIG. 6 are examples of the vehicle front image or the rear image of the vehicle, and various images are used as learning data. can be used The discrimination model is the vehicle front image 201-1 or the vehicle based on a feature object that can be distinguished from the image for a part of the vehicle rear 201-2 included in the image for a part of the vehicle front 201-1. It can be learned to distinguish the back image 201 - 2 from each other. The feature object may be a visual feature that can distinguish the front from the rear of the vehicle. For example, the feature object may be distinguished by a visual feature included only in the front or rear of the vehicle, ie, shape and/or color. According to an embodiment, the electronic device 101 may determine whether the second image is a front image or a rear image of the vehicle based on whether a feature object exists within a predetermined range based on the license plate 301 . According to an embodiment, the electronic device 101 determines whether at least a partial area of the feature object is included within a certain range (eg, the second area 330 of FIG. 7 ) based on the license plate 301 . Based on this, it may be determined whether the second image 340 is an image corresponding to a part of the front of the vehicle or an image corresponding to a part of the rear of the vehicle. For example, only a part of the feature object may be included in the second image 340. Referring to FIGS. 5, 6, and 7, the vehicle front image 201-1 is the vehicle rear image 201- The feature object that can be distinguished from 2) may include a front grill 510 and/or an air intake 520 . In addition, the vehicle rear image 201-2 is a feature object that can be distinguished from the vehicle front image 201-1, the intaglio 610 area formed around the rear license plate 301 and the lighting formed at the top of the license plate 301 The illuminance change due to 620 may be included. In the case of the light 620 at the top of the license plate 301, in the case of the second image 340 taken at night, it may be a feature object that can be distinguished from other areas. According to an embodiment, the second image 340 may include a part of the area formed by the feature object. For example, the second image 340 may include a portion of the front grill 510 and/or a portion of the air intake 520 . For another example, the second image 340 may include a portion of the intaglio 610 and/or a portion of the rear license plate 301 top light 620 .

According to various embodiments, the electronic device 101 uses the discrimination model to generate a second image 340 corresponding to the front side of the vehicle 201-1 and/or a second image corresponding to the rear side 201-2 of the vehicle ( 201 - 2 ). 340), it may be determined whether the input image (eg, the first image) is the front or rear of the vehicle based on the difference between the feature objects.

The discriminant model may be trained to identify any one of the feature objects of the vehicle front 201-1 and/or rear 201-2 based on a convolutional neural network (CNN) algorithm. For example, the discriminant model may be trained using a plurality of vehicle image data including at least a portion of any one of the feature objects of the front of the vehicle (eg, the front grill 510 and/or the air intake 520 ). can According to an embodiment, the discrimination model may be trained to determine whether the vehicle is the front or the rear using the second image 340 including only a partial region of the feature object. For example, the discrimination model may determine whether the vehicle is the front or the rear using the second image 340 including only a partial area of the air intake 520 . The training data used for training the discrimination model may include an image in which at least a portion of the vehicle (eg, the second image 340 ) including at least one of the feature objects in the front of the vehicle is captured. Since the discrimination model can preferentially detect the license plate area to identify the vehicle in the first image, the learning data includes at least the license plate 301, and at least one of the feature objects on the front of the vehicle is included ( For example, the second image 340 of FIG. 3) may be included. According to an embodiment, the discrimination model may be learned through a plurality of second images.

For another example, the discrimination model includes at least a portion of any one of the feature objects on the rear of the vehicle (eg, the intaglio 610 around the license plate 301 and/or the license plate 301 top light 620). It can be trained using a plurality of vehicle image data. The training data used for training the discrimination model may include an image of at least a portion of the vehicle including at least one of the feature objects on the rear of the vehicle. Since the discrimination model can detect the license plate area preferentially to identify the vehicle in the first image, the learning data includes at least the license plate 301, and at least one of the feature objects on the rear of the vehicle is included ( For example, the second image 340 of FIG. 3) may be included. According to an embodiment, the discrimination model may be learned through the plurality of second images 340 . According to an embodiment, the electronic device 101 may train the discrimination model by using the plurality of second images 340 including a partial region of the feature object.

According to an embodiment, the electronic device 101 acquires a second image (eg, the second image 340 of FIG. 3 ), and determines whether the acquired second image 340 using the discrimination model is the front of the vehicle. It can be determined whether it is the rear of the vehicle.

Referring to FIG. 7 , the second image 340 may be at least a part of the first image 310 . The electronic device 101 may check whether the vehicle is front or rear using only a part of the first area 310 (eg, the second image 340 ) for accurate and efficient determination. The first area 310 may be an area formed by the license plate 301 identified by the electronic device 101 . According to an embodiment, the second image 340 may be a region selected based on the first region 310 . For example, the second image 340 may include a region in which the first region 310 is expanded by a predetermined ratio. As another example, the second image 340 may include a region formed at a position spaced apart from the first region 320 by a predetermined ratio. According to an embodiment, the second image 340 may include the first region 320 . According to an embodiment, the electronic device 101 may detect the first area 320 in which the license plate 301 exists in the first image 310 by using the license plate area detection model. The electronic device 101 may select the second area 330 based on the first area 320 . According to various embodiments, the electronic device 101 may select the second image 340 based on the location, area, and/or length of the first region 320 . According to an embodiment, the electronic device 101 may check the location, area, and/or length of the first region 320 . For example, the first region 320 may be a region including a rectangle or a shape similar to a rectangle. According to an embodiment, the electronic device 101 may select an area extending in the left and right directions of the first area 320 as the second image 340 . The electronic device 101 selects the second image 340 by setting d3 expanded by a predetermined ratio based on the horizontal length (eg, d1 ) of the first region 320 as the horizontal length of the second image 340 . can The second image 340 selected by extending the horizontal length may include at least a portion of the engraved area 610 around the rear license plate 301 of the vehicle. According to an embodiment, the electronic device 101 may select an area extending in the vertical direction of the first area 320 as the second area 330 . The electronic device 101 selects the second image 340 by setting d4 expanded by a predetermined ratio based on the vertical length (eg, d2 ) of the first region 320 as the vertical length of the second image 340 . can The second image 340 selected by extending the vertical length is the front grill 510 and/or air intake 520 area, and the rear license plate 301 upper lighting area 620 and/or license plate 301 area ( For example, the first region 320) may be included. The electronic device 101 engraves the vehicle front image 201-1 and/or the vehicle rear image 201-2 around characteristic objects (eg, the front grill 510, the air intake 520, and the license plate 301). 610 and / or information about the enlargement ratio of the second image 340 for the first area 320 to include at least one of the ambient lighting 620) of the license plate 301 is stored in a memory (eg: It may be stored in advance in the memory 220 of Fig. 2. The electronic device 101 expands the first area 320 based on a preset ratio, so as to display the vehicle front image 201-1 and/or the vehicle rear side. At least one of the feature objects of the image 201-2 (eg, the front grill 510, the air intake 520, the license plate 301 surrounding intaglio 610 and/or the license plate 301 ambient lighting 620) may select the second region 330 including the region formed by .The electronic device 101 selects the second region 330 and generates a second image corresponding to the selected second region 330 . can

An electronic device according to various embodiments disclosed in this document includes a camera module and a processor operatively connected to the camera module, wherein the processor controls the camera module to capture a moving picture to capture a vehicle image The operation of obtaining a first image including the operation of detecting a first area including at least one license plate image in the first image using the license plate detection model stored in the memory, the first image in the An operation of acquiring a second image corresponding to a second area extended by a certain range from the first area, determining whether at least one feature object is included in the second image using a discrimination model, and the determination An operation of determining whether the second image is a front image or a rear image of a vehicle based on can do.

In addition, the discrimination model performs an operation of learning to determine whether the at least one feature object exists in the second image, and the discrimination model is the license plate image and the second image in the second image through learning An operation of checking whether all feature objects exist may be performed.

In addition, the discrimination model may be learned to determine whether the air intake area and the license plate image are present in the input image at the same time.

In addition, the discrimination model may be learned to determine whether there is an intaglio region formed around the license plate in the input image.

In addition, the discrimination model can be learned to determine whether the license plate upper lighting area is present in the input image.

In addition, the discrimination model, based on a convolutional neural network (CNN) algorithm, performs an operation of learning to determine whether the input image is the front or rear of the vehicle, and the learning operation is a license plate Labeling learning using a plurality of image data including a plurality of vehicle front images and a plurality of vehicle rear images including a license plate and information on whether the front or the rear corresponding to each of the plurality of image data is labeled techniques may be included.

Also, the second area may include at least an area extended by a predetermined multiple in the vertical direction with respect to the first area.

The processor may further perform an operation of determining whether the vehicle is traveling in the reverse direction based on the traveling direction of the vehicle, and an operation of storing the first image based on whether the vehicle is traveling in the reverse direction.

In addition, the processor recognizes the characters in the license plate included in the first image, determines whether the vehicle is traveling in the reverse direction based on the driving direction of the vehicle, and stores the recognized characters based on whether the vehicle is traveling in the reverse direction. more actions can be performed.

A method for an electronic device to determine a driving direction of a vehicle according to various embodiments disclosed in this document includes: acquiring a first image including a vehicle image by shooting a video; Detecting a first area including at least one license plate image in the first image, obtaining a second image corresponding to a second area extended by a predetermined area from the first area in the first image, determining determining whether at least one feature object is included in the second image using a model, and determining whether the second image is a front image or a rear image of a vehicle based on the determination, and the front image The method may include confirming the driving direction of the vehicle included in the first image based on whether it is a recognized rear image.

In addition, the discrimination model performs an operation of learning to determine whether the at least one feature object exists in the second image, and the discrimination model is the license plate image and the second image in the second image through learning An operation of checking whether all feature objects exist may be performed.

In addition, the discrimination model may be learned to determine whether the air intake area and the license plate image are present in the input image at the same time.

In addition, the discrimination model, based on a convolutional neural network (CNN) algorithm, is trained to determine whether the input image is the front or rear of the vehicle, the learning is a plurality of vehicle front including a license plate A plurality of image data including a plurality of rear images of the vehicle including an image and a license plate, and information on whether the front or rear corresponding to each of the plurality of image data may include a labeling learning technique using the labeled data.

Also, the second area may include at least an area extended by a predetermined multiple in the vertical direction with respect to the first area.

The method may further include determining whether the vehicle is traveling in the reverse direction based on the traveling direction of the vehicle, and storing the first image based on whether the vehicle is traveling in the reverse direction.

In addition, the steps of recognizing a character in the license plate included in the first image, determining whether the vehicle is traveling in the reverse direction based on the driving direction of the vehicle, and storing the recognized character based on whether the vehicle is traveling in the reverse direction may include.

Claims (16)

In an electronic device,
camera module;
memory for storing data; and
a processor operatively connected to the camera module;
The processor is
An operation of controlling the camera module to shoot a video and acquiring a first image including a vehicle image from the captured video;
Detecting a first area including a license plate image in the first image using the license plate detection model stored in the memory,
From the first image, a second region including the first region, having the same center as the first region, and corresponding to a second region having an area extended by a predetermined amount with respect to the area of the first region the act of acquiring an image;
The second image is analyzed using a discrimination model to determine whether a feature object exists in the second image, and based on the identified feature object in the second image, whether the second image is a front image of a vehicle or a rear image action to determine whether it is recognized, and
Performs an operation of confirming the driving direction of the vehicle included in the first image based on a combination of the data on the driving direction of the road stored in the memory and whether it is the front image or the rear image,
The second image is obtained from a portion of the first image,
The discriminant model analyzes only the second image. According to claim 1,
The discriminant model is
performing an operation of learning to determine whether the feature object exists in the second image,
The discrimination model is an electronic device for performing an operation to check whether both the license plate image and the feature object in the second image through learning. According to claim 1,
The discriminant model is
An electronic device trained to determine whether the air intake area and the license plate image exist simultaneously in the input image. According to claim 1,
The discriminant model is
An electronic device trained to determine whether an intaglio region formed around the license plate exists in the input image. According to claim 1,
The discriminant model is
An electronic device that has been trained to determine whether a license plate top lighting area is present in the input image. According to claim 1,
The discriminant model is
Based on a convolutional neural network (CNN) algorithm, the operation of learning to determine whether the input image is the front or rear of the vehicle is performed,
In the learning operation, a plurality of image data including a plurality of vehicle front images including a license plate and a plurality of vehicle rear images including a license plate and information on whether the front or rear corresponding to each of the plurality of image data are labeled An electronic device including a labeling learning technique using the data. According to claim 1,
The first region includes a rectangular shape,
The second area is
including an area corresponding to a range in which a horizontal length and/or a vertical length of the first region is extended by a multiple of a predetermined length,
The multiple of the predetermined length is within three times. According to claim 1,
The processor is
Determining whether the vehicle is traveling in the reverse direction based on the traveling direction of the vehicle;
The electronic device further performing an operation of storing the first image based on whether the vehicle is traveling in the reverse direction. According to claim 1,
The processor is
Recognizing the characters in the license plate included in the first image,
Determining whether the vehicle is traveling in the reverse direction based on the traveling direction of the vehicle;
The electronic device further performing an operation of storing the recognized character based on whether the vehicle travels in the reverse direction. A method for an electronic device to determine a driving direction of a vehicle, the method comprising:
acquiring a first image including a vehicle image from the captured video by taking a video;
detecting a first area including a license plate image in the first image using a license plate detection model;
From the first image, a second area including the first area, having the same center as the first area, and corresponding to a second area having an area extended by a predetermined multiple of the area of the first area 2 acquiring an image;
The second image is analyzed using a discrimination model to determine whether a feature object exists in the second image, and based on the identified feature object in the second image, whether the second image is a front image of a vehicle or a rear image determining whether it is recognized; and
determining the driving direction of the vehicle included in the first image based on a combination of data regarding the traveling direction of the road stored in the electronic device and whether the image is the front image or the rear image,
The second image is obtained from a portion of the first image,
The discriminant model analyzes only the second image. 11. The method of claim 10,
The discriminant model is
performing an operation of learning to determine whether the feature object exists in the second image,
The discriminant model is a method of performing an operation of confirming whether both the license plate image and the feature object are present in the second image through learning. 11. The method of claim 10,
The discriminant model is
A method learned to determine whether an air intake area and a license plate image exist simultaneously in the input image. 11. The method of claim 10,
The discriminant model is
Based on a convolutional neural network (CNN) algorithm, it is learned to determine whether the input image is the front or rear of the vehicle,
The learning is a plurality of image data including a plurality of vehicle front images including a license plate and a plurality of vehicle rear images including a license plate and information on whether the front or rear corresponding to each of the plurality of image data is labeled data A method including a labeling learning technique using 11. The method of claim 10,
The first region includes a rectangular shape,
The second area is
including an area corresponding to a range in which a horizontal length and/or a vertical length of the first region is extended by a multiple of a predetermined length,
A multiple of the predetermined length is within three times. 11. The method of claim 10,
determining whether the vehicle is traveling in the reverse direction based on the traveling direction of the vehicle; and
and storing the first image based on whether the vehicle travels in the reverse direction. 11. The method of claim 10,
Recognizing the characters in the license plate included in the first image;
determining whether the vehicle is traveling in the reverse direction based on the traveling direction of the vehicle; and
and storing the recognized character based on whether the vehicle travels in the reverse direction.

Images