KR102603424B1 - Method, Apparatus, and Computer-readable Medium for Determining Image Classification using a Neural Network Model - Google Patents

Abstract

This disclosure relates to a method, apparatus, and computer-readable medium for determining image event classification using a neural network model. A method according to an embodiment of the present disclosure includes inputting an image as input data to a first neural network model, obtaining an event classification result for the image as output data of the first neural network model, event classification results, and event classification. A new event classification for the image can be determined based on the reliability information of the landmark point that matches the result.

Description

{Method, Apparatus, and Computer-readable Medium for Determining Image Classification using a Neural Network Model}

The present invention relates to a method, apparatus, and computer-readable medium for determining image event classification using a neural network model.

A neural network model is a computer system that implements human-level intelligence and is a model in which machines learn and make decisions on their own.

The neural network model consists of machine learning (deep learning) technology that uses an algorithm that classifies/learns the characteristics of input data on its own, and element technologies that mimic the functions of the human brain such as cognition and judgment using machine learning algorithms.

Element technologies include, for example, linguistic understanding technology that recognizes human language/characters, visual understanding technology that recognizes objects as if they were human eyes, reasoning/prediction technology that judges information and makes logical inferences and predictions, and human experience information. It may include at least one of a knowledge expression technology that processes knowledge data and a motion control technology that controls the autonomous driving of a vehicle and the movement of a robot.

Visual understanding is a technology that recognizes and processes objects like human vision, and includes object recognition, object tracking, image search, person recognition, scene understanding, spatial understanding, and image improvement.

Inferential prediction is a technology that judges information to make logical inferences and predictions, and includes knowledge/probability-based reasoning, optimization prediction, preference-based planning, and recommendations.

Recently, electronic devices that classify events in images using artificial intelligence technology have been developed, and research is needed to improve the accuracy of event classification.

The above-mentioned background technology is technical information that the inventor possessed for deriving the present invention or acquired in the process of deriving the present invention, and cannot necessarily be said to be known art disclosed to the general public before filing the application for the present invention.

Republic of Korea Patent Publication No. 10-2005150 (2019.07.23)

The present invention provides a method, apparatus, and computer-readable medium for determining image event classification using a neural network model. The problem to be solved by the present invention is not limited to the problems mentioned above, and other problems and advantages of the present invention that are not mentioned can be understood through the following description and can be understood more clearly through the examples of the present invention. It will be. In addition, it can be seen that the problems and advantages to be solved by the present invention can be realized by the means and combinations thereof indicated in the patent claims.

As a technical means for achieving the above-mentioned technical problem, a first aspect of the present disclosure is a method of determining image event classification using a neural network model, inputting an image as input data to a first neural network model, Obtaining an event classification result for the image using output data of the first neural network model; and determining a new event classification for the image based on the event classification result and reliability information of a landmark point matching the event classification result. A method may be provided, wherein the new event classification is related to the event classification result.

A second aspect of the present disclosure provides an apparatus for determining image event classification using a neural network model, comprising: a memory storing at least one program; and at least one processor that drives a neural network model by executing the at least one program, wherein the at least one processor inputs an image as input data to a first neural network model, and operates the first neural network model. Obtain an event classification result for the image as output data, determine a new event classification for the image based on the event classification result and reliability information of a landmark point matching the event classification result, and determine a new event classification for the image. Classification may provide a device that is associated with the event classification results.

A third aspect of the present disclosure may provide a computer-readable recording medium recording a program for executing the method according to the first aspect on a computer.

In addition, another method for implementing the present invention, another system, and a computer-readable recording medium storing a computer program for executing the method may be further provided.

Other aspects, features and advantages in addition to those described above will become apparent from the following drawings, claims and detailed description of the invention.

According to the problem solving means of the present disclosure described above, the present disclosure can overcome the limitations of the event classification model and classify events in images with higher accuracy.

According to the problem-solving means of the present disclosure described above, the present disclosure determines image event classification by combining the information of the event classification model and the landmark detection model, thereby overcoming the limitations of the event classification model and classifying events in the image with high accuracy. can do.

1 is a block diagram of a system according to one embodiment.
2A and 2B are exemplary diagrams for explaining a method of detecting a face area in an image according to an embodiment.
FIG. 3 is an exemplary diagram illustrating a method for detecting landmarks in a face area according to an embodiment.
FIG. 4 is an exemplary diagram illustrating a method of classifying events in an image according to an embodiment.
Figure 5 is a diagram for explaining an example of an event classification result for an image according to an embodiment.
6A and 6B are exemplary diagrams for explaining a method of calculating reliability information of a landmark point according to an embodiment.
7A and 7B are exemplary diagrams for explaining a method of determining a new event classification for an image according to an embodiment.
8A and 8B are exemplary diagrams for explaining a method of calculating reliability of a landmark point according to an embodiment.
Figure 9 is a flowchart illustrating a method of determining image event classification using a neural network model according to an embodiment.
Figure 10 is a block diagram of an image event classification determination device according to an embodiment.

The advantages and features of the present invention and methods for achieving them will become clear by referring to the embodiments described in detail together with the accompanying drawings. However, the present invention is not limited to the embodiments presented below, but may be implemented in various different forms, and should be understood to include all conversions, equivalents, and substitutes included in the spirit and technical scope of the present invention. . The embodiments presented below are provided to ensure that the disclosure of the present invention is complete and to fully inform those skilled in the art of the scope of the invention. In describing the present invention, if it is determined that a detailed description of related known technologies may obscure the gist of the present invention, the detailed description will be omitted.

The terms used in this application are only used to describe specific embodiments and are not intended to limit the invention. Singular expressions include plural expressions unless the context clearly dictates otherwise. In this application, terms such as “comprise” or “have” are intended to designate the presence of features, numbers, steps, operations, components, parts, or combinations thereof described in the specification, but are not intended to indicate the presence of one or more other features. It should be understood that this does not exclude in advance the possibility of the existence or addition of elements, numbers, steps, operations, components, parts, or combinations thereof.

Some embodiments of the present disclosure may be represented by functional block configurations and various processing steps. Some or all of these functional blocks may be implemented in various numbers of hardware and/or software configurations that perform specific functions. For example, the functional blocks of the present disclosure may be implemented by one or more microprocessors, or may be implemented by circuit configurations for certain functions. Additionally, for example, functional blocks of the present disclosure may be implemented in various programming or scripting languages. Functional blocks may be implemented as algorithms running on one or more processors. Additionally, the present disclosure may employ conventional technologies for electronic environment setup, signal processing, and/or data processing. Terms such as “mechanism,” “element,” “means,” and “configuration” may be used broadly and are not limited to mechanical and physical configurations.

Additionally, connection lines or connection members between components shown in the drawings merely exemplify functional connections and/or physical or circuit connections. In an actual device, connections between components may be represented by various replaceable or additional functional connections, physical connections, or circuit connections.

Hereinafter, the present disclosure will be described in detail with reference to the attached drawings.

1 is a block diagram of a system according to one embodiment.

System 100 may include a face detection model 110, a landmark detection model 121, an event classification model 122, and a post-processing module 130.

The face detection model 110 may acquire an image and detect a face area from the acquired image.

The landmark detection model 121 may receive the face area detected in the image from the face detection model 110. The landmark detection model 121 can detect landmarks within the face area.

The event classification model 122 can classify events in the acquired image. When the face detection model 110 detects a face area in an image, the event classification model 122 may classify the event of the image in which the face area is detected.

The post-processing module 130 may perform post-processing using landmarks in the face area obtained from the landmark detection model 121 or event classification results of the image obtained from the event classification model 122. For example, the post-processing module 130 may determine a head pose using a landmark, or may perform voting using the landmark and event classification results.

The system 100 may provide information to users using the system 100 through the results output from the post-processing module 130. For example, system 100 may be a driving monitoring system, in which case system 100 may include a face detection model 110, a landmark detection model 121, an event classification model 122, and By processing the image captured of the driver using the post-processing module 130, information on the driver's status can be provided. For example, the driver's status information may include distraction, drowsy, smoking, and phone.

2A and 2B are exemplary diagrams for explaining a method of detecting a face area in an image according to an embodiment.

The face detection model can acquire an image 200 for detecting a face area. The face detection model may be implemented as a software module, a hardware module, or various combinations thereof to detect a face area from the image 200.

Various algorithms can be used as a method for detecting the face area within the image 200. For example, a face detection model can be implemented as a neural network model using the Adaboost algorithm, but the algorithm that can be used to implement the neural network model is not limited to this.

Referring to FIG. 2A, a sliding window 210 may be used to detect a facial area within image 200. The face detection model can scan the image 200 using the sliding window 210 and determine whether each area of the scanned image 200 corresponds to a face area or a background area.

Referring to FIG. 2B, the face detection model can detect a predetermined area within the image 200 as the face area 220 through learning.

FIG. 3 is an exemplary diagram illustrating a method for detecting landmarks in a face area according to an embodiment.

The landmark detection model may be implemented as a software module, a hardware module, or various combinations thereof to detect the landmark 310 within the face area 300. The landmark detection model may receive the face area 300 detected in the face detection model and detect the landmark 310 within the face area 300.

The landmark 310 refers to a feature point indicating a position representing each feature within the face area 300. For example, the landmark 310 may represent feature points corresponding to eyes, nose, mouth, and eyebrows, which are representative features of a person's face. At least one landmark 310 may be set for each feature, and a plurality of landmarks 310 may be set for one feature.

The landmark detection model can be implemented as a neural network model using learning and statistical algorithms (e.g., AAM (active appearance model), SDM (Supervised Descent Method), etc.) for the previously acquired face area 300. , the algorithms that can be used to implement the neural network model are not limited to this.

Taking the right eyebrow as an example among the features of the face area 300 in FIG. 3, the number of landmarks 310 corresponding to the right eyebrow may be set to 9.

Meanwhile, the number of landmarks 310 corresponding to each feature of the face area 300 may be set to various numbers depending on the application in which the landmarks 310 are used.

FIG. 4 is an exemplary diagram illustrating a method of classifying events in an image according to an embodiment.

The event classification model may be implemented as a software module, a hardware module, or various combinations thereof to classify events in an image. When the face detection model detects a face area in a predetermined image, the event classification model can classify the event of the predetermined image in which the face area is detected.

The event classification model can classify which of the preset events the image corresponds to based on the face area included in the image and objects around the face area.

The event classification model can be implemented as a convolutional neural network (CNN) model, but the algorithm that can be used to implement the neural network model is not limited to this.

Specifically, the event classification model can be implemented as an architecture with multiple layers including an input image, feature maps, and output. In the event classification model, a convolution operation is performed on the input image with a filter called a kernel, and as a result, feature maps are output. The output feature maps generated at this time are input feature maps, and a convolution operation with the kernel is performed again, and new feature maps are output. As a result of this convolution operation being performed repeatedly, event results based on the features of the input image can ultimately be output through an event classification model. Through the above-described process, the event classification model can classify events in a given image.

Referring to FIG. 4, the event classification model can classify image events into normal (410), phone (420), smoke (430), and mask (440). phone(420) is an event indicating that the person in the image is using a cell phone, smoke(430) is an event indicating that the person in the image is smoking, and mask(440) is an event indicating that the person in the image is wearing a mask. , and normal(410) may be an event representing all cases other than the above-mentioned events. However, the types of events that the event classification model can classify are not limited to the above examples.

Figure 5 is a diagram for explaining an example of an event classification result for an image according to an embodiment.

A device for determining image event classification using a neural network model (hereinafter referred to as an image event classification determination device) inputs an image as input data to a first neural network model and determines the image event as output data of the first neural network model. Event classification results can be obtained.

The first neural network model may be the event classification model described in FIG. 4, and the event classification result for the image may be any one of the events for the image in FIG. 4.

According to one embodiment, the event classification result for an image may include confidence information about the event classification result. The higher the reliability of the image event classification results, the lower the uncertainty, and conversely, the lower the reliability of the image event classification results, the greater the uncertainty.

Referring to FIG. 5, the images 510 in the group with clear image event classification results were classified into mask events and normal events, respectively, and the reliability of the classification results was high at 97.3% and 93.1%, respectively, while the image event classification results were high at 97.3% and 93.1%, respectively. The image 530 of the group with unclear classification results was classified as a mask event, but it can be seen that the reliability of the classification results was somewhat low at 71.1% and 63.7%.

Meanwhile, even if the event classification model is well-trained, the event classification results cannot have 100% reliability. For example, referring to the image 530 of the group for which the image event classification result is unclear, it can be seen that although the person in the image is wearing a mask, it is worn on the chin rather than the mouth, so the reliability of mask event classification is low. In addition, for example, it is determined that a person is using a cell phone in the image, but since the person is using it close to the mouth rather than the ear, the reliability of phone event classification may be low, and although it is classified as a normal event, the person is using it with his or her hand in the image. There may be cases where the reliability of event classification is low because the eyes are covered.

In this disclosure, in order to overcome the limitations of such event classification models and classify events in images with higher accuracy, the image event classification results are based on the reliability information of landmark points matching the image event classification results. A method for determining a new event classification is disclosed. A more detailed method will be described later.

6A and 6B are exemplary diagrams for explaining a method of calculating reliability information of a landmark point according to an embodiment.

The image event classification determination device may input an image as input data to a second neural network model, detect a plurality of landmark points in the image, and calculate reliability information for each of the plurality of landmark points.

In one embodiment, the image input to the second neural network model may be a related image (hereinafter, the second image) related to the image (hereinafter, the first image) input to the first neural network model. The second image may be a modified image of the first image. Specifically, the second image may be an image that has been resized, cropped, or channel converted to the first image. For example, the second image may be an image with a margin expanded by 1 pixel compared to the first image, an image with the rgb channel order changed, or an image whose size is changed through resize.

The second neural network model may be the landmark detection model described in FIG. 3, and the plurality of landmark points may be landmarks within the face area detected through the landmark detection model. A plurality of landmark points may represent feature points corresponding to features representing a person's face, and a plurality of landmark points may be set for each feature.

Reliability information for each of a plurality of landmark points may mean accuracy of the location of the landmark point. Reliability information may be determined according to the distribution of scores calculated in a regression process for each point of a plurality of landmarks. A method of calculating reliability information through a regression analysis process for each of a plurality of landmark points will be described below with reference to FIGS. 8A and 8B.

Referring to FIG. 6A, the image event classification determination device inputs the original image 610 into the second neural network model and detects both eyes, nose, and both ends of the mouth using a plurality of landmark points 631. Thus, it can be confirmed that reliability information 651 for each of the plurality of landmark points 631 has been calculated.

According to one embodiment, the reliability information of the landmark point may be reliability information of a target landmark point determined among a plurality of landmark points based on an event classification result for the image.

The image event classification determination device may determine some landmark points as target landmarks among a plurality of landmark points detected through the second neural network model, based on the image event classification result obtained through the first neural network model. .

For example, the image event classification determination device may determine a landmark point corresponding to the mouth or chin as the target landmark based on the mask event classification result. As another example, the image event classification determination device may determine the landmark point corresponding to the ear as the target landmark based on the phone event classification result, and may determine the landmark point corresponding to the mouth as the target landmark based on the smoke event classification result. You can.

Accordingly, the image event classification determination device according to the present disclosure will be able to organically combine information between the event classification model and the landmark detection model. It will be understood that by determining a new event classification for an image by combining the reliability information of landmark points that have an organic connection with the event classification result, it is possible to maintain the correlation between the event classification result and the new event classification.

According to another embodiment, the image event classification determination device divides a plurality of landmark points into one or more areas, selects at least one area from among the areas based on the event classification result for the image, and selects at least one area included in the selected area. The landmark point can be determined as the target landmark point.

As described above in FIG. 3, the number of landmarks 310 corresponding to each feature of the face area 300 may be set to plural, and may be set to various numbers depending on the application in which the landmarks 310 are used. there is. At this time, the image event classification determination device may group a plurality of landmarks corresponding to one feature and distinguish an area for each feature. For example, taking the face area 300 of FIG. 3 as an example, nine landmarks 310 corresponding to the right eyebrow may be divided into one area. The 9 landmarks 310 corresponding to the nose and the 17 landmarks 310 corresponding to the mouth can also be divided into one area.

The image event classification determination device may select an area among the divided areas based on the event classification result for the image, and determine a landmark point included in the selected area as a target landmark point. Depending on the image event classification results, which of one or more areas should be selected may follow a predetermined setting value or may be learned. For example, when a mask event classification result is obtained, the mouth and chin areas can be selected, and when a phone event classification result is obtained, the ear area can be selected.

Referring to FIG. 6B, the image event classification determination device determines a target landmark point 632 included in the mouth area among a plurality of landmark points 631 based on the mask event classification result 620, and determines the target landmark point 632 included in the mouth area. It can be confirmed that the reliability information 652 of the landmark point has been calculated.

7A and 7B are exemplary diagrams for explaining a method of determining a new event classification for an image according to an embodiment.

The image event classification determination device may determine a new event classification for the image based on the event classification result for the image and the reliability information of the landmark point matching the classification result. The new event classification for the image may be related to the event classification result for the image.

In order to classify image events with high accuracy, the present disclosure can determine a new event classification for an image based on reliability information of landmark points that match the image event classification results.

Since the new event classification for the image is determined based on the event classification result for the image, it may be related to the event classification result for the image. For example, a new event classification for an image may mean a new event that the event classification model was unable to classify for the image. In this disclosure, image event classification is determined by combining the image event classification results through an event classification model and the reliability information of landmark points through a landmark detection model, thereby overcoming the limitations of the event classification model and identifying events in the image with high accuracy. can be classified.

According to one embodiment, the new event classification for an image may be a sub-concept of the event classification result for the image. For example, in the hand event classification, eye rubbing, covering one's mouth, etc. may be determined as a new event classification as sub-concepts, and in the mask event classification, a sub-concept such as pull a mask under the chin may be determined as a new event classification. there is.

Referring to FIG. 7A, the image event classification determination device obtained a mask event with a reliability of 63.7% as the image event classification result 710. The image event classification determination device determines the image event classification as a tusk event as a new event classification 750 for the image based on the reliability information 730 of the landmark point matching the image event classification result 710. You can check it.

According to one embodiment, the image event classification determination device may obtain occlusion information of the landmark using reliability information of the landmark point that matches the image event classification result. The image event classification determination device may determine a new event classification for the image based on the event classification result for the image, the reliability information of the landmark point matching the classification result, and the occlusion information of the landmark.

For example, when the reliability information of a landmark point is greater than a predetermined value, the image event classification determination device may obtain occlusion information indicating that the corresponding landmark is not occluded. As another example, when the reliability information of a landmark point is less than a predetermined value, the image event classification determination device may obtain occlusion information indicating that the landmark is occluded. The predetermined value may be determined as 50%, 60%, etc. depending on the application.

Specifically, the image event classification determination device may determine a new event classification for the image in response to the fact that the result of the event classification result for the image is greater than or equal to the first result value and the reliability information of the landmark point is greater than or equal to the first reliability value. there is. Additionally, the image event classification device may determine a new event classification for the image in response to the fact that the result of the event classification result for the image is greater than or equal to the second result value and the reliability information of the landmark point is less than the second reliability value. The result of the event classification result may be reliability information about the classification result.

For example, as an example, an image event classification device obtains A information that the reliability of the hand event classification result is more than 0.6, and B information that the reliability information of the landmark point in the eye area, which is the target landmark point, is less than 0.6. Listen. The image event classification device can obtain information that a human hand appears in the image through A information, and obtain occlusion information that an eye landmark in the face area in the image is occluded through B information. The image event classification device can determine a new event classification of eye rubbing by combining information that a human hand appears in the image and occlusion information that the eye landmark is occluded.

Referring to FIG. 7B, the image event classification determination device obtained a mask event classification result (711) with a reliability of 97.3% and a mask event classification result (712) with a reliability of 63.7% as the image event classification result (710). The image event classification determination device may obtain landmark occlusion information using the reliability information 730 of the landmark point that matches each of the image event classification results 710. For example, the image event classification determination device may use the reliability information 731 of the first landmark point to obtain occlusion information 732 indicating that the mouth is covered, and the reliability information of the second landmark point ( Using 733), occlusion information 734 indicating that the mouth is not covered can be obtained.

Continuing to refer to FIG. 7B, the image event classification determination device is based on the mask event classification result 711 with 97.3% reliability, reliability information 731 of the first landmark point, and occlusion information 732 indicating that the mouth is covered. Accordingly, a new event classification 751 for the image showing that a mask is being worn can be determined. Likewise, the image event classification decision device determines whether the mask is being worn based on the mask event classification result 712 with a reliability of 63.7%, the reliability information 733 of the second landmark point, and the occlusion information 734 indicating that the mouth is not covered. A new event classification 752 for the image can be determined.

8A and 8B are exemplary diagrams for explaining a method of calculating reliability of a landmark point according to an embodiment.

The image event classification determination device may calculate reliability information of the landmark point through a regression process for each of the plurality of landmark points.

According to one embodiment, the image event classification determination device determines a plurality of coordinates for each of a plurality of landmark points, calculates the reliability for each of the plurality of coordinates, and determines the highest reliability among the reliability for each of the plurality of coordinates. The reliability information of the landmark point can be calculated through a regression analysis process that determines the reliability information calculated for each of the plurality of landmark points.

Referring to FIG. 8A, an example of calculating reliability information of a landmark point corresponding to an eye area is shown. Six coordinates and reliability for each of the landmark points corresponding to the eye area can be calculated. It can be seen that the reliability of the landmark point corresponding to the coordinates of (49, 67) is the highest at 97%. The image event classification decision device can determine the highest reliability of 97% among six coordinates and reliability information as reliability information calculated for the landmark point corresponding to the eye area. The image event classification determination device may calculate reliability information for each landmark point corresponding to another face area through a regression analysis process according to the embodiment.

According to another embodiment, the image event classification determination device finds a heat map distribution for each of a plurality of landmark points, calculates the central coordinate on the heat map distribution, and calculates the central coordinate based on the central coordinate. Reliability information for landmark points can be calculated through a regression analysis process that determines reliability information calculated for each of a plurality of landmark points according to the degree of spread of the heatmap distribution.

Referring to FIG. 8B, an example of calculating reliability information of a landmark point corresponding to an eye area is shown. The image event classification decision device can find the heatmap distribution for the landmark point corresponding to the eye area, and the central coordinate in the heatmap distribution can be calculated as (49.67). The image event classification decision device can determine 97% reliability based on (49, 67) and the degree of spread of the heatmap distribution as reliability information calculated for the landmark point corresponding to the eye area. The image event classification determination device may calculate reliability information for each landmark point corresponding to another face area through a regression analysis process according to the embodiment.

Figure 9 is a flowchart illustrating a method of determining image event classification using a neural network model according to an embodiment.

Referring to FIG. 9, in step 910, the image event classification determination device may input an image as input data to a first neural network model and obtain an event classification result for the image as output data of the first neural network model. .

According to one embodiment, the event classification result for an image may include reliability information about the classification result. The higher the reliability of the image event classification results, the lower the uncertainty, and conversely, the lower the reliability of the image event classification results, the greater the uncertainty.

In step 920, the image event classification determination device may determine a new event classification for the image based on the event classification result for the image and the reliability information of the landmark point matching the classification result. The new event classification for the image may be related to the event classification result for the image.

According to one embodiment, the image event classification determination device may obtain occlusion information of the landmark using reliability information of the landmark point that matches the image event classification result. The image event classification determination device may determine a new event classification for the image based on the event classification result for the image, the reliability information of the landmark point matching the classification result, and the occlusion information of the landmark.

According to one embodiment, the image event classification determination device inputs an image as input data to a second neural network model, detects a plurality of landmark points in the image, and calculates reliability information for each of the plurality of landmark points. can do.

According to one embodiment, the reliability information of the landmark point may be reliability information of a target landmark point determined among a plurality of landmark points based on an event classification result for the image.

According to another embodiment, the image event classification determination device divides a plurality of landmark points into one or more areas, selects at least one area from among the areas based on the event classification result for the image, and selects at least one area included in the selected area. The landmark point can be determined as the target landmark point.

According to one embodiment, the image event classification determination device may calculate reliability information of the landmark point through a regression process for each of the plurality of landmark points.

According to one embodiment, the image event classification determination device determines a plurality of coordinates for each of a plurality of landmark points, calculates the reliability for each of the plurality of coordinates, and determines the highest reliability among the reliability for each of the plurality of coordinates. The reliability information of the landmark point can be calculated through a regression analysis process that determines the reliability information calculated for each of the plurality of landmark points.

According to another embodiment, the image event classification determination device finds a heat map distribution for each of a plurality of landmark points, calculates the central coordinate on the heat map distribution, and calculates the central coordinate based on the central coordinate. Reliability information for landmark points can be calculated through a regression analysis process that determines reliability information calculated for each of a plurality of landmark points according to the degree of spread of the heatmap distribution.

Figure 10 is a block diagram of an image event classification determination device according to an embodiment.

Referring to FIG. 10, the image event classification determination device 1000 may include a communication unit 1010, a processor 1020, and a DB 1030. In the image event classification determination device 1000 of FIG. 10, only components related to the embodiment are shown. Accordingly, those skilled in the art can understand that other general-purpose components may be included in addition to the components shown in FIG. 10.

The communication unit 1010 may include one or more components that enable wired/wireless communication with an external server or external device. For example, the communication unit 1010 may include at least one of a short-range communication unit (not shown), a mobile communication unit (not shown), and a broadcast receiver (not shown).

The DB 1030 is hardware that stores various data processed within the image event classification determination device 1000, and can store programs for processing and control of the processor 1020.

The DB 1030 is a random access memory (RAM) such as dynamic random access memory (DRAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), CD- It may include ROM, Blu-ray or other optical disk storage, a hard disk drive (HDD), a solid state drive (SSD), or flash memory.

The processor 1020 controls the overall operation of the image event classification determination device 1000. For example, the processor 1020 can generally control the input unit (not shown), display (not shown), communication unit 1010, DB 1030, etc. by executing programs stored in the DB 1030. The processor 1020 may control the operation of the image event classification determination device 1000 by executing programs stored in the DB 1030.

The processor 1020 may control at least some of the operations of the image event classification determination device 1000 described above with reference to FIGS. 1 to 9 . The image event classification determination device 1000 may be the same as the system 100 of FIG. 1 or may be implemented as a device that performs some of the operations of the system 100.

The processor 1020 includes application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, and microcontrollers. It may be implemented using at least one of micro-controllers, microprocessors, and other electrical units for performing functions.

In one embodiment, the image event classification determination device 1000 may be a mobile electronic device. For example, the image event classification determination device 1000 is implemented in smartphones, tablet PCs, PCs, smart TVs, personal digital assistants (PDAs), laptops, media players, navigation devices, devices equipped with cameras, and other mobile electronic devices. It can be. Additionally, the image event classification determination device 1000 may be implemented as a wearable device such as a watch, glasses, hair band, or ring equipped with a communication function and data processing function.

In another embodiment, the image event classification determination device 1000 may be an electronic device embedded in a vehicle. For example, the image event classification determination device 1000 may be an electronic device inserted into a vehicle through tuning after the production process.

In another embodiment, the image event classification determination device 1000 may be a server located outside the vehicle. A server may be implemented as a computer device or a plurality of computer devices that communicate over a network to provide commands, codes, files, content, services, etc. The server may receive data necessary to classify image events from devices mounted on the vehicle, and classify image events based on the received data.

In another embodiment, the process performed by the image event classification determination device 1000 may be performed by at least some of a mobile electronic device, an electronic device embedded in a vehicle, and a server located outside the vehicle.

Embodiments according to the present invention may be implemented in the form of a computer program that can be executed through various components on a computer, and such a computer program may be recorded on a computer-readable medium. At this time, the media includes magnetic media such as hard disks, floppy disks, and magnetic tapes, optical recording media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and ROM. , RAM, flash memory, etc., may include hardware devices specifically configured to store and execute program instructions.

Meanwhile, the computer program may be designed and configured specifically for the present invention, or may be known and available to those skilled in the art of computer software. Examples of computer programs may include not only machine language code such as that created by a compiler, but also high-level language code that can be executed by a computer using an interpreter or the like.

According to one embodiment, methods according to various embodiments of the present disclosure may be included and provided in a computer program product. Computer program products are commodities and can be traded between sellers and buyers. The computer program product may be distributed in the form of a machine-readable storage medium (e.g. compact disc read only memory (CD-ROM)) or through an application store (e.g. Play StoreTM) or between two user devices. It may be distributed in person or online (e.g., downloaded or uploaded). In the case of online distribution, at least a portion of the computer program product may be at least temporarily stored or temporarily created in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

Unless there is an explicit order or statement to the contrary regarding the steps constituting the method according to the invention, the steps may be performed in any suitable order. The present invention is not necessarily limited by the order of description of the above steps. The use of any examples or illustrative terms (e.g., etc.) in the present invention is merely to describe the present invention in detail, and unless limited by the claims, the scope of the present invention is limited by the examples or illustrative terms. It doesn't work. Additionally, those skilled in the art will recognize that various modifications, combinations and changes may be made depending on design conditions and factors within the scope of the appended claims or their equivalents.

Therefore, the spirit of the present invention should not be limited to the above-described embodiments, and the scope of the patent claims described below as well as all scopes equivalent to or equivalently changed from the scope of the claims are within the scope of the spirit of the present invention. It will be said to belong to

100: System
110: Face detection model
121: Landmark detection model
122: Event classification model
130: Post-processing module
200: image
210: sliding window
220: Face area
300: face area
310: Landmark
410: normal
420: phone
430: smoke
440: mask
510: Images in a group with clear image event classification results
530: Images in a group where the image event classification result is unclear
610: Original image
631: Multiple landmark points
651: Reliability information
620: Image event classification results
632: Target landmark point
652: Reliability information of target landmark point
710: Image event classification results
711: mask event classification result
712: mask event classification result
730: Reliability information of landmark points
731: Reliability information of the first landmark point
732: Occlusion information that the mouth is covered
733: Reliability information of the second landmark point
734: Occlusion information that the mouth is not covered
750: New event classification
751: Classification of new events while wearing a mask
752: New event classification while wearing tusk
1000: Image event classification decision device
1010: Ministry of Communications
1020: Processor
1030:DB

Claims (11)

In a method of determining image event classification using a neural network model,
Inputting an image as input data to a first neural network model, classifying an event for the image and obtaining reliability regarding the classification of the event using output data of the first neural network model; and
In response to the obtained event classification being a first event, obtaining reliability information of a landmark point matching the first event, and obtaining occlusion information of the landmark point based on the reliability information; - The occlusion information is obtained based on first occlusion information obtained based on high reliability information of the landmark point matching the first event and low reliability information of the landmark point matching the first event. Contains second occlusion information that is - and
determining an event for the image as one of the first event or a second event associated with the first event by combining the event classification, reliability regarding the event classification, and occlusion information of the landmark; Including,
The determining step is,
When the reliability of the event for the image being classified as the first event by the first neural network model is greater than or equal to a predetermined value, second occlusion information of the landmark point matching the first event In further consideration, change the event for the image to the second event, or
When the reliability of being classified as the first event is less than a predetermined value, the event for the image is converted to the second event by further considering the first occlusion information of the landmark point matching the first event. It includes the step of changing to,
The second event is a new event classification that the first neural network model could not classify for the image. delete delete According to claim 1,
The reliability information of the landmark point is,
A method of inputting a related image related to the image as input data to a second neural network model, detecting a plurality of landmark points in the related image, and calculating reliability information for each of the plurality of landmark points. According to clause 4,
The reliability information of the landmark point is,
Reliability information of a target landmark point determined among the plurality of landmark points based on the event classification result. According to clause 5,
The target landmark point is,
The method includes dividing the plurality of landmark points into one or more areas, selecting at least one area from among the areas based on the event classification result, and determining the landmark point included in the selected area. According to clause 4,
The reliability information of the landmark point is,
A method of reliability information calculated through a regression process for each of the plurality of landmark points. According to clause 7,
The regression analysis process is,
Determine a plurality of coordinates for each of the plurality of landmark points, calculate a reliability for each of the plurality of coordinates, and assign the highest reliability among the reliability for each of the plurality of coordinates to each of the plurality of landmark points. A method that is a process of deciding with reliability information calculated for . According to clause 7,
The regression analysis process is,
Find a heat map distribution for each of the plurality of landmark points, calculate the coordinates at the center of the heat map distribution, and calculate the coordinates at the center according to the degree of spread of the heat map distribution based on the coordinates at the center. A method of determining reliability information calculated for each of the plurality of landmark points. In a device for determining image event classification using a neural network model,
a memory in which at least one program is stored; and
At least one processor that runs a neural network model by executing the at least one program,
The at least one processor,
Input an image as input data to a first neural network model, classify an event for the image and obtain reliability regarding the classification of the event as output data of the first neural network model,
In response to the obtained event classification being a first event, reliability information of the landmark point matching the first event is obtained, and occlusion information of the landmark point based on the reliability information - the occlusion information First occlusion information obtained based on high reliability information of the landmark point matching the first event and second occlusion obtained based on low reliability information of the landmark point matching the first event. Contains baggage information - Obtain,
Combining the event classification, the reliability of the event classification, and the occlusion information of the landmark to determine the event for the image as one of the first event or a second event associated with the first event,
The processor,
When the reliability of the event for the image being classified as the first event by the first neural network model is greater than or equal to a predetermined value, second occlusion information of the landmark point matching the first event In further consideration, change the event for the image to the second event, or
When the reliability of being classified as the first event is less than a predetermined value, the event for the image is converted to the second event by further considering the first occlusion information of the landmark point matching the first event. Including changing to,
The second event is a new event classification that the first neural network model was unable to classify for the image. A computer-readable recording medium that records a program for executing the method of claim 1 on a computer.

Images