KR102633937B1 - Electronic device and method for obtain attribute information related to object using the same - Google Patents

Abstract

According to various embodiments of the present invention, an electronic device includes a memory and a processor, wherein the processor performs at least one task related to color information among a plurality of tasks representing a plurality of attribute information related to an object using a multi-task model. Perform learning of one task, apply a first loss function to at least one task related to the color information, and make predictions based on the actual color information of the object and the learning based on predefined representative color information. Calculate a first loss value between the predicted color information of the object, calculate a loss weight based on the calculated first loss value and a specified weight value, and perform at least one task to which the first loss function is applied. By applying the calculated loss weight, the loss function value related to the color information can be adjusted.
Various other embodiments other than those disclosed in this document may be possible.

Description

Method of obtaining attribute information related to electronic devices and objects using them {ELECTRONIC DEVICE AND METHOD FOR OBTAIN ATTRIBUTE INFORMATION RELATED TO OBJECT USING THE SAME}

Embodiments of the present invention relate to a method of obtaining attribute information related to an electronic device and an object using the same.

Recently, with the increase in the use of image capture functions, various studies on image processing technology are being conducted. For example, image processing technology may refer to technology that analyzes images according to purpose.

As an example, an electronic device can process images acquired through a camera using an artificial intelligence model. The artificial intelligence model may include a learning model learned by, for example, machine learning. For example, machine learning can be a technology that analyzes collected big data to learn on its own and make decisions. The electronic device uses the above-described machine learning to detect an object, such as a means of transportation (e.g., a vehicle), in an image, and collects a plurality of attribute information related to the detected means of transportation, such as the type of the means of transportation and the type of the means of transportation. Attribute information such as color can be obtained.

The above information may be provided as background art for the purpose of facilitating understanding of the present invention. No claim or decision is made as to whether any of the foregoing can be applied as prior art to the present invention.

Korean Patent Publication No. 10-2023-0070224.

However, when obtaining attribute information such as color among a plurality of attribute information related to the detected means of transportation, only predefined color information, for example, predefined color information based on RGB values, can be obtained. there is.

When learning a task representing attribute information related to the color information of an object, for example, a means of transportation, the electronic device according to an embodiment of the present invention sets a loss weight based on the difference between the actual color of the means of transportation and the predicted color. can be calculated. The electronic device may adjust the loss function value related to color information using the calculated loss weight and learn a multi-task model for acquiring color information related to the means of transportation using the adjusted loss function value.

An electronic device according to an embodiment of the present invention includes a memory and a processor, wherein the processor performs at least one task related to color information among a plurality of tasks representing a plurality of attribute information related to an object using a multi-task model. Perform learning of a task, apply a first loss function to at least one task related to the color information, and based on predefined representative color information, determine the actual color information of the object and the color predicted by the learning. Calculating a first loss value between the predicted color information of an object, calculating a loss weight based on the calculated first loss value and a specified weight value, and calculating the first loss function for at least one task to which the first loss function is applied. By applying the loss weight, the loss function value related to the color information can be adjusted.

A method of obtaining attribute information related to an object of an electronic device according to an embodiment of the present invention includes at least one task related to color information among a plurality of tasks representing a plurality of attribute information related to the object using a multi-task model. An operation of performing learning, an operation of applying a first loss function to at least one task related to the color information, based on predefined representative color information, the actual color information of the object and the predicted by the learning An operation of calculating a first loss value between the predicted color information of the object, an operation of calculating a loss weight based on the calculated first loss value and a specified weight value, and at least one task to which the first loss function is applied It may include an operation of adjusting a loss function value related to the color information by applying the calculated loss weight.

The electronic device according to an embodiment of the present invention learns a multi-task model for obtaining color information of an object, for example, a means of transportation using a loss weight based on the difference between the actual color of the means of transportation and the predicted color. , color information close to the actual color of the means of transportation can be obtained. Accordingly, for example, when tracking a suspect's means of transportation in a crime prevention environment, color information close to the actual color of the means of transportation can be obtained by the above-described method, which not only shortens the tracking time when tracking the suspect. However, it may be possible to accurately track the suspect.

1 is a block diagram illustrating an electronic device according to an embodiment of the present invention.
Figure 2 is a flowchart illustrating a method of learning a multi-task model to obtain attribute information related to an object, according to an embodiment of the present invention.
Figure 3 is a diagram for explaining a method of learning color information among attribute information related to an object, according to an embodiment of the present invention.
Figure 4 is a diagram for explaining a method of augmenting learning data of a plurality of tasks according to an embodiment of the present invention.
Figure 5 is a flowchart illustrating a method of obtaining attribute information related to an object, according to an embodiment of the present invention.
Figure 6 is a diagram for explaining a method of obtaining attribute information related to an object according to an embodiment of the present invention.

Hereinafter, with reference to the drawings, embodiments of the present invention will be described in detail so that those skilled in the art can easily implement the present invention. However, the present invention may be implemented in many different forms and is not limited to the embodiments described herein. In relation to the description of the drawings, identical or similar reference numerals may be used for identical or similar components. Additionally, in the drawings and related descriptions, descriptions of well-known functions and configurations may be omitted for clarity and brevity.

Figure 1 is a block diagram showing an electronic device 101 according to an embodiment of the present invention.

Referring to FIG. 1 , the electronic device 101 may include a camera 110, a memory 120, and/or a processor 130. The components shown in FIG. 1 are some of the components included in the electronic device 101, and the electronic device 101 includes various components other than the components shown in FIG. 1 (e.g., communication circuits and/or or display) may be further included.

According to an embodiment of the present invention, the camera 110 may acquire captured images (eg, still images and/or moving images). Camera 110 may include one or more lenses, image sensors, image signal processors, or flashes. According to one embodiment, the camera 110 may include a speed dome camera (SPEED DOME camera) and at least one camera capable of photographing the entire azimuth area. However, it is not limited to this.

According to one embodiment of the present invention, the memory 120 functions to store programs for processing and control of the processor 130, an operating system (OS), various applications, and/or input/output data. A program that controls the overall operation of the electronic device 101 can be stored. Memory 120 may store various instructions that can be performed by processor 130.

In one embodiment, the memory 120 may store one or more instructions and/or a program that controls the processor 130 to perform task learning based on a multi-task model. The memory 120 may store instructions and/or a program for the processor 130 to obtain at least one attribute information related to at least one object from image frames acquired through the camera 110.

According to one embodiment of the present invention, the processor 130 may include a microcontroller unit (MCU), and runs an operating system (OS) or embedded software program to operate a plurality of devices connected to the processor 130. Hardware components can be controlled. The processor 130 may control multiple hardware components according to instructions stored in the memory 120, for example.

In various embodiments, processor 130 may include a neural network model. The processor 130 may include a hardware structure and/or a software structure specialized for processing artificial intelligence models. Artificial intelligence models can be created through machine learning. Machine learning may include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but the examples described above It is not limited to An artificial intelligence model may include multiple artificial neural network layers. Artificial neural networks include deep neural network (DNN), convolutional neural network (CNN), recurrent neural network (RNN), restricted boltzmann machine (RBM), belief deep network (DBN), bidirectional recurrent deep neural network (BRDNN), It may be one of deep Q-networks, or a combination of two or more of the above, but is not limited to the examples described above.

In one embodiment, the processor 130 executes one or more instructions stored in the memory 120, overall operations for performing task learning using a multi-task model, and/or the camera 110. ) can control overall operations for acquiring at least one attribute information related to at least one object from image frames obtained through ).

For example, the processor 130 may classify a plurality of tasks representing a plurality of attribute information related to an object based on characteristics. For example, an object may include a means of movement. However, it is not limited to this. In one embodiment, a plurality of attribute information related to an object, for example, a means of transportation, includes color information of the object (e.g., red (R) value, green (G, green) value, and blue (B) value), second to fourteenth attribute information related to the type of object (e.g., second attribute information related to a passenger car (e.g., sedan), third attribute information related to SUV, van ( For example: 4th attribute information related to a van, 5th attribute information related to a taxi, 6th attribute information related to a light vehicle, 7th attribute information related to a truck, 8th attribute information related to a bus, ambulance (e.g., ambulance) and 9th attribute information related, 10th attribute information related to fire trucks, 11th attribute information related to police cars, 12th attribute information related to bicycles, 13th attribute information related to motorcycles, 14th attribute information related to object non-detection), and It may include fifteenth attribute information related to whether an object is cut out of the image. The processor 130 may label (eg, map) a plurality of tasks representing a plurality of attribute information related to an object.

In one embodiment, the processor 130 may check the characteristics of each task representing each attribute information related to the object and classify the tasks based on this. For example, each task can have at least one output node. In this case, the characteristics of each task may include the number of output nodes each task has. For example, the first task among the plurality of tasks is a task for acquiring color information of an object (eg, means of transportation), for example, predefined representative color information, and may have nine output nodes. For example, predefined representative color information may include blue, green, black, silver, red, white, orange, yellow, and light pink. However, it is not limited to this. For another example, among the plurality of tasks, the second to fourteenth tasks are tasks for obtaining the type of an object (e.g., a means of transportation), and each of the first to fourteenth tasks has one output node. You can. For another example, the 15th task among the plurality of tasks is a task for checking whether an object (eg, a means of transportation) has been cut and may have one output node.

In one embodiment, a model for acquiring (or predicting) a plurality of attribute information (e.g., first to fifteenth attribute information) related to an object includes a sigmoid (or prediction) having an output value between 0 and 1. You can use the sigmoid) function.

In one embodiment, the processor 130 may learn at least one task having the first characteristic among a plurality of tasks using a first loss function and a second loss function. For example, the processor 130 may execute at least one task having a first characteristic, for example, at least one task having nine output nodes (e.g., a first task) using a first loss function and a second loss function. You can learn using . The first loss function may include a binary cross entropy (BCE) loss function. The second loss function may include a mean absolute error (MAE) loss function.

In one embodiment, the processor 130 may calculate a first loss value between the actual color information of the object and the predicted color information of the object predicted through learning, based on predefined representative color information. For example, predefined representative color information may be defined as color embedding values. The processor 130 may calculate a first loss value between the actual color information of the object and the predicted color information of the object predicted through learning, based on the defined color embedding values. The processor 130 may calculate a loss weight based on the calculated first loss value and the designated weight value. The processor 130 may calculate a loss weight by adding the calculated first loss value and the designated weight value. For example, the specified weight value may be 1.0. However, it is not limited to this.

In one embodiment, the processor 130 may adjust the first loss function value related to color information by applying the calculated loss weight to at least one task to which the first loss function is applied. The adjusted first loss function value may be used as a value calculated by learning at least one task having the first characteristic using the first loss function and the second loss function.

In one embodiment, the processor 130 may learn at least one other task having the second characteristic among the plurality of tasks using the first loss function. For example, the processor 130 may execute at least one other task having a second characteristic, for example, at least one task having one output node (e.g., second to fifteenth tasks) using the first loss function. You can learn using . The processor 130 may adjust the second loss function value based on the learning result. The adjusted second loss function value can be used to learn a multitask model.

Figure 2 is a flowchart illustrating a method of learning a multi-task model to obtain attribute information related to an object, according to an embodiment of the present invention.

In the following embodiments, each operation may be performed sequentially, but is not necessarily performed sequentially. For example, the order of each operation may be changed, and at least two operations may be performed in parallel.

According to one embodiment, operations 205 to 220 may be understood as being performed by a processor (e.g., processor 130 of FIG. 1) of an electronic device (e.g., electronic device 101 of FIG. 1).

In one embodiment, operations 205 to 220, which are not shown but will be described later, may be performed using images related to the object. For example, an image related to an object includes an image related to the object stored in memory (e.g., memory 120 in FIG. 1) and/or an image acquired through a camera (e.g., camera 110 in FIG. 1). can do.

Referring to FIG. 2, in operation 205, the processor 130 may classify a plurality of tasks representing a plurality of attribute information related to an object based on characteristics.

In one embodiment, the object may include a means of movement. However, it is not limited to this. In the following embodiments, the object will be described assuming that it is a means of movement.

In one embodiment, a plurality of attribute information related to an object, for example, a means of transportation, includes color information of the object (e.g., red (R) value, green (G, green) value, and blue (B) value), second to fourteenth attribute information related to the type of object (e.g., second attribute information related to a passenger car (e.g., sedan), third attribute information related to SUV, van ( For example: 4th attribute information related to a van, 5th attribute information related to a taxi, 6th attribute information related to a light vehicle, 7th attribute information related to a truck, 8th attribute information related to a bus, ambulance (e.g. an ambulance) and 9th attribute information related, 10th attribute information related to fire trucks, 11th attribute information related to police cars, 12th attribute information related to bicycles, 13th attribute information related to motorcycles, 14th attribute information related to object non-detection), and It may include a 15th attribute information related to whether an object is cut out of the image.

In various embodiments, the plurality of attribute information related to an object has been described as including 15 pieces of attribute information, but is not limited thereto. For example, the plurality of attribute information related to an object may include less than 15 pieces of attribute information or more than 15 pieces of attribute information.

In one embodiment, the processor 130 may label (eg, map) a plurality of tasks representing a plurality of attribute information related to an object. For example, the processor 130 may label the first task representing the first attribute information with a first label. The processor 130 may label the second task representing the second attribute information with a second label. The processor 130 may label the third task representing the third attribute information with a third label. The processor 130 may label the fourth task representing the fourth attribute information with a fourth label. The processor 130 may label the fifth task representing the fifth attribute information with a fifth label. The processor 130 may label the sixth task representing the sixth attribute information with a sixth label. The processor 130 may label the seventh task representing the seventh attribute information with a seventh label. The processor 130 may label the eighth task representing the eighth attribute information with an eighth label. The processor 130 may label the ninth task representing the ninth attribute information with a ninth label. The processor 130 may label the tenth task representing the tenth attribute information with the tenth label. The processor 130 may label the 11th task representing the 11th attribute information with an 11th label. The processor 130 may label the twelfth task representing the twelfth attribute information with a twelfth label. The processor 130 may label the 13th task representing the 13th attribute information with a 13th label. The processor 130 may label the 14th task representing the 14th attribute information with a 14th label. The processor 130 may label the 15th task with a 15th label.

In one embodiment, the processor 130 may check the characteristics of each task representing each attribute information related to the object and classify the tasks based on this. For example, each task can have at least one output node. In this case, the characteristics of each task may include the number of output nodes each task has. However, it is not limited to this.

In one embodiment, the first task among the plurality of tasks includes color information of an object (e.g., a means of transportation), such as predefined representative color information, such as blue, green, black, silver, red, This task is for obtaining white, orange, yellow, and light pink colors and can have 9 output nodes. The previously defined representative color information described above is an example and is not limited thereto.

In one embodiment, the second to fourteenth tasks among the plurality of tasks are tasks for obtaining the type of an object (e.g., a means of transportation), and each of the first to fourteenth tasks has one output node. You can.

In one embodiment, the 15th task among the plurality of tasks is a task for checking whether an object (eg, a means of transportation) is cut, and may have one output node.

Accordingly, the number of output nodes of the final output layer of the model for acquiring (or predicting) a plurality of attribute information (e.g., first to fifteenth attribute information) related to an object (e.g., means of transportation) is 24. It could be a dog.

In one embodiment, a model for acquiring (or predicting) a plurality of attribute information (e.g., first to fifteenth attribute information) related to an object includes a sigmoid (or prediction) having an output value between 0 and 1. You can use the sigmoid) function.

In one embodiment, the processor 130 may learn at least one task having the first characteristic among a plurality of tasks using a first loss function and a second loss function in operation 210. For example, the processor 130 may execute at least one task having a first characteristic, for example, at least one task having nine output nodes (e.g., a first task) using a first loss function and a second loss function. You can learn using .

In one embodiment, the first loss function may include a binary cross entropy (BCE) loss function. The second loss function may include a mean absolute error (MAE) loss function. For example, the processor 130 may apply a binary cross-entropy and an average absolute error loss function (e.g., 9 output nodes) to 9 output nodes associated with a first task having a first characteristic (e.g., 9 output nodes) among the plurality of tasks. ) can be applied.

In one embodiment, the processor 130 may learn at least one other task having the second characteristic among the plurality of tasks using the first loss function in operation 215. For example, the processor 130 may execute at least one other task having a second characteristic, for example, at least one task having one output node (e.g., second to fifteenth tasks) using the first loss function. You can learn using .

In one embodiment, the first loss function may include a binary cross entropy (BCE) loss function. For example, the processor 130 applies a binary cross-entropy loss function (e.g., A binary cross-entropy loss function applied to the output nodes associated with each of the second to fourteenth tasks (e.g. ), a binary cross-entropy loss function applied to the output nodes associated with the 15th task, e.g. ) can be applied.

The binary cross entropy loss function described above as the first loss function and the average absolute error loss function described as the second loss function according to various embodiments are an example and are not limited thereto.

In one embodiment, the final loss function obtained by applying the loss function to 24 output nodes related to the plurality of tasks described above may be expressed as Equation 1 below. For example, the processor 130 may apply different loss functions (e.g., binary cross-entropy loss function and/or mean absolute error loss function) to the output nodes (e.g., 24 output nodes) of each task depending on the characteristics of each task. ) can be applied and the final loss function can be determined by summing each output node to which the loss function is applied.

In one embodiment, the processor 130 may adjust the loss function value based on the learning result in operation 220. For example, the processor 130 may calculate the loss function value (Loss) based on the above-described Equation 1. The calculated loss function value (Loss) can be adjusted (e.g., adjusted to an optimal loss function value) by repeatedly performing operations 210 and 215 described above. Although not shown, the adjusted loss function value can be used to learn a multi-task model. For example, the processor 130 may learn a multi-task model by applying a loss function value adjusted through back-propagation to the multi-task model.

Figure 3 is a diagram for explaining a method of learning color information among attribute information related to an object, according to an embodiment of the present invention.

FIG. 3 according to one embodiment is a diagram specifying operation 210 of FIG. 2.

In the following embodiments, each operation may be performed sequentially, but is not necessarily performed sequentially. For example, the order of each operation may be changed, and at least two operations may be performed in parallel.

According to one embodiment, operations 305 to 325 may be understood as being performed by a processor (e.g., processor 130 of FIG. 1) of an electronic device (e.g., electronic device 101 of FIG. 1).

In one embodiment, operations 305 to 325, which are not shown but will be described later, may be performed using images related to the object. For example, an image related to an object may be an image related to the object stored in memory (e.g., memory 120 in FIG. 1) and/or an image related to the object acquired through a camera (e.g., camera 110 in FIG. 1). Can include images. In one embodiment, the object may include a means of movement. However, it is not limited to this.

Referring to FIG. 3, in operation 305, the processor 130 may perform learning of at least one task related to color information among a plurality of tasks representing a plurality of attribute information related to an object using a multi-task model. . For example, as seen in FIG. 2, at least one task related to color information includes predefined representative color information, for example, blue, green, black, silver ( You can have 9 output nodes for silver, red, white, orange, yellow, and light pink. Accordingly, at least one task related to color information may be a task having the first characteristic.

In one embodiment, the processor 130 may apply the first loss function to at least one task related to color information in operation 310. For example, the first loss function may include a binary cross entropy (BCE) loss function.

In one embodiment, in operation 315, the processor 130 determines the difference between the actual color information of the object and the predicted color information of the object predicted by learning (e.g., learning in operation 305), based on predefined representative color information. The first loss value can be calculated.

In one embodiment, predefined representative color information (e.g., blue, green, black, silver, red, white, orange) , yellow, and light pink) can be defined by color embedding values, as shown in Table 1 below.

EMB[9][3] = {
{0, 0, 255}, // blue
{20, 180, 60}, // green
{0, 0, 0}, // black
{130, 130, 130}, // silver
{255, 0, 0}, // red
{255, 255, 255}, // white
{255, 100, 0}, // orange
{255, 255, 0}, // yellow
{255, 224, 239} // light pink
}

In one embodiment, as shown in <Table 1>, defining predefined representative color information as color embedding values uses a sigmoid function with the distance between colors (e.g., difference between colors) as a weight. This may be to evenly distribute the confidence of similar color information and prevent overconfidence.

In one embodiment, the first loss value between the calculated actual color information of the object and the predicted color information of the object predicted by learning may be as shown in <Table 2> below.

M.A.E. BLUE GREEN BLACK GRAY RED WHITE ORANGE YELLOW PINK BLUE 0 510 255 385 510 510 610 765 296 GREEN 510 0 255 515 510 510 410 255 270 BLACK 255 255 0 375 255 765 355 510 718 GRAY 385 385 375 0 385 375 285 380 328 RED 510 510 255 385 0 510 100 255 463 WHITE 510 510 765 375 510 0 410 255 47 ORANGE 610 410 355 285 100 410 0 155 363 YELLOW 765 255 510 380 255 255 155 0 270 PINK 296 270 718 328 463 47 363 270 0

In one embodiment, the processor 130 may calculate a loss weight based on the calculated first loss value and the designated weight value in operation 320. For example, the processor 130 may calculate the loss weight by adding the calculated first loss value and the designated weight value. For example, the specified weight value may be 1.0. However, it is not limited to this.

In one embodiment, the processor 130 may adjust the loss function value related to color information by applying the calculated loss weight to at least one task to which the first loss function is applied in operation 325. For example, the processor 130 may calculate a loss function value (Loss) for color information (eg, first attribute information) related to an object based on Equation 2 below. The loss function value (Loss) can be adjusted (e.g., adjusted to an optimal loss function value) by repeatedly performing operations 305 and 325 described above. In one embodiment, the adjusted loss function value may be used as a value calculated by learning using the first loss function and the second loss function in operation 210 of FIG. 2 described above.

In various embodiments, by applying the loss weight calculated by operation 320 to the first loss function to learn (or train) color information related to the object, rather than learning color information using only the first loss function, Learning about color information can be done flexibly. For example, when the model learns color information using only the first loss function, the loss function when predicting an object with a white color for an object whose actual color is black and when predicting an object with a gray color The values may be the same. In embodiments of the present invention, when color information is learned by applying the loss weight calculated by operation 320 to the first loss function, when the model predicts an object whose actual color is black as an object whose color is white The loss function value of may be higher than the loss function value when predicted as an object with a gray color. In other words, if a gray object is predicted, which is a color close to the actual color of black, a lower loss function value may be produced than if a white object is predicted. Accordingly, a color close to the actual color of the object may be calculated as the actual color of the object. It is possible to obtain relatively large model confidence compared to colors that are distant from the color.

Figure 4 is a diagram for explaining a method of augmenting learning data of a plurality of tasks according to an embodiment of the present invention.

The operation of FIG. 4 described later according to various embodiments may be performed while learning each task of operation 210 and operation 220 of FIG. 2, and/or after adjusting the loss function value of operation 220. You can. However, it is not limited to this.

Referring to FIG. 4, a processor (e.g., processor 130 of FIG. 1) of an electronic device (e.g., electronic device 101 of FIG. 1) changes the first object image (e.g., an object image without transformation). Learning data can be augmented by generating a second object image (e.g., a transformed object image).

In one embodiment, the processor 130 may learn the characteristics of the camera that acquires the first object image, the shooting time, and variables that may occur at the time of shooting. For example, the processor 130 may increase the quantity of learning data through a data augmentation algorithm that reflects variables that may actually occur with respect to the first object image. For example, the processor 130 may reflect variables that may occur at the time of shooting in the first object image. Variables that may occur at the time of shooting according to one embodiment may be variables related to objects that may obscure the object, such as other cars, street lights, signboards, and/or electric power lines. Depending on the variable, at least one attribute information related to the object may be incorrectly obtained. For example, variables may cause the user to focus on the color of the surrounding environment rather than the color related to the object, and accordingly, the 15th attribute information related to the color value among the plurality of attribute information related to the object may be incorrectly obtained.

In one embodiment of the present invention, the processor 130 processes the first object images 411, 431, and 451 (e.g., untransformed images) as shown in reference numerals <410>, <430>, and <450>. A transformation (413, 415, 433, 435, 453, 455) such as drawing a line with random color and thickness can be applied to the object image, and the transformed image can be generated as a second object image. The processor 130 may learn each attribute information related to the object based on the generated second object image.

4 according to an embodiment, by using data augmentation to increase the quantity of learning data through a data augmentation algorithm reflecting the variable, and learning a task related to the fifteenth attribute information, color information (e.g., color information related to the object) 15 attribute information) can be obtained accurately.

Figure 5 is a flowchart illustrating a method of obtaining attribute information related to an object, according to an embodiment of the present invention.

In the following embodiments, each operation may be performed sequentially, but is not necessarily performed sequentially. For example, the order of each operation may be changed, and at least two operations may be performed in parallel.

According to one embodiment, operations 510 and 520 may be understood as being performed by a processor (e.g., processor 130 of FIG. 1) of an electronic device (e.g., electronic device 101 of FIG. 1).

Referring to FIG. 5, the processor 130 may detect at least one object from image frames acquired through a camera (eg, camera 110 of FIG. 1) in operation 510.

In one embodiment, at least one object may include at least one means of movement. In one embodiment, the processor 130 may crop an area containing at least one object detected in image frames. The processor 130 may generate an area including at least one cropped object as one image and perform operation 520, which will be described later. However, it is not limited to this.

In one embodiment, the processor 130 may obtain at least one attribute information related to each of the at least one detected object based on the multitask model in operation 520.

In one embodiment, the processor 130 acquires at least one attribute information related to each of at least one object included in the image using a multi-task model learned by the operations of FIG. 2 and/or FIG. 3 described above. can do. For example, the at least one attribute information related to the object may include first information related to color information of the object (e.g., a red (R, red) value, a green (G, green) value, and a blue (B, blue) value). Attribute information, second to fourteenth attribute information related to the type of object (e.g., second attribute information related to a passenger car (e.g., sedan), third attribute information related to an SUV, and related to a passenger vehicle (e.g., van) Fourth attribute information, fifth attribute information related to taxis, sixth attribute information related to light vehicles, seventh attribute information related to trucks, eighth attribute information related to buses, ninth attribute information related to ambulances (e.g. ambulances), 10th attribute information related to fire trucks, 11th attribute information related to police cars, 12th attribute information related to bicycles, 13th attribute information related to motorcycles, 14th attribute information related to object non-detection), and whether an object is cut out in the image. It may include 15th attribute information related to whether or not.

Figure 6 is a diagram for explaining a method of obtaining attribute information related to an object according to an embodiment of the present invention.

Referring to FIG. 6, the processor (e.g., processor 130 of FIG. 1) of an electronic device (e.g., electronic device 101 of FIG. 1) transmits at least one signal through a camera (e.g., camera 110 of FIG. 1). Image frames containing one means of movement can be acquired. The processor 130 crops the area containing at least one object detected in the image frames, and converts the area containing the cropped at least one object into one image (e.g., the first image of reference numeral <610>). 611), the second image 631 with reference number <630>, and the third image 651 with reference number <650>).

In one embodiment, the processor 130 may obtain at least one attribute information related to each of the at least one detected object based on the multitask model.

In one embodiment, looking at the color information of the object, the processor 130 calculates the color value calculated by sigmoid based on color embedding values for predefined representative colors according to <Table 1>. Final color information can be obtained by using the confidence of each representative color as a weight. For example, if the confidence level for green is 0.7, the confidence level for black is 0.3, and the confidence level for the remaining colors is close to 0, the processor 130 generates “{0, 1.0, 0} * 0.7 + {0.0 “{0.2, 1.0, 0.2}” calculated based on “, 0.0, 0.0} * 0.3” can be obtained as the final color information.

For example, referring to reference numeral <610>, the processor 130 determines the type of vehicle, for example, SUV (e.g., third attribute information) from the first image 611, based on the multi-task model. 613) and color information (e.g., first attribute information) 615 (e.g., reliability for black: 0.20, reliability for gray: 0.8) can be obtained.

For another example, referring to reference numeral <630>, the processor 130 selects a type of transportation vehicle (e.g., a sedan) from the second image 631 (e.g., a second image) based on a multi-task model. Attribute information) 633 and color information, light gray (e.g., first attribute information) 635 (e.g., reliability for white: 0.20, reliability for gray: 0.8) can be obtained.

As another example, referring to reference numeral <650>, the processor 130 determines a type of transportation vehicle (e.g., seventh attribute information) 653 from the third image 651, based on the multi-task model. ) and color information of bright yellow (e.g., first attribute information) 655 (e.g., reliability for white: 0.3, reliability for yellow: 0.7) can be obtained.

The various embodiments of this document and the terms used herein are not intended to limit the technical features described in this document to specific embodiments, and should be understood to include various changes, equivalents, or replacements of the embodiments. In connection with the description of the drawings, similar reference numbers may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the above items, unless the relevant context clearly indicates otherwise. As used herein, “A or B”, “at least one of A and B”, “at least one of A or B”, “A, B or C”, “at least one of A, B and C”, and “A Each of phrases such as “at least one of , B, or C” may include any one of the items listed together in the corresponding phrase, or any possible combination thereof. Terms such as "first", "second", or "first" or "second" may be used simply to distinguish one element from another, and may be used to distinguish such elements in other respects, such as importance or order) is not limited.

Various embodiments of this document may be implemented as software including one or more instructions stored in a storage medium (e.g., memory 120) that can be read by a machine (e.g., electronic device 101). You can. For example, a processor (e.g., processor 130) of a device (e.g., electronic device 101) may call at least one command among one or more commands stored from a storage medium and execute it. This allows the device to be operated to perform at least one function according to the at least one instruction called. The one or more instructions may include code generated by a compiler or code that can be executed by an interpreter. A storage medium that can be read by a device may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' only means that the storage medium is a tangible device and does not contain signals (e.g. electromagnetic waves), and this term refers to cases where data is semi-permanently stored in the storage medium. There is no distinction between temporary storage cases.

According to one embodiment, methods according to various embodiments disclosed in this document may be provided and included in a computer program product. Computer program products are commodities and can be traded between sellers and buyers. The computer program product may be distributed in the form of a machine-readable storage medium (e.g. compact disc read only memory (CD-ROM)) or via an application store (e.g. Play Store ^TM ) or on two user devices (e.g. It can be distributed (e.g. downloaded or uploaded) directly between smart phones) or online. In the case of online distribution, at least a portion of the computer program product may be at least temporarily stored or temporarily created in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

According to various embodiments, each component (e.g., module or program) of the above-described components may include a single or plural entity, and some of the plurality of entities may be separately placed in other components. there is. According to various embodiments, one or more of the components or operations described above may be omitted, or one or more other components or operations may be added. Alternatively or additionally, multiple components (eg, modules or programs) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each component of the plurality of components in the same or similar manner as those performed by the corresponding component of the plurality of components prior to the integration. . According to various embodiments, operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, or omitted. Alternatively, one or more other operations may be added.

101: Electronic device 110: Camera
120: memory 130: processor

Claims (20)

In electronic devices,
Memory; and
Includes a processor,
The processor,
Perform learning of at least one task related to color information among a plurality of tasks representing a plurality of attribute information related to an object using a multi-task model,
Applying a binary cross entropy (BCE) loss function to at least one task related to the color information for which the learning was performed,
Based on the color embedding values for the predefined representative color information, a color embedding value corresponding to the actual color information of the object among the predefined representative color information and predicted color information of the object predicted by the learning. A mean absolute error (MAE) loss function that calculates a first loss value related to the difference between the corresponding color embedding values, and calculates a loss weight related to the color information by applying a specified weight value to the calculated first loss value. Apply ,
Generating a final loss function related to the color information based on a second loss value related to the color information calculated by applying the BCE loss function and a loss weight related to the color information calculated by applying the MAE loss function; , and
An electronic device that applies the generated final loss function to the multi-task model through back-propagation. delete According to claim 1,
The electronic device where the specified weight value is 1.0. delete According to claim 1,
The processor,
Learning at least one task other than the task related to color information among the plurality of tasks using the BCE loss function, and
An electronic device that generates a third loss function for the plurality of tasks based on the generated final loss function and at least one other task learned using the BCE loss function. delete delete According to claim 5,
The processor,
Classifying a plurality of tasks representing a plurality of attribute information related to the object based on characteristics,
Each of the plurality of tasks includes at least one output node, and
The characteristic includes the number of output nodes included in each of the plurality of tasks. According to claim 5,
The processor,
An electronic device that learns the plurality of tasks by applying the third loss function generated through the back-propagation to the multi-task model. According to clause 9,
Contains more cameras,
The processor,
detecting at least one object from image frames obtained through the camera, and
An electronic device that obtains at least one attribute information related to each of the at least one detected object using the multi-task model. In a method of obtaining attribute information related to an object of an electronic device,
An operation of performing learning of a task related to color information among a plurality of tasks representing a plurality of attribute information related to an object using a multi-task model;
An operation of applying a binary cross entropy (BCE) loss function to a task related to the learned color information;
Based on color embedding values for predefined representative color information, a color embedding value corresponding to actual color information of the object among the predefined representative color information and predicted color information of the object predicted by the learning. A mean absolute error (MAE) loss function that calculates a first loss value related to the difference between the corresponding color embedding values, and calculates a loss weight related to the color information by applying a specified weight value to the calculated first loss value. The action of applying;
Based on the second loss value related to the color information calculated by applying the BCE loss function and the loss weight related to the color information calculated by applying the MAE loss function, a final loss function value related to the color information is generated. action; and
A method comprising applying the generated final loss function to the multi-task model through back-propagation. delete According to claim 11,
The method where the specified weight value is 1.0. delete According to claim 11,
Learning at least one task other than the task related to color information among the plurality of tasks using the BCE loss function; and
The method further includes generating a third loss function for the plurality of tasks based on the generated final loss function and at least one other task learned using the BCE loss function. delete delete According to claim 15,
Further comprising classifying a plurality of tasks representing a plurality of attribute information related to the object based on characteristics,
Each of the plurality of tasks includes at least one output node, and
The characteristic includes the number of output nodes included in each of the plurality of tasks. According to claim 15,
The method further includes a method of learning the plurality of tasks by applying the generated third loss function to the multi-task model through the back-propagation. According to claim 19,
An operation of detecting at least one object from image frames obtained through a camera; and
The method further includes obtaining at least one attribute information related to each of the at least one detected object using the multi-task model.

Images