KR20240028909A - Method of creating training data for neural network model - Google Patents

Abstract

The present disclosure is a method of generating training data for a neural network model, which is performed by at least one computing device. The method includes acquiring a plurality of images in which an object is photographed in a situation in which a first environment and a second environment are repeatedly switched. generating training data for the neural network model based on a first image in which the object is captured in the first environment, and based on a second image in which the object is captured in the second environment, It may include the step of generating correct answer data for learning a neural network model.

Description

Method for generating training data for neural network model {METHOD OF CREATING TRAINING DATA FOR NEURAL NETWORK MODEL}

This disclosure relates to a method for generating training data for a neural network model. More specifically, it relates to a method for generating training data and correct answer data from an environment that is repeatedly switched for learning a neural network model.

Today, when learning a neural network model, using high-quality learning data is an important factor in improving the performance of the neural network model, so securing high-quality learning data has emerged as an important task when creating a neural network model. It is becoming.

However, while there are areas where it is easy to secure learning data (for example, when existing accumulated data is sufficient, where public data is sufficient, and where data is easy to generate), there are also cases where it is relatively difficult to secure learning data. do. For example, there may be cases where existing accumulated data is insufficient, there may be insufficient public data, the cost of data generation may be high, or the speed of data generation may be slow.

In particular, in the fields of video production, virtual reality, and augmented reality, in order to create and learn a neural network model that classifies objects (objects or people) and backgrounds, training data, which is images with various backgrounds and various objects, and background Correct answer data with the area of the object and object indicated separately is required. At this time, when securing learning data using conventional methods, copyright and personal information issues and cost issues due to manpower mobilization act as constraints, and even when generating correct answer data, people must determine the background and object areas for each image. Since each data must be displayed, there is a risk that defective data may be generated due to human error.

For example, augmented reality graphics used in broadcasting are implemented by synthesizing graphics generated by a computing device on top of captured images. At this time, it is necessary to separate the objects included in the captured image from the background in order to synthesize a natural sense of space. However, in special situations such as movie or broadcast production where the background is a display device, the accuracy of the conventional method is not high. There is.

Therefore, in recent field work, there is a need for a method that improves the costs and risks mentioned above compared to the conventional method in relation to generating learning data and answer data for learning neural network models.

Meanwhile, the present disclosure has been derived based at least on the technical background examined above, but the technical problem or purpose of the present disclosure is not limited to solving the problems or shortcomings examined above. In other words, in addition to the technical issues discussed above, the present disclosure can cover various technical issues related to the content to be described below.

The present disclosure aims to solve the problem of generating learning data and correct answer data to learn a neural network model. For example, the present disclosure aims to generate learning data and correct answer data for training a neural network model based on environments that are repeatedly switched.

Meanwhile, the technical problem to be achieved by the present disclosure is not limited to the technical problems mentioned above, and may include various technical problems within the scope of what is apparent to those skilled in the art from the contents described below.

A method for generating training data for a neural network model, which is performed by at least one computing device to solve the above-described problem, is disclosed. The method includes: an object in a situation where a first environment and a second environment are repeatedly switched Obtaining a plurality of images in which the object is photographed, generating training data for the neural network model based on the first image in which the object is photographed in the first environment, and generating training data for the neural network model in which the object is photographed in the second environment Based on the second image, it may include generating ground truth for training the neural network model.

Alternatively, the neural network model may be a model for segmenting objects and backgrounds included in input data.

Alternatively, the first environment includes an environment in which a background image is displayed, the second environment includes an environment in which a chromakey image is displayed, and the plurality of images include the background image and the chromakey image. It may include a plurality of images in which the object is captured in a situation where the marquee image is repeatedly switched.

Alternatively, the background image may include at least one of a playing image or a still image, and the chromakey image may include at least one of a continuous pattern or a solid plain pattern.

Alternatively, the background image and the chroma key image may be repeatedly switched and displayed by a display device, and the plurality of images may be generated by the display device and a photographing device that photographs the object.

Alternatively, the display device and the photographing device may be set to have the same number of frames per second (fps).

Alternatively, the imaging device may include a depth sensor that measures depth data related to the perspective of the object, and the plurality of images may include depth data related to the perspective of the object.

Alternatively, the second image may be captured in the frame immediately before or after the frame in which the first image was captured.

Alternatively, the plurality of images may include a plurality of images in which two or more objects are captured simultaneously, and the two or more objects may include at least one of a moving object or a motionless object.

Alternatively, the two or more objects may include at least one of an object or a person.

In addition, in the method of classifying objects and backgrounds included in image data based on a neural network model, which is performed by at least one computing device for solving the above-described problem, the method includes: Inputting data into a neural network model and classifying objects and backgrounds based on the image data, wherein the neural network model includes a plurality of images in which the learning object is photographed in a situation where the first environment and the second environment are repeatedly switched. corresponds to a model learned based on the images, and the image in which the learning object is captured in the first environment corresponds to learning data for learning the neural network model, and the image in which the learning object is captured in the second environment The image may correspond to the correct answer data for learning the neural network model.

In addition, an apparatus for solving the above-described problem includes at least one processor and a memory, wherein the processor acquires a plurality of images in which an object is photographed in a situation in which a first environment and a second environment are repeatedly switched, Based on the first image in which the object is photographed in the first environment, training data for the neural network model is generated, and based on the second image in which the object is photographed in the second environment, the neural network model Correct answer data for learning can be generated.

A computer program stored in a computer-readable storage medium for solving the above-described problem, wherein the program causes a processor included in the computing device to perform operations for generating training data for a neural network model, and the operations include: a first environment and An operation of acquiring a plurality of images in which an object is photographed in a situation in which a second environment is repeatedly switched; an operation of generating training data for the neural network model based on a first image in which the object is photographed in the first environment; and It may include generating correct answer data for learning the neural network model based on a second image in which the object is captured in a second environment.

The present disclosure can generate learning data for a neural network model that separates objects and models without human manual work. For example, the present disclosure utilizes a plurality of images captured based on repeatedly switching environments to provide “learning data including objects and backgrounds” and “answer data separating objects and backgrounds” without human manual work. "can be created.

Meanwhile, the effects of the present disclosure are not limited to the effects mentioned above, and various effects may be included within the range apparent to those skilled in the art from the contents described below.

1 is a block diagram of a computing device performing operations according to an embodiment of the present disclosure.
Figure 2 is a schematic diagram showing a neural network model, according to an embodiment of the present disclosure.
Figure 3 is a flowchart showing a method of generating training data and correct answer data for learning a neural network model, according to an embodiment of the present disclosure.
Figure 4 is a schematic diagram showing a method for generating learning data and correct answer data based on a first environment and a second environment, according to an embodiment of the present disclosure.
Figure 5 is a schematic diagram showing a method of generating learning data and correct answer data based on a display device, according to an embodiment of the present disclosure.
FIG. 6 is a schematic diagram illustrating a neural network model for separating an object and a background from an image, according to an embodiment of the present disclosure.
7 is a brief, general schematic diagram of an example computing environment in which embodiments of the present disclosure may be implemented.

Various embodiments are now described with reference to the drawings. In this disclosure, various descriptions are presented to provide an understanding of the disclosure. However, it is clear that these embodiments may be practiced without these specific descriptions.

As used in this disclosure, the terms “component,” “module,” “system,” and the like refer to a computer-related entity, hardware, firmware, software, a combination of software and hardware, or an implementation of software. For example, a component may be, but is not limited to, a process running on a processor, a processor, an object, a thread of execution, a program, and/or a computer. For example, both an application running on a computing device and the computing device can be a component. One or more components may reside within a processor and/or thread of execution. A component may be localized within one computer. A component may be distributed between two or more computers. Additionally, these components can execute from various computer-readable media having various data structures stored thereon. Components can transmit signals, for example, with one or more data packets (e.g., data and/or signals from one component interacting with other components in a local system, a distributed system, to other systems and over a network such as the Internet). Depending on the data being transmitted, they may communicate through local and/or remote processes.

Additionally, the term “or” is intended to mean an inclusive “or” and not an exclusive “or.” That is, unless otherwise specified or clear from context, “X utilizes A or B” is intended to mean one of the natural implicit substitutions. That is, either X uses A; X uses B; Or, if X uses both A and B, “X uses A or B” can apply to either of these cases. Additionally, the term “and/or” as used in this disclosure should be understood to refer to and include all possible combinations of one or more of the related listed items.

Additionally, the terms “comprise” and/or “comprising” should be understood to mean that the corresponding feature and/or element is present. However, the terms “comprise” and/or “comprising” should be understood as not excluding the presence or addition of one or more other features, elements and/or groups thereof. Additionally, unless otherwise specified or the context is clear to indicate a singular form, the singular forms in this disclosure and claims should generally be construed to mean “one or more.”

And, the term “at least one of A or B” should be interpreted to mean “when it contains only A,” when it contains only B, and “when it is a combination of A and B.”

Those skilled in the art will additionally understand that the various illustrative logical blocks, configurations, modules, circuits, means, logics, and algorithm steps described in connection with the embodiments disclosed herein may be implemented in electronic hardware, computer software, or both. It should be recognized that it can be implemented with combinations of. To clearly illustrate the interchangeability of hardware and software, various illustrative components, blocks, configurations, means, logics, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented in hardware or software will depend on the specific application and design constraints imposed on the overall system. A skilled technician can implement the described functionality in a variety of ways for each specific application. However, such implementation decisions should not be construed as causing a departure from the scope of the present disclosure.

The description of the presented embodiments is provided to enable anyone skilled in the art to use or practice the present disclosure. Various modifications to these embodiments will be apparent to those skilled in the art. The general principles defined herein may be applied to other embodiments without departing from the scope of the disclosure. Therefore, the present disclosure is not limited to the embodiments presented herein. This disclosure is to be interpreted in the broadest scope consistent with the principles and novel features presented herein.

The configuration of the computing device 100 shown in FIG. 1 is only a simplified example. In one embodiment of the present disclosure, the computing device 100 may include different configurations for performing the computing environment of the computing device 100, and only some of the disclosed configurations may configure the computing device 100.

The computing device 100 may include a processor 110, a memory 130, and a network unit 150.

The processor 110 may be composed of one or more cores, and may include a central processing unit (CPU), a general purpose graphics processing unit (GPGPU), and a tensor processing unit (TPU) of a computing device. unit) may include a processor for data analysis and deep learning. The processor 110 may read a computer program stored in the memory 130 and perform data processing for machine learning according to an embodiment of the present disclosure. According to an embodiment of the present disclosure, the processor 110 may perform an operation for learning a neural network model. The processor 110 processes input data for learning in deep learning (DL), extracts features from input data, calculates errors, and learns neural network models such as updating the weights of the neural network model using backpropagation. You can perform calculations for . At least one of the CPU, GPGPU, and TPU of the processor 110 may process learning of the neural network model. For example, the CPU and GPGPU can work together to process neural network model learning and data classification using the neural network model. Additionally, in one embodiment of the present disclosure, the processors of a plurality of computing devices can be used together to process learning of a neural network model and data classification using the neural network model. Additionally, a computer program executed in a computing device according to an embodiment of the present disclosure may be a CPU, GPGPU, or TPU executable program.

According to an embodiment of the present disclosure, the memory 130 may store any type of information generated or determined by the processor 110 and any type of information received by the network unit 150.

According to an embodiment of the present disclosure, the memory 130 is a flash memory type, hard disk type, multimedia card micro type, or card type memory (e.g. (e.g. SD or -Only Memory), and may include at least one type of storage medium among magnetic memory, magnetic disk, and optical disk. The computing device 100 may operate in connection with web storage that performs a storage function of the memory 130 on the Internet. The description of the memory described above is merely an example, and the present disclosure is not limited thereto.

The network unit 150 according to an embodiment of the present disclosure includes Public Switched Telephone Network (PSTN), x Digital Subscriber Line (xDSL), Rate Adaptive DSL (RADSL), Multi Rate DSL (MDSL), and VDSL ( A variety of wired communication systems can be used, such as Very High Speed DSL), Universal Asymmetric DSL (UADSL), High Bit Rate DSL (HDSL), and Local Area Network (LAN).

In addition, the network unit 150 presented in the present disclosure includes Code Division Multi Access (CDMA), Time Division Multi Access (TDMA), Frequency Division Multi Access (FDMA), Orthogonal Frequency Division Multi Access (OFDMA), and SC-FDMA ( A variety of wireless communication systems can be used, such as Single Carrier-FDMA) and other systems.

In the present disclosure, the network unit 150 may be configured regardless of the communication mode, such as wired or wireless, and may be composed of various communication networks such as a personal area network (PAN) and a wide area network (WAN). You can. Additionally, the network may be the well-known World Wide Web (WWW), or may use wireless transmission technology used for short-distance communication, such as Infrared Data Association (IrDA) or Bluetooth. The techniques described in this disclosure can also be used in other networks mentioned above.

Figure 2 is a schematic diagram showing a neural network according to an embodiment of the present disclosure.

Throughout this disclosure, neural network and neural network may be used interchangeably. A neural network can generally consist of a set of interconnected computational units, which can be referred to as nodes. These nodes may also be referred to as neurons. A neural network consists of at least one node. Nodes (or neurons) that make up neural networks may be interconnected by one or more links.

Within a neural network, one or more nodes connected through a link can form a relative input node and output node relationship. The concepts of input node and output node are relative, and any node in an output node relationship with one node may be in an input node relationship with another node, and vice versa. As described above, input node to output node relationships can be created around links. One or more output nodes can be connected to one input node through a link, and vice versa.

In a relationship between an input node and an output node connected through one link, the value of the data of the output node may be determined based on the data input to the input node. Here, the link connecting the input node and the output node may have a weight (here, the terms parameter and weight may be used with the same meaning throughout the present disclosure). Weights may be variable and may be varied by the user or algorithm in order for the neural network to perform the desired function. For example, when one or more input nodes are connected to one output node by respective links, the output node is set to the values input to the input nodes connected to the output node and the links corresponding to each input node. The output node value can be determined based on the weight.

As described above, in a neural network, one or more nodes are interconnected through one or more links to form an input node and output node relationship within the neural network. The characteristics of the neural network can be determined according to the number of nodes and links within the neural network, the correlation between the nodes and links, and the value of the weight assigned to each link. For example, if the same number of nodes and links exist and two neural networks with different weight values of the links exist, the two neural networks may be recognized as different from each other.

A neural network may consist of a set of one or more nodes. A subset of nodes that make up a neural network can constitute a layer. Some of the nodes constituting the neural network may form one layer based on the distances from the first input node. For example, a set of nodes with a distance n from the initial input node may constitute an n layer. The distance from the initial input node can be defined by the minimum number of links that must be passed to reach the node from the initial input node. However, this definition of a layer is arbitrary for illustrative purposes, and the order of a layer within a neural network may be defined in a different way than described above. For example, a tier of nodes may be defined by their distance from the final output node.

The initial input node may refer to one or more nodes in the neural network through which data is directly input without going through links in relationships with other nodes. Alternatively, in a neural network network, in the relationship between nodes based on links, it may mean nodes that do not have other input nodes connected by links. Similarly, the final output node may refer to one or more nodes that do not have an output node in their relationship with other nodes among the nodes in the neural network. Additionally, hidden nodes may refer to nodes constituting a neural network other than the first input node and the last output node.

A neural network model including a neural network may be learned by at least one of supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning. Learning a neural network model may be a process of applying knowledge for the neural network model to perform a specific operation to the neural network model.

Neural network models can be trained to minimize output errors. In the learning of a neural network model, learning data is repeatedly input into the neural network model, the output of the neural network model and the error of the target for the learning data are calculated, and the error of the neural network model is calculated from the output layer of the neural network model to the input layer in the direction of reducing the error. This is the process of updating the weight of each node in the neural network model through backpropagation. In the case of teacher learning, learning data in which the correct answer is labeled in each learning data is used (i.e., labeled learning data), and in the case of non-teacher learning, the correct answer may not be labeled in each learning data. That is, for example, in the case of teacher learning regarding data classification, the learning data may be data in which each learning data is labeled with a category. Labeled training data is input to the neural network model, and the error can be calculated by comparing the output (category) of the neural network model and the label of the training data. As another example, in the case of non-teachable learning for data classification, the error can be calculated by comparing the input learning data with the neural network model output. The calculated error is back-propagated in the neural network model in the reverse direction (i.e., from the output layer to the input layer), and the connection weight of each node in each layer of the neural network model can be updated according to the back-propagation. The amount of change in the connection weight of each updated node may be determined according to the learning rate. The calculation of a neural network model for input data and backpropagation of errors can constitute an epoch. The learning rate may be applied differently depending on the number of repetitions of the epoch of the neural network model. For example, in the early stages of training a neural network model, a high learning rate can be used to ensure that the neural network model quickly achieves a certain level of performance to increase efficiency, and in the latter stages of training, a low learning rate can be used to increase accuracy.

In the training of a neural network model, the training data can generally be a subset of real data (i.e., the data to be processed using the learned neural network model), and therefore, the error for the training data is reduced, but the error for the real data is reduced. There may be increasing epochs. Over-fitting is a phenomenon in which errors in actual data increase due to excessive learning on training data. For example, a phenomenon in which a neural network model that learned a cat by showing a yellow cat fails to recognize that it is a cat when it sees a non-yellow cat may be a type of overfitting. Overfitting can cause errors in machine learning algorithms to increase. To prevent such overfitting, various optimization methods can be used. To prevent overfitting, methods such as increasing the learning data, regularization, dropout that disables some of the network nodes during the learning process, and use of batch normalization can be applied. .

According to an embodiment of the present disclosure, a computer-readable medium storing a data structure is disclosed.

Data structure can refer to the organization, management, and storage of data to enable efficient access and modification of data. Data structure can refer to the organization of data to solve a specific problem (e.g., retrieving data, storing data, or modifying data in the shortest possible time). A data structure may be defined as a physical or logical relationship between data elements designed to support a specific data processing function. Logical relationships between data elements may include connection relationships between user-defined data elements. Physical relationships between data elements may include actual relationships between data elements that are physically stored in a computer-readable storage medium (e.g., a persistent storage device). A data structure may specifically include a set of data, relationships between data, and functions or instructions applicable to the data. Effectively designed data structures allow computing devices to perform computations while minimizing the use of the computing device's resources. Specifically, computing devices can increase the efficiency of operations, reading, insertion, deletion, comparison, exchange, and search through effectively designed data structures.

Data structures can be divided into linear data structures and non-linear data structures depending on the type of data structure. A linear data structure may be a structure in which only one piece of data is connected to another piece of data. Linear data structures may include List, Stack, Queue, and Deque. A list can refer to a set of data that has an internal order. The list may include a linked list. A linked list may be a data structure in which data is connected in such a way that each data is connected in a single line with a pointer. In a linked list, a pointer may contain connection information to the next or previous data. Depending on its form, a linked list can be expressed as a singly linked list, a doubly linked list, or a circularly linked list. A stack may be a data listing structure that allows limited access to data. A stack can be a linear data structure in which data can be processed (for example, inserted or deleted) at only one end of the data structure. Data stored in the stack may have a data structure (LIFO-Last in First Out) where the later it enters, the sooner it comes out. A queue is a data listing structure that allows limited access to data. Unlike the stack, it can be a data structure (FIFO-First in First Out) where data stored later is released later. A deck can be a data structure that can process data at both ends of the data structure.

A non-linear data structure may be a structure in which multiple pieces of data are connected behind one piece of data. Nonlinear data structures may include graph data structures. A graph data structure can be defined by vertices and edges, and an edge can include a line connecting two different vertices. Graph data structure may include a tree data structure. A tree data structure may be a data structure in which there is only one path connecting two different vertices among a plurality of vertices included in the tree. In other words, it may be a data structure that does not form a loop in the graph data structure.

The data structure may include a neural network model. And the data structure including the neural network model may be stored in a computer-readable medium. The data structure including the neural network model includes preprocessed data for processing by the neural network model, data input to the neural network model, weights of the neural network model, hyperparameters of the neural network model, data acquired from the neural network model, and each node or layer of the neural network model. It may include an activation function associated with, a loss function for learning a neural network model, etc. A data structure containing a neural network model may include any of the components disclosed above. In other words, the data structure including the neural network model includes data preprocessed for processing by the neural network model, data input to the neural network model, weights of the neural network model, hyperparameters of the neural network model, data acquired from the neural network model, each node of the neural network model, or It may be composed of all or any combination of the activation function associated with the layer, the loss function for learning the neural network model, etc. In addition to the configurations described above, the data structure containing the neural network model may include any other information that determines the characteristics of the neural network model. Additionally, the data structure may include all types of data used or generated in the computational process of the neural network model and is not limited to the above. Computer-readable media may include computer-readable recording media and/or computer-readable transmission media. A neural network model can generally consist of a set of interconnected computational units, which can be referred to as nodes. These nodes may also be referred to as neurons. A neural network model is composed of at least one node.

The data structure may include data input to the neural network model. A data structure containing data input to the neural network model may be stored in a computer-readable medium. Data input to the neural network model may include learning data input during the learning process of the neural network model and/or input data input to the neural network model on which training has been completed. Data input to the neural network model may include data that has undergone pre-processing and/or data subject to pre-processing. Preprocessing may include a data processing process to input data into a neural network model. Therefore, the data structure may include data subject to preprocessing and data generated by preprocessing. The above-described data structure is only an example and the present disclosure is not limited thereto.

The data structure may include weights of the neural network model. (In the present disclosure, weights and parameters may be used with the same meaning.) And the data structure including the weights of the neural network model may be stored in a computer-readable medium. A neural network model may include multiple weights. Weights may be variable and may be varied by the user or algorithm in order for the neural network model to perform the desired function. For example, when one or more input nodes are connected to one output node by respective links, the output node is set to the values input to the input nodes connected to the output node and the links corresponding to each input node. Based on the weight, the data value output from the output node can be determined. The above-described data structure is only an example and the present disclosure is not limited thereto.

As an example and not a limitation, the weight may include a weight that changes during the learning process of the neural network model and/or a weight that has completed training of the neural network model. Weights that vary during the learning process of a neural network model may include weights that change at the start of an epoch and/or weights that change during the epoch. Weights for which training of the neural network model has been completed may include weights for which the epoch has been completed. Therefore, the data structure including the weights of the neural network model may include weights that change during the learning process of the neural network model and/or the data structure including weights that have completed training of the neural network model. Therefore, the above-described weights and/or combinations of each weight are included in the data structure including the weights of the neural network model. The above-described data structure is only an example and the present disclosure is not limited thereto.

The data structure including the weights of the neural network model may be stored in a computer-readable storage medium (e.g., memory, hard disk) after going through a serialization process. Serialization can be the process of converting a data structure into a form that can be stored on the same or a different computing device and later reorganized and used. Computing devices can transmit and receive data over a network by serializing data structures. The data structure containing the weights of the serialized neural network model can be reconstructed on the same computing device or on a different computing device through deserialization. The data structure including the weights of the neural network model is not limited to serialization. Furthermore, the data structure including the weights of the neural network model is a data structure to increase computational efficiency while minimizing the use of computing device resources (e.g., B-Tree, Trie, m-way search tree, AVL tree in non-linear data structures). , Red-Black Tree). The foregoing is merely an example and the present disclosure is not limited thereto.

The data structure may include hyper-parameters of the neural network model. And the data structure including the hyper-parameters of the neural network model may be stored in a computer-readable medium. A hyperparameter may be a variable that can be changed by the user. Hyperparameters include, for example, learning rate, cost function, number of epoch repetitions, weight initialization (e.g., setting the range of weight values subject to weight initialization), It may include the number of hidden units (e.g., number of hidden layers, number of nodes in the hidden layer). The above-described data structure is only an example and the present disclosure is not limited thereto.

Before describing embodiments of the present disclosure, terms used throughout the present disclosure are explained.

'First environment' and 'second environment' used throughout this disclosure may refer to environments in which different images or images are displayed. For example, the 'first environment' and 'second environment' may refer to environments in which different images or images are displayed in relation to photographing an object. At this time, the video or image may be used interchangeably and may include a still video or image or a video or image that is being played. Additionally, the video or image may include virtual graphics or graphics corresponding to a real environment. Meanwhile, photographing the object is not limited to photographing a real object, and may also include an operation of synthesizing a virtual object. For example, photographing the object may include photographing a real object in an environment where different images or images are displayed, compositing a virtual object on different images or images, etc.

Additionally, the first environment may include an environment in which various background images are displayed, and the second environment may include an environment in which an image including a continuous pattern or a plain pattern is displayed. You can. In this case, the first environment may include an environment in which a background image of an actual photograph of nature, a cityscape, etc. is displayed, or a virtual background image generated by a computer is displayed.

Additionally, objects used throughout this disclosure may be understood as semantic or individually classifiable objects, including people or objects. For example, if one image includes one employee, one customer, and one desk, the image may be understood as including three objects, or it may be understood as including two people and one object. However, the method for classifying objects is not limited to this and can be implemented in various ways.

From now on, with reference to steps S301 to S303 of FIG. 3, a method of generating training data and correct answer data for learning a neural network model based on the first environment and the second environment according to an embodiment of the present disclosure. The general process is explained.

Referring to FIG. 3, the processor 110 acquires a plurality of images in which an object is photographed in a situation in which the first environment and the second environment are repeatedly switched (S301), and a first environment in which the object is photographed is captured in the first environment. 1. Generating training data for the neural network model based on the image (S302), and generating correct answer data for learning the neural network model based on the second image in which the object is photographed in the second environment. The step (S303) can be performed. (At this time, the processor 110 needs to perform step S301 before performing step S302 and step S303, but after step S301, step S302 and step S303 can be freely performed as needed. .)

At this time, the neural network model may include a model for separating objects and backgrounds included in the input data (a detailed description of this will be discussed in detail later with reference to FIG. 6). In addition, the plurality of images include, It may include a plurality of images in which a single object is captured in a situation where the first environment and the second environment are repeatedly switched. Alternatively, the plurality of images may include a plurality of images in which two or more objects are captured simultaneously in a situation where the first environment and the second environment are repeatedly switched. Here, the two or more objects may include at least one of a moving object or a motionless object, and the two or more objects may include at least one of an object or a person.

As mentioned earlier, the first environment may include an environment in which a background image is displayed, and the second environment may include an environment in which a chroma key image is displayed. Additionally, the plurality of images may include a plurality of images in which a single object or two or more objects are photographed in a situation where the background image and the chroma key image are repeatedly switched.

At this time, the background image may include at least one of a playing background image or a stationary background image, and the chroma key image may include a continuous pattern or a continuous pattern for a computing device or person to easily classify the background and object. It may include at least one solid color plain pattern. For example, the background image and the chroma key image may be repeatedly switched and displayed by various display devices such as LEDs, and the plurality of images may be generated by the display device and a photographing device that photographs the object. there is. In addition, the display device and the photographing device may be set to have the same number of frames per second (FPS), and “the second environment and the second image in which the object is captured” may be “the first environment and the second image.” The “first image in which the object is captured” may be an image captured in the frame immediately before or in the frame immediately after the frame in which the image was captured. (A detailed description of the above example will be discussed in detail later with reference to FIGS. 4 and 5.)

Meanwhile, the photographing device may further include additional sensors in addition to parts that capture visual information. For example, the photographing device may include a depth sensor that measures depth data related to the perspective of the object, and the plurality of images captured based on the photographing device may include depth data associated with the perspective of the object. It may include, and a plurality of images including the depth data can be used for learning neural network models related to the field of virtual reality or augmented reality, including the 'apple AR kit'.

However, sensors that the imaging device may additionally include are not limited to the depth sensor, and may additionally include various sensors such as an infrared sensor, a tilt sensor, a gyroscope sensor, an acceleration sensor, and a SAR sensor.

Figure 4 is a schematic diagram showing a method for generating learning data and correct answer data based on a first environment and a second environment, according to an embodiment of the present disclosure.

Previously, in the general process of the present disclosure, it has been mentioned that the processor 110 can perform steps S301 to S303. Referring to FIG. 4, an embodiment of steps S301 to S303 performed by a processor is disclosed.

First, in the step (S301) of the processor 110 acquiring a plurality of images in which an object is photographed in a situation where the first environment and the second environment are repeatedly switched, the plurality of images are image_1 (400), image It may include image_2 (410), image_3 (420), and image_4 (430). At this time, the image_1 (400) to image_4 (430) display objects every 1/60 second in a situation where the first environment and the second environment are repeatedly switched every 1/60 second (i.e., switched at 60 FPS). When shooting (i.e., shooting at 60 FPS), it may correspond to an image shot at 1/60 second to 4/60 second. Accordingly, each of Image_1 (400) to Image_4 (430) includes the same object, but may include either a background image or a chroma key image. However, the FPS is not limited to 60 as the example above and can be set in various ways.

In the step (S302) of the processor 110 generating training data for the neural network model based on the first image in which the object is captured in the first environment, the first environment may correspond to a background image. And the first image in which the object is captured in the first environment may include the image_1 (400) and the image_3 (420). That is, the processor 110 may generate training data for the neural network model based on image_1 (400) and image_3 (420).

Next, in the step (S303) of the processor 110 generating correct answer data for the neural network model based on the second image in which the object is captured in the second environment, the second environment is a chroma key image. The second image in which the object is captured in the second environment may include image_2 (410) and image_4 (430). That is, the processor 110 may generate correct answer data for the neural network model based on image_2 (410) and image_4 (430).

At this time, the processor 110 converts the second image (i.e., the second image, which is the correct answer data corresponding to the first image, which is learning data) into an image captured in the frame immediately before or in the frame immediately after the frame in which the first image was captured. You can decide. For example, when the processor 110 determines the image_3 (420) as learning data, the processor 110 selects image_2 (410) captured in the image frame immediately before image_3 (420) or image_4 captured immediately after. (430) can be determined as the correct answer data. In terms of effectiveness, when using the method of the above embodiment, 30 learning data and 30 correct answer data can be generated every second, based on the use of filming equipment that shoots at a speed of 60 FPS.

Figure 5 is a schematic diagram showing a method of generating learning data and correct answer data based on a display device, according to an embodiment of the present disclosure.

Referring to FIG. 5, a specific embodiment of step S301 performed by a processor is disclosed.

First, in the step (S301) of the processor 110 acquiring a plurality of images in which an object is photographed in a situation where the first environment and the second environment are repeatedly switched, referring to FIG. 5, the plurality of images are, The display device 500, in which images corresponding to the first environment and the second environment are repeatedly switched and displayed, can be created by continuously photographing the object 510 together with the photographing device 520. At this time, the first environment and the second environment displayed by the display device 500 may correspond to a background image and a chroma key image, respectively, and the display device 500 switches between the first environment and the second environment. The frequency may be set to have the same number of frames per second (FPS) as the frequency at which the photographing device 520 captures images. Additionally, the display device 500 may be a variety of image display devices including LCD, LED, projector, etc.

In addition, the photographing device 520 can perform photographing by modifying the photographing angle and the position of the photographing device, and the object 510 that is the object of photographing is a moving object. It may include at least one of 510 or an object 510 without movement. For example, the photographing device 520 may generate a plurality of images by continuously photographing at least one actor (i.e., the object 510) performing a walking motion from various angles. Specifically, assuming that the four movements of walking are "1.stride, 2.squash, 3.passing position, 4.stretch", the actor 510 performs the walking movement together with the display device 500. Based on multiple videos filmed continuously, learning data and correct answer data containing actors performing the four movements mentioned above (i.e., 1.stride, 2.squash, 3.passing position, 4.stretch) are generated. can be created.

When generating a plurality of images using the method of the above example, the duplication of data is reduced compared to the situation in which the object 510 is stationary, and the generated learning data and answer data are enriched, thereby A neural network model learned based on data can be learned relatively robustly.

FIG. 6 is a schematic diagram illustrating a neural network model for separating an object and a background from an image, according to an embodiment of the present disclosure.

Referring to FIG. 6 , image data 600 is input and the image data 600 is received using a neural network model 610 learned based on learning data and correct answer data generated by a method according to an embodiment of the present disclosure. ) An embodiment of a method for separating an object 600a and a background 600b is disclosed.

The processor 110 inputs image data 600 including a background 600b and an object 600a into a neural network model 610, and creates an object 600a and a background based on the image data 600 ( The step of classifying 600b) can be performed. At this time, the neural network model 610 corresponds to a model learned based on a plurality of images in which the learning object 600a is photographed in a situation where the first environment and the second environment are repeatedly switched, and in the first environment The image in which the learning object 600a is captured may correspond to learning data for training the neural network model 610. Additionally, an image of the learning object 600a captured in the second environment may correspond to correct answer data for learning the neural network model 610.

In addition, when the processor 110 learns the neural network model 610 using the learning data and answer data generated by the method of the above embodiment, high-quality learning data can be secured at a lower cost compared to the conventional method. Therefore, a neural network model 610 with relatively excellent performance can be created with the same resources. In addition, in the field of synthesizing images and graphics, including broadcasting, movie augmented reality, and virtual reality, the processor 110 uses the neural network model 610 to perform tasks related to augmented reality occlusion. can do.

7 is a brief, general schematic diagram of an example computing environment in which embodiments of the present disclosure may be implemented.

Although the disclosure has been described above as being generally capable of being implemented by a computing device, those skilled in the art will understand that the disclosure can be implemented in combination with computer-executable instructions and/or other program modules that can be executed on one or more computers and/or hardware. It will be well known that it can be implemented as a combination of software.

Typically, program modules include routines, programs, components, data structures, etc. that perform specific tasks or implement specific abstract data types. Additionally, those skilled in the art will understand that the methods of the present disclosure can be used on single-processor or multiprocessor computer systems, minicomputers, mainframe computers, as well as personal computers, handheld computing devices, microprocessor-based or programmable consumer electronics, etc. It will be appreciated that other computer system configurations may be implemented, including, but not limited to, each of which may operate in connection with one or more associated devices.

The described embodiments of the present disclosure may be practiced in a distributed computing environment where certain tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote memory storage devices.

Computers typically include a variety of computer-readable media. Computer-readable media can be any medium that can be accessed by a computer, and such computer-readable media includes volatile and non-volatile media, transitory and non-transitory media, removable and non-transitory media. Includes removable media. By way of example, and not limitation, computer-readable media may include computer-readable storage media and computer-readable transmission media. Computer-readable storage media refers to volatile and non-volatile media, transient and non-transitory media, removable and non-removable, implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Includes media. Computer readable storage media may include RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital video disk (DVD) or other optical disk storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage. This includes, but is not limited to, a device, or any other medium that can be accessed by a computer and used to store desired information.

A computer-readable transmission medium typically implements computer-readable instructions, data structures, program modules, or other data on a modulated data signal, such as a carrier wave or other transport mechanism. Includes all information delivery media. The term modulated data signal refers to a signal in which one or more of the characteristics of the signal have been set or changed to encode information within the signal. By way of example, and not limitation, computer-readable transmission media includes wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, RF, infrared, and other wireless media. Combinations of any of the above are also intended to be included within the scope of computer-readable transmission media.

An example environment 1100 is shown that implements various aspects of the present disclosure, including a computer 1102, which includes a processing unit 1104, a system memory 1106, and a system bus 1108. do. System bus 1108 couples system components, including but not limited to system memory 1106, to processing unit 1104. Processing unit 1104 may be any of a variety of commercially available processors. Dual processors and other multiprocessor architectures may also be used as processing unit 1104.

System bus 1108 may be any of several types of bus structures that may further be interconnected to a memory bus, peripheral bus, and local bus using any of a variety of commercial bus architectures. System memory 1106 includes read only memory (ROM) 1110 and random access memory (RAM) 1112. The basic input/output system (BIOS) is stored in non-volatile memory 1110, such as ROM, EPROM, and EEPROM, and is a basic input/output system that helps transfer information between components within the computer 1102, such as during startup. Contains routines. RAM 1112 may include high-speed RAM, such as static RAM for caching data.

Computer 1102 includes an internal hard disk drive (HDD) 1114 (e.g., EIDE, SATA), which internal hard disk drive 1114 may be configured for external use within a suitable chassis (not shown). , a magnetic floppy disk drive (FDD) 1116 (e.g., for reading from or writing to a removable diskette 1118), and an optical disk drive 1120 (e.g., a CD-ROM disk ( 1122) or for reading from or writing to other high-capacity optical media such as DVDs). Hard disk drive 1114, magnetic disk drive 1116, and optical disk drive 1120 are connected to system bus 1108 by hard disk drive interface 1124, magnetic disk drive interface 1126, and optical drive interface 1128, respectively. ) can be connected to. The interface 1124 for implementing an external drive includes at least one or both of Universal Serial Bus (USB) and IEEE 1394 interface technologies.

These drives and their associated computer-readable media provide non-volatile storage of data, data structures, computer-executable instructions, and the like. For computer 1102, drive and media correspond to storing any data in a suitable digital format. Although the description of computer-readable media above refers to removable optical media such as HDDs, removable magnetic disks, and CDs or DVDs, those of ordinary skill in the art will also recognize zip drives, magnetic cassettes, flash memory cards, and cartridges. It will be appreciated that other types of computer-readable media may also be used in the example operating environment, and that any such media may contain computer-executable instructions for performing the methods of the present disclosure. .

A number of program modules may be stored in drives and RAM 1112, including an operating system 1130, one or more application programs 1132, other program modules 1134, and program data 1136. All or portions of the operating system, applications, modules and/or data may be cached in RAM 1112. It will be appreciated that the present disclosure may be implemented on various commercially available operating systems or combinations of operating systems.

A user may enter commands and information into computer 1102 through one or more wired/wireless input devices, such as a keyboard 1138 and a pointing device such as mouse 1140. Other input devices (not shown) may include microphones, IR remote controls, joysticks, game pads, stylus pens, touch screens, etc. These and other input devices are connected to the processing unit 1104 through an input device interface 1142, which is often connected to the system bus 1108, but may also include a parallel port, an IEEE 1394 serial port, a game port, a USB port, an IR interface, It can be connected by other interfaces, etc.

A monitor 1144 or other type of display device is also connected to system bus 1108 through an interface, such as a video adapter 1146. In addition to monitor 1144, computers typically include other peripheral output devices (not shown) such as speakers, printers, etc.

Computer 1102 may operate in a networked environment using logical connections to one or more remote computers, such as remote computer(s) 1148, via wired and/or wireless communications. Remote computer(s) 1148 may be a workstation, computing device computer, router, personal computer, portable computer, microprocessor-based entertainment device, peer device, or other conventional network node, and is generally connected to computer 1102. For simplicity, only memory storage device 1150 is shown, although it includes many or all of the components described. The logical connections depicted include wired/wireless connections to a local area network (LAN) 1152 and/or a larger network, such as a wide area network (WAN) 1154. These LAN and WAN networking environments are common in offices and companies and facilitate enterprise-wide computer networks, such as intranets, all of which can be connected to a worldwide computer network, such as the Internet.

When used in a LAN networking environment, computer 1102 is connected to local network 1152 through wired and/or wireless communication network interfaces or adapters 1156. Adapter 1156 may facilitate wired or wireless communication to a local area network (LAN) 1152, which may include a wireless access point installed thereon for communicating with wireless adapter 1156. When used in a WAN networking environment, the computer 1102 may include a modem 1158 or be connected to a communicating computing device on the WAN 1154 or to establish communications over the WAN 1154, such as via the Internet. Have other means. Modem 1158, which may be internal or external and a wired or wireless device, is coupled to system bus 1108 via serial port interface 1142. In a networked environment, program modules described for computer 1102, or portions thereof, may be stored in remote memory/storage device 1150. It will be appreciated that the network connections shown are exemplary and that other means of establishing a communications link between computers may be used.

Computer 1102 may be associated with any wireless device or object deployed and operating in wireless communications, such as a printer, scanner, desktop and/or portable computer, portable data assistant (PDA), communications satellite, wirelessly detectable tag. Performs actions to communicate with any device or location and telephone. This includes at least Wi-Fi and Bluetooth wireless technologies. Accordingly, communication may be a predefined structure as in a conventional network or may simply be ad hoc communication between at least two devices.

Wi-Fi (Wireless Fidelity) allows connection to the Internet, etc. without wires. Wi-Fi is a wireless technology, like cell phones, that allows these devices, such as computers, to send and receive data indoors and outdoors, anywhere within the coverage area of a base station. Wi-Fi networks use wireless technology called IEEE 802.11 (a, b, g, etc.) to provide secure, reliable, and high-speed wireless connections. Wi-Fi can be used to connect computers to each other, the Internet, and wired networks (using IEEE 802.3 or Ethernet). Wi-Fi networks can operate in the unlicensed 2.4 and 5 GHz wireless bands, for example, at data rates of 11 Mbps (802.11a) or 54 Mbps (802.11b), or in products that include both bands (dual band). .

Those skilled in the art will understand that information and signals may be represented using any of a variety of different technologies and techniques. For example, data, instructions, commands, information, signals, bits, symbols and chips that may be referenced in the above description include voltages, currents, electromagnetic waves, magnetic fields or particles, optical fields. It can be expressed by particles or particles, or any combination thereof.

Those skilled in the art will understand that the various illustrative logical blocks, modules, processors, means, circuits and algorithm steps described in connection with the embodiments disclosed herein may be used in electronic hardware, (for convenience) It will be understood that it may be implemented by various forms of program or design code (referred to herein as software) or a combination of both. To clearly illustrate this interoperability of hardware and software, various illustrative components, blocks, modules, circuits and steps have been described above generally with respect to their functionality. Whether this functionality is implemented as hardware or software depends on the specific application and design constraints imposed on the overall system. A person skilled in the art of this disclosure may implement the described functionality in various ways for each specific application, but such implementation decisions should not be construed as departing from the scope of this disclosure.

The various embodiments presented herein may be implemented as a method, apparatus, or article of manufacture using standard programming and/or engineering techniques. The term article of manufacture includes a computer program, carrier, or media accessible from any computer-readable storage device. For example, computer-readable storage media include magnetic storage devices (e.g., hard disks, floppy disks, magnetic strips, etc.), optical disks (e.g., CDs, DVDs, etc.), smart cards, and flash. Includes, but is not limited to, memory devices (e.g., EEPROM, cards, sticks, key drives, etc.). Additionally, various storage media presented herein include one or more devices and/or other machine-readable media for storing information.

It is to be understood that the specific order or hierarchy of steps in the processes presented is an example of illustrative approaches. It is to be understood that the specific order or hierarchy of steps in processes may be rearranged within the scope of the present disclosure, based on design priorities. The appended method claims present elements of the various steps in a sample order but are not meant to be limited to the particular order or hierarchy presented.

The description of the presented embodiments is provided to enable any person skilled in the art to make or use the present disclosure. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be applied to other embodiments without departing from the scope of the disclosure. Thus, the present disclosure is not limited to the embodiments presented herein but is to be interpreted in the broadest scope consistent with the principles and novel features presented herein.

Claims (1)

A method of generating training data for a neural network model, performed by at least one computing device.