KR20210096347A - Laundry machine and laundry machine operating method - Google Patents

Abstract

Disclosed are a washing apparatus and a washing apparatus driving method, which can determine a washing process in a 5G communication environment by executing an artificial intelligence (AI) algorithm and/or a machine learning algorithm. According to one embodiment of the present invention, the washing apparatus driving method comprises: a step of acquiring first laundry information as an inside image of a washing tank based on a vision sensor for photographing the inside of the washing tank; a step of acquiring second laundry information as weight information of laundry included in the washing tank based on a weight detection sensor for detecting the weight of the inside of the washing tank; a step of using the first laundry information and the second laundry information to determine a property of the laundry; and a step of determining a washing process of the washing apparatus based on the property of the laundry. According to the present invention, the washing apparatus can effectively perform targeted washing processing of a user by accurately identifying the property of the laundry.

Description

LAUNDRY MACHINE AND LAUNDRY MACHINE OPERATING METHOD

The present invention relates to a washing machine and a washing machine driving method for determining the operation of the washing machine by measuring the density of laundry.

As a washing machine, an electronic product for washing laundry such as clothes, bedding or towels, or other textile products is typical, and a dryer for drying clothes can also be called a washing machine. Of course, a washing machine and dryer capable of washing and drying clothes may also be referred to as washing machines. Recently, a refresher that refreshes clothes using hot air or steam rather than washing with water has been released, and this can also be referred to as a washing machine.

A washing machine as an author's product for washing laundry is provided with a washing tub capable of accommodating laundry and washing water, and washing can be performed according to operations of various parts installed in the washing tub.

When the laundry is put into the washing tub, the user determines the washing course according to the type of laundry, the volume or weight of the laundry, and the degree of contamination of the laundry, and determines, for example, whether to soak, the number of times of washing, rinsing, and spin-drying, and whether to dry. When a parameter value corresponding to a predetermined course is input to the washing machine, the washing machine operates according to the input parameter value.

Although a technique for detecting the weight of laundry by using a weight sensor or the like has been disclosed for such a washing machine, there is a problem in that the volume of laundry cannot be detected. As such, due to the problem that the washing machine does not detect the volume of the laundry, there may be a problem of not accurately grasping the properties of the laundry and performing wrong processing.

The above-mentioned background art is technical information possessed by the inventor for derivation of the present invention or acquired in the process of derivation of the present invention, and cannot necessarily be said to be a known technique disclosed to the general public prior to the filing of the present invention.

SUMMARY OF THE INVENTION One object of the present invention is to solve a problem in that the laundry apparatus does not detect the volume of laundry and thus does not accurately grasp the properties of the laundry and performs wrong processing.

An object of the present invention is to accurately understand the properties of laundry by detecting the volume of laundry put into the washing tub based on a vision sensor.

An object of the present invention is to detect a volume of laundry put into the washing tub based on a vision sensor and accurately determine the properties of laundry by using the weight of the laundry put into the washing tub based on a weight sensor.

An object of the present invention is to detect the volume of laundry put into the washing tub based on a vision sensor, accurately determine the properties of laundry by using the weight of the laundry put into the washing tub based on the weight sensor, and determine the properties of the laundry It is based on driving the washing machine.

In a washing machine driving method according to an embodiment of the present invention, the property of laundry is determined by using first laundry information based on an image inside the washing tub and second laundry information based on the weight of laundry put into the washing tub. and determining a washing process based on the properties of the laundry.

Specifically, the method of driving a washing machine according to an embodiment of the present invention includes the steps of: acquiring first laundry information as an image inside the washing tub based on a vision sensor photographing the inside of the washing tub; Acquiring second laundry information as weight information of the laundry included in the washing tub based on the detection sensor, determining the properties of laundry using the first laundry information and the second laundry information; determining a washing process of the washing machine based on the attribute.

Through the washing machine driving method according to an embodiment of the present invention, the washing machine can accurately identify the properties of the laundry to effectively perform the washing process for the user, and also automatically determine the washing process compared to the laundry. Therefore, the user does not need to manually determine the washing process, which is convenient.

A laundry device according to an embodiment of the present invention includes a first acquisition unit that acquires first laundry information as an image inside the washing tub based on a vision sensor that captures the inside of the washing tub, and a weight detection sensor that detects the weight of the inside of the washing tub A second acquisition unit for acquiring second laundry information as weight information of laundry included in the washing tub based on It may include a determining unit that determines a washing process of the washing machine based on the attribute of the laundry.

In addition to this, another method for implementing the present invention, another system, and a computer-readable recording medium storing a computer program for executing the method may be further provided.

Other aspects, features and advantages other than those described above will become apparent from the following drawings, claims, and detailed description of the invention.

Advantageous Effects of Invention According to the present invention, the laundry apparatus accurately grasps the properties of laundry so that the user can effectively perform the desired laundry treatment.

In addition, since the laundry process is automatically determined compared to the laundry, the user does not need to manually determine the washing process, which is convenient.

In addition, it is possible to determine the properties of the laundry by relatively accurately measuring the volume of the laundry put into the washing tub, and accordingly perform a suitable washing treatment on the laundry.

In addition, it is possible to determine the properties of the laundry by relatively accurately measuring the volume and weight of the laundry put into the washing tub, and accordingly perform a suitable washing treatment on the laundry.

In addition, although the washing machine itself is a mass-produced uniform product, the user recognizes the washing machine as a personalized device, so that a user-customized product can be created.

In addition, power efficiency of the washing machine may be improved by driving the washing machine using only the optimal processor resources.

Effects of the present invention are not limited to those mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the following description.

1 is an exemplary diagram of a laundry environment including a laundry apparatus, a user terminal, a server, and a network connecting them according to the present embodiment.
2 is an external perspective view of the washing machine according to the present embodiment.
3 is a block diagram schematically illustrating the configuration of a washing machine according to the present embodiment.
4 is a block diagram schematically illustrating the configuration of an information processing unit in the washing machine of FIG. 3 .
5 is a view showing the inside of the washing tub of the washing machine according to the present embodiment.
6 is an exemplary view for determining the volume information of laundry according to the laundry put into the washing tub according to the present embodiment.
7 is an exemplary diagram of a result of determining laundry properties according to the present embodiment.
8 is a view for explaining a deep neural network model for estimating the volume of laundry according to the present embodiment.
9 is a flowchart illustrating a method of driving a washing machine according to the present embodiment.

Advantages and features of the present invention, and a method of achieving them will become apparent with reference to the embodiments described in detail in conjunction with the accompanying drawings. However, it should be understood that the present invention is not limited to the embodiments presented below, but may be implemented in a variety of different forms, and includes all transformations, equivalents, and substitutes included in the spirit and scope of the present invention. . The embodiments presented below are provided to complete the disclosure of the present invention, and to fully inform those of ordinary skill in the art to which the present invention pertains to the scope of the invention. In describing the present invention, if it is determined that a detailed description of a related known technology may obscure the gist of the present invention, the detailed description thereof will be omitted.

The terms used in the present application are only used to describe specific embodiments, and are not intended to limit the present invention. The singular expression includes the plural expression unless the context clearly dictates otherwise. In the present application, terms such as “comprise” or “have” are intended to designate that a feature, number, step, operation, component, part, or combination thereof described in the specification exists, but one or more other features It should be understood that it does not preclude the possibility of the presence or addition of numbers, steps, operations, components, parts, or combinations thereof. Terms such as first, second, etc. may be used to describe various elements, but the elements should not be limited by the terms. The above terms are used only for the purpose of distinguishing one component from another.

Hereinafter, embodiments according to the present invention will be described in detail with reference to the accompanying drawings, and in the description with reference to the accompanying drawings, the same or corresponding components are given the same reference numerals, and overlapping descriptions thereof are omitted. decide to do

1 is an exemplary diagram of a laundry environment including a laundry apparatus, a user terminal, a server, and a network connecting them according to the present embodiment. Referring to FIG. 1 , the laundry environment may include a laundry device 100 , a user terminal 200 , a server 300 , and a network 400 .

The washing machine 100 may include a device for processing laundry through various actions such as washing, dehydration and/or drying. The washing machine 100 includes a washing machine that removes contamination from laundry (hereinafter also referred to as "cloth") using water and detergent, a dehydrator that spins a drum into which wet laundry is placed at high speed to dehydrate the laundry; It may include a dryer for drying the laundry by supplying drying air into the drum into which the laundry is put, and a washing machine for drying with both a drying function and a washing function, and the like.

The laundry device 100 may support object intelligence communication (internet of things (IoT), internet of everything (IoE), internet of small things (IoST), etc.), and M2M (machine to machine) communication, D2D (device to device) communication, and the like.

The washing machine 100 may acquire first laundry information as an image inside the washing tub based on the vision sensor ( 131 in FIG. 2 ) photographing the inside of the washing tub ( 102 in FIG. 2 ). In this embodiment, the washing machine 100 may acquire the first laundry information as an image including the characteristic shapes (102a to 102f of FIG. 5B) inside the washing tub 102. In an optional embodiment, the washing machine 100 includes a first image of the inside of the washing tub 102 before the laundry is put into the washing tub 102 and a second image of the inside of the washing tub 102 after the laundry is put into the washing tub 102 . It is possible to obtain first laundry information including.

The washing machine 100 may obtain the second laundry information as laundry weight information included in the washing tub 102 based on the weight detection sensor (132 in FIG. 3 ) that detects the weight inside the washing tub 102 . there is. In this embodiment, the washing machine 100 is based on the current detection sensor (133 in FIG. 3) for detecting the driving current of the motor (not shown) rotating the washing tub 102, laundry corresponding to the driving current of the motor It is possible to obtain the second laundry information as the weight information of the.

The laundry apparatus 100 may determine the attributes of laundry by using the first laundry information and the second laundry information. The laundry apparatus 100 may estimate the volume information of the laundry in order to determine the properties of the laundry, and estimate the density information of the laundry using the volume information of the laundry and the weight information of the laundry. The laundry apparatus 100 may determine the attribute of the laundry by using the density information of the laundry.

The laundry apparatus 100 may estimate the laundry volume information based on the first laundry information. The washing machine 100 determines the number of exposed feature shapes 102a to 102f of the washing tub 102 into which laundry is placed from the image including the feature shapes 102a to 102f inside the washing tub or the washing tub hidden by the laundry ( The volume information of the laundry may be estimated based on the number of the feature shapes 102a to 102f of 102).

As an optional embodiment, the washing machine 100 uses a deep neural network model trained in advance to estimate the volume of laundry using images including feature shapes inside the washing tub, and collects information about the volume of laundry put into the washing tub 102 . can be estimated Here, the deep neural network model is a CNN-based model, and a neural network model trained in a supervised learning method using images of a plurality of washing tubs in a state in which various amounts of laundry are labeled for each image as training data. can

In an optional embodiment, the washing machine 100 includes a first image of the inside of the washing tub 102 before the laundry is put into the washing tub 102 and a second image of the inside of the washing tub 102 after the laundry is put into the washing tub 102 . can be compared, and the volume information of the laundry can be estimated based on the comparison result of the first image and the second image. Here, since the volume information of the first image is zero, it is possible to estimate the volume information of the laundry included in the second image based on the volume information of the first image.

The laundry apparatus 100 may estimate the density information of the laundry by using the volume information of the laundry and the weight information of the laundry, and determine the properties of the laundry. In this embodiment, the laundry property may include a first property and a second property. The first attribute may include a low volume and heavy load (eg, pre-washing). The laundry apparatus 100 may determine that laundry having a predetermined number or more of the feature shapes 102a to 102f from the inside image of the washing tub 102 and having weight information equal to or greater than a predetermined value is the laundry having the first attribute. The second attribute may include a special load (eg, a cotton blanket, a padded jumper, etc.) that is bulky and light in weight. The laundry apparatus 100 may determine laundry having less than a predetermined number of feature shapes 102a to 102f from the inside image of the washing tub 102 and having weight information less than a predetermined value as laundry having the second attribute. In the present embodiment, the laundry apparatus 100 classifies the laundry property into the first property and the second property, but is not limited thereto and may include various properties based on the result of estimating the density of the laundry.

The washing machine 100 may determine the washing process of the washing machine based on the property of the laundry. Here, determining the washing process may include determining a washing operation and washing factors. Determination of washing strokes may include, for example, determination of a course (eg, standard course, preparatory course, duvet course, etc.) and frequency of washing-rinsing-drying. The determination of the washing factor may include determination of a water supply amount (eg, large, medium, small), water supply temperature, detergent amount, and administration time during a washing operation.

The laundry apparatus 100 may output detergent input amount guide information based on the laundry process determination result, and may output the laundry property determination result and the laundry process determination result before the laundry apparatus 100 is driven.

In an optional embodiment, the washing machine 100 uses another deep neural network model trained in advance to determine the washing process of the washing machine using the density (volume and weight) of the laundry put into the washing tub, 100) can be determined. Here, another deep neural network model is a CNN-based model, where the deep neural network model is a CNN-based model, and is based on the density (volume and weight) of various laundry put into the washing tub and the density of each laundry. A suitable washing process can be trained in a supervised learning manner using the labeled training data.

In this way, the laundry device 100 uses big data, artificial intelligence (AI) algorithms, and/or machine learning algorithms in a 5G environment connected for the Internet of Things (IoT) to provide laundry volume information and/or washing process can be decided

Here, artificial intelligence (AI) is a field of computer engineering and information technology that studies how computers can do the thinking, learning, and self-development that can be done with human intelligence. It could mean making it possible to imitate intelligent behavior.

Also, AI does not exist by itself, but has many direct and indirect connections with other fields of computer science. In particular, in modern times, attempts are being made to introduce artificial intelligence elements in various fields of information technology and use them to solve problems in that field.

Machine learning is a branch of artificial intelligence, a field of study that gives computers the ability to learn without an explicit program.

Specifically, machine learning can be said to be a technology that studies and builds a system and an algorithm for learning based on empirical data, making predictions, and improving its own performance. Algorithms in machine learning can take the approach of building specific models to make predictions or decisions based on input data, rather than executing strictly set static program instructions.

The term 'machine learning' may be used interchangeably with the term 'machine learning'.

With regard to how to classify data in machine learning, many machine learning algorithms have been developed. A typical machine learning algorithm may include a decision tree, a Bayesian network, a support vector machine (SVM), and an artificial neural network (ANN).

The decision tree may include an analysis method for performing classification and prediction by charting decision rules into a tree structure.

The Bayesian network may include a model that expresses a probabilistic relationship (conditional independence) between a plurality of variables in a graph structure. Bayesian networks may be suitable for data mining through unsupervised learning.

The support vector machine is a model of supervised learning for pattern recognition and data analysis, and can be mainly used for classification and regression analysis.

The artificial neural network models the operation principle of biological neurons and the connection relationship between neurons. there is.

Artificial neural networks are models used in machine learning, and can include statistical learning algorithms inspired by neural networks in biology (especially the brain in the central nervous system of animals) in machine learning and cognitive science.

Specifically, the artificial neural network may refer to an overall model having problem-solving ability by changing the bonding strength of synapses through learning in which artificial neurons (nodes) formed a network by combining synapses.

The term artificial neural network may be used interchangeably with the term neural network.

The artificial neural network may include a plurality of layers, and each of the layers may include a plurality of neurons. Also, the artificial neural network may include neurons and synapses connecting neurons.

In general, artificial neural networks calculate the output value from the following three factors: (1) the connection pattern between neurons in different layers (2) the learning process that updates the weight of the connection (3) the weighted sum of the input received from the previous layer It can be defined by the activation function it creates.

The artificial neural network may include network models such as a deep neural network (DNN), a recurrent neural network (RNN), a bidirectional recurrent deep neural network (BRDNN), a multilayer perceptron (MLP), and a convolutional neural network (CNN). , but not limited thereto.

In this specification, the term 'layer' may be used interchangeably with the term 'layer'.

Artificial neural networks can be classified into single-layer neural networks and multi-layer neural networks according to the number of layers.

A general single-layer neural network may be composed of an input layer and an output layer.

In addition, a general multilayer neural network may be composed of an input layer, one or more hidden layers, and an output layer.

The input layer is a layer that receives external data. The number of neurons in the input layer is the same as the number of input variables, and the hidden layer is located between the input layer and the output layer. can The output layer may receive a signal from the hidden layer and output an output value based on the received signal. The input signal between neurons is multiplied by each connection strength (weight) and then summed. If the sum is greater than the threshold of the neuron, the neuron is activated and the output value obtained through the activation function can be output.

Meanwhile, a deep neural network including a plurality of hidden layers between an input layer and an output layer may be a representative artificial neural network that implements deep learning, which is a type of machine learning technology.

Meanwhile, the term 'deep learning' may be used interchangeably with the term 'deep learning'.

The artificial neural network may be trained using training data. Here, learning refers to a process of determining parameters of an artificial neural network using learning data to achieve the purpose of classifying, regressing, or clustering input data. can As a representative example of parameters of an artificial neural network, a weight applied to a synapse or a bias applied to a neuron may be mentioned.

The artificial neural network learned by the training data may classify or cluster the input data according to a pattern of the input data.

On the other hand, an artificial neural network trained using training data may be called a trained model in this embodiment.

The following describes the learning method of the artificial neural network.

A learning method of an artificial neural network may be largely classified into supervised learning, unsupervised learning, semi-supervised learning, and reinforcement learning.

Supervised learning can be a method of machine learning to infer a function from training data. And among these inferred functions, outputting continuous values is called regression, and predicting and outputting the class of an input vector can be called classification.

In supervised learning, an artificial neural network can be trained in a state in which a label for training data is given. Here, the label may mean a correct answer (or a result value) that the artificial neural network should infer when training data is input to the artificial neural network.

In this embodiment, when training data is input, the correct answer (or result value) that the artificial neural network must infer may be called a label or labeling data.

Also, in this embodiment, setting a label on the training data for learning of the artificial neural network may be named as labeling the labeling data on the training data.

In this case, the training data and the label corresponding to the training data) constitute one training set, and may be input to the artificial neural network in the form of a training set.

On the other hand, training data represents a plurality of features, and labeling the training data may mean that the features represented by the training data are labeled. In this case, the training data may represent the features of the input object in a vector form.

The artificial neural network may infer a function for the relationship between the training data and the labeling data by using the training data and the labeling data. In addition, parameters of the artificial neural network may be determined (optimized) through evaluation of the function inferred from the artificial neural network.

Unsupervised learning is a type of machine learning where no labels are given to training data.

Specifically, the unsupervised learning may be a learning method for learning the artificial neural network to find and classify patterns in the training data itself, rather than the association between the training data and the labels corresponding to the training data.

Examples of unsupervised learning include clustering or independent component analysis.

In this specification, the term 'clustering' may be used interchangeably with the term 'clustering'.

Examples of artificial neural networks using unsupervised learning include generative adversarial networks (GANs) and auto encoders (AEs).

A generative adversarial neural network may include a machine learning method in which two different artificial intelligences compete to improve performance, a generator and a discriminator.

In this case, the generator is a model that creates new data, and can generate new data based on the original data.

In addition, the discriminator is a model for recognizing patterns in data, and may play a role of discriminating whether input data is original data or new data generated by the generator.

In addition, the generator learns by receiving data that has not been deceived by the discriminator, and the discriminator can learn by receiving data deceived from the generator. Accordingly, the generator can evolve to deceive the discriminator as best as possible, and the discriminator can evolve to distinguish the original data and the data generated by the generator well.

Autoencoders may include neural networks that aim to reproduce the input itself as an output. The auto-encoder may include an input layer, at least one hidden layer, and an output layer. In this case, since the number of nodes in the hidden layer is smaller than the number of nodes in the input layer, the dimension of data is reduced, and thus compression or encoding may be performed.

Also, data output from the hidden layer can enter the output layer. In this case, since the number of nodes of the output layer is greater than the number of nodes of the hidden layer, the dimension of data increases, and accordingly, decompression or decoding may be performed.

On the other hand, the auto-encoder adjusts the neuron's connection strength through learning, so that input data can be expressed as hidden layer data. The hidden layer expresses information with fewer neurons than the input layer, and being able to reproduce the input data as an output may mean that the hidden layer found and expressed hidden patterns from the input data.

Semi-supervised learning is a type of machine learning, and may refer to a learning method using both labeled and unlabeled training data.

As one of the techniques of semi-supervised learning, there is a technique of inferring a label of unlabeled training data and then performing learning using the inferred label. can

Reinforcement learning can include the theory that, given an environment where an agent can decide what action to take at every moment, it can find the best way through experience without data.

Reinforcement learning can be mainly performed by a markov decision process (MDP).

To explain the Markov decision process, first, an environment is given in which the information necessary for the agent to take the next action is given, secondly, how the agent behaves in that environment is defined, and thirdly, the agent is rewarded ( reward) and what penalty points are given for failing to do so, and fourthly, the optimal policy can be derived by repeating experiences until the future reward reaches the highest point.

The structure of the artificial neural network is specified by the model configuration, activation function, loss function or cost function, learning algorithm, optimization algorithm, etc., and hyperparameters are It is set in advance, and then, a model parameter is set through learning and the content can be specified.

For example, factors determining the structure of the artificial neural network may include the number of hidden layers, the number of hidden nodes included in each hidden layer, an input feature vector, a target feature vector, and the like.

The hyperparameter may include several parameters to be initially set for learning, such as initial values of model parameters. And, the model parameter may include several parameters to be determined through learning.

For example, the hyperparameter may include an initial weight value between nodes, an initial value of bias between nodes, a mini-batch size, a number of learning repetitions, a learning rate, and the like. In addition, the model parameters may include inter-node weights, inter-node biases, and the like.

The loss function may be used as an index (reference) for determining the optimal model parameter in the learning process of the artificial neural network. In artificial neural networks, learning refers to the process of manipulating model parameters to reduce the loss function, and the purpose of learning can be seen to determine the model parameters that minimize the loss function.

The loss function may mainly use mean squared error (MSE) or cross entropy error (CEE), but the present disclosure is not limited thereto.

The cross-entropy error can be used when the correct answer label is one-hot encoded. One-hot encoding may include an encoding method in which only the correct answer label value is set to 1 for neurons corresponding to the correct answer and the correct answer label value is set to 0 for neurons that do not have the correct answer.

In machine learning or deep learning, learning optimization algorithms can be used to minimize the loss function, and learning optimization algorithms include gradient descent (GD), stochastic gradient descent (SGD), and momentum (momentum). ), NAG (nesterov accelerate gradient), Adagrad, AdaDelta, RMSProp, Adam, Nadam, and the like.

The gradient descent method may include a technique of adjusting a model parameter in a direction to reduce the loss function value in consideration of the gradient of the loss function in the current state.

The direction in which the model parameter is adjusted may be referred to as a step direction, and the size to be adjusted may be referred to as a step size.

In this case, the step size may mean a learning rate.

In the gradient descent method, a gradient may be obtained by partial differentiation of the loss function into each model parameter, and the model parameters may be updated by changing the learning rate in the obtained gradient direction.

The stochastic gradient descent method may include a technique in which the frequency of gradient descent is increased by dividing the training data into mini-batch and performing the gradient descent method for each mini-batch.

Adagrad, AdaDelta, and RMSProp may include techniques to increase optimization accuracy by adjusting the step size in SGD. In SGD, momentum and NAG may include a technique to increase optimization accuracy by adjusting the step direction. Adam can include a technique to increase optimization accuracy by adjusting the step size and step direction by combining momentum and RMSProp. Nadam may include a technique to increase optimization accuracy by adjusting the step size and step direction by combining NAG and RMSProp.

The learning speed and accuracy of the artificial neural network may include features that largely depend on hyperparameters as well as the structure of the artificial neural network and the type of learning optimization algorithm. Therefore, in order to obtain a good learning model, it may be important not only to determine an appropriate artificial neural network structure and learning algorithm, but also to set appropriate hyperparameters.

In general, the hyperparameter can be set to an optimal value that provides a stable learning speed and accuracy as a result of training an artificial neural network by experimentally setting it to various values.

The neural network model to which the artificial intelligence technology as described above is applied may first be generated through a training step by a training calculation system (not shown), and then stored in the server 300 and then stored in the washing machine 100 through the network 400 . may have been sent to

The neural network model is a learning model trained to analyze an image inside a container or a washing tub of the washing machine 100, a learning model trained to determine the operating state of the washing machine 100, or to determine the amount of laundry such as the volume of laundry It may be a model trained for Here, the training method may be performed using supervised learning and unsupervised learning.

Typically, such a neural network model may be stored in the washing machine 100 in a state that can be applied during the operation of the washing machine 100 after the training step in the training calculation system is completed, but in some embodiments, the neural network model is the washing machine ( 100) may also be updated or upgraded through additional training.

Meanwhile, the neural network model stored in the washing machine 100 may be some of the models generated by the training calculation system, and if necessary, new neural network models may be generated in the training calculation system and transmitted to the washing machine 100 . .

As another example, the neural network models are stored in the server 300 instead of being stored in the laundry machine 100 , and may provide a function necessary for the laundry machine 100 in the form of a web service.

The user terminal 200 may receive a service for driving or controlling the washing machine 100 through an authentication process after accessing the washing machine driving application or the washing machine driving site. In this embodiment, the user terminal 200 that has completed the authentication process may drive the washing machine 100 and control the operation of the washing machine 100 .

In this embodiment, the user terminal 200 is a desktop computer, a smartphone, a notebook computer, a tablet PC, a smart TV, a mobile phone, a personal digital assistant (PDA), a laptop, a media player, a micro server, a global positioning (GPS) operated by a user. system) devices, e-book terminals, digital broadcast terminals, navigation devices, kiosks, MP4 players, digital cameras, home appliances, and other mobile or non-mobile computing devices, but is not limited thereto. Also, the user terminal 200 may be a wearable terminal such as a watch, glasses, a hair band, and a ring having a communication function and a data processing function. The user terminal 200 is not limited to the above description, and a terminal capable of web browsing may be borrowed without limitation.

The server 300 may be a database server that provides big data necessary for applying various artificial intelligence algorithms and data processed by the request of the laundry apparatus 100 to the laundry apparatus 100 . In addition, the server 300 may include a web server or an application server for remotely controlling the operation of the washing machine 100 using a laundry machine driving application installed in the user terminal 200 or a laundry machine driving web browser. can

The server 300 may receive the first laundry information and the second laundry information from the laundry device 100 and determine the properties of laundry. The server 300 may determine a washing process by using the attribute determination result of the laundry and transmit it to the washing apparatus 100 . The server 300 may execute an artificial intelligence (AI) algorithm and/or a machine learning algorithm in a 5G communication environment to determine the properties of laundry and determine a washing process.

The server 300 may have a larger processing power and a larger memory capacity than the laundry machine 100 , which is a household appliance in general. Therefore, according to the system implementation, a heavy neural network model requiring more processing power for application is stored in the server 300 , and a lightweight neural network model requiring less processing power for application is stored in the washing machine 100 . It may be configured to be stored.

The network 400 may serve to connect the washing machine 100 , the user terminal 200 , and the server 300 . The network 400 is, for example, wired networks such as local area networks (LANs), wide area networks (WANs), metropolitan area networks (MANs), integrated service digital networks (ISDNs), wireless LANs, CDMA, Bluetooth, and satellite communication. It may cover a wireless network such as, but the scope of the present invention is not limited thereto. Also, the network 400 may transmit/receive information using short-distance communication and/or long-distance communication. Here, the short-distance communication may include Bluetooth, radio frequency identification (RFID), infrared data association (IrDA), ultra-wideband (UWB), ZigBee, and wireless fidelity (Wi-Fi) technologies. Communication may include code division multiple access (CDMA), frequency division multiple access (FDMA), time division multiple access (TDMA), orthogonal frequency division multiple access (OFDMA), single carrier frequency division multiple access (SC-FDMA) technology. can

Network 400 may include connections of network elements such as hubs, bridges, routers, switches, and gateways. Network 400 may include one or more connected networks, eg, multiple network environments, including public networks such as the Internet and private networks such as secure enterprise private networks. Access to network 400 may be provided via one or more wired or wireless access networks. Furthermore, the network 400 may support an Internet of Things (IoT) network and/or 5G communication that exchanges and processes information between distributed components such as things.

2 is an external perspective view of the washing machine according to the present embodiment. In the following description, the part overlapping with the description of FIG. 1 will be omitted.

Referring to FIG. 2 , the washing machine 100 includes a main body 101 forming an exterior, a washing tub 102 for containing laundry, and a door 103 installed to open and close a passage through which laundry is introduced into the washing tub 102 . and a vision sensor 131 (camera) disposed on the door 103 to photograph the inside of the washing tub 102 and a user interface (140 in FIG. 3 ) for the user to interact with the washing machine 100, wherein the user It may include input interfaces 141a and 141b for inputting a control command, and an output interface 142 for displaying control information according to the control command input by the user.

The input interfaces 141a and 141b may include a jog shuttle that can determine one of preset washing paths and/or a button that can adjust the time or number of times, and the output interface 142 visually displays laundry-related information. It may include a display to provide. The user may transmit a command to the washing machine 100 through the input interfaces 141a and 141b , and the washing machine 100 may transmit information to the user through the output interface 142 .

The main body 101 is a case that forms the exterior of the washing machine 100, and inside the main body 101, a washing tub (for example, an outer tub and an inner tub), which is a container for storing laundry, a motor rotating the washing tub, and a washing operation are performed. It may include one or more processors to control, a memory connected to the processor, and the like.

Laundry that is put into the main body 101 may be an object to be treated that is put into the washing tub 102 in the main body 101 , and a part of the main body 101 may have laundry put into the washing tub 102 in the main body 101 . An opening may be formed.

In the washing machine 100 , a door 103 for opening and closing a passage through which an article is inserted through the opening of the main body 101 may be rotatably disposed. A vision sensor 131 capable of photographing the inside of the washing tub 102 may be disposed inside the door 103 .

The vision sensor 131 may be disposed to face the inside instead of the outside of the washing machine 100 , and may photograph the washing tub 102 containing laundry. The vision sensor 131 may be selected from among various types of cameras, such as a 2D camera, a 3D camera, a stereo camera, and an infrared camera. In an optional embodiment, the vision sensor 131 may be disposed to face the outside rather than the inside of the washing machine 100 , and may photograph a motion of a user existing around the washing machine 100 .

Meanwhile, although not shown in FIG. 2 , the washing machine 100 may include a light disposed to illuminate the inside of the washing tub 102 , which is a container. The light is turned on when the door 103 is closed after the laundry is put in, so that an image inside the washing tub 102 can be captured more clearly.

Meanwhile, the washing machine 100 may further include a weight detection sensor ( 132 in FIG. 3 ) capable of detecting the weight of the laundry contained in the washing tub 102 . The weight sensor 132 may be a load sensor that measures a change in the weight of the washing tub 102 . In addition, the washing machine 100 further includes a current sensing sensor (133 in FIG. 3 ) for detecting a driving current of a motor (not shown) that rotates the washing tub 102 in response to the weight of the laundry contained in the washing tub 102 . can do.

3 is a block diagram schematically illustrating the configuration of a washing machine according to the present embodiment. In the following description, descriptions of parts overlapping with those of FIGS. 1 and 2 will be omitted. Referring to FIG. 3 , the washing machine 100 includes a communication unit 110 , a driving unit 120 , a sensor 130 , a user interface 140 , an audio processing unit 150 , an information processing unit 160 , a memory 170 and The controller 180 may be included.

The communication unit 110 may provide a communication interface necessary to provide a transmission/reception signal between the laundry apparatus 100 , the user terminal 200 and/or the server 300 in the form of packet data in connection with the network 400 . Furthermore, the communication unit 110 may serve to receive a predetermined information request signal from the user terminal 200 , and may serve to transmit information processed by the washing machine 100 to the user terminal 200 . there is. In addition, the communication unit 110 transmits a predetermined information request signal from the washing machine 100 and/or the user terminal 200 to the server 300, and receives the response signal processed by the server 300, 100) and/or to the user terminal 200 . In addition, the communication unit 110 may be a device including hardware and software necessary for transmitting and receiving signals such as control signals or data signals through wired/wireless connection with other network devices.

In addition, the communication unit 110 may support various things intelligent communication (Internet of things (IoT), Internet of everything (IoE), Internet of small things (IoST), etc.), and M2M (machine to machine) communication, V2X ( Vehicle to everything communication) communication, D2D (device to device) communication, etc. may be supported.

The driving unit 120 may adjust water supply and drainage according to the control signal of the control unit 180 , generate driving force, and transmit the driving force to the washing tub 102 to perform washing in the washing tub 102 .

The sensor 130 may include various sensors for sensing a situation inside and/or around the washing machine 100 , and may include a vision sensor 131 , a weight detection sensor 132 , and a current detection sensor 133 . there is.

The vision sensor 131 may include a camera provided on the door 103 to capture an image of the inside of the washing tub 102 , and a plurality of vision sensors 131 may be installed for photographing efficiency. For example, the camera includes at least one optical lens and an image sensor (eg, CMOS image sensor) configured to include a plurality of photodiodes (eg, pixels) that are imaged by light passing through the optical lens. ) and a digital signal processor (DSP) that configures an image based on signals output from the photodiodes. The digital signal processor may generate a still image as well as a moving picture composed of frames composed of still images. Meanwhile, an image obtained by photographing by a camera as the vision sensor 131 may be stored in the memory 170 .

The weight sensor 132 may detect the weight of laundry put into the washing tub 102 and store it in the memory 170 . The current detection sensor 133 may detect a driving current of a motor driving the washing tub 102 and may obtain information on the weight of the laundry corresponding to the driving current of the motor.

In this embodiment, the sensor 130 is limited to the vision sensor 131 , the weight detection sensor 132 , and the current detection sensor 133 , but is not limited thereto, and the inside and/or surrounding conditions of the washing machine 100 are monitored. For example, although not shown, a sensor capable of detecting a lidar sensor, an illumination sensor, a touch sensor, an acceleration sensor, a magnetic sensor, Gravity sensor (G-sensor), gyroscope sensor, motion sensor, RGB sensor, infrared sensor (infrared sensor), fingerprint sensor (finger scan sensor), ultrasonic sensor ), optical sensor, microphone, battery gauge, environmental sensor (eg barometer, hygrometer, thermometer, radiation sensor, thermal sensor, gas sensor, etc.), chemical sensor (eg, an electronic nose, a healthcare sensor, a biometric sensor, etc.). Meanwhile, in the present embodiment, the washing machine 100 may combine and utilize information sensed by at least two or more of these sensors.

The input interface 141 of the user interface 140 may include a plurality of operation buttons and/or jog shuttles, and may transmit signals corresponding to the plurality of operation buttons and/or jog shuttles to the controller 180 . The input interface 141 may be configured as a sensor, button, or switch structure capable of recognizing a user's touch or press operation. In this embodiment, the input interface 141 may transmit an operation signal operated by the user to the control unit 180 to check or change various information related to the operation of the washing machine 100 displayed on the output interface 142 . there is.

The output interface 142 of the user interface 140 may display the driving state of the washing machine 100 under the control of the controller 180 . According to an embodiment, the output interface 142 may be configured as a touch screen by forming a layer structure with the touch pad. In this case, the output interface 142 may also be used as the input interface 141 capable of inputting information by a user's touch. To this end, the output interface 142 may be configured as a touch-sensitive display controller or other various input/output controllers. As an example, a touch-sensitive display controller may provide an output interface and an input interface between the device and the user. The touch recognition display controller may transmit and receive electrical signals to and from the controller 180 . In addition, the touch-aware display controller displays a visual output to the user, and the visual output may include text, graphics, images, video, and combinations thereof. Such an output interface 142 may be, for example, a predetermined display member such as an organic light emitting display (OLED), a liquid crystal display (LCD), or a light emitting display (LED) capable of recognizing a touch.

The audio input unit 151 of the audio processing unit 150 may receive the user's uttered voice (eg, starting words and uttered sentences (voice command)) and transmit it to the controller 180, and the controller 180 receives the user's uttered voice. By processing, the operation of the washing machine 100 may be controlled. To this end, the audio input unit 151 may include one or more microphones (not shown). In addition, a plurality of microphones (not shown) may be provided to more accurately receive the user's spoken voice. Here, each of the plurality of microphones may be disposed to be spaced apart from each other, and may process the received user's uttered voice as an electrical signal.

In an optional embodiment, the audio input unit 151 may use various noise removal algorithms for removing noise generated in the process of receiving the user's spoken voice. In an optional embodiment, the audio input unit 151 includes various components for processing a voice signal, such as a filter (not shown) that removes noise when receiving a user's spoken voice, and an amplifier (not shown) that amplifies and outputs a signal output from the filter. may include

The audio output unit 152 of the audio processing unit 150 includes a notification message such as a warning sound, an operation mode, an operation state, and an error state, response information corresponding to the user's utterance information, and a user's voice command according to the control of the controller 180 . It is possible to output the processing result and the like corresponding to the . The audio output unit 152 may convert the electrical signal from the control unit 180 into an audio signal and output it. For this, a speaker or the like may be provided.

The information processing unit 160 may acquire first laundry information as an image inside the washing tub 102 based on the vision sensor 131 photographing the inside of the washing tub 102 . The information processing unit 160 may acquire the second laundry information as weight information of the laundry included in the washing tub 102 based on the weight detection sensor 132 that detects the weight inside the washing tub 102 . The information processing unit 160 may determine the attributes of laundry by using the first laundry information and the second laundry information. The information processing unit 160 may determine the washing process of the washing machine 100 based on the attribute of the laundry.

In this embodiment, the information processing unit 160 may perform learning in conjunction with the control unit 180 or receive a learning result from the control unit 180 . In this embodiment, the information processing unit 160 may be provided outside the control unit 180 as shown in FIG. 3 , or may be provided inside the control unit 180 to operate like the control unit 180 , or the server of FIG. 1 . 300 may be provided inside. Hereinafter, details of the information processing unit 160 will be described with reference to FIGS. 4 to 7 .

The memory 170 stores various types of information necessary for the operation of the washing machine 100 , and may include a volatile or nonvolatile recording medium. For example, the memory 170 may store a preset starting word for determining the existence of the starting word from the user's spoken voice. Meanwhile, the starting word may be set by the manufacturer. For example, "high LG" may be set as the starting word, and the setting may be changed by the user. This start word is input to activate the washing machine 100 , and the washing machine 100 recognizing the start word uttered by the user may switch to the voice recognition activation state.

In addition, the memory 170 may store the user's spoken voice (startup word and uttered voice) received through the audio input unit 151 , and may store information detected by the sensor 130 . In addition, the memory 170 may store various types of information necessary for the operation of the washing machine 100 and control software capable of operating the washing machine 100 .

In addition, the memory 170 determines the properties of laundry by using a command to be executed by the information processing unit 160, for example, a command to obtain first laundry information and second laundry information, first laundry information, and second laundry information command, a command for determining a washing process of the washing machine 100 based on the property of laundry, and the like may be stored. Also, the memory 170 may store various types of information processed by the information processing unit 160 .

Here, the memory 170 may include magnetic storage media or flash storage media, but the scope of the present invention is not limited thereto. Such memory 170 may include internal memory and/or external memory, volatile memory such as DRAM, SRAM, or SDRAM, one time programmable ROM (OTPROM), PROM, EPROM, EEPROM, mask ROM, flash ROM, Non-volatile memory, such as NAND flash memory, or NOR flash memory, SSD. It may include a flash drive such as a compact flash (CF) card, an SD card, a Micro-SD card, a Mini-SD card, an Xd card, or a memory stick, or a storage device such as an HDD.

Here, simple voice recognition may be performed by the washing machine 100 , and high-level voice recognition such as natural language processing may be performed by the server 300 . For example, when the user's spoken voice includes only a preset start word, the washing machine 100 may activate the voice recognition function and switch to the ready state for receiving the spoken message. In this case, the washing machine 100 may perform only a voice recognition process up to whether or not a starting word is inputted, and voice recognition for a subsequent utterance may be performed through the server 300 . Since the system resources of the washing machine 100 are limited, complex natural language recognition and processing may be performed through the server 300 .

The controller 180 may control the overall operation of the washing machine 100 . The controller 180 receives a user's command through the input interface 141 and/or the audio input unit 151 and outputs the user's command processing result through the output interface 142 and/or the audio output unit 152 . can do.

The controller 180 is a kind of central processing unit, and may control the operation of the washing machine 100 as a whole by driving the control software mounted on the memory 170 . The controller 180 may include all kinds of devices capable of processing data, such as a processor. Here, the 'processor' may refer to a data processing device embedded in hardware, for example, having a physically structured circuit to perform a function expressed as a code or an instruction included in a program. As an example of the data processing apparatus embedded in the hardware as described above, a microprocessor, a central processing unit (CPU), a processor core, a multiprocessor, an application-specific integrated (ASIC) circuit) and a processing device such as a field programmable gate array (FPGA), but the scope of the present invention is not limited thereto.

4 is a block diagram schematically illustrating the configuration of an information processing unit in the washing machine of FIG. 3 , FIG. 5 is a view showing the inside of the washing tub of the washing machine according to the present embodiment, and FIG. 6 is this embodiment It is an exemplary view of determining the volume information of laundry according to the laundry put into the washing tub according to the present invention, and FIG. 7 is an exemplary view of the attribute determination result of laundry according to the present embodiment. In the following description, parts overlapping with those of FIGS. 1 to 3 will be omitted.

4 to 7 , the information processing unit 160 includes a first obtaining unit 161 , a second obtaining unit 162 , a determining unit 163 , a determining unit 164 , and an output unit 165 . can do. In an optional embodiment, the information processing unit 160 may include one or more processors. In an optional embodiment, the first acquisition unit 161 to the output unit 165 may correspond to one or more processors. In an optional embodiment, the first acquisition unit 161 to the output unit 165 may correspond to software components configured to be executed by one or more processors.

The first acquisition unit 161 may acquire first laundry information as an image inside the washing tub based on the vision sensor 131 . The first acquisition unit 161 may acquire the first laundry information as an image including the feature shapes 102a to 102f inside the washing tub 102 .

5A is a view showing an image inside the washing tub 102 . The washing tub 102 of the washing machine 100 may be captured by the vision sensor 131 as an image as shown in FIG. 5A . The washing tub 102 rotates during washing, rinsing, and dehydration processes, and may have a symmetrical shape about a rotation axis as shown in FIG. 5A .

Meanwhile, although not shown in FIG. 5A , the washing tub 102 is surrounded by an outer tub, and since the washing tub 102 has a plurality of holes on the bottom and side surfaces, the washing tub 102 enters the washing tub 102 . Water can pass through the holes to an outer tub. Due to the above characteristics, most of the washing tubs 102 have repeated characteristic shapes.

FIG. 5B is a diagram illustrating characteristic shapes captured in the washing tub 102 . In the image inside the washing tub 102 , the feature shapes 102a , 102b , 102c , 102d , 102e , 102f can be found as described above. The feature shapes 102a , 102b , 102c , 102d , 102e , 102f may all be the same shapes of the first shape, and only some may be the same shapes of the first shape.

An image analysis neural network model may be used to identify features in an image inside the washing tub 102 . In an embodiment, such an image analysis neural network model may be a model trained in advance to detect repeated specific shapes. In another embodiment, the image analysis neural network model may be a pre-trained model to detect a shape specified by a developer or a user.

The image analysis neural network model may extract repeated feature shapes 102a, 102b, 102c, 102d, 102e, 102f from the image inside the washing tub 102, and record the location and/or number of the extracted feature shapes. .

Meanwhile, in FIG. 5B , six characteristic shapes are exemplarily displayed, but the number of the characteristic shapes may vary depending on the shape of the washing tub 102 and the method of capturing the characteristic shapes.

5C is a view for explaining the correlation between the feature shapes captured in the washing tub 102 and the volume of the laundry. The feature shapes identified in FIG. 5B may be associated with the volume of the laundry. For example, as shown in FIG. 5C , six feature shapes may be referred to as A-1, A-2, B-1, B-2, C-1, C-2.

When all of the six feature shapes are exposed, the washing tub 102 may be associated with an empty state without laundry.

When some of A-1 and A-2 and B-1, B-2, C-1, and C-2 among all six feature shapes are exposed, the volume of laundry is level 1, and one of the six feature shapes is When A-1 and A-2 are hidden, the volume of laundry is level 2, when A-1, A-2, B-1, and B-2 are hidden, the volume of laundry is level 3, A- If 1, A-2, B-1, B-2, C-1, C-2 are concealed, the volume of laundry can be associated with level 4. Here, the higher the number of levels, the greater the volume (amount) of laundry.

The image analysis neural network model may identify a portion having a predetermined feature shape in the image inside the washing tub 102 , and estimate the amount of laundry according to which feature shapes among all the feature shapes are identified.

For example, when A-1, A-2, B-1, B-2, C-1, and C-2 are exposed in the image inside the washing tub 102, the neural network model may determine that the washing tub is empty. .

When parts of A-1 and A-2 and B-1, B-2, C-1, and C-2 are exposed in the image inside the washing tub 102, the neural network model determines that level 1 laundry is present in the washing tub. can do.

When B-1, B-2, C-1, and C-2 are exposed in the image inside the washing tub 102, the neural network model may determine that level 2 laundry is present in the washing tub.

When C-1 and C-2 are exposed in the image inside the washing tub 102 , the neural network model may determine that level 3 laundry is present in the washing tub 102 .

When all of the feature shapes are concealed, that is, not visible at all, in the image inside the washing tub 102 , the neural network model may determine that level 4 laundry is present in the washing tub 102 .

Conversely, the neural network model may determine the volume of laundry based on concealed feature shapes.

On the other hand, in another embodiment, the level of each laundry volume is associated with a plurality of (eg, two) feature shapes, and the neural network model shows the level of the corresponding level even if only one of the plurality of feature shapes associated with the level is seen. It may be judged that there is laundry.

For example, when A-1 is hidden and A-2 is exposed in the image inside the washing tub 102, the neural network model may determine that level 2 laundry exists.

Conversely, the neural network model may be configured to determine that there is laundry of the corresponding level only when all of the plurality of feature shapes associated with each level of the amount of laundry must be concealed.

Meanwhile, although it has been described in the above description that the image analysis and laundry volume estimation are performed by a neural network model, other types of learning models or algorithms may be used according to embodiments.

In an optional embodiment, the first acquisition unit 161 may include a first image of the inside of the washing tub 102 before the laundry is put into the washing tub 102, and the first image of the inside of the washing tub 102 after the laundry is put into the washing tub 102. 2 It is possible to obtain the first laundry information including the image. Later, the volume of the laundry may be estimated by comparing the first image and the second image.

The second acquisition unit 162 may acquire second laundry information as laundry weight information included in the washing tub 102 based on the weight detection sensor 132 that detects the weight inside the washing tub 102 . .

In an optional embodiment, the second acquisition unit 162 is based on the current detection sensor 133 for detecting the driving current of the motor (not shown) rotating the washing tub 102, the weight of the laundry corresponding to the driving current of the motor Second laundry information as information can be obtained.

The determination unit 163 may determine the attributes of laundry by using the first laundry information and the second laundry information. In this embodiment, the determiner 163 may include a first estimator 163-1, a second estimator 163-2, and an attribute determiner 163-3.

The first estimator 163-1 may estimate the laundry volume information based on the first laundry information. The first estimator 163-1 determines the number of features 102a to 102f exposed by the laundry from the first laundry information as an image including the feature features 102a to 102f inside the washing tub 102, and Volume information of the laundry may be estimated based on the number of feature shapes 102a to 102f hidden by the laundry.

When a certain amount of laundry is placed in the washing tub 102 , some feature shapes are hidden in the washing tub 102 internal image as shown in FIG. 6 . The image analysis neural network model may determine the volume of laundry based on concealment or exposure of feature features.

Referring to FIG. 6A , A-1 and A2 are hidden, B-1, B-2, C-1, and C-2 are exposed, and the first estimator 163-1 or the neural network model determines the volume of laundry. can be determined to be level 2 (small).

Referring to FIG. 6B , A-1, A2, B1, and B2 are hidden, C-1 and C-2 are exposed, and the first estimator 163-1 or the neural network model indicates that the volume of laundry is level 3 ( middle) can be considered.

Referring to FIG. 6C , A-1, A2, B1, B2, C-1, and C-2 are all hidden, so that the first estimator 163-1 or the neural network model shows that the volume of laundry is level 4 (large). can be judged that

The reason why the features below are hidden according to the amount of laundry in FIGS. 6A to 6C is that the present embodiment relates to a drum washing machine and gravity acts in the downward direction of the image.

On the other hand, when the present disclosure is used in a washing machine, the entire floor surface is covered even with a small amount of laundry. will be.

In an optional embodiment, the first estimator 163-1 may include a first image of the inside of the washing tub 102 before the laundry is put into the washing tub 102 and the inside of the washing tub 102 after the laundry is put into the washing tub 102 . may compare the second image of and estimate the volume information of the laundry based on the comparison result of the first image and the second image. Since the volume information of the first image is zero, the first estimator 163-1 may estimate the volume information of the laundry included in the second image based on the volume information of the first image.

The second estimator 163-2 performs laundry based on the laundry volume information estimated by the first estimator 163-1 and the laundry weight information as the second laundry information acquired by the second acquirer 162. The density (mass (weight)/volume) information of can be estimated.

The attribute determination unit 163-3 may determine the attribute of the laundry based on the density information of the laundry. In this embodiment, the laundry property may include a first property and a second property. FIG. 7A shows a pre-wash as laundry having a first property, and FIG. 7B shows a duvet as laundry having a second property.

The first attribute may include a low volume and heavy load (eg, pre-washing). Here, the small volume and heavy weight may be laundry with high density.

The small volume means that A-1 and A-2 of the six feature shapes are hidden from the inside image of the washing tub 102, B-1, B-2, C-1, C-2 are exposed, or 6 Among the features, some of A-1 and A-2 are hidden, and the rest of A-1 and A-2 and B-1, B-2, C-1, and C-2 are exposed. can do.

The heavy weight refers to a case in which the weight of the laundry relative to the volume of the laundry is equal to or greater than the first reference value, and may include a case in which the laundry is a wet cloth. Here, the wet cloth (濕布) may include wet laundry put into the washing tub 102 after hand washing or preliminary washing is performed. A poultice can be heavier than a raisin. For example, when B-1, B-2, C-1, and C-2 among the six feature shapes are exposed and the volume information of the laundry is <small>, whereas the weight information of the laundry is <weight>, It can be seen that the laundry is a wet cloth.

The second attribute may include a special load (eg, a cotton blanket, a padded jumper, etc.) that is bulky and light in weight. Here, the large volume and light weight may be laundry with a low density.

The bulkiness may include a case in which all six feature shapes A-1, A-2, B-1, B-2, C-1, and C-2 are hidden from the washing tub 102 internal image. there is.

Light weight refers to a case in which the weight of the laundry relative to the volume of the laundry is equal to or less than the second reference value, and may include a case in which the laundry is dry cloth. Here, raisins (乾布) may include dry laundry put into the washing tub 102 without hand washing or primary washing. Raisins can be lighter in weight than poultices. For example, all of the six feature shapes A-1, A-2, B-1, B-2, C-1, and C-2 are hidden so that the volume information of the laundry is <large>, whereas the weight of the laundry is If the information is <lightweight>, it can be seen that the laundry is raisin.

The poultice may be wet because it has been hand-washed or pre-washed. The raisin may not hold any moisture at all because it has not been hand washed or prewashed, or it may be barely wet because it is in a low state. Optionally, raisins may contain laundry that is slightly wet to the extent that it may occur in daily life.

In this embodiment, the attribute determining unit 163-3 determines whether the laundry is wet or dry compared to the volume of laundry by setting the reference weight when the laundry is a wet cloth and a reference weight when the laundry is a dry cloth according to the volume of laundry. can do.

In an optional embodiment, the attribute determination unit 163-3 compares the first image and the second image, the laundry included in the first image compared to the second image is less than or equal to the first reference level (for example, 30%), The laundry determined as a wet cloth from the weight information may be determined as laundry having the first attribute.

In addition, as a result of the comparison of the first image and the second image, the attribute determination unit 163-3 determines that the laundry included in the second image compared to the first image is greater than or equal to the second reference level (eg, 80%), and the weight information The laundry determined to be dry cloth may be determined as laundry having the second attribute.

In the present embodiment, the attribute determiner 163-3 classifies the attributes of laundry into the first attribute and the second attribute, but is not limited thereto and may include various attributes based on the result of estimating the density of the laundry. For example, the properties of medium-volume and medium-weight laundry may be classified as normal laundry. Here, the medium volume may include a case in which C-1 and C-2 among the six feature shapes are exposed. As an optional embodiment, the medium volume may include a case in which the laundry included in the second image exceeds the first reference level and is less than the second reference level as a result of comparing the first image with the second image. In addition, the middle weight may include a case in which the weight of the laundry relative to the volume of the laundry is between the first reference value and the second reference value.

In an alternative embodiment, the attribute determining unit 163-3 may determine whether the laundry is a wet cloth or dry cloth based on the first laundry information (image information of the laundry). For example, the wet cloth may have a more vivid color than the dry cloth, or the wet cloth may contain moisture in the washing tub 102 . Accordingly, it is possible to determine whether the laundry put into the washing tub 102 is a wet cloth or a dry cloth based on the first laundry information.

In an optional embodiment, the determination unit 163 may determine the type of laundry. The determination unit 163 may be more sophisticated through an object recognition model. In addition to the laundry density estimated in the above-described manner, the type of laundry may be more accurately determined through object recognition of the laundry image captured by the vision sensor 131 .

For example, if the density of laundry is high and the color of the laundry based on the image captured by the vision sensor 131 is blue and the type is estimated as pants, the laundry estimation algorithm may estimate that the laundry is jeans.

As another example, if the object identification technology through the image taken when the density of the laundry is low, it can be more accurately determined whether the laundry with the low density is a padding or a blanket.

Accordingly, the laundry apparatus 100 may transmit to the user terminal 200 that the input laundry is determined as a leather jacket and/or that the laundry mode is selected as laundry for leather, and may receive feedback from the user.

The determination unit 164 may determine the washing process of the washing machine based on the laundry property determined by the determination unit 163 . Here, determining the washing process may include determining a washing operation and washing factors. Determination of washing strokes may include, for example, determination of a course (eg, standard course, preparatory course, duvet course, etc.) and frequency of washing-rinsing-drying. The determination of the washing factor may include determining the amount of water supplied during the washing operation (eg, large, medium, small), the amount of detergent, the administration time, and the like.

In the case of the laundry having the first attribute from the above-described example, since the primary washing is performed, the washing cycle is set as the primary course, so that washing-rinsing-drying may be simpler than the standard course. For example, when the standard course is soaking-washing-rinsing-drying-rinsing-drying, it may be laundry-rinsing-drying-rinsing-drying in the case of laundry having the first property. In addition, the amount of water supplied, the amount of detergent, and the administration time included in the washing factor may be different from the standard course, for example, the amount of water supplied and the amount of detergent may be smaller than that of the standard course, and the stroke time (washing time) may be shorter than that of the standard course.

In addition, in the case of laundry having the second property, since it is a bulky raisin such as a cotton duvet or a padded jumper, the washing cycle is set to the duvet course, and washing-rinsing-drying may be more complicated than the standard course. For example, when the standard course is soaking-washing-rinsing-drying-rinsing-drying, in the case of laundry having the second property, it may be soaking-washing-rinsing-drying-rinsing-drying-rinsing-drying. In addition, the amount of water supplied, the amount of detergent, and the administration time included in the washing factor may be different from the standard course. For example, the amount of water and detergent included in the washing factor is greater than that of the standard course, and the administration time (soaking time, washing time, rinsing time, and spin-drying time) It can be longer than this standard course.

In an optional embodiment, the washing machine 100 uses another deep neural network model trained in advance to determine the washing process of the washing machine using the density (volume and weight) of the laundry put into the washing tub, 100) can be determined. Here, another deep neural network model may be pre-trained using training data including densities (volume and weight) of various laundry put into the washing tub and a washing process suitable for the density of each laundry.

The output unit 165 may output detergent input amount guide information based on the washing process determination result. In the case of a washing machine in which detergent is not automatically added, the user must directly input the detergent. In this case, the output unit 165 outputs the detergent input amount guide information to the output interface 142 and/or audio based on the washing process determination result. It may output to the output unit 152 and/or the user terminal 200 . Detergent information used by a user to output detergent input amount guide information may be stored in the washing machine 100 or the user terminal 200 .

In addition, the output unit 165 outputs the laundry property determination result and the laundry process determination result to the output interface 142 and/or the audio output unit 152 and/or the user terminal 200 before the washing machine 100 is driven. can be printed out.

8 is a view for explaining a deep neural network model for estimating the volume of laundry according to the present embodiment.

In some embodiments, an initial neural network model may be prepared to generate a neural network model capable of determining the volume of laundry by analyzing an image, and training data capable of training the neural network model may be prepared.

A neural network model that can be used here may generally be a convolutional neural network.

The training data may include an image of the inside of the washing tub 102 before the laundry is put in, images of the inside of the washing tub 102 into which various amounts of laundry are put, and data labeled by the volume of laundry for each image.

The neural network model that has undergone supervised learning using such training data may estimate the volume of laundry in the washing tub 102 when an image of the washing tub 102 is input.

In another embodiment, an algorithm capable of determining the volume of laundry by analyzing the image extracts feature shapes from the image inside the empty washing tub 102, and associates the concealment or exposure state of the features with the volume of the laundry, and then When the laundry is put into the washing tub 102, it may be configured to determine the volume of laundry based on the hidden features or exposed features in the image.

On the other hand, the image analysis neural network model and algorithm may be configured to determine the amount of laundry based on the number of features identified in the image inside the washing tub 102 photographed by the vision sensor 131 .

For example, if only four feature shapes are identified as shown in FIG. 6A after laundry is put into the washing tub 102 set to have all six feature shapes, it can be determined that the laundry has a volume of level 2.

As another example, if only two characteristic shapes are identified as shown in FIG. 6B after the laundry is put into the washing tub 102 set to have all 6 characteristic shapes, the laundry may be determined to have a volume of level 3.

When the volume of laundry is determined in the manner described above, the laundry apparatus 100 determines the laundry property corresponding to the density of the laundry by using the determined volume and the weight of the laundry, and washes the laundry appropriately to the laundry property. process can be determined.

9 is a flowchart illustrating a method of driving a washing machine according to the present embodiment. In the following description, descriptions of parts overlapping with those of FIGS. 1 to 8 will be omitted.

Referring to FIG. 9 , in step S910 , the washing apparatus 100 acquires first laundry information as an image inside the washing tub, based on the vision sensor 131 photographing the inside of the washing tub 102 . The washing machine 100 may acquire the first laundry information as an image including the feature shapes 102a to 102f inside the washing tub 102 . In an optional embodiment, the washing machine 100 includes a first image of the inside of the washing tub 102 before the laundry is put into the washing tub 102 and a second image of the inside of the washing tub 102 after the laundry is put into the washing tub 102 . It is possible to obtain first laundry information including.

In step S920 , the washing machine 100 acquires second laundry information as laundry weight information included in the washing tub 102 based on the weight detection sensor 132 that detects the weight inside the washing tub 102 . . In an optional embodiment, the laundry device 100 is based on the current detection sensor 133 for detecting the driving current of the motor (not shown) rotating the washing tub 102, the weight information of the laundry corresponding to the driving current of the motor It is possible to obtain the second laundry information as

In step S930, the laundry apparatus 100 determines the properties of laundry by using the first laundry information and the second laundry information. The laundry apparatus 100 may estimate the volume information of the laundry in order to determine the properties of the laundry, and estimate the density information of the laundry using the volume information of the laundry and the weight information of the laundry. The laundry apparatus 100 may determine the attribute of the laundry by using the density information of the laundry.

The laundry apparatus 100 may estimate the laundry volume information based on the first laundry information. The washing machine 100 determines the number of exposed feature shapes 102a to 102f of the washing tub 102 into which laundry is placed from the image including the feature shapes 102a to 102f inside the washing tub or the washing tub hidden by the laundry ( The volume information of the laundry may be estimated based on the number of the feature shapes 102a to 102f of 102).

As an optional embodiment, the washing machine 100 uses a deep neural network model trained in advance to estimate the volume of laundry using images including feature shapes inside the washing tub, and collects information about the volume of laundry put into the washing tub 102 . can be estimated Here, the deep neural network model is a CNN-based model, and a neural network model trained in a supervised learning method using images of a plurality of washing tubs in a state in which various amounts of laundry are labeled for each image as training data. can

In an optional embodiment, the washing machine 100 includes a first image of the inside of the washing tub 102 before the laundry is put into the washing tub 102 and a second image of the inside of the washing tub 102 after the laundry is put into the washing tub 102 . can be compared, and the volume information of the laundry can be estimated based on the comparison result of the first image and the second image. Here, since the volume information of the first image is zero, it is possible to estimate the volume information of the laundry included in the second image based on the volume information of the first image.

The laundry apparatus 100 may estimate the density information of the laundry by using the volume information of the laundry and the weight information of the laundry, and determine the properties of the laundry. In this embodiment, the laundry property may include a first property and a second property. The first attribute may include a load condition with a small volume and heavy weight (eg, pre-washing), and the second attribute may include a special load with a large volume and light weight (eg, cotton quilt, padded jumper, etc.). may include In the present embodiment, the laundry apparatus 100 classifies the laundry property into the first property and the second property, but is not limited thereto and may include various properties based on the result of estimating the density of the laundry.

In step S940 , the washing machine 100 determines a washing process of the washing machine based on the properties of the laundry. Here, determining the washing process may include determining a washing operation and washing factors. Determination of washing strokes may include, for example, determination of a course (eg, standard course, preparatory course, duvet course, etc.) and frequency of washing-rinsing-drying. The determination of the washing factor may include determination of a water supply amount (eg, large, medium, small), water supply temperature, detergent amount, and administration time during a washing operation.

As an optional embodiment, the laundry apparatus 100 may output detergent input amount guide information based on the laundry process determination result, and before driving the laundry apparatus 100 , output the laundry property determination result and the laundry process determination result there is.

The embodiment according to the present invention described above may be implemented in the form of a computer program that can be executed through various components on a computer, and such a computer program may be recorded in a computer-readable medium. In this case, the medium includes a hard disk, a magnetic medium such as a floppy disk and a magnetic tape, an optical recording medium such as CD-ROM and DVD, a magneto-optical medium such as a floppy disk, and a ROM. , RAM, flash memory, and the like, hardware devices specially configured to store and execute program instructions.

Meanwhile, the computer program may be specially designed and configured for the present invention, or may be known and used by those skilled in the computer software field. Examples of the computer program may include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like.

In the specification of the present invention (especially in the claims), the use of the term "above" and similar referential terms may be used in both the singular and the plural. In addition, when a range is described in the present invention, each individual value constituting the range is described in the detailed description of the invention as including the invention to which individual values belonging to the range are applied (unless there is a description to the contrary). same as

The steps constituting the method according to the present invention may be performed in an appropriate order, unless there is an explicit order or description to the contrary. The present invention is not necessarily limited to the order in which the steps are described. The use of all examples or exemplary terms (eg, etc.) in the present invention is merely for the purpose of describing the present invention in detail, and unless defined by the claims, the scope of the present invention is limited by the examples or exemplary terminology. it's not going to be In addition, those skilled in the art will recognize that various modifications, combinations, and changes may be made in accordance with design conditions and factors within the scope of the appended claims or their equivalents.

Therefore, the spirit of the present invention should not be limited to the above-described embodiments, and the scope of the spirit of the present invention is not limited to the scope of the scope of the present invention. will be said to belong to

100: washing machine
200: user terminal
300: server
400: network

Claims (20)

A method of driving a washing machine, comprising:
acquiring first laundry information as an image inside the washing tub based on a vision sensor photographing the inside of the washing tub;
obtaining second laundry information as weight information of laundry included in the washing tub based on a weight detection sensor that detects the weight of the inside of the washing tub;
determining an attribute of the laundry by using the first laundry information and the second laundry information; and
determining a washing process of the washing machine based on the attribute of the laundry;
How to drive a washing machine. The method of claim 1,
The step of obtaining the second laundry information,
Comprising the step of obtaining the second laundry information as weight information of the laundry corresponding to the driving current of the motor based on a current detection sensor that senses the driving current of the motor rotating the washing tub,
How to drive a washing machine. The method of claim 1,
The step of determining the properties of the laundry,
estimating the volume information of the laundry based on the first laundry information; and
Comprising the step of determining the attribute of the laundry based on the volume information of the laundry and the weight information of the laundry,
How to drive a washing machine. 4. The method of claim 3,
The step of estimating the volume information of the laundry,
Estimating the volume information of the laundry based on the number of exposed feature shapes of the washing tub into which the laundry is placed or the number of feature shapes of the washing tub hidden by the laundry from the image including the feature shapes inside the washing tub containing,
How to drive a washing machine. The method of claim 1,
The step of determining the properties of the laundry,
determining, from the image, laundry having a predetermined number or more of the feature shapes in the washing tub and having the weight information equal to or greater than a predetermined value as laundry having a first attribute; and
Comprising the step of determining, from the image, a laundry having a predetermined number of features in the washing tub exposed to less than a predetermined value and the weight information of which is less than a predetermined value as laundry having a second attribute,
How to drive a washing machine. 4. The method of claim 3,
The step of determining the volume information of the laundry,
Using a deep neural network model trained in advance to estimate the volume of laundry using images including feature shapes inside the washing tub, estimating the volume information of the laundry put into the washing tub,
The deep neural network model is
It is a neural network model trained in advance using training data including images including feature shapes inside the washing tub after various laundry is input and training data including labels for laundry volume information for each image,
How to drive a washing machine. 4. The method of claim 3,
The step of determining the volume information of the laundry,
comparing the first image inside the washing tub before the laundry is put into the washing tub and the second image inside the washing tub after the laundry is put into the washing tub;
Comprising the step of estimating the volume information of the laundry based on the comparison result of the first image and the second image,
How to drive a washing machine. 8. The method of claim 7,
The step of determining the properties of the laundry,
As a result of comparing the first image and the second image, the laundry included in the first image compared to the second image is less than or equal to the first reference level and the weight information is equal to or greater than a predetermined value, the laundry having the first attribute judging by; and
As a result of comparing the first image and the second image, the laundry included in the second image compared to the first image is the second reference level or higher and the weight information is less than a predetermined value, the laundry having the second attribute comprising the step of judging by
How to drive a washing machine. The method of claim 1,
The step of determining the washing process of the washing machine,
determining a washing operation of the washing machine based on the attribute of the laundry; and
Comprising the step of determining a washing factor of the washing machine based on the determination result of the washing operation of the washing machine,
How to drive a washing machine. The method of claim 1,
Further comprising the step of outputting detergent input amount guide information based on the washing process determination result of the washing machine,
How to drive a washing machine. A washing machine comprising:
a first acquisition unit configured to acquire first laundry information as an image inside the washing tub based on a vision sensor photographing the inside of the washing tub;
a second acquisition unit configured to acquire second laundry information as weight information of laundry included in the washing tub based on a weight detection sensor that detects the weight of the inside of the washing tub;
a determination unit configured to determine an attribute of the laundry by using the first laundry information and the second laundry information; and
Comprising a determining unit for determining a washing process of the washing machine based on the attribute of the laundry,
laundry device. 12. The method of claim 11,
The second acquisition unit,
configured to obtain the second laundry information as weight information of the laundry corresponding to the driving current of the motor based on a current detection sensor that senses a driving current of a motor rotating the washing tub,
laundry device. 12. The method of claim 11,
The judging unit,
a first estimator for estimating the volume information of the laundry based on the first laundry information; and
Comprising a property determination unit for determining the property of the laundry based on the volume information of the laundry and the weight information of the laundry,
laundry device. 14. The method of claim 13,
The first estimation unit,
Configured to estimate the volume information of the laundry based on the number of exposed feature shapes of the washing tub in which the laundry is put or the number of feature shapes of the washing tub hidden by the laundry from the image including the feature shapes inside the washing tub felled,
laundry device. 12. The method of claim 11,
The judging unit,
From the image, a predetermined number or more of the feature shapes inside the washing tub are exposed and the laundry having the weight information of a predetermined value or more is determined as laundry having a first attribute,
configured to determine laundry having less than a certain number of feature shapes in the washing tub from the image and having the weight information less than a certain value as laundry having a second attribute,
laundry device. 14. The method of claim 13,
The first estimation unit,
It is configured to estimate the volume information of the laundry put into the washing tub by using a deep neural network model trained in advance to estimate the volume of the laundry using images including the feature shapes inside the washing tub,
The deep neural network model is
It is a neural network model trained in advance using training data including images including feature shapes inside the washing tub after various laundry is input and training data including labels for laundry volume information for each image,
laundry device. 14. The method of claim 13,
The first estimation unit,
The first image inside the washing tub before the laundry is put into the washing tub and the second image inside the washing tub after the laundry is put into the washing tub are compared, and based on the comparison result of the first image and the second image configured to estimate the volume information of the laundry as
laundry device. 18. The method of claim 17,
The attribute determination unit,
As a result of comparing the first image and the second image, the laundry included in the first image compared to the second image is less than or equal to the first reference level and the weight information is equal to or greater than a predetermined value, the laundry having the first attribute judged by
As a result of comparing the first image and the second image, the laundry included in the second image compared to the first image is the second reference level or higher and the weight information is less than a predetermined value, the laundry having the second attribute constituted to be judged by
laundry device. 12. The method of claim 11,
The determining unit is
and determine a washing cycle of the washing machine based on the attribute of the laundry, and determine a washing factor of the washing machine based on a result of the determination of the washing cycle of the washing machine,
laundry device. 12. The method of claim 11,
Further comprising an output unit for outputting detergent input amount guide information based on the washing process determination result of the washing machine,
laundry device.

Images