US11908175B2 - Electronic device training image recognition model and operation method for same - Google Patents
Electronic device training image recognition model and operation method for same Download PDFInfo
- Publication number
- US11908175B2 US11908175B2 US17/693,095 US202217693095A US11908175B2 US 11908175 B2 US11908175 B2 US 11908175B2 US 202217693095 A US202217693095 A US 202217693095A US 11908175 B2 US11908175 B2 US 11908175B2
- Authority
- US
- United States
- Prior art keywords
- virtual
- image
- dvs
- recognition model
- information
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active, expires
Links
- 238000012549 training Methods 0.000 title claims abstract description 37
- 238000000034 method Methods 0.000 title claims abstract description 33
- 238000012545 processing Methods 0.000 description 15
- 238000013473 artificial intelligence Methods 0.000 description 12
- 238000013528 artificial neural network Methods 0.000 description 12
- 238000010586 diagram Methods 0.000 description 10
- 238000005516 engineering process Methods 0.000 description 8
- 230000005540 biological transmission Effects 0.000 description 5
- 230000008569 process Effects 0.000 description 5
- 238000004891 communication Methods 0.000 description 4
- 238000013527 convolutional neural network Methods 0.000 description 4
- 230000005236 sound signal Effects 0.000 description 4
- 230000008901 benefit Effects 0.000 description 3
- 230000000306 recurrent effect Effects 0.000 description 3
- 230000008859 change Effects 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 239000004973 liquid crystal related substance Substances 0.000 description 2
- 230000001537 neural effect Effects 0.000 description 2
- 238000011017 operating method Methods 0.000 description 2
- 238000012546 transfer Methods 0.000 description 2
- 230000001133 acceleration Effects 0.000 description 1
- 210000003423 ankle Anatomy 0.000 description 1
- 230000002457 bidirectional effect Effects 0.000 description 1
- 238000004140 cleaning Methods 0.000 description 1
- 238000004590 computer program Methods 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 239000004744 fabric Substances 0.000 description 1
- 239000011521 glass Substances 0.000 description 1
- 238000010295 mobile communication Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000004091 panning Methods 0.000 description 1
- 230000002787 reinforcement Effects 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 238000010897 surface acoustic wave method Methods 0.000 description 1
- 230000002123 temporal effect Effects 0.000 description 1
- 239000010409 thin film Substances 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/10—Image acquisition
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
- G06V10/774—Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
- G06V10/7747—Organisation of the process, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/20—Analysis of motion
- G06T7/246—Analysis of motion using feature-based methods, e.g. the tracking of corners or segments
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/20—Analysis of motion
- G06T7/246—Analysis of motion using feature-based methods, e.g. the tracking of corners or segments
- G06T7/251—Analysis of motion using feature-based methods, e.g. the tracking of corners or segments involving models
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
- G06V10/774—Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/94—Hardware or software architectures specially adapted for image or video understanding
- G06V10/945—User interactive design; Environments; Toolboxes
Definitions
- the disclosure relates to an electronic device for training an image recognition model that recognizes an image, and an operating method thereof.
- a dynamic vision sensor (DVS) image captured by a DVS may include pixel data indicating an amount of change in light sensed by a moving object, unlike existing images including image data with respect to the entire region of the image. Therefore, unlike existing images, a DVS image has the advantage of a fast processing speed because an amount of data to be processed is small, and may be utilized in various operations that may be performed by sensing a moving object.
- An image recognition model is an artificial intelligence model for recognizing a DVS image, and may be used, for example, to recognize a moving object captured from a DVS image, and to extract data related to the moving object.
- the image recognition model may be trained based on data related to a plurality of DVS images.
- Embodiments of the disclosure provide an electronic device for training an image recognition model and an operating method thereof.
- Embodiments of the disclosure provide a computer-readable recording medium having recorded thereon a program for executing the method on a computer.
- the disclosure is not limited to the above aspects, and there may be other aspects of the disclosure.
- a method of training an image recognition model including: generating a virtual dynamic vision sensor (DVS) image using a virtual simulator; generating label information including information about a correct answer to a result of recognizing the DVS image by the image recognition model with respect to the DVS image; and training the image recognition model by modifying the image recognition model so that a difference between the result of recognizing the DVS image by the image recognition model and the label information is minimized.
- DVD virtual dynamic vision sensor
- an electronic device configured to train an image recognition model including: a memory storing the image recognition model; and at least one processor configured to: generate a virtual dynamic vision sensor (DVS) image using a virtual simulator, generate label information comprising information about a correct answer to a result of recognizing the DVS image by the image recognition model with respect to the DVS image, and train the image recognition model by modifying the image recognition model so that a difference between the result of recognizing the DVS image by the image recognition model and the label information is minimized.
- DVD virtual dynamic vision sensor
- a recording medium having stored therein a program for performing the method according to the above method.
- FIG. 1 is a block diagram illustrating an example of training an image recognition model according to various embodiments
- FIG. 2 is a block diagram illustrating an example configuration of an electronic device according to various embodiments
- FIG. 3 is a block diagram illustrating an example configuration of an electronic device according to various embodiments
- FIG. 4 is a flowchart illustrating an example method of training an image recognition model according to various embodiments
- FIG. 5 is a diagram illustrating an example of a virtual environment generated by a virtual simulator according to various embodiments
- FIG. 6 is a flowchart illustrating an example method of training an image recognition model based on virtual data according to various embodiments.
- FIG. 7 is a block diagram illustrating an example of an electronic device and an external device according to various embodiments.
- the element when an element is referred to as being “connected to” another element, the element may be “directly connected to” the other element, or the element may also be “electrically connected to” the other element with an intervening element therebetween.
- the element when an element is referred to as “including” or “comprising” another element, unless otherwise stated, the element may further include or comprise yet another element rather than preclude the yet other element.
- the processor may include at least one processor.
- the at least one processor may be a general-purpose processor such as a central processing unit (CPU), an application processor (AP), or a digital signal processor (DSP), a dedicated graphics processor such as a graphics processing unit (GPU) or a vision processing unit (VPU), or an artificial intelligence-dedicated processor such as a neural processing unit (NPU).
- the at least one processor may be controlled to process input data according to a predefined operation rule stored in the memory or an artificial intelligence model.
- the at least one processor is an artificial intelligence-dedicated processor
- the artificial intelligence-dedicated processor may be designed in a hardware structure specialized for processing a specific artificial intelligence model.
- the predefined operation rule or the artificial intelligence model are made through training.
- the expression “made through training” may refer, for example, to an existing artificial intelligence model being trained based on a learning algorithm using a large number of pieces of training data and thus made into a predefined operation rule or an artificial intelligence model, which is set to fulfill an intended feature (or purpose).
- the training may be performed by a device itself, in which artificial intelligence according to the disclosure is performed, or may be performed through a separate server and/or system.
- Examples of the learning algorithm include supervised learning, unsupervised learning, semi-supervised learning, and reinforcement learning, but are not limited thereto.
- An artificial intelligence model may include a plurality of neural network layers.
- Each of the neural network layers has a plurality of weight values and performs a neural network operation through an operation between an operation result of a previous layer and the plurality of weight values.
- the plurality of weight values that the neural network layers have may be optimized by a result of training of the artificial intelligence model. For example, the plurality of weight values may be refined to minimize a loss value or cost value obtained by the artificial intelligence model during a training process.
- An artificial neural network may include a deep neural network (DNN), and may be, for example, a convolutional neural network (CNN), a deep neural network (DNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), or a deep Q-network, but is not limited thereto.
- DNN deep neural network
- CNN convolutional neural network
- DNN deep neural network
- RNN recurrent neural network
- RBM restricted Boltzmann machine
- DBN deep belief network
- BNN bidirectional recurrent deep neural network
- BDN bidirectional recurrent deep neural network
- a deep Q-network but is not limited thereto.
- FIG. 1 is a block diagram illustrating an example of training an image recognition model 130 according to various embodiments.
- an electronic device 1000 for training the image recognition model 130 may obtain a virtual dynamic vision sensor (DVS) image and label information 120 with respect to the virtual DVS image through a virtual simulator 110 , and train the image recognition model 130 based on the obtained virtual DVS image and label information 120 .
- the electronic device 1000 according to an embodiment of the disclosure may train the image recognition model 130 by modifying the image recognition model 130 so that a difference between label information and a result of recognizing the virtual DVS image by the image recognition model 130 is minimized.
- a method of training the image recognition model 130 according to an embodiment of the disclosure may be performed on various types of images as well as a DVS image.
- the image recognition model 130 may be trained based on various types of images and label information generated through the virtual simulator 110 .
- the electronic device 1000 may recognize at least one DVS image using the image recognition model 130 trained according to an embodiment of the disclosure. For example, as a result of recognizing the DVS image using the image recognition model 130 , the electronic device 1000 may obtain information (e.g., size, location, identification information, movement direction, movement speed, state, etc. of an object included in the DVS image) about the DVS image, and perform various operations based on the obtained information.
- information e.g., size, location, identification information, movement direction, movement speed, state, etc. of an object included in the DVS image
- the electronic device 1000 may be implemented in various forms.
- the electronic device 1000 described herein may include a digital camera, a smart phone, a laptop computer, a tablet PC, an electronic book terminal, a digital broadcasting terminal, a personal digital assistant (PDA), a portable multimedia player (PMP), a navigation system, an MP3 player, etc., but is not limited thereto.
- PDA personal digital assistant
- PMP portable multimedia player
- MP3 player etc.
- the electronic device 1000 described herein may include a wearable device that may be worn by a user.
- the wearable device may include, but is not limited to, at least one of an accessory type device (for example, a watch, a ring, a wristband, an ankle band, a necklace, glasses, or contact lenses), a head-mounted device (HMD), a fabric or clothing-integrated device (for example, electronic clothes), a body-attached device (for example, a skin pad), or a bio-implantable device (for example, an implantable circuit).
- an accessory type device for example, a watch, a ring, a wristband, an ankle band, a necklace, glasses, or contact lenses
- HMD head-mounted device
- a fabric or clothing-integrated device for example, electronic clothes
- a body-attached device for example, a skin pad
- a bio-implantable device for example, an implantable circuit
- the DVS image according to an embodiment of the disclosure may be an image generated by a DVS sensor.
- the DVS sensor according to an embodiment is an image sensor employing a method in which an iris of a person receives information, and is a sensor capable of obtaining image data of a moving object.
- the DVS sensor may transmit the image data to a processor only when there is a local change due to a movement in a pixel unit.
- the DVS sensor according to an embodiment of the disclosure may transmit the image data to the processor only when a moving event occurs.
- the DVS sensor may not process data when the object is not moving but measures the moving object when the object is moving and transmits the data to the processor, thereby preventing and/or reducing wastage of data due to frames continuously transmitted by general image sensors to the image processor.
- the DVS sensor may address a problem that a general visual recognition system is vulnerable to a fast movement.
- the DVS sensor may overcome a blur phenomenon because the DVS sensor receives data on a per-pixel basis rather than on a frame-unit basis.
- the DVS sensor may have a resolution of microseconds.
- the DVS sensor may have a better temporal resolution than an ultra-high-speed camera that shoots thousands of frames per second (e.g., ultra-high-speed frames>1 K FPS).
- a dynamic range a range of brightness that s sensor is capable of distinguish
- the DVS sensor may generate the DVS image by sensing the movement of an object even in a dark place with little light.
- An image recognition model for recognizing the DVS image may be trained based on the DVS image generated by the DVS sensor and label information about the DVS image.
- the label information according to an embodiment of the disclosure may include information about a correct answer for a result of recognizing the DVS image by the image recognition model.
- the label information according to an embodiment of the disclosure may include a plurality of pieces of information different from each other according to a type of information to be recognized from the DVS image.
- the label information is information that may be recognized with respect to the same DVS image, and may include different types of information, such as a state of an object and a moving direction of the object.
- the image recognition model may be trained based on a pair of at least one DVS image and label information corresponding to each DVS image.
- the image recognition model may be trained by modifying the image recognition model so that a difference between the result of recognizing the DVS image by the image recognition model and the label information with respect to the DVS image is minimized.
- the image recognition model may be trained to output the same or similar result as “a state in which a person is walking” as the result of recognizing the DVS image by the image recognition model.
- the image recognition model may not be sufficiently trained, which may deteriorate the accuracy and performance of the image recognition model.
- the electronic device 1000 may use the virtual simulator 110 to sufficiently generate the pair of virtual DVS image and label information, and thus the image recognition model may be sufficiently trained.
- a virtually generated DVS image and a real DVS image captured by the DVS sensor may be substantially similar in appearance. Therefore, according to the image recognition model trained based on the pair of DVS image and label information generated using the virtual simulator 110 according to an embodiment of the disclosure, image recognition may be performed even on the real DVS image with high performance and accuracy.
- the virtual simulator 110 may generate various types of virtual environments in which DVS images may be captured.
- the virtual simulator 110 may generate various virtual environments such as a house, an office, and a road.
- the virtual simulator 110 may generate various types of virtual objects that may exist in a virtual environment, and may place the generated virtual objects in the virtual environment.
- an object that may be captured as the real DVS image may be placed as a virtual object in the virtual environment.
- virtual objects such as people, furniture, home appliances, and pets may be placed in a virtual environment of a house.
- virtual objects such as signs, cars, and lanes may be placed in a virtual environment of the road.
- the virtual DVS image according to an embodiment of the disclosure may be generated based on the virtual environment generated by the virtual simulator 110 and at least one virtual object placed in the virtual environment. Also, the virtual DVS image according to an embodiment of the disclosure may be obtained as a plurality of image sequences in which scenes in which a virtual object moves or changes over time are captured.
- the virtual simulator 110 may generate the virtual DVS image that may be captured by the DVS sensor in the virtual environment.
- the virtual simulator 110 may determine from least one camera view point in which an actual DVS sensor may be located in the virtual environment, and generate at least one virtual DVS image captured from each camera view point.
- the at least one virtual DVS image may be an image simultaneously captured from each camera view point.
- the virtual simulator 110 may generate the virtual DVS image captured in the virtual environment by generating a vision image that may be captured from the at least one camera view point in the virtual environment and generating the DVS image from the vision image.
- the DVS image is not limited to the vision image, and may be generated from other types of images.
- the virtual simulator 110 may generate a virtual DVS image captured in a virtual environment through various methods.
- the label information with respect to the virtual DVS image according to an embodiment of the disclosure may be generated based on information about at least one of the virtual environment or the at least one virtual object placed in the virtual environment.
- the label information according to an embodiment of the disclosure may be obtained based on previously set information about the virtual environment and the at least one virtual object used to generate the virtual environment and the at least one virtual object.
- the virtual simulator 110 may arrange a virtual object in the virtual environment so that the virtual object is placed or moved according to the previously set information.
- the label information according to an embodiment of the disclosure may be obtained based on information about predefined characteristic information with respect to at least one of the virtual environment or the virtual object.
- the virtual simulator 110 may arrange a vehicle in the virtual environment so that the vehicle moves along a previously designated path in the virtual environment.
- the vehicle may be placed in the virtual environment according to the set information. Accordingly, label information with respect to a virtual DVS image in which the vehicle is captured may be obtained based on a path, location, speed, etc., which is information about characteristic information of the vehicle, which is previously set by the virtual simulator 110 to arrange the vehicle.
- the label information according to an embodiment of the disclosure may be obtained based on characteristic information predefined by the virtual simulator 110 when the virtual environment and the virtual object are generated. Accordingly, the label information with respect to the virtual DVS image according to an embodiment of the disclosure may include more accurate information than label information directly input by the person with respect to a real image.
- the label information according to an embodiment of the disclosure may be automatically obtained based on characteristic information predefined by the virtual simulator 110 .
- the label information according to an embodiment of the disclosure may be automatically obtained based on the predefined characteristic information, when the virtual environment or the virtual object is generated by the virtual simulator 110 . Accordingly, the label information according to an embodiment of the disclosure may be easily and quickly obtained by the electronic device 1000 .
- the label information may be automatically obtained whenever a virtual DVS image is generated based on the characteristic information predefined by the virtual simulator 110 . Accordingly, according to an embodiment of the disclosure, as data for training the image recognition model 130 , a large number of pairs of virtual DVS images and label information may be quickly and accurately generated in a short time.
- the image recognition model 130 may be trained based on the virtual DVS image generated by the virtual simulator 110 and the label information with respect to the virtual DVS image.
- the electronic device 1000 may train the image recognition model 130 by modifying at least one component included in the image recognition model 130 so that a difference between the label information and the information about the result of recognizing the DVS image by the image recognition model 130 is minimized and/or reduced.
- a structure of a node, a weight value, a bias value, etc. included in the image recognition model 130 may be modified as the image recognition model 130 is trained.
- the image recognition model 130 may be a data recognition model based on a neural network such as a convolutional neural network (CNN), a deep neural network (DNN), a recurrent neural network (RNN), and a multi-layer perceptron (MLP) used to classify and detect an object in an image.
- a neural network such as a convolutional neural network (CNN), a deep neural network (DNN), a recurrent neural network (RNN), and a multi-layer perceptron (MLP) used to classify and detect an object in an image.
- CNN convolutional neural network
- DNN deep neural network
- RNN recurrent neural network
- MLP multi-layer perceptron
- the image recognition result may include, for example, information about an object recognized from an image, information about a location of the object recognized from the image, and information about a movement of the object recognized from the image.
- a recognition result of the image of the data recognition model may include the “vehicle”.
- the image recognition model 130 may be trained so that a result close to the “vehicle” may be output when a real DVS image similar to a virtual DVS image is input to the image recognition model 130 based on the virtual DVS image including the “vehicle” and label information.
- the image recognition model 130 may be trained based on a large number of pairs of virtual DVS images and label information generated quickly and accurately by the virtual simulator 110 . Accordingly, even when the real DVS image and label information for training the image recognition model 130 are insufficient, the electronic device 1000 according to an embodiment of the disclosure may quickly obtain the image recognition model 130 having high accuracy and performance, based on pairs of virtual DVS images and label information.
- the electronic device 1000 may use an external server (not shown) to train the image recognition model 130 for recognizing a DVS image.
- the external server may be implemented as at least one computer device.
- the external server may be distributed in the form of a cloud and may provide commands, codes, files, contents, etc.
- the external server may perform operations that the electronic device 1000 may execute. For example, the external server may generate the virtual DVS image and the label information for training the image recognition model 130 according to a request of the electronic device 1000 . The external server may train the image recognition model 130 based on the generated virtual DVS image and label information, and transmit the trained image recognition model 130 to the electronic device 1000 . The external server may transmit, to the electronic device 1000 , a result of recognizing a real DVS image by the trained image recognition model 130 according to an embodiment of the disclosure.
- the image recognition model 130 may be trained on the electronic device 1000 and a result of recognizing an image by the image recognition model 130 may be output without data transmission and reception with an external server (not shown), according to an on-device AI technology.
- the electronic device 1000 may train the image recognition model 130 according to an embodiment of the disclosure according to the generated DVS image, based on various types of information collected by the electronic device 1000 in real time, without having to use big data stored in the external server.
- the electronic device 1000 may be trained by itself based on data collected by itself, and may make a decision by itself based on a trained AI model. According to the on-device AI technology, the electronic device 1000 operates by itself without transmitting the collected data to the outside, and thus there is an advantage in terms of protection of personal information of a user and a data processing speed.
- the electronic device 1000 may operate using the on-device AI technology without a connection with the external server, according to whether a network environment of the electronic device 1000 is unstable, or whether only the information collected by the electronic device 1000 is sufficient to perform an operation according to an embodiment of the disclosure according to the AI model trained by the electronic device 1000 without having to use big data.
- the electronic device 1000 is not limited to operating according to the on-device AI technology, and may perform the operation according to an embodiment of the disclosure through data transmission and reception with an external server or an external device.
- the electronic device 1000 may perform the operation according to an embodiment of the disclosure by combining the above-described on-device AI technology and a method through data transmission and reception with the external server.
- the operation according to an embodiment of the disclosure may be performed according to the method through the external server when an operation through the external server is more advantageous in terms of the data processing speed according to a network environment and a computing power of the electronic device 1000 , or when the method through the external server is more advantageous than the on-device AI technology such as data that does not include the personal information of the user is transmitted to the external server.
- FIG. 2 is a block diagram illustrating an example configuration of the electronic device 1000 according to various embodiments.
- FIG. 3 is a block diagram illustrating an example configuration of the electronic device 1000 according to various embodiments.
- the electronic device 1000 may include a processor (e.g., including processing circuitry) 1300 and a memory 1700 .
- a processor e.g., including processing circuitry
- memory 1700 e.g., a random access memory
- the electronic device 1000 of FIG. 2 may be implemented by more components or less components than the components shown in FIG. 2 .
- the electronic device 1000 may further include a communicator (e.g., including communication circuitry) 1520 , an outputter (e.g., including output circuitry) 1020 , a user inputter (e.g., including input circuitry) 1100 , a sensing unit (e.g., including at least one sensor) 1400 , and an audio/video (A/V) inputter (e.g., including A/V input circuitry) 1600 , in addition to the processor 1300 and the memory 1700 .
- a communicator e.g., including communication circuitry
- an outputter e.g., including output circuitry
- a user inputter e.g., including input circuitry
- a sensing unit e.g., including at least one sensor
- an audio/video (A/V) inputter e.g., including A/V input circuitry
- the user inputter 1100 may include various circuitry for inputting data for a user to control the electronic device 1000 .
- the user inputter 1100 may include a keypad, a dome switch, a touch pad (a touch capacitive type, a pressure resistive type, an infrared beam sensing type, a surface acoustic wave type, an integral strain gauge type, a piezoelectric type, etc.), a jog wheel, a jog switch, etc. but is not limited thereto.
- the user inputter 1100 may receive a user input for training the image recognition model 130 .
- the outputter 1200 may include various output circuitry and output an audio signal, a video signal, or a vibration signal, and the outputter 1200 may include a display 1210 , a sound outputter 1220 , and a vibration motor 1230 .
- the display 1210 displays and outputs information processed by the electronic device 1000 .
- the display 1210 may display a virtual DVS image generated by the virtual simulator 110 .
- the display 1210 according to an embodiment of the disclosure may display a result of recognizing the DVS image by the image recognition model 130 .
- the display 1210 may be used as an input device in addition to an output device.
- the display 1210 may include at least one of a liquid crystal display, a thin film transistor-liquid crystal display, an organic light-emitting diode, a flexible display, a three-dimensional (3D) display, or an electrophoretic display.
- the electronic device 1000 may include two or more displays 1210 .
- the sound outputter 1220 may include various circuitry that outputs audio data received from the communicator 1500 or stored in the memory 1700 .
- the vibration motor 1230 may output a vibration signal. Also, the vibration motor 1230 may output a vibration signal when a touch is input to the touch screen.
- the sound outputter 1220 and the vibration motor 1230 may output information related to a result of training the image recognition model 130 based on the virtual DVS image and label information, or the result of recognizing the DVS image by the image recognition model 130 .
- the processor 1300 may include various processing circuitry and generally controls the overall operation of the electronic device 1000 .
- the processor 1300 may generally control the user inputter 1100 , the outputter 1200 , the sensing unit 1400 , the communicator 1500 , and the A/V inputter 1600 by executing programs stored in the memory 1700 .
- the electronic device 1000 may include at least one processor 1300 .
- the electronic device 1000 may include various types of processor such as a central processing unit (CPU), a graphics processing unit (GPU), and a neural processing unit (NPU).
- CPU central processing unit
- GPU graphics processing unit
- NPU neural processing unit
- the processor 1300 may be configured to process commands of a computer program by performing basic arithmetic, logic, and input/output operations.
- the commands may be provided to the processor 1300 from the memory 1700 or may be received through the communicator 1500 and provided to the processor 1300 .
- the processor 1300 may be configured to execute the commands according to program codes stored in a recording device such as memory.
- the processor 1300 may generate a virtual DVS image using the virtual simulator 110 and may generate label information with respect to the DVS image.
- the label information according to an embodiment of the disclosure may include information about a correct answer to the result of recognizing the DVS image by the image recognition model 130 .
- the label information according to an embodiment of the disclosure may be obtained whenever the DVS image is generated, based on predefined characteristic information about at least one of a virtual environment or a virtual object placed in the virtual environment, which is previously set by the virtual simulator 110 when the DVS image is generated.
- the processor 1300 may train the image recognition model 130 by modifying at least one component included in the image recognition model 130 so that a difference between the label information and the information about the result of recognizing the DVS image by the image recognition model 130 is minimized.
- the processor 1300 may determine at least one camera view point in the virtual environment generated by the virtual simulator 110 , and generate at least one virtual DVS image simultaneously captured from each camera view point. Accordingly, the image recognition model according to an embodiment of the disclosure may be trained based on the at least one virtual DVS image with respect to the at least one camera view point.
- the processor 1300 may generate a virtual environment based on information about a surrounding environment in which a real DVS image that may be recognized by the image recognition model may be captured, and, based on the virtual environment, generate a virtual DVS image.
- the virtual DVS image may be generated based on changed information about the surrounding environment whenever the information about the surrounding environment changes by more than a reference value.
- the processor 1300 may recognize a DVS image using the image recognition model 130 and output a result thereof.
- the sensing unit 1400 may include various sensors and sense a state of the electronic device 1000 or a state around the electronic device 1000 , and may transfer sensed information to the processor 1300 .
- the sensing unit 1400 may include at least one of a geomagnetic sensor 1410 , an acceleration sensor 1420 , a temperature/humidity sensor 1430 , an infrared sensor 1440 , a gyroscope sensor 1450 , and a position sensor. (e.g., GPS) 1460 , a barometric pressure sensor 1470 , a proximity sensor 1480 , or an RGB sensor (illuminance sensor) 1490 , but is not limited thereto.
- a geomagnetic sensor 1410 e.g., GPS 1460 , a barometric pressure sensor 1470 , a proximity sensor 1480 , or an RGB sensor (illuminance sensor) 1490 , but is not limited thereto.
- the sensing unit 1400 may further include a DVS sensor for capturing a DVS image.
- a virtual environment may be generated or an object of the virtual environment may be generated and placed.
- the electronic device 1000 may generate a virtual environment and an object that are highly likely to be captured as real DVS images by the DVS sensor of the electronic device 1000 , based on the information sensed by the sensing unit 1400 .
- the electronic device 1000 may predict an environment in which the electronic device 1000 is currently placed, and based on the predicted environment, generate the virtual environment and the object of the virtual environment that are highly likely to be captured as real DVS images
- the image recognition model 130 may be trained based on a virtual DVS image that is highly likely to be captured as a real DVS image.
- the communicator 1500 may include one or more components, each including various communication circuitry, that allow the electronic device 1000 to communicate with the server 2000 or an external device (not shown).
- the communicator 1500 may include a short-range communicator 1510 , a mobile communicator 1520 , and a broadcast receiver 1530 .
- the short-range wireless communicator 1510 may include a Bluetooth communicator, a Bluetooth Low Energy (BLE) communicator, a near field communicator, a wireless local area network (WLAN) (Wi-Fi) communicator, a Zigbee communicator, an Infrared Data Association (IrDA) communicator (not shown), a Wi-Fi Direct (WFD) communicator, an ultra wideband (UWB), and an Ant+ communicator, but is not limited thereto.
- BLE Bluetooth Low Energy
- Wi-Fi wireless local area network
- Zigbee Zigbee communicator
- IrDA Infrared Data Association
- WFD Wi-Fi Direct
- UWB ultra wideband
- Ant+ communicator but is not limited thereto.
- the mobile communicator 1520 transmits and receives wireless signals to and from at least one of a base station, an external terminal, or a server on a mobile communication network.
- the wireless signal may include various types of data according to transmission/reception of a voice call signal, a video call signal, or a text/multimedia message.
- the broadcast receiver 1530 receives a broadcast signal and/or broadcast-related information from the outside through a broadcast channel.
- the broadcast channel may include a satellite channel and a terrestrial channel.
- the electronic device 1000 may not include the broadcast receiver 1530 .
- the communicator 1500 may transmit and receive data required for training the image recognition model 130 .
- the A/V inputter 1600 may include various components including various circuitry for inputting an audio signal or a video signal, and may include a camera 1610 , a microphone 1620 , etc.
- the camera 1610 may obtain an image frame such as a still image or a moving image through an image sensor in a video call mode or a photographing mode.
- the image captured through the image sensor may be processed through the processor 1300 or a separate image processing unit (not shown).
- the microphone 1620 receives an external sound signal and processes the sound signal as electrical speech data.
- the memory 1700 may store a program for processing and controlling the processor 1300 , and may also store data input to or output from the electronic device 1000 .
- the memory 1700 may store information necessary for generating a virtual DVS image and label information and training the image recognition model 130 .
- the memory 1700 may store the image recognition model 130 .
- the memory 1700 according to an embodiment of the disclosure may store information about the virtual simulator 110 that generates the virtual DVS image.
- the memory 1700 may include at least one of a flash memory type storage medium, a hard disk type storage medium, a multimedia card micro type storage medium, card type memory (e.g., secure digital (SD) memory, eXtreme Digital (XD) memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, a magnetic disk, or an optical disk.
- card type memory e.g., secure digital (SD) memory, eXtreme Digital (XD) memory, etc.
- RAM random access memory
- SRAM static random access memory
- ROM read-only memory
- EEPROM electrically erasable programmable read-only memory
- PROM programmable read-only memory
- magnetic memory a magnetic disk, or an optical disk.
- the programs stored in the memory 1700 may be classified into a plurality of modules, for example, a UI module 1710 , a touch screen module 1720 , a notification module 1730 , etc. according to functions thereof.
- the UI module 1710 may provide a specialized UI, a graphic user interface (GUI), etc. that interact with the electronic device 1000 for each application.
- the touch screen module 1720 may sense a touch gesture of the user on a touch screen and may transfer information about the touch gesture to the processor 1300 .
- the touch screen module 1720 according to some embodiments may recognize and analyze a touch code.
- the touch screen module 1720 may be configured as separate hardware including a controller.
- Various sensors may be provided inside or near the touch screen to sense a touch or a proximity touch of the touch screen.
- An example of a sensor for sensing a touch with respect to the touch screen includes a tactile sensor.
- the tactile sensor refers to a sensor that senses a touch of a specific object to the extent or higher than that felt by a human.
- the tactile sensor may sense various information such as roughness of a contact surface, hardness of a contact object, and a temperature of a contact point.
- the touch gesture of the user may include tap, touch and hold, double tap, drag, panning, flick, drag and drop, swipe, etc.
- the notification module 1730 may generate a signal for notifying the occurrence of an event in the electronic device 1000 .
- FIG. 4 is a flowchart illustrating an example method of training the image recognition model 130 according to various embodiments.
- the electronic device 1000 may generate a virtual DVS image using the virtual simulator 110 .
- the electronic device 1000 according to an embodiment of the disclosure may generate a virtual environment through the virtual simulator 110 and may generate the virtual DVS image based on the virtual environment.
- the virtual simulator 110 may generate the virtual environment based on information about a surrounding environment in which a real DVS image recognized by the image recognition model may be captured. For example, based on information sensed by various sensors provided in the electronic device 1000 , the electronic device 1000 may obtain information about the surrounding environment, and transmit the information about the surrounding environment to the virtual simulator 110 .
- the information about the surrounding environment may include information about the surrounding environment of the electronic device 1000 including a DVS sensor that captures a DVS image, or a device other than the electronic device 1000 .
- the virtual simulator 110 may generate the virtual environment based on the information about the surrounding environment of the electronic device 1000 and may generate the virtual DVS image based on the generated virtual environment.
- the information about the surrounding environment may include a variety of information that may be sensed by a sensor of the electronic device 1000 , such as a location and a movement state of the electronic device 1000 , information sensed with respect to an object around the electronic device 1000 , etc.
- the electronic device 1000 may determine whether the information about the surrounding environment obtained by the electronic device 1000 has changed by more than a reference value.
- the electronic device 1000 according to an embodiment of the disclosure may transmit the changed information about the surrounding environment to the virtual simulator 110 whenever the information about the surrounding environment changes by more than the reference value.
- the virtual simulator 110 may generate the virtual environment whenever the changed information about the surrounding environment is obtained from the electronic device 1000 , and generate the virtual DVS image based on the generated virtual environment. Accordingly, according to an embodiment of the disclosure, whenever the information about the surrounding environment changes by more than the reference value, the virtual DVS image may be generated based on the changed information about the surrounding environment. Also, according to an embodiment of the disclosure, the image recognition model may be continuously trained based on the changed information about the surrounding environment of the electronic device 1000 .
- the electronic device 1000 may generate label information with respect to the virtual DVS image.
- the label information according to an embodiment of the disclosure may include information about at least one correct answer to a result of recognizing the virtual DVS image by the image recognition model.
- the image recognition model according to an embodiment of the disclosure may be modified so that a difference between the result of the image recognition model and the correct answer information is minimized, so that the image recognition model may be updated.
- the label information according to an embodiment of the disclosure may be determined based on information about the virtual environment generated by the virtual simulator 110 .
- the information about the virtual environment according to an embodiment of the disclosure may be determined according to predefined characteristic information about the virtual environment and the virtual object.
- the label information may be determined based on identification information and movement information of the virtual object.
- the label information may be automatically generated whenever the virtual DVS image is generated.
- the label information may include accurate and detailed information.
- the electronic device 1000 may train the image recognition model for recognizing the real DVS image based on the virtual DVS image and the label information corresponding to the virtual DVS image.
- the image recognition model may be sufficiently trained based on the virtual DVS image generated by the virtual simulator 110 .
- the label information with respect to the virtual DVS image may be determined according to accurate and detailed information previously set by the virtual simulator 110 , the image recognition model is trained based on the label information, and thus the performance of the image recognition model may be better.
- FIG. 5 is a diagram illustrating an example of a virtual environment generated by the virtual simulator 110 according to various embodiments.
- the virtual simulator 110 may generate a virtual environment 500 in which, for example, an environment inside a house is implemented, and place a plurality of objects 511 , 512 , and 520 in the virtual environment 500 .
- the plurality of objects 511 , 512 , and 520 that may be placed in the virtual environment 500 may be objects that are likely to be captured as a real DVS image in a real environment corresponding to the virtual environment 500 .
- a sofa 512 and a table 511 which are fixed objects
- a robot cleaner 520 which is an object that captures the virtual DVS image while moving
- the sofa 512 , the table 511 , and the robot cleaner 520 may be placed in previously designated positions in the virtual environment 500 .
- the robot cleaner 520 may be placed to move in the virtual environment 500 according to previously designated path and speed.
- the virtual DVS image according to an embodiment of the disclosure may be generated at each camera view point with respect to the plurality of DVS sensors 521 , 522 , and 523 placed in the virtual environment 500 .
- the image recognition model 330 may be trained based on a plurality of virtual DVS images simultaneously captured from different camera view points.
- the virtual DVS image according to an embodiment of the disclosure may be obtained as a plurality of image sequences including a scene in which the robot cleaner 520 moves captured from different camera view points over time.
- the DVS sensors 522 and 523 placed in fixed positions may obtain a virtual DVS image in which the robot cleaner 520 which is a moving object is sensed, except for the table 511 and sofa 512 which are fixed objects.
- the DVS sensor 521 disposed in the moving robot cleaner 520 may sense even a fixed object as a moving object in the DVS image due to the movement of the DVS sensor 521 . Accordingly, the DVS sensor 521 disposed in the robot cleaner 520 may obtain the virtual DVS image in which not only the moving object but also the table 511 and the sofa 512 which are fixed objects are sensed.
- the image recognition model 330 may be trained based on more precise and a lot of data, according to a plurality of virtual DVS images captured from different view points and in different moving states.
- FIG. 6 is a flowchart illustrating an example method of training an image recognition model based on virtual data according to various embodiments.
- the virtual data may include a pair of a virtual DVS image and label information generated based on a virtual environment.
- the electronic device 1000 may generate the virtual environment with respect to a DVS image in order to obtain the virtual DVS image.
- the virtual environment according to an embodiment of the disclosure may be generated by previously defining characteristic information of the virtual environment (e.g., size, object, brightness, etc. of the virtual environment). For example, based on surrounding environment information sensed by the electronic device 1000 , characteristic information related to the virtual environment may be previously set.
- the electronic device 1000 may generate the virtual DVS image based on the virtual environment. After generating the virtual environment, the electronic device 1000 according to an embodiment of the disclosure may determine at least one view point from which the virtual DVS image is captured.
- the at least one view point from which the virtual DVS image is captured according to an embodiment of the disclosure may be fixed or moved. According to an embodiment of the disclosure, when the at least one view point from which the virtual DVS image is captured is a moving view point, information about movement, such as a movement path and speed, may be set, and the virtual DVS image may be generated according to the set information.
- a movement path and speed of the robot cleaner may be set based on characteristic information of the robot cleaner. According to the set movement information, the virtual DVS image captured by the DVS sensor of the robot cleaner may be generated.
- the electronic device 1000 may generate label information with respect to the virtual DVS image generated in operation 620 based on the virtual environment.
- the label information according to an embodiment of the disclosure may include correct answer information indicating a correct answer to a result of recognizing the virtual DVS image by the image recognition model.
- the label information according to an embodiment of the disclosure may be obtained when the correct answer information is determined, based on the above-described previously set characteristic information with respect to the virtual environment, previously set information about the moving path and speed of the DVS sensor that captures the DVS image, and previously set information regarding the virtual object included in the DVS image (e.g., location, movement speed, movement path, identification information, and state), etc.
- the label information according to an embodiment of the disclosure may be determined based on information that needs to be previously set in order to generate the virtual environment and place the virtual object. Accordingly, at a time point at which the virtual DVS image is generated, the information used to determine the label information may be information already stored in the electronic device 1000 to generate the virtual environment and place the virtual object. The label information according to an embodiment of the disclosure may be automatically determined without another input from a user based on the information already stored in the electronic device 1000 in relation to the virtual environment.
- the label information may include at least one correct answer information indicating a correct answer to information that may be output by the image recognition model.
- the label information may include at least one correct answer information respectively corresponding to the object recognition information and object state information, object movement information, etc.
- the electronic device 1000 may generate virtual data including the at least one virtual DVS image and the label information corresponding to each DVS image.
- the electronic device 1000 according to an embodiment of the disclosure may train the image recognition model based on the virtual data.
- the image recognition model may be trained by modifying the image recognition model so that a recognition result by the image recognition model with respect to the virtual DVS image has a value close to the correct answer information of the label information.
- FIG. 7 is a block diagram illustrating an example of the electronic device 1000 and an external device 700 according to various embodiments.
- an image recognition model trained by the electronic device 1000 may be transmitted to the external device 700 .
- the external device 700 may include a device including the image recognition model trained by the electronic device 1000 and recognizing a real DVS image obtained by the external device 700 based on the image recognition model.
- the external device 700 may include a robot cleaner, a smart refrigerator, a smart TV, a camera, etc. that may be connected to the electronic device 1000 , and is not limited to the above-described example, but may include various types of devices.
- the electronic device 1000 may generate the virtual DVS image that may be captured by each external device 700 and train the image recognition model based on the virtual DVS image.
- the electronic device 1000 may generate the virtual DVS image that may be captured by a DVS sensor provided in the robot cleaner according to a movement path and speed of the robot cleaner among the external devices 700 .
- the electronic device 1000 may generate a virtual DVS image that may be captured by a DVS sensor provided in each of the smart TV and the smart refrigerator according to locations of the smart TV and the smart refrigerator among the external devices 700 .
- the electronic device 1000 may obtain label information with respect to the virtual DVS image that may be captured by each of the external devices 700 based on previously set information so as to generate the virtual environment.
- the previously set information may be characteristic information predefined with respect to at least one of a virtual environment or a virtual object of the virtual environment.
- the electronic device 1000 may train the image recognition model based on the virtual DVS image and the label information generated with respect to the at least one external device 700 , and transmit the trained image recognition model to the at least one external device 700 .
- the at least one external device 700 may recognize the DVS image obtained by each external device 700 based on the image recognition model received from the electronic device 1000 .
- the smart TV among the external devices 700 may sense a movement of a user from the DVS image captured by the smart TV using the image recognition model and determine whether the user watches the TV.
- the smart TV according to an embodiment of the disclosure may perform various operations, for example, an operation of turning off or on the power of the smart TV, based on whether the user watches the TV.
- the robot cleaner among the external devices 700 may predict a movement path of the user from the DVS image captured by the robot cleaner using the image recognition model.
- the robot cleaner according to an embodiment of the disclosure may determine the movement path of the robot cleaner based on the movement path of the user, and perform floor cleaning while moving according to the determined path.
- the electronic device 1000 may receive the DVS image obtained by each external device 700 rather than transmitting the image recognition model to the at least one external device 700 .
- the electronic device 1000 according to an embodiment of the disclosure may recognize the DVS image received from each external device 700 based on the trained image recognition model according to an embodiment of the disclosure.
- the electronic device 1000 may perform an operation according to a result of recognizing the DVS image received from each external device 700 or transmit information related to the result to each external device 700 .
- the image recognition model may be sufficiently trained even when data for training the image recognition model is not sufficient for training the image recognition model, based on a pair of DVS image and label information generated using a virtual simulator.
- An embodiment of the disclosure may also be realized in a form of a recording medium including instructions executable by a computer, such as a program module executed by a computer.
- a computer-readable recording medium may be an arbitrary available medium accessible by a computer, and examples thereof include all volatile and non-volatile media and separable and non-separable media.
- examples of the computer-readable recording medium may include a computer storage medium and a communication medium. Examples of the computer storage medium include all volatile and non-volatile media and separable and non-separable media, which have been implemented by an arbitrary method or technology, for storing information such as computer-readable instructions, data structures, program modules, and other data.
- the communication medium typically includes a computer-readable instruction, a data structure, or a program module, and includes an arbitrary information transmission medium.
- the term such as “ . . . unit” or “ . . . portion” used herein may refer to a hardware component such as a processor or a circuit, and/or a software component executed by the hardware component such as a processor.
Abstract
Description
Claims (15)
Applications Claiming Priority (3)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
KR1020190113023A KR20210031269A (en) | 2019-09-11 | 2019-09-11 | Electronic device and operating method for training an image recognition model |
KR10-2019-0113023 | 2019-09-11 | ||
PCT/KR2020/004610 WO2021049730A1 (en) | 2019-09-11 | 2020-04-06 | Electronic device training image recognition model and operation method for same |
Related Parent Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/KR2020/004610 Continuation WO2021049730A1 (en) | 2019-09-11 | 2020-04-06 | Electronic device training image recognition model and operation method for same |
Publications (2)
Publication Number | Publication Date |
---|---|
US20220198786A1 US20220198786A1 (en) | 2022-06-23 |
US11908175B2 true US11908175B2 (en) | 2024-02-20 |
Family
ID=74867030
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US17/693,095 Active 2040-10-04 US11908175B2 (en) | 2019-09-11 | 2022-03-11 | Electronic device training image recognition model and operation method for same |
Country Status (3)
Country | Link |
---|---|
US (1) | US11908175B2 (en) |
KR (1) | KR20210031269A (en) |
WO (1) | WO2021049730A1 (en) |
Families Citing this family (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
KR20210031269A (en) * | 2019-09-11 | 2021-03-19 | 삼성전자주식회사 | Electronic device and operating method for training an image recognition model |
KR102633177B1 (en) * | 2023-01-20 | 2024-02-02 | 주식회사 핀텔 | The Method and System for Labeled Machine-Learning Data By Digital Twin |
Citations (22)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JPH0248691A (en) | 1988-08-10 | 1990-02-19 | Mita Ind Co Ltd | Method and device for measuring variance of discharging of corona discharger |
US8510283B2 (en) * | 2006-07-31 | 2013-08-13 | Ricoh Co., Ltd. | Automatic adaption of an image recognition system to image capture devices |
US8825682B2 (en) * | 2006-07-31 | 2014-09-02 | Ricoh Co., Ltd. | Architecture for mixed media reality retrieval of locations and registration of images |
US8856108B2 (en) * | 2006-07-31 | 2014-10-07 | Ricoh Co., Ltd. | Combining results of image retrieval processes |
US8868555B2 (en) * | 2006-07-31 | 2014-10-21 | Ricoh Co., Ltd. | Computation of a recongnizability score (quality predictor) for image retrieval |
US9020966B2 (en) * | 2006-07-31 | 2015-04-28 | Ricoh Co., Ltd. | Client device for interacting with a mixed media reality recognition system |
KR20170074413A (en) | 2015-12-22 | 2017-06-30 | 연세대학교 산학협력단 | 2d image data generation system using of 3d model, and thereof method |
US20180174323A1 (en) | 2016-12-20 | 2018-06-21 | Samsung Electronics Co., Ltd. | Multiscale weighted matching and sensor fusion for dynamic vision sensor tracking |
KR20180093582A (en) | 2017-02-14 | 2018-08-22 | 한국전자통신연구원 | Method and apparatus for indexing event sections from video using end-to-end learning |
US20190090786A1 (en) | 2017-09-27 | 2019-03-28 | Samsung Electronics Co., Ltd. | Method and device for detecting dangerous situation |
US20190096068A1 (en) | 2017-09-28 | 2019-03-28 | Samsung Electronics Co., Ltd. | Camera pose and plane estimation using active markers and a dynamic vision sensor |
US20190096081A1 (en) | 2017-09-28 | 2019-03-28 | Samsung Electronics Co., Ltd. | Camera pose determination and tracking |
JP6548691B2 (en) | 2016-10-06 | 2019-07-24 | 株式会社アドバンスド・データ・コントロールズ | Image generation system, program and method, simulation system, program and method |
KR20190096866A (en) | 2019-07-31 | 2019-08-20 | 엘지전자 주식회사 | A Monitoring Method and Apparatus in the Vehicle, and a 3D Modeling Unit for Generating an Object Detection Mmodel therefor |
KR20190099185A (en) | 2017-05-25 | 2019-08-26 | 삼성전자주식회사 | Method and system for detecting a dangerous situation |
US20200034524A1 (en) * | 2019-08-22 | 2020-01-30 | Lg Electronics Inc. | Method for performing user authentication and function execution simultaneously and electronic device for the same |
US11106273B2 (en) * | 2015-10-30 | 2021-08-31 | Ostendo Technologies, Inc. | System and methods for on-body gestural interfaces and projection displays |
US11256954B2 (en) * | 2020-06-18 | 2022-02-22 | Tala Consultancy Services Limited | System and method of gesture recognition using a reservoir based convolutional spiking neural network |
US20220198786A1 (en) * | 2019-09-11 | 2022-06-23 | Samsung Electronics Co., Ltd. | Electronic device training image recognition model and operation method for same |
US11431660B1 (en) * | 2020-09-25 | 2022-08-30 | Conversation Processing Intelligence Corp. | System and method for collaborative conversational AI |
US11574470B2 (en) * | 2017-05-16 | 2023-02-07 | Google Llc | Suggested actions for images |
US11676269B2 (en) * | 2017-10-20 | 2023-06-13 | Genfit | Automated pattern recognition and scoring method of histological images |
-
2019
- 2019-09-11 KR KR1020190113023A patent/KR20210031269A/en active Search and Examination
-
2020
- 2020-04-06 WO PCT/KR2020/004610 patent/WO2021049730A1/en active Application Filing
-
2022
- 2022-03-11 US US17/693,095 patent/US11908175B2/en active Active
Patent Citations (25)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JPH0248691A (en) | 1988-08-10 | 1990-02-19 | Mita Ind Co Ltd | Method and device for measuring variance of discharging of corona discharger |
US8510283B2 (en) * | 2006-07-31 | 2013-08-13 | Ricoh Co., Ltd. | Automatic adaption of an image recognition system to image capture devices |
US8825682B2 (en) * | 2006-07-31 | 2014-09-02 | Ricoh Co., Ltd. | Architecture for mixed media reality retrieval of locations and registration of images |
US8856108B2 (en) * | 2006-07-31 | 2014-10-07 | Ricoh Co., Ltd. | Combining results of image retrieval processes |
US8868555B2 (en) * | 2006-07-31 | 2014-10-21 | Ricoh Co., Ltd. | Computation of a recongnizability score (quality predictor) for image retrieval |
US9020966B2 (en) * | 2006-07-31 | 2015-04-28 | Ricoh Co., Ltd. | Client device for interacting with a mixed media reality recognition system |
US11106273B2 (en) * | 2015-10-30 | 2021-08-31 | Ostendo Technologies, Inc. | System and methods for on-body gestural interfaces and projection displays |
KR20170074413A (en) | 2015-12-22 | 2017-06-30 | 연세대학교 산학협력단 | 2d image data generation system using of 3d model, and thereof method |
JP6548691B2 (en) | 2016-10-06 | 2019-07-24 | 株式会社アドバンスド・データ・コントロールズ | Image generation system, program and method, simulation system, program and method |
US10565458B2 (en) | 2016-10-06 | 2020-02-18 | Advanced Data Controls Corp. | Simulation system, simulation program and simulation method |
US20180174323A1 (en) | 2016-12-20 | 2018-06-21 | Samsung Electronics Co., Ltd. | Multiscale weighted matching and sensor fusion for dynamic vision sensor tracking |
KR20180093582A (en) | 2017-02-14 | 2018-08-22 | 한국전자통신연구원 | Method and apparatus for indexing event sections from video using end-to-end learning |
US11574470B2 (en) * | 2017-05-16 | 2023-02-07 | Google Llc | Suggested actions for images |
KR20190099185A (en) | 2017-05-25 | 2019-08-26 | 삼성전자주식회사 | Method and system for detecting a dangerous situation |
US11080891B2 (en) | 2017-05-25 | 2021-08-03 | Samsung Electronics Co., Ltd. | Method and system for detecting dangerous situation |
US20190090786A1 (en) | 2017-09-27 | 2019-03-28 | Samsung Electronics Co., Ltd. | Method and device for detecting dangerous situation |
US20190096068A1 (en) | 2017-09-28 | 2019-03-28 | Samsung Electronics Co., Ltd. | Camera pose and plane estimation using active markers and a dynamic vision sensor |
US20190096081A1 (en) | 2017-09-28 | 2019-03-28 | Samsung Electronics Co., Ltd. | Camera pose determination and tracking |
US11676269B2 (en) * | 2017-10-20 | 2023-06-13 | Genfit | Automated pattern recognition and scoring method of histological images |
KR20190096866A (en) | 2019-07-31 | 2019-08-20 | 엘지전자 주식회사 | A Monitoring Method and Apparatus in the Vehicle, and a 3D Modeling Unit for Generating an Object Detection Mmodel therefor |
US20190392256A1 (en) | 2019-07-31 | 2019-12-26 | Lg Electronics Inc. | Monitoring method and apparatus in the vehicle, and a 3d modeling unit for generating an object detection model therefor |
US20200034524A1 (en) * | 2019-08-22 | 2020-01-30 | Lg Electronics Inc. | Method for performing user authentication and function execution simultaneously and electronic device for the same |
US20220198786A1 (en) * | 2019-09-11 | 2022-06-23 | Samsung Electronics Co., Ltd. | Electronic device training image recognition model and operation method for same |
US11256954B2 (en) * | 2020-06-18 | 2022-02-22 | Tala Consultancy Services Limited | System and method of gesture recognition using a reservoir based convolutional spiking neural network |
US11431660B1 (en) * | 2020-09-25 | 2022-08-30 | Conversation Processing Intelligence Corp. | System and method for collaborative conversational AI |
Also Published As
Publication number | Publication date |
---|---|
KR20210031269A (en) | 2021-03-19 |
WO2021049730A1 (en) | 2021-03-18 |
US20220198786A1 (en) | 2022-06-23 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US11470385B2 (en) | Method and apparatus for filtering video | |
US11699213B2 (en) | Image-capturing device and method for controlling same | |
US11386600B2 (en) | System and method for displaying virtual image through HMD device | |
CN110168618B (en) | Augmented reality control system and method | |
US10412379B2 (en) | Image display apparatus having live view mode and virtual reality mode and operating method thereof | |
US10374648B2 (en) | Wearable device for transmitting a message comprising strings associated with a state of a user | |
US11908175B2 (en) | Electronic device training image recognition model and operation method for same | |
US11317034B2 (en) | Electronic device and operating method of controlling brightness of light source | |
CN110622108B (en) | Method of providing haptic feedback and electronic device performing the same | |
US20170201676A1 (en) | Image processing apparatus and control method thereof | |
KR102592907B1 (en) | Method and device for recognizing a text | |
US11598976B1 (en) | Object recognition for improving interfaces on an eyewear device and other wearable and mobile devices | |
US11954595B2 (en) | Electronic device for recognizing object in image, and operation method thereof | |
CN109547696B (en) | Shooting method and terminal equipment | |
KR20200094607A (en) | Electronic device and operating method for generating caption information for a image sequence | |
US20220343156A1 (en) | Server for refining model in terminal and operation method thereof | |
US11531912B2 (en) | Electronic apparatus and server for refining artificial intelligence model, and method of refining artificial intelligence model | |
US20200327451A1 (en) | Electronic apparatus and server for refining artificial intelligence model, and method of refining artificial intelligence model | |
US11617957B2 (en) | Electronic device for providing interactive game and operating method therefor | |
US20220187092A1 (en) | Electronic device for acquiring location information on basis of image, and method for operating same | |
KR20210158695A (en) | Electronic device and operating method for detecting a plane in an image | |
US11217267B2 (en) | Electronic device for identifying location information of external device and operating method thereof | |
US20220138628A1 (en) | Electronic device for updating artificial intelligence model and operating method thereof | |
KR20210116838A (en) | Electronic device and operating method for processing a voice input based on a gesture | |
CN114554069A (en) | Terminal, task running method and device thereof, and storage medium |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AS | Assignment |
Owner name: SAMSUNG ELECTRONICS CO., LTD., KOREA, REPUBLIC OF Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:JANG, SEOWOO;REEL/FRAME:059246/0436 Effective date: 20210817 |
|
FEPP | Fee payment procedure |
Free format text: ENTITY STATUS SET TO UNDISCOUNTED (ORIGINAL EVENT CODE: BIG.); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION |
|
AS | Assignment |
Owner name: SAMSUNG ELECTRONICS CO., LTD., KOREA, REPUBLIC OF Free format text: EMPLOYMENT AGREEMENT;ASSIGNOR:YI, SANGUNG;REEL/FRAME:060813/0788 Effective date: 20190605 |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: NOTICE OF ALLOWANCE MAILED -- APPLICATION RECEIVED IN OFFICE OF PUBLICATIONS |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: PUBLICATIONS -- ISSUE FEE PAYMENT VERIFIED |
|
STCF | Information on status: patent grant |
Free format text: PATENTED CASE |