WO2023239043A1

WO2023239043A1 - Method for detecting object, and electronic device supporting same

Info

Publication number: WO2023239043A1
Application number: PCT/KR2023/005184
Authority: WO
Inventors: 이종원; 강인표; 윤도일; 이동욱; 이영기; 이주봉; 이주헌; 전창민
Original assignee: 삼성전자주식회사; 서울대학교산학협력단
Priority date: 2022-06-08
Filing date: 2023-04-17
Publication date: 2023-12-14

Abstract

An electronic device according to an embodiment may include a memory and at least one processor operatively connected to the memory. The at least one processor may be configured to acquire at least one image. The at least one processor may be configured to acquire at last one region of interest, related to an object in the at least one image. The at least one processor may be configured to acquire at least one first region corresponding to the at least one region of interest. The at least one processor may be configured to acquire a frame including the at least one first region. The at least one processor may be configured to acquire information related to the object from the frame by using an artificial intelligence model.

Description

Method for detecting objects and electronic device supporting the same

This disclosure relates to a method for detecting an object and an electronic device supporting the same.

Internet of things (IoT) technology can provide intelligent Internet technology services that create new value in human life by collecting and analyzing data generated from devices. Through the convergence and combination of existing Internet technology and various industries, IoT technology can be applied to fields such as smart homes, smart buildings, smart cities, smart cars, and smart home appliances.

IoT technology is used in conjunction with one or more external electronic devices placed in a location (or place) (e.g., home or company) registered on a server (e.g., a cloud server supporting IoT technology). , can provide a variety of services. For example, IoT technology analyzes images acquired through a camera (e.g. IoT camera) placed at a location (or place) registered on the server, such as movement of an object detected at a location registered on the server (e.g. Information related to the movement of a child or an elderly person, or movement related to an intrusion) may be provided (e.g., information related to an accident of a child or an elderly person, or intrusion information).

Images acquired through a camera placed at a location (or place) registered on the server may be related to the user's sensitive information. If the operation of analyzing the image and/or obtaining information related to the movement of an object detected at a location registered in the server is performed on the server, there may be a risk that the user's sensitive information may be leaked to the outside.

Additionally, in order to analyze images and obtain information related to the movement of objects, artificial intelligence platforms and/or various algorithms that require higher performance and/or more computing resources may be used. Accordingly, lower-performance IoT devices (e.g. home appliances, or smartphones not in use by the user) (also referred to as “edge devices”) analyze images and obtain information related to the movement of objects. It may be inappropriate to perform the action. For example, when an IoT device with lower performance performs actions such as analyzing images and obtaining information related to the movement of objects, the user is provided with information of the quality desired (e.g., more accurate information about the object). It is difficult, and the operation execution time may be longer.

One embodiment of the present disclosure provides a method for detecting an object, using an artificial intelligence model to obtain information related to the object from a frame including at least one region corresponding to at least one region of interest obtained from an image. It relates to a method and an electronic device supporting the same.

The technical problems to be achieved by the present disclosure are not limited to the technical problems mentioned above, and other technical problems not mentioned can be clearly understood by those skilled in the art related to this document from the description below. There will be.

An electronic device according to an embodiment may include a memory and at least one processor operatively connected to the memory. The at least one processor may be configured to acquire at least one image. The at least one processor may be configured to obtain at least one region of interest related to an object within the at least one image. The at least one processor may be configured to obtain at least one first region corresponding to the at least one region of interest. The at least one processor may be configured to obtain a frame including the at least one first area. The at least one processor may be configured to obtain information related to the object from the frame using an artificial intelligence model.

A method for detecting an object in an electronic device according to an embodiment may include acquiring at least one image. The method may include obtaining at least one region of interest associated with an object within the at least one image. The method may include obtaining at least one first area corresponding to the at least one area of interest. The method may include obtaining a frame including the at least one first region. The method may include an operation of obtaining information related to the object from the frame using an artificial intelligence model.

In one embodiment, a non-transitory computer-readable medium recording computer-executable instructions, wherein the computer-executable instructions, when executed, cause an electronic device including at least one processor to acquire at least one image. It can be. The computer-executable instructions, when executed, may configure an electronic device including at least one processor to obtain at least one region of interest associated with an object within the at least one image. The computer-executable instructions, when executed, may configure an electronic device including at least one processor to acquire at least one first region corresponding to the at least one region of interest. The computer-executable instructions, when executed, may configure an electronic device including at least one processor to obtain a frame including the at least one first area. The computer-executable instructions, when executed, may be configured to cause an electronic device including at least one processor to obtain information related to the object from the frame using an artificial intelligence model.

A method for detecting an object and an electronic device supporting the same according to an embodiment include, using an artificial intelligence model, from a frame including at least one area corresponding to at least one area of interest obtained from an image, By performing an operation to obtain related information, an operation to detect an object can be performed more quickly and more accurately.

The effects that can be obtained from the present disclosure are not limited to the effects mentioned above, and other effects not mentioned can be clearly understood by those skilled in the art from the description below. will be.

1 shows an internet of things (IoT) system, according to one embodiment.

Figure 2 is a block diagram of an electronic device in a network environment, according to one embodiment.

Figure 3 is a block diagram of an electronic device, according to one embodiment.

Figure 4 is a diagram for explaining a processor, according to one embodiment.

Figure 5 is a flowchart explaining a method for detecting an object, according to one embodiment.

Figure 6 is an example diagram for explaining a region of interest, according to an embodiment.

FIG. 7 is a flowchart illustrating a method of determining the size of a first area, according to an embodiment.

FIG. 8 is an example diagram illustrating a method of determining the size of a first area, according to an embodiment.

FIG. 9 is an example diagram illustrating a method of obtaining a frame including at least one first area, according to an embodiment.

Figure 10 is an example diagram explaining a method of obtaining information related to an object, according to an embodiment.

FIG. 11 is a flowchart illustrating a method of profiling the relationship between the execution time of an inference operation and the size of a mixed frame for each type of artificial intelligence model, according to an embodiment.

Figure 12 is an example diagram illustrating a method of profiling the relationship between the execution time of an inference operation and the size of a mixed frame for each type of artificial intelligence model, according to an embodiment.

Figure 13 is an example diagram explaining a method of learning the relationship between accuracy and image size, according to an embodiment.

Figure 14 is an example diagram explaining a method for detecting an object according to an embodiment.

Figure 15 is an example diagram explaining a method for detecting an object according to an embodiment.

Figure 16 is an example diagram explaining a method of providing information related to an object, according to an embodiment.

1 shows an Internet of things (IoT) system 100 according to one embodiment. Meanwhile, at least some of the components in FIG. 1 may be omitted, and may be implemented to include additional components not shown.

Referring to FIG. 1, the IoT system 100 according to one embodiment includes a plurality of electronic devices connectable to the

data network

116 or 146. For example, the IoT system 100 includes a first IoT server 110, a first node 120, a voice assistance server 130, a second IoT server 140, and a second node. 150, or may include at least one of the

devices

121, 122, 123, 124, 125, 136, 137, 151, 152, and 153.

According to one embodiment, the first IoT server 110 may include at least one of a communication interface 111, a processor 112, or a storage unit 113. The second IoT server 140 may include at least one of a communication interface 141, a processor 142, or a storage unit 143. “IoT server” in this document refers to a relay device (e.g., first node 120 or second node (120), for example, based on a data network (e.g., data network 116 or data network 146). One or more devices (e.g.,

devices

121, 122, 123, 124, 125, 151, 152, 153) can be remotely controlled and/or monitored via 150)) or directly without a relay device. “Device” herein refers to a sensor, appliance, office electronic device, or It is a device for performing processes, and there are no restrictions on its type. A device that receives a control command and performs an operation corresponding to the control command may be named a “target device.” The IoT server may be called a central server in that it selects a target device among a plurality of devices and provides control commands.

According to one embodiment, the first IoT server 110 may communicate with the

devices

121, 122, and 123 through the data network 116. Data network 116 may refer to a network for long-distance communication, such as the Internet or a computer network (e.g., LAN or WAN), or may include a cellular network.

According to one embodiment, the first IoT server 110 may be connected to the data network 116 through the communication interface 111. The communication interface 111 may include a communication device (or communication module) to support communication of the data network 116, and may be integrated into one component (e.g., a single chip), or may be integrated into a plurality of separate components. It can be implemented with components (e.g., multiple chips). The first IoT server 110 may communicate with the

devices

121, 122, and 123 through the first node 120. The first node 120 may receive data from the first IoT server 110 through the data network 116 and transmit the received data to at least some of the

devices

121, 122, and 123. Alternatively, the first node 120 may receive data from at least some of the

devices

121, 122, and 123, and transmit the received data to the first IoT server 110 through the data network 116. The first node 120 may function as a bridge between the data network 116 and the

devices

121, 122, and 123. Meanwhile, in FIG. 1, it is shown as if there is only one first node 120, but this is simply an example and there is no limit to the number.

A “node” in this document may be an edge computing system, or may be a hub device. According to one embodiment, the first node 120 supports wired and/or wireless communication of the data network 116, and may also support wired and/or wireless communication with the

devices

121, 122, and 123. . For example, the first node 120 may be configured to communicate via a short-range communication network such as at least one of Bluetooth, Wi-Fi, Wi-Fi direct, Z-wave, Zig-bee, INSETEON, X10, or IrDA (infrared data association). It can be connected to

devices

121, 122, and 123, but there is no limitation on the type of communication. The first node 120 is placed in an environment such as a home, office, factory, building, external location, or other types of premises. (or, location). Accordingly, the

devices

121, 122, and 123 may be monitored and/or controlled by the service provided by the first IoT server 110, and the

devices

121, 122, and 123 may be connected to the first IoT server 110. It may not be required to have the capability of complete network communication (e.g., Internet communication) for direct connection to the IoT server 110.

Devices

121, 122, and 123 may include, for example, a light switch, a proximity sensor, Alternatively, it is shown as being implemented as an electronic device in a home environment, such as a temperature sensor, but this is by way of example and not limiting.

According to one embodiment, the first IoT server 110 may support direct communication with the

devices

124 and 125. Here, “direct communication” may mean communication that does not go through a relay device such as the first node 120, for example, communication through a cellular communication network and/or a data network.

According to one embodiment, the first IoT server 110 may transmit a control command to at least some of the

devices

121, 122, 123, 124, and 125. Here, “control command” may mean data that causes a controllable device to perform a specific operation, and the specific operation is an operation performed by the device, such as outputting information, sensing information, reporting information, Alternatively, it may include management of information (e.g. deletion or creation), and there is no limit to the type. For example, the processor 112 generates a control command from an external source (e.g., the voice assistant server 130, the second IoT server 140, the external system 160, or at least some of the

devices

121, 122, 123, 124, and 125). Information (or request) to do so may be obtained, and a control command may be generated based on the obtained information. Alternatively, the processor 112 may generate a control command based on the monitoring results of at least some of the

devices

121, 122, 123, 124, and 125 satisfying specified conditions. The processor 112 may control the communication interface 111 to transmit control commands to the target device.

According to one embodiment, the processor 112, or the processor 132, or the processor 142 includes a central processing unit (CPU), a digital signal processor (DSP), an application processor (AP), or a communication processor (CP). It may be implemented as a combination of one or more of a general-purpose processor, a graphical processing unit (GPU), a graphics-specific processor such as a vision processing unit (VPU), or an artificial intelligence-specific processor such as a neural processing unit (NPU). Those skilled in the art will understand that the above-described processing unit is merely an example, and that the processor 112 is not limited to any computational means that can, for example, execute instructions stored in the memory 113 and output the executed result.

According to one embodiment, the processor 112 may configure a web-based interface based on the API 114 or expose resources managed by the first IoT server 110 to the outside. . The web-based interface may support communication between the first IoT server 110 and an external web service, for example. The processor 112 may, for example, allow the external system 160 to control and/or access the

devices

121, 122, and 123. External system 160 may be, for example, an independent system that is not related to or part of system 100. External system 160 may be, for example, an external server or a website. However, security is required for access to the

devices

121, 122, and 123 from the external system 160 or the resources of the first IoT server 110. According to one embodiment, the processor 112 and the automation application may expose an API endpoint (eg, a universal resource locator (URL)) based on the API 114 to the outside. According to the above description, the first IoT server 110 may transmit a control command to the target device among the

devices

121, 122, and 123. Meanwhile, the description of the communication interface 141, the processor 142, and the API 144 and database 145 of the storage unit 143 of the second IoT server 140 are those of the first IoT server 110. It may be substantially the same as the description of the communication interface 111, the processor 112, the API 114 of the storage unit 113, and the database 115. In addition, the description of the second node 150 may be substantially the same as the description of the first node 120. The second IoT server 140 may transmit a control command to a target device among the

devices

151, 152, and 153. The first IoT server 110 and the second IoT server 140 may be operated by the same service provider in one embodiment, but may be operated by different service providers in another embodiment.

According to one embodiment, the voice assistant server 130 may transmit and receive data with the first IoT server 110 through the data network 116. The voice assistant server 130 according to one embodiment may include at least one of a communication interface 131, a processor 132, and a storage unit 133. The communication interface 131 may communicate with the smart phone 136 or the AI speaker 137 through a data network (not shown) and/or a cellular network (not shown). The smart phone 136 or the AI speaker 137 may include a microphone, acquire a user voice, convert it into a voice signal, and transmit the voice signal to the voice assistant server 130. The processor 132 may receive a voice signal from the smart phone 136 or the AI speaker 137 through the communication interface 131. The processor 132 may process the received voice signal based on the stored model 134. The processor 132 may generate (or confirm) a control command using the processing result based on information stored in the database 135. According to one embodiment, the

storage units

113, 133, and 143 include flash memory type, hard disk type, multimedia card micro type, and card type memory (e.g. SD or ), it may include at least one type of non-transitory storage medium among magnetic memory, magnetic disk, or optical disk, and there is no limitation on its type.

Figure 2 is a block diagram of an electronic device 201 in a network environment 200, according to one embodiment.

Referring to FIG. 2, in the network environment 200, the electronic device 201 communicates with the electronic device 202 through the first network 298 (e.g., a short-range wireless communication network) or through the second network 299. It is possible to communicate with at least one of the electronic device 204 or the server 208 through (e.g., a long-distance wireless communication network). According to one embodiment, the electronic device 201 may communicate with the electronic device 204 through the server 208. According to one embodiment, the electronic device 201 includes a processor 220, a memory 230, an input module 250, an audio output module 255, a display module 260, an audio module 270, and a sensor module ( 276), interface 277, connection terminal 278, haptic module 279, camera module 280, power management module 288, battery 289, communication module 290, subscriber identification module 296 , or may include an antenna module 297. In some embodiments, at least one of these components (eg, the connection terminal 278) may be omitted, or one or more other components may be added to the electronic device 201. In some embodiments, some of these components (e.g., sensor module 276, camera module 280, or antenna module 297) are integrated into one component (e.g., display module 260). It can be.

Processor 220, for example, executes software (e.g., program 240) to operate at least one other component (e.g., hardware or software component) of electronic device 201 connected to processor 220. It can be controlled and various data processing or calculations can be performed. According to one embodiment, as at least part of data processing or computation, the processor 220 stores instructions or data received from another component (e.g., sensor module 276 or communication module 290) in volatile memory 232. The commands or data stored in the volatile memory 232 can be processed, and the resulting data can be stored in the non-volatile memory 234. According to one embodiment, the processor 220 may include a main processor 221 (e.g., a central processing unit or an application processor) or an auxiliary processor 223 (e.g., a graphics processing unit, a neural network processing unit) that can operate independently or together with the main processor 221. It may include a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor). For example, if the electronic device 201 includes a main processor 221 and a auxiliary processor 223, the auxiliary processor 223 may be set to use lower power than the main processor 221 or be specialized for a designated function. You can. The auxiliary processor 223 may be implemented separately from the main processor 221 or as part of it.

The auxiliary processor 223 may, for example, act on behalf of the main processor 221 while the main processor 221 is in an inactive (e.g., sleep) state, or while the main processor 221 is in an active (e.g., application execution) state. ), together with the main processor 221, at least one of the components of the electronic device 201 (e.g., the display module 260, the sensor module 276, or the communication module 290) At least some of the functions or states related to can be controlled. According to one embodiment, coprocessor 223 (e.g., image signal processor or communication processor) may be implemented as part of another functionally related component (e.g., camera module 280 or communication module 290). there is. According to one embodiment, the auxiliary processor 223 (eg, neural network processing device) may include a hardware structure specialized for processing artificial intelligence models. Artificial intelligence models can be created through machine learning. For example, such learning may be performed in the electronic device 201 itself on which the artificial intelligence model is performed, or may be performed through a separate server (e.g., server 208). Learning algorithms may include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but It is not limited. An artificial intelligence model may include multiple artificial neural network layers. Artificial neural networks include deep neural network (DNN), convolutional neural network (CNN), recurrent neural network (RNN), restricted boltzmann machine (RBM), belief deep network (DBN), bidirectional recurrent deep neural network (BRDNN), It may be one of deep Q-networks or a combination of two or more of the above, but is not limited to the examples described above. In addition to hardware structures, artificial intelligence models may additionally or alternatively include software structures.

The memory 230 may store various data used by at least one component (eg, the processor 220 or the sensor module 276) of the electronic device 201. Data may include, for example, input data or output data for software (e.g., program 240) and instructions related thereto. Memory 230 may include volatile memory 232 or non-volatile memory 234.

The program 240 may be stored as software in the memory 230 and may include, for example, an operating system 242, middleware 244, or application 246.

The input module 250 may receive commands or data to be used in a component of the electronic device 201 (e.g., the processor 220) from outside the electronic device 201 (e.g., a user). The input module 250 may include, for example, a microphone, mouse, keyboard, keys (eg, buttons), or digital pen (eg, stylus pen).

The sound output module 255 may output sound signals to the outside of the electronic device 201. The sound output module 255 may include, for example, a speaker or a receiver. Speakers can be used for general purposes such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. According to one embodiment, the receiver may be implemented separately from the speaker or as part of it.

The display module 260 can visually provide information to the outside of the electronic device 201 (eg, a user). The display module 260 may include, for example, a display, a hologram device, or a projector, and a control circuit for controlling the device. According to one embodiment, the display module 260 may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of force generated by the touch.

The audio module 270 can convert sound into an electrical signal or, conversely, convert an electrical signal into sound. According to one embodiment, the audio module 270 acquires sound through the input module 250, the sound output module 255, or an external electronic device (e.g., directly or wirelessly connected to the electronic device 201). Sound may be output through an electronic device 202 (e.g., speaker or headphone).

The sensor module 276 detects the operating state (e.g., power or temperature) of the electronic device 201 or the external environmental state (e.g., user state) and generates an electrical signal or data value corresponding to the detected state. can do. According to one embodiment, the sensor module 276 includes, for example, a gesture sensor, a gyro sensor, an air pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, It may include a temperature sensor, humidity sensor, or light sensor.

The interface 277 may support one or more designated protocols that can be used to connect the electronic device 201 directly or wirelessly with an external electronic device (eg, the electronic device 202). According to one embodiment, the interface 277 may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

The connection terminal 278 may include a connector through which the electronic device 201 can be physically connected to an external electronic device (eg, the electronic device 202). According to one embodiment, the connection terminal 278 may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (eg, a headphone connector).

The haptic module 279 can convert electrical signals into mechanical stimulation (e.g., vibration or movement) or electrical stimulation that the user can perceive through tactile or kinesthetic senses. According to one embodiment, the haptic module 279 may include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

The camera module 280 can capture still images and moving images. According to one embodiment, the camera module 280 may include one or more lenses, image sensors, image signal processors, or flashes.

The power management module 288 can manage power supplied to the electronic device 201. According to one embodiment, the power management module 288 may be implemented as at least a part of, for example, a power management integrated circuit (PMIC).

Battery 289 may supply power to at least one component of electronic device 201. According to one embodiment, the battery 289 may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

Communication module 290 provides a direct (e.g., wired) communication channel or wireless communication channel between electronic device 201 and an external electronic device (e.g., electronic device 202, electronic device 204, or server 208). It can support establishment and communication through established communication channels. Communication module 290 operates independently of processor 220 (e.g., an application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module 290 may be a wireless communication module 292 (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module 294 (e.g., : LAN (local area network) communication module, or power line communication module) may be included. Among these communication modules, the corresponding communication module is a first network 298 (e.g., a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network 299 (e.g., legacy It may communicate with an external electronic device 204 through a telecommunication network such as a cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., LAN or WAN). These various types of communication modules may be integrated into one component (e.g., a single chip) or may be implemented as a plurality of separate components (e.g., multiple chips). The wireless communication module 292 uses subscriber information (e.g., International Mobile Subscriber Identifier (IMSI)) stored in the subscriber identification module 296 within a communication network such as the first network 298 or the second network 299. The electronic device 201 can be confirmed or authenticated.

The wireless communication module 292 may support 5G networks after 4G networks and next-generation communication technologies, for example, NR access technology (new radio access technology). NR access technology provides high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimization of terminal power and access to multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency (URLLC (ultra-reliable and low latency). -latency communications)) can be supported. The wireless communication module 292 may support high frequency bands (e.g., mmWave bands), for example, to achieve high data rates. The wireless communication module 292 uses various technologies to secure performance in high frequency bands, for example, beamforming, massive array multiple-input and multiple-output (MIMO), and full-dimensional multiplexing. It can support technologies such as input/output (FD-MIMO: full dimensional MIMO), array antenna, analog beam-forming, or large scale antenna. The wireless communication module 292 may support various requirements specified in the electronic device 201, an external electronic device (e.g., electronic device 204), or a network system (e.g., second network 299). According to one embodiment, the wireless communication module 292 supports Peak data rate (e.g., 20 Gbps or more) for realizing eMBB, loss coverage (e.g., 164 dB or less) for realizing mmTC, or U-plane latency (e.g., 164 dB or less) for realizing URLLC. Example: Downlink (DL) and uplink (UL) each of 0.5 ms or less, or round trip 1 ms or less) can be supported.

The antenna module 297 may transmit or receive signals or power to or from the outside (e.g., an external electronic device). According to one embodiment, the antenna module 297 may include an antenna including a radiator made of a conductor or a conductive pattern formed on a substrate (eg, PCB). According to one embodiment, the antenna module 297 may include a plurality of antennas (eg, an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network such as the first network 298 or the second network 299 is, for example, connected to the plurality of antennas by the communication module 290. can be selected. Signals or power may be transmitted or received between the communication module 290 and an external electronic device through the at least one selected antenna. According to some embodiments, in addition to the radiator, other components (eg, radio frequency integrated circuit (RFIC)) may be additionally formed as part of the antenna module 297.

According to various embodiments, the antenna module 297 may form a mmWave antenna module. According to one embodiment, a mmWave antenna module includes: a printed circuit board, an RFIC disposed on or adjacent to a first side (e.g., bottom side) of the printed circuit board and capable of supporting a designated high frequency band (e.g., mmWave band); And a plurality of antennas (e.g., array antennas) disposed on or adjacent to the second side (e.g., top or side) of the printed circuit board and capable of transmitting or receiving signals in the designated high frequency band. can do.

At least some of the components are connected to each other through a communication method between peripheral devices (e.g., bus, general purpose input and output (GPIO), serial peripheral interface (SPI), or mobile industry processor interface (MIPI)) and signal ( (e.g. commands or data) can be exchanged with each other.

According to one embodiment, commands or data may be transmitted or received between the electronic device 201 and the external electronic device 204 through the server 208 connected to the second network 299. Each of the external

electronic devices

202 or 204 may be of the same or different type as the electronic device 201. According to one embodiment, all or part of the operations performed in the electronic device 201 may be executed in one or more of the external

electronic devices

202, 204, or 208. For example, when the electronic device 201 needs to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device 201 may perform the function or service instead of executing the function or service on its own. Alternatively, or additionally, one or more external electronic devices may be requested to perform at least part of the function or service. One or more external electronic devices that have received the request may execute at least part of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device 201. The electronic device 201 may process the result as is or additionally and provide it as at least part of a response to the request. For this purpose, for example, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology can be used. The electronic device 201 may provide an ultra-low latency service using, for example, distributed computing or mobile edge computing. In another embodiment, the external electronic device 204 may include an Internet of Things (IoT) device. Server 208 may be an intelligent server using machine learning and/or neural networks. According to one embodiment, the external electronic device 204 or server 208 may be included in the second network 299. The electronic device 201 may be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

Electronic devices according to various embodiments disclosed in this document may be of various types. Electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, wearable devices, or home appliances. Electronic devices according to embodiments of this document are not limited to the above-described devices.

The various embodiments of this document and the terms used herein are not intended to limit the technical features described in this document to specific embodiments, and should be understood to include various changes, equivalents, or replacements of the embodiments. In connection with the description of the drawings, similar reference numbers may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the above items, unless the relevant context clearly indicates otherwise. As used herein, “A or B”, “at least one of A and B”, “at least one of A or B”, “A, B or C”, “at least one of A, B and C”, and “A Each of phrases such as “at least one of , B, or C” may include any one of the items listed together in the corresponding phrase, or any possible combination thereof. Terms such as "first", "second", or "first" or "second" may be used simply to distinguish one component from another, and to refer to that component in other respects (e.g., importance or order) is not limited. One (e.g., first) component is said to be “coupled” or “connected” to another (e.g., second) component, with or without the terms “functionally” or “communicatively.” When mentioned, it means that any of the components can be connected to the other components directly (e.g. wired), wirelessly, or through a third component.

The term “module” used in various embodiments of this document may include a unit implemented in hardware, software, or firmware, and is interchangeable with terms such as logic, logic block, component, or circuit, for example. It can be used as A module may be an integrated part or a minimum unit of the parts or a part thereof that performs one or more functions. For example, according to one embodiment, the module may be implemented in the form of an application-specific integrated circuit (ASIC).

Various embodiments of the present document are one or more instructions stored in a storage medium (e.g., built-in memory 236 or external memory 238) that can be read by a machine (e.g., electronic device 201). It may be implemented as software (e.g., program 240) including these. For example, a processor (e.g., processor 220) of a device (e.g., electronic device 201) may call at least one command among one or more commands stored from a storage medium and execute it. This allows the device to be operated to perform at least one function according to the at least one instruction called. The one or more instructions may include code generated by a compiler or code that can be executed by an interpreter. A storage medium that can be read by a device may be provided in the form of a non-transitory storage medium. Here, “non-transitory” simply means that the storage medium is a tangible device and does not contain signals (e.g. electromagnetic waves), and this term refers to cases where data is semi-permanently stored in the storage medium. There is no distinction between temporary storage cases.

According to one embodiment, methods according to various embodiments disclosed in this document may be provided and included in a computer program product. Computer program products are commodities and can be traded between sellers and buyers. The computer program product may be distributed in the form of a machine-readable storage medium (e.g. compact disc read only memory (CD-ROM)), or through an application store (e.g. Play Store ^TM ) or on two user devices (e.g. It can be distributed (e.g. downloaded or uploaded) directly between smart phones) or online. In the case of online distribution, at least a portion of the computer program product may be at least temporarily stored or temporarily created in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

According to various embodiments, each component (e.g., module or program) of the above-described components may include a single or plural entity, and some of the plurality of entities may be separately placed in other components. there is. According to various embodiments, one or more of the components or operations described above may be omitted, or one or more other components or operations may be added. Alternatively or additionally, multiple components (eg, modules or programs) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each component of the plurality of components in the same or similar manner as those performed by the corresponding component of the plurality of components prior to the integration. . According to various embodiments, operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, or omitted. Alternatively, one or more other operations may be added.

Figure 3 is a block diagram of an electronic device 301, according to one embodiment.

Referring to Figure 3, in one embodiment, the electronic device 301 is one of the

devices

121, 122, 123, 124, 125, 136, 137, 151, 152, and 153 of Figure 1, or a hub ( hub) device (eg, the first node 120 or the second node 150). In one embodiment, the electronic device 301 may be the same or similar to the electronic device 201 of FIG. 2 .

In one embodiment, the electronic device 301 may include a communication module 310, a display module 320, a camera module 330, a memory 340, and/or a processor 350.

In one embodiment, communication module 310 may be communication module 290 of FIG. 2 .

In one embodiment, the communication module 310 may allow the electronic device 301 to communicate with an external electronic device (e.g., at least one of the electronic device 202, the electronic device 204, or the server 208). there is. For example, the communication module 310 may receive an image from an external electronic device (eg, a camera device). For example, the communication module 310 may transmit information related to an object to an external electronic device (eg, at least one of the electronic device 202, the electronic device 204, or the server 208).

In one embodiment, the display module 320 may be the display module 260 of FIG. 2 .

In one embodiment, display module 320 may display information related to an image or object. However, the present invention is not limited to this, and the information displayed by the display module 320 will be described later.

In one embodiment, the camera module 330 may be the camera module 280 of FIG. 2 .

In one embodiment, camera module 330 may acquire images. For example, the camera module 330 may acquire a still image or a dynamic image (eg, a video). The camera module 330 may transmit the acquired image to the processor 350.

In one embodiment, memory 340 may be memory 230 of FIG. 2 .

In one embodiment, the memory 340 may store information for performing an operation to detect an object. Information for performing an operation to detect an object stored in the memory 340 will be described later.

In one embodiment, processor 350 may be processor 220 of FIG. 2 .

In one embodiment, the processor 350 may generally control operations for detecting objects. The processor 350 may include a plurality of components to perform an operation to detect an object. A plurality of components included in the processor 350 will be described with reference to FIG. 4.

In one embodiment, processor 350 may include one or more processors to perform an operation to detect an object.

In Figure 3, the electronic device 301 is illustrated as including a communication module 310, a display module 320, a camera module 330, a memory 340, and/or a processor 350, but is not limited thereto. . For example, the electronic device 301 may not include a communication module 310, a display module 320, and/or a camera module 330. For example, the electronic device 301 may further include at least one configuration shown in FIG. 2 in addition to the configurations shown in FIG. 3 .

FIG. 4 is a diagram for explaining the processor 350 according to one embodiment.

Referring to Figure 4, in one embodiment, the processor 350 includes an image acquisition module 410, a region of interest acquisition module 420, a patch generation module 430, a patch mixing module 440, and a mixing It may include a frame size determination module 450, an inference engine 460, and/or a reconstruction module 470.

In one embodiment, the image acquisition module 410 may acquire at least one image (eg, an image frame).

In one embodiment, the image acquisition module 410 may acquire at least one image from an external electronic device through the communication module 310. For example, the image acquisition module 410 may receive at least one image acquired from an external electronic device (eg, a camera device) through the communication module 310. For example, the image acquisition module 410 is configured to stream (e.g., real-time) or download from a server (e.g., web server) through the communication module 310, at least One image can be received.

In one embodiment, the image acquisition module 410 may acquire at least one image from the memory 340 . In one embodiment, the image acquisition module 410 may acquire at least one image through a camera module (eg, camera module 280) included in the electronic device 301.

In one embodiment, the image acquisition module 410 may continuously acquire a plurality of images (eg, video). However, the image acquisition module 410 is not limited to this and may acquire one image.

In one embodiment, the image acquisition module 410 may acquire a plurality of images (eg, a plurality of images) from a plurality of external electronic devices (eg, a plurality of camera devices). In one embodiment, the image acquisition module 410 may acquire at least one image from one external electronic device.

In one embodiment, the region of interest acquisition module 420 selects at least one region of interest associated with an object within at least one image (hereinafter referred to as “at least one image”) acquired through the image acquisition module 410. (region of interest; ROI) can be acquired (e.g. extracted). For example, the region of interest acquisition module 420 may acquire at least one region of interest from each of at least one image.

In one embodiment, the region of interest may be an area containing an object whose movement is detected within the image. For example, the region of interest may be an area in which movement of at least a portion of an object is detected within an image (eg, within a plurality of image frames acquired continuously). For example, the region of interest may be an area surrounding an object whose movement is detected within an image. For example, a region of interest is an area in the form of a bounding box that contains the outline of an object whose movement has been detected in the image (e.g., 2 parallel to the width of the image and passing through the outermost upper and lower points of the outline). It may be a bounding box formed by two horizontal lines and two vertical lines parallel to the length of the image and passing through the outermost left and right points of the outline.

In one embodiment, the region of interest acquisition module 420 may acquire at least one region of interest within at least one image using a designated algorithm.

In one embodiment, the region-of-interest acquisition module 420 uses a region-of-interest extraction algorithm (referred to as “pixel difference based POI extraction”) based on pixel differences between consecutive images to obtain at least one POI within at least one image. The area of interest can be obtained. For example, the region of interest acquisition module 420 may calculate differences in pixel values for each corresponding pixel in continuously acquired images (eg, continuously acquired image frames). The region of interest acquisition module 420 may detect a region in the image that has a difference greater than or equal to a threshold difference among the differences between calculated pixel values. The region of interest acquisition module 420 may detect the edge of the detected region using an edge detection algorithm (eg, Canny edge detection algorithm). The region of interest acquisition module 420 may acquire a contour from the detected edge. The region of interest acquisition module 420 may acquire the region of interest based on the acquired outline.

In one embodiment, the region of interest acquisition module 420 uses a region of interest extraction algorithm based on a previously detected region of interest (referred to as “inference history based POI extraction”) to extract at least one POI within at least one image. Areas of interest can be obtained. For example, the region of interest acquisition module 420 may track a previously detected region of interest (e.g., bounding box) using an optical flow algorithm (e.g., through a previously performed region of interest detection operation). there is. For example, the region of interest acquisition module 420 uses an optical flow algorithm to extract the direction in which the object moves within the previous image (e.g., previous image frame) and the current image (e.g., current image frame), thereby The detected area of interest can be tracked. The region of interest acquisition module 420 may acquire a region of interest within an image (eg, the current image frame) by tracking a previously detected region of interest.

However, the algorithm used by the region of interest acquisition module 420 is not limited to the region of interest extraction algorithm based on pixel differences between consecutive images and the region of interest extraction algorithm based on previously detected regions of interest.

In one embodiment, the patch generation module 430 (also referred to as a “patch generator”) may obtain at least one first region corresponding to at least one region of interest. For example, the patch generation module 430 may generate at least one first region (hereinafter referred to as “at least one”), each corresponding to at least one region of interest and included (e.g., to be mixed) in a mixed frame to be described later. (referred to as “the first area of”) can be obtained.

In one embodiment, the patch creation module 430 may obtain a first region (hereinafter also referred to as a “patch”) by adjusting the size of the region of interest (eg, enlarging or reducing it).

In one embodiment, patch generation module 430 adjusts the size of the region of interest to a size that corresponds to a specified accuracy (also referred to as “target accuracy”) (e.g., a preset accuracy), thereby You can obtain a patch corresponding to the area. In one embodiment, accuracy (e.g., accuracy value) may refer to the degree of agreement (or similarity rate) between the output data and the ground truth for the input data of the inference engine 460.

In one embodiment, the accuracy may be higher as the size of the image (e.g., patch size) input to the inference engine 460 is larger, and may be lower as the size of the image input to the inference engine 460 is smaller. In one embodiment, the smaller the size of the image input to the inference engine 460, the faster the speed at which the inference engine 460 performs the inference operation. The larger the size of the image input to the inference engine 460, the faster the speed at which the inference engine 460 performs the inference operation. , the speed at which the inference engine 460 performs inference operations may be slow.

In one embodiment, the relationship between accuracy and image size may vary depending on the artificial intelligence model (e.g., data it was trained on). In one embodiment, the relationship between accuracy and image size may vary depending on the type of object being detected. For example, the relationship between accuracy and image size when the object to be detected through an inference operation is a person, and the relationship between accuracy and image size when the object to be detected through an inference operation is an animal (e.g., a companion animal). The relationship between the sizes may be different.

In one embodiment, a relationship between accuracy and image size (e.g., a graph between accuracy and image size) may be stored in memory 340. For example, the relationship between accuracy and image size may be learned in advance and stored in memory 340. For example, the relationship between accuracy and image size may be received from an external electronic device and then stored in memory 340.

In one embodiment, the patch generation module 430 may determine the size of the image corresponding to the specified accuracy based on the relationship between the accuracy and the size of the image.

In one embodiment, the patch creation module 430 enlarges or reduces the size of each of the at least one region of interest to the size of the image corresponding to a specified accuracy (also referred to as “resizing”), At least one first area can be acquired.

In one embodiment, when the type of object included in the region of interest is not confirmed, the patch creation module 430 selects the largest size among the sizes of the image (patch) for each object corresponding to the specified accuracy, as the first It can be determined by the size of the area. For example, the inference engine 460 may be configured to perform the following inference operations: detecting a person, detecting a cat, and detecting a dog. The patch generation module 430 performs inference before the inference engine 460 performs an inference operation (e.g., when the patch creation module 430 first performs an operation on a region of interest) or prior inference of the inference engine 460. If an object is not detected in motion, the type of object included in the area of interest may not be confirmed. In this case, the patch creation module 430 determines the relationship between accuracy and image size when the type of object is a person, the relationship between accuracy and size of the image when the type of object is a cat, and the relationship between accuracy and image size when the type of object is a dog. In each relationship between sizes, the sizes of the image corresponding to the specified accuracy can be confirmed. The patch creation module 430 may determine the largest size among the confirmed sizes as the size of the first area (patch).

In one embodiment, when the type of object included in the region of interest is identified, the patch creation module 430 responds to the specified accuracy based on the relationship between the size of the image and the accuracy for detecting the identified type of object. You can check the size of the image. The patch creation module 430 may determine the size of the confirmed image as the size of the first area. For example, the inference engine 460 may be set to perform an operation to detect a person. The patch creation module 430 may confirm that the type of object detected in the region of interest as a result of the previous inference operation is a person. For example, the patch creation module 430 performs the previous inference operation of the inference engine 460, which is fed back to the patch creation module 430 via the reconstruction module 470 and the region of interest acquisition module 420. Based on the results of , it can be confirmed that the type of object detected in the area of interest as a result of the previous inference operation is a person. If the type of object is a person, the patch creation module 430 may check the size of the image corresponding to the specified accuracy based on the relationship between accuracy and image size. The processor 350 may determine the size of the confirmed image as the size of the first area.

In one embodiment, the size of the image corresponding to the specified accuracy is the smallest among the sizes of the image corresponding to the accuracies that satisfy the specified accuracy (e.g., accuracies that are greater than or equal to the specified accuracy) in the relationship between the accuracy and the size of the image. It could be size. For example, the smaller the size of the image (e.g., the size of the patch), the faster the processing speed of the operation to detect the object. Accordingly, the size of the image corresponding to the specified accuracy may be determined as the size of the first area so that an operation for detecting the object is performed at the fastest speed while satisfying the specified accuracy.

In one embodiment, the patch mixing module 440 (also referred to as a “patch mixer”) may obtain a frame (hereinafter referred to as a “mixed frame”) that includes at least one first region. there is. For example, the patch mixing module 440 may mix (or combine) at least one first region within a bin.

In one embodiment, the patch mixing module 440 may obtain a mixed frame including at least one first area using a bin packing algorithm. For example, the patch mixing module 440 may include at least one first region in one bin as much as possible using a bin packing algorithm.

In one embodiment, the mixed frame may include at least one first area obtained based on one image. In one embodiment, the mixed frame may include at least one first area obtained based on a plurality of images. For example, the mixed frame may include at least one first region in which at least one region of interest obtained from a plurality of sequentially acquired images is enlarged or reduced.

In one embodiment, the patch mixing module 440 may assign priority to (eg, allocate) at least one first region.

In one embodiment, the patch mixing module 440 may give priority to at least one first region based on the reverse order of the order in which the at least one image (image frame) was acquired. For example, the patch mixing module 440 assigns the highest priority to the first region obtained from the currently acquired image and the lowest priority to the first region obtained from the earliest acquired image. It can be granted.

In one embodiment, the patch mixing module 440 may give priority to at least one first region based on the number of first regions (or regions of interest) obtained from at least one image. For example, the patch mixing module 440 assigns higher priority to the first region obtained from images with a smaller number of first regions, and gives higher priority to the first region obtained from images with a larger number of first regions. A lower priority may be given to the first area obtained from . Through this prioritization operation, the patch mixing module 440 can accurately perform an object detection operation even for images in which a smaller first area is obtained among a plurality of images.

In one embodiment, the patch mixing module 440 may include at least one first region in one bin using a bin packing algorithm based on priority. For example, the patch mixing module 440 may obtain a mixed frame using a bin packing algorithm so that the first region with high priority is preferentially included in the bin.

In one embodiment, the patch mixing module 440 may not include some of the at least one first region in the first region if it is not possible to include all of the at least one first region in one bin. . For example, the patch mixing module 440 includes the first region in one bin using a bin packing algorithm, in order of the first region with high priority, and the first region that cannot be included in one bin. 1 Areas can be deleted (or discarded).

In one embodiment, the bin packing algorithm may include a guillotine bin packing algorithm, a shelf algorithm, or a maximal rectangle algorithm. However, the bin packing algorithm is not limited to the above-described algorithm.

In one embodiment, mixed frame size determination module 450 (also referred to as a “mixed frame size planner”) may determine the size of a mixed frame.

In one embodiment, the mixed frame size determination module 450 may determine the size of the mixed frame based on the type and number of artificial intelligence models used by the inference engine 460.

In one embodiment, the mixed frame size determination module 450 is based on the performance (and/or computing resources) of the electronic device 301 and the type and/or number of artificial intelligence models used by the inference engine 460, The size of the mixed frame may be determined so that the inference engine 460 performs the inference operation within a specified time. For example, when the inference engine 460 is set to perform a face detection operation, the mixed frame size determination module 450 performs a face detection operation for one mixed frame at a specified time ( Example: You can determine the size of the blending frame so that it executes in about 1 second). For example, when the inference engine 460 is set to perform a face detection operation and an animal detection operation simultaneously (or sequentially), the mixed frame size determination module 450 may perform the inference engine 460 for one mixed frame. The size of the blended frame can be determined so that the face detection operation and the animal detection operation are performed within a specified time (e.g., about 1 second). In one embodiment, the designated time may be a time requested (or designated) by an application that performs an operation to detect an object.

In one embodiment, the mixed frame size determination module 450 is based on the performance (and/or computing resources) of the electronic device 301 and the type and/or number of artificial intelligence models used by the inference engine 460, The maximum size of a mixed frame that can enable the inference engine 460 to perform an inference operation within a specified time can be determined as the size of the mixed frame. For example, when the inference engine 460 is controlled by a GPU (graphics processing unit), the larger the size of the mixed frame, the more data the inference engine 460 can process per hour. For example, rather than the speed at which the inference engine 460 processes five mixed frames of size 100*100 (e.g., 100 pixels horizontally and 100 pixels vertically), it processes 5 mixed frames of size 100*500. Branches can be fast in processing one mixed frame. Accordingly, the mixed frame size determination module 450 determines the object based on the performance (and/or computing resources) of the electronic device 301 and the type and/or number of artificial intelligence models used by the inference engine 460. The maximum size of the mixed frame that allows the application performing the detection operation to perform the operation within the required time can be determined by the size of the mixed frame.

In one embodiment, the mixed frame size determination module 450 may store in advance the relationship between the performance time of the inference operation and the size of the mixed frame for each type of artificial intelligence model in the memory 340. For example, the mixed frame size determination module 450 may profile the relationship between the execution time of an inference operation and the size of the mixed frame for each type of artificial intelligence model. For example, when information about the type of artificial intelligence model and the performance of the electronic device is confirmed, the mixed frame size determination module 450 considers the performance of the electronic device 301 and makes inferences for each type of artificial intelligence model. The relationship between the execution time of an operation and the size of the mixed frame can be profiled.

In one embodiment, the mixed frame size determination module 450 performs an operation to detect an object when the relationship between the performance time of the inference operation and the size of the mixed frame is stored in advance in the memory 340 for each type of artificial intelligence model. The type and number of artificial intelligence models set for this purpose can be checked, and the size of the mixed frame can be determined based on the type and number of confirmed artificial intelligence models.

In one embodiment, when the size of the mixed frame is determined, the mixed frame size determination module 450 may transmit the determined size of the mixed frame to the patch mixing module 440. The patch mixing module 440 may perform the above-described operations based on the determined size of the mixing frame.

In one embodiment, the inference engine 460 may perform an inference operation using an artificial intelligence model based on a mixed frame. For example, the inference engine 460 may perform an operation to detect an object (e.g., a person, an animal, and/or a human face) using an artificial intelligence model based on a mixed frame as input data. .

In one embodiment, the inference engine 460 provides, as result data of the inference operation, a region in which an object was detected within the mixed frame (hereinafter referred to as “at least one second region”) and a region within the at least one second region. You can obtain the type of object detected. For example, when the inference engine 460 is set to perform an operation of detecting a human face, the inference engine 460 may select at least one second area (e.g., at least one second area) where a face is detected within the mixed frame. coordinates of area 2) may be obtained, and it may be determined that a human face is detected within at least one second area. In one embodiment, each of the at least one second area may include coordinates indicating a bounding box corresponding to the area in which the object was detected.

In one embodiment, the inference engine 460 (or inference framework) may include tensorflow-lite or a mobile neural network (MNN). However, it is not limited to this.

In one embodiment, the reconstruction module 470 (reconstruction) may obtain information related to an object detected in at least one image based on at least one second area in which the object was detected within the mixed frame.

In one embodiment, the reconstruction module 470 stores at least one second region (and the type of object detected in the at least one second region) as result data of the inference operation to at least one second region included in the mixed frame. It can be compared to area 1. For example, the reconstruction module 470 may check intersection over unions (IoUs) between each of all first areas included in the mixed frame and each of at least one second area. The reconstruction module 470 may match the first area and the second area with the highest IoU among the confirmed IoUs. For example, when the plurality of second areas include the 2-1 area and the 2-2 area, the reconfiguration module 470 determines that the IoU with the 2-1 area is the highest among the plurality of first areas. The high 1-1 area can be matched with the 2-1 area, and the 1-2 area with the highest IoU with the 2-2 area among the plurality of first areas can be matched with the 2-2 area. .

In one embodiment, the reconstruction module 470 compares the first area and the second area to determine an area corresponding to at least one second area among the at least one first area included in the mixed frame (hereinafter, " Referred to as “at least one third area”) (e.g., at least one area each including at least one second area) can be identified.

In one embodiment, the reconstruction module 470 is configured to reconstruct within at least one image based on metadata of the at least one third region and the at least one third region (or at least one first region). The area where the object is detected and/or the type of the detected object may be obtained. For example, metadata includes identification information of the image from which the third region is obtained among at least one image (image frame) (e.g., identifier of the image including the image and index of the image within the image), Includes the location of a region of interest (e.g., a region of interest enlarged or reduced to obtain the third region) corresponding to the third region in the image in which the third region is acquired, and the position of the third region within the blended frame. can do. The reconstruction module 470, based on the metadata, the coordinates of the at least one second area, and the coordinates of the at least one third area, at least within each of the at least one image that was acquired by the image acquisition module 410 An area in which one object is detected (hereinafter referred to as “at least one fourth area”) (eg, coordinates of at least one fourth area) may be obtained. Additionally, the reconstruction module 470 may obtain the type of object detected in at least one fourth area.

In one embodiment, the information related to the object may include at least one fourth area and the type of object detected in the at least one fourth area. In one embodiment, at least some of the information related to the object output from the reconstruction module 470 may be transmitted (eg, feedback) to the region of interest acquisition module 420. For example, the type of object output from the reconstruction module 470 may be transmitted to the region of interest acquisition module 420.

In one embodiment, the processor 350 may periodically perform an operation to detect an object.

In one embodiment, when an object is detected in at least one image as a result of the inference operation, the processor 350 tracks the region of interest in the acquired at least one image without performing some of the operations for detecting the object. (tracking) operations can be performed. For example, when an object is detected in at least one image, the processor 350 performs the patch generation module 430, the patch mixing module 440, the mixed frame size determination module 450, and the inference engine 460. , or controlling at least some of the reconstruction modules 470 not to operate, and tracking a region of interest within at least one subsequently acquired image based on the type of the object and/or the at least one fourth region that was acquired. You can only perform the following actions.

In one embodiment, when an operation for detecting an object is performed on a plurality of images, the processor 350 performs some of the operations for detecting the object on the image in which the object is detected among the plurality of images. An operation of tracking a region of interest within at least one acquired image may be performed without performing the operation. The processor 350 may cause all components included in the processor 350 to perform an operation on an image in which an object is not detected among the plurality of images.

In one embodiment, the processor 350 may transmit at least part of the information related to the object to an external electronic device through the communication module 310. For example, when an electronic device operates as an IoT device or hub device on an IoT network, the processor 350 provides at least one image and the type of object detected within the at least one image to the IoT service provider. Transfers may be made to electronic devices (e.g., electronic devices of users whose accounts are registered with the server).

In one embodiment, the processor 350 may display at least a portion of information related to an object through the display module 320. For example, the processor 350, through the display module 320, provides an indication (e.g., bounding box) indicating a fourth area in which an object is detected in at least one image, and an indication within the fourth area. Information indicating the type of object detected can be displayed.

An electronic device (e.g., electronic device 301) according to an embodiment includes a memory (e.g., memory 340) and at least one processor (e.g., memory 340) operatively connected to the memory (e.g., memory 340). It may include a processor 350). The at least one processor (eg, processor 350) may be configured to acquire at least one image. The at least one processor (eg, processor 350) may be configured to obtain at least one region of interest related to an object within the at least one image. The at least one processor (eg, processor 350) may be configured to obtain at least one first region corresponding to the at least one region of interest. The at least one processor (eg, processor 350) may be configured to obtain a frame including the at least one first area. The at least one processor (eg, processor 350) may be configured to obtain information related to the object from the frame using an artificial intelligence model.

In one embodiment, the at least one processor (eg, processor 350) may be configured to obtain the at least one first region by enlarging or reducing the at least one region of interest.

In one embodiment, the at least one processor (eg, processor 350) may determine the size of the image corresponding to the specified accuracy based on the relationship between the accuracy and the size of the image. The at least one processor (eg, processor 350) may be configured to obtain the at least one first region by enlarging or reducing the size of the at least one region of interest to the confirmed size.

In one embodiment, the relationship between the accuracy and the size of the image may vary depending on the type of artificial intelligence model and/or the object.

In one embodiment, the at least one processor (eg, processor 350) may check the performance of the electronic device, the type of the artificial intelligence model, and/or the number of the artificial intelligence model. The at least one processor (e.g., processor 350) may be configured to determine the size of the frame based on the performance of the electronic device, the type of the artificial intelligence model, and/or the number of the artificial intelligence model. You can.

In one embodiment, the at least one processor (e.g., processor 350) obtains the frame by mixing the at least one first region using a bin packing algorithm based on the size of the frame. It can be configured.

In one embodiment, the at least one processor (e.g., processor 350) may assign priority to the at least one first area based on the reverse order of the order in which the at least one image was acquired. You can. The at least one processor (eg, processor 350) may be configured to mix the at least one first region based on the priority.

In one embodiment, the at least one processor (e.g., processor 350) uses the artificial intelligence model to determine at least one second area where the object is detected within the frame and the object detected within the frame. The type of the object can be obtained. The at least one processor (eg, processor 350) may identify at least one third area corresponding to the at least one second area among the at least one first area. The at least one processor (e.g., processor 350) based on the at least one second area, the at least one third area, and metadata of the at least one third area, the at least one It may be configured to obtain at least one fourth area where the object is detected within one image.

In one embodiment, the at least one image may include a plurality of images acquired from at least one external electronic device.

In one embodiment, the electronic device may be an IoT (internet of things) device or a hub device that can provide information related to the acquired object to an external electronic device.

FIG. 5 is a flowchart 500 illustrating a method for detecting an object, according to one embodiment.

Referring to Figure 5, in operation 501, in one embodiment, processor 350 may acquire at least one image.

In one embodiment, the processor 350 may acquire at least one image from an external electronic device through the communication module 310. For example, the processor 350 may receive at least one image from an external electronic device (eg, a camera device) through the communication module 310. For example, the processor 350 may stream (e.g., real-time) or download from a server (e.g., a web server) through the communication module 310, at least one Images can be received.

In one embodiment, the processor 350 may obtain at least one image from the memory 340.

In one embodiment, the processor 350 may continuously acquire a plurality of images (eg, video). However, the present invention is not limited to this, and the processor 350 may acquire one image.

In one embodiment, the processor 350 may acquire a plurality of images (eg, a plurality of images) from a plurality of external electronic devices (eg, a plurality of camera devices). In one embodiment, the processor 350 may acquire at least one image from one external electronic device.

At operation 503, in one embodiment, processor 350 may obtain at least one region of interest associated with an object within at least one image.

In one embodiment, the processor 350 selects at least one region of interest associated with an object within at least one image acquired through the image acquisition module 410 (hereinafter referred to as “at least one image”). of interest (ROI) can be acquired (e.g. extracted). For example, the processor 350 may obtain at least one region of interest from each of at least one image.

Hereinafter, the region of interest will be described with reference to FIG. 6.

FIG. 6 is an example diagram 600 for explaining a region of interest according to an embodiment.

Referring to FIG. 6, in one embodiment, the region of interest may be an area containing an object whose movement has been detected within the image. For example, the region of interest may be an area in which movement of at least a portion of an object is detected within an image (eg, within a plurality of frames). For example, the region of interest may be an area surrounding an object whose movement is detected within an image. For example, a region of interest is an area in the form of a bounding box that contains the outline of an object whose movement has been detected in the image (e.g., 2 parallel to the width of the image and passing through the outermost upper and lower points of the outline). It may be a bounding box formed by two horizontal lines and two vertical lines parallel to the length of the image and passing through the outermost left and right points of the outline. For example, in FIG. 6 , image 1 610 may be an image acquired from a first external electronic device, and image 2 620 may be an image obtained from a second external electronic device. The processor 350 extracts region of interest 1 (613) in the form of a bounding box associated with object 1 (611) in image 1 (610) and extracts a region of interest 1 (613) associated with object 2 (621) in image 2 (620). Area of interest 2 (623) of the shape can be extracted.

Referring back to FIG. 5 , in one embodiment, the processor 350 may obtain at least one region of interest within at least one image using a designated algorithm.

In one embodiment, the processor 350 may obtain at least one region of interest within at least one image using a region of interest extraction algorithm based on pixel differences between consecutive images. For example, the processor 350 may calculate differences in pixel values for each corresponding pixel in continuously acquired images (eg, continuously acquired image frames). The processor 350 may detect an area within the image that has a difference greater than or equal to a threshold difference among the differences between calculated pixel values. The processor 350 may detect the edge of the detected area using an edge detection algorithm (eg, Canny edge detection algorithm). The processor 350 may obtain a contour from the detected edge. The processor 350 may obtain a region of interest based on the obtained outline.

In one embodiment, the processor 350 may obtain at least one region of interest within at least one image using a region of interest extraction algorithm based on a previously detected region of interest. For example, the processor 350 may track a previously detected region of interest (eg, bounding box) (eg, through a previously performed region of interest detection operation) using an optical flow algorithm. For example, the processor 350 uses an optical flow algorithm to extract the direction in which the object moves within the previous image (e.g., previous image frame) and the current image (e.g., current image frame), thereby detecting the previously detected object. Areas of interest can be tracked. The processor 350 may obtain a region of interest within an image (eg, the current image frame) by tracking a previously detected region of interest.

However, the algorithm used by the processor 350 is not limited to the region-of-interest extraction algorithm based on pixel differences between consecutive images and the region-of-interest extraction algorithm based on previously detected regions of interest.

At operation 505, in one embodiment, the processor 350 may obtain at least one first region corresponding to at least one region of interest.

In one embodiment, the processor 350 may obtain at least one first region corresponding to at least one region of interest. For example, the processor 350 may select at least one first region (hereinafter referred to as “at least one first region”), each corresponding to at least one region of interest, to be included (e.g., to be mixed) in a mixed frame. ") can be obtained.

In one embodiment, the processor 350 may obtain a first region (hereinafter also referred to as a “patch”) by adjusting the size of the region of interest (eg, enlarging or reducing it).

In one embodiment, processor 350 adjusts the size of the region of interest to a size that corresponds to a specified accuracy (also referred to as “target accuracy”) (e.g., a preset accuracy) to determine the region of interest. You can obtain the corresponding patch. In one embodiment, accuracy (e.g., accuracy value) may refer to the degree of agreement (or similarity rate) between the output data and the ground truth for the input data of the inference engine 460.

Hereinafter, a method for determining the size of the first area will be described with reference to FIGS. 7 and 8.

FIG. 7 is a flowchart 700 illustrating a method of determining the size of a first area, according to one embodiment.

FIG. 8 is an example diagram 800 illustrating a method of determining the size of a first area, according to an embodiment.

Referring to FIGS. 7 and 8 , in one embodiment, determining the size of the first area may be included in the operation of obtaining at least one first area in operation 505 .

At operation 701, in one embodiment, processor 350 may confirm the type of artificial intelligence model and/or object. For example, the processor 350 may check an artificial intelligence model (eg, learned data) set to be used in an object detection operation. For example, if an inference operation was performed based on a previously acquired image, the processor 350 may confirm the type of object based on the result of the inference operation.

At operation 703, in one embodiment, the processor 350 may determine the size of the first area based on the artificial intelligence model and/or type of object.

In one embodiment, Figure 8 shows a graph representing relationships between object-specific accuracy and image size in an artificial intelligence model (e.g., YOLO-v4 model). In the graph of Figure 8, the X-axis represents the size of the image (e.g., the longer of the horizontal and vertical lengths of the image), and the Y-axis may represent accuracy. In the graph of Figure 8, line 1 (811) represents the relationship between accuracy and image size when the type of object is a human, and line 2 (813) represents the relationship between accuracy and size of the image when the type of object is a cat. , Line 3 (813) may represent the relationship between accuracy and image size when there are two types of objects.

In one embodiment, as shown in FIG. 8, the accuracy is higher as the size of the image (e.g., patch size) input to the inference engine 460 is larger, and the size of the image input to the inference engine 460 increases. The smaller it can be, the lower it can be.

In one embodiment, as shown in FIG. 8, the relationship between accuracy and image size may vary depending on the type of object to be detected. For example, if the accuracy (or accuracy value) required by the system or user is n, and the type of object is a person, the size of the image (e.g., the longer of the horizontal and vertical lengths of the patch) is determined as m1; If the type of object is a cat, the size of the image may be determined as m2, and if the type of object is a dog, the size of the image may be determined as m3.

In one embodiment, a relationship between accuracy and image size (e.g., a graph between accuracy and image size) may be stored in memory 340. For example, the relationship between accuracy and image size may be learned in advance and stored in memory 340. For example, the relationship between accuracy and image size may be received from an external electronic device and then stored in memory 340. A method of learning the relationship between accuracy and image size will be described later with reference to FIG. 13.

In one embodiment, processor 350 may determine the size of the image that corresponds to the specified accuracy based on the relationship between the accuracy and the size of the image.

In one embodiment, the processor 350 may acquire at least one first region by enlarging or reducing the size of each of the at least one region of interest to the size of the image corresponding to a specified accuracy.

In one embodiment, when the type of object included in the region of interest is not confirmed, the processor 350 determines the largest size among the sizes of images for each object corresponding to the specified accuracy as the size of the first region. You can. For example, the processor 350 may be set to perform an operation for detecting a person, an operation for detecting a cat, and an operation for detecting a dog as inference operations. The processor 350 may not be able to confirm the type of the object if the object is not detected before performing the inference operation or in the previous inference operation. In this case, the processor 350 determines the relationship between accuracy and image size when the type of object is a person, the relationship between accuracy and size of the image when the type of object is a cat, and the relationship between accuracy and size of the image when the type of object is a dog. For each relationship, we can identify the sizes of the image that correspond to the specified accuracy. The processor 350 may determine the largest size among the confirmed sizes as the size of the first area (patch). For example, in FIG. 8, if the specified accuracy is n, the processor 350 may generate an image on line 1 811, line 2 812, and line 3 813, where the specified accuracy corresponds to n. You can check the sizes (m1, m2, m3). The processor 350 may determine the size (m3) of the image with the largest size among the image sizes (m1, m2, and m3) as the size of the first area.

In one embodiment, when the type of object included in the region of interest is confirmed, the processor 350 determines the size of the image corresponding to the specified accuracy in the relationship between the accuracy for detecting the identified type of object and the size of the image. You can check. The processor 350 may determine the size of the confirmed image as the size of the first area. For example, the inference engine 460 may be set to perform an operation to detect a person. The processor 350 may confirm that the type of object detected in the region of interest as a result of the previous inference operation is a person. If the type of object is a person, the processor 350 may check the size of the image corresponding to the specified accuracy in the relationship between accuracy and image size. The processor 350 may determine the size of the confirmed image as the size of the first area. For example, in FIG. 8, when the specified accuracy is n, the processor 350 determines that the specified accuracy corresponds to n on line 1 811, which represents the relationship between accuracy and the size of the image when the type of object is a person. , you can check the size (m1) of the image. The processor 350 may determine the size (m1) of the image as the size of the first area.

In one embodiment, the size of the image corresponding to the specified accuracy is the smallest among the sizes of the image corresponding to the accuracies that satisfy the specified accuracy (e.g., accuracies that are greater than or equal to the specified accuracy) in the relationship between the accuracy and the size of the image. It could be size. For example, the smaller the size of the image (e.g., the size of the patch), the faster the processing speed of the operation to detect the object. Accordingly, so that an operation for detecting an object is performed at the fastest speed while satisfying the specified accuracy, the smallest image size among the image sizes corresponding to the specified accuracy may be determined as the size of the first area.

At operation 507, in one embodiment, processor 350 may obtain a frame that includes at least one first region.

In one embodiment, the processor 350 may determine the size of a frame (hereinafter referred to as a “mixed frame”) including at least one first region.

In one embodiment, the processor 350 may determine the size of the mixed frame based on the type and number of artificial intelligence models used in the inference operation.

In one embodiment, the processor 350 makes inferences within a specified time based on the performance (and/or computing resources) of the electronic device 301 and the type and/or number of artificial intelligence models used by the inference engine 460. For engine 460 to perform inference operations, the size of the mixed frame may be determined. For example, when the processor 350 (e.g., the inference engine 460) is set to perform a face detection operation as an inference operation, the processor 350 detects a face for one mixed frame. The size of the mixing frame can be determined so that the operation is performed within a specified time (e.g., about 1 second). For example, when the processor 350 is set to perform a face detection operation and an animal detection operation simultaneously (or sequentially) as an inference operation, the processor 350 detects a face for one mixed frame. The size of the mixing frame can be determined so that the detection operation and the animal detection operation are performed within a specified time (e.g., approximately 1 second). In one embodiment, the designated time may be a time requested (or designated) by an application that performs an operation to detect an object.

In one embodiment, the processor 350 runs an inference engine ( The maximum size of the mixed frame that can enable 460) to perform the inference operation can be determined by the size of the mixed frame. For example, the processor 350 (eg, GPU) can process more data per hour as the size of the mixed frame becomes larger. For example, the speed at which the processor 350 processes five mixed frames with a size of 100*100 (e.g., 100 pixels horizontally and 100 pixels vertically) is faster than the speed at which the processor 350 processes five mixed frames with a size of 100*500. Processing of one mixed frame can be fast. Accordingly, the processor 350 is an application that performs an operation to detect an object based on the performance (and/or computing resources) of the electronic device and the type and/or number of artificial intelligence models used in the inference operation. The maximum size of the mixing frame that can allow an operation to be performed within the required time can be determined by the size of the mixing frame.

In one embodiment, the processor 350 may store in advance the relationship between the performance time of the inference operation and the size of the mixed frame for each type of artificial intelligence model in the memory 340. For example, the processor 350 may profile the relationship between the performance time of an inference operation and the size of the mixed frame for each type of artificial intelligence model. For example, when information about the type of artificial intelligence model and the performance of the electronic device 301 is confirmed, the processor 350 performs an inference operation for each type of artificial intelligence model, taking into account the performance of the electronic device 301. The relationship between the execution time of and the size of the mixed frame can be profiled. A method of profiling the relationship between the performance time of the inference operation and the size of the mixed frame for each type of artificial intelligence model will be described later with reference to FIGS. 11 and 12.

In one embodiment, the processor 350 stores the relationship between the execution time of the inference operation and the size of the mixed frame for each type of artificial intelligence model in advance in the memory 340, and the artificial intelligence set for the operation to detect the object You can check the type and number of models and determine the size of the mixed frame based on the type and number of confirmed artificial intelligence models.

In one embodiment, the processor 350 may mix (or combine) at least one first area into a bin having the determined size of the mixed frame.

In one embodiment, the processor 350 may obtain a mixed frame including at least one first area using a bin packing algorithm. For example, the processor 350 may include at least one first region in one bin as much as possible using a bin packing algorithm.

In one embodiment, the processor 350 may assign priority to (eg, allocate) at least one first region.

In one embodiment, the processor 350 may give priority to at least one first area based on the reverse order of the order in which at least one image (image frame) was acquired. For example, the processor 350 may assign the highest priority to the first region obtained from the currently acquired image and the lowest priority to the first region obtained from the earliest acquired image. You can.

In one embodiment, the processor 350 may give priority to at least one first region based on the number of first regions (or regions of interest) obtained from at least one image. For example, the processor 350 assigns higher priority to the first region obtained from images with a smaller number of first regions, and to the first region obtained from images with a larger number of first regions. A lower priority may be given to the first area. Through this prioritization operation, the processor 350 can ensure that the operation of detecting an object is accurately performed even for an image in which a smaller first area is obtained among a plurality of images.

In one embodiment, the processor 350 may include at least one first region in one bin using a bin packing algorithm based on priority. For example, the processor 350 may obtain a mixed frame using a bin packing algorithm so that the first region with high priority is preferentially included in the bin.

In one embodiment, if the processor 350 cannot include all of the at least one first area in one bin, the processor 350 may not include part of the at least one first area in the first area. For example, the processor 350 uses a bin packing algorithm to include the first region in one bin, in order of the first region having a high priority, and to include the first region that cannot be included in one bin. can be deleted (or discarded).

At operation 509, in one embodiment, the processor 350 may obtain information related to an object from the frame (mixed frame) using an artificial intelligence model.

In one embodiment, the processor 350 may perform an inference operation using an artificial intelligence model based on the mixed frame. For example, the processor 350 may perform an operation to detect an object (eg, a person, an animal, and/or a human face) using a mixed frame and an artificial intelligence model as input data.

In one embodiment, the processor 350 provides, as result data of the inference operation, a region in which an object was detected within the mixed frame (hereinafter referred to as “at least one second region”) and within the at least one second region. The type of detected object can be obtained. For example, when the processor 350 is set to perform an operation of detecting a human face, the processor 350 may detect a face in at least one second area (e.g., at least one second area) in the mixed frame. coordinates) and determine that a human face is detected within at least one second area. In one embodiment, each of the at least one second area may include coordinates indicating a bounding box corresponding to the area in which the object was detected.

In one embodiment, the processor 350 may obtain information related to an object detected in at least one image based on at least one second area in which the object is detected within the mixed frame.

In one embodiment, the processor 350 combines at least one second area (and the type of object detected in the at least one second area) as result data of the inference operation with at least one first area included in the mixed frame. It can be compared to area. For example, the processor 350 may check intersection over unions (IoUs) between each of all first areas included in the mixed frame and each of at least one second area. The processor 350 may match the first area and the second area with the highest IoU to each other among the confirmed IoUs. For example, when the plurality of second areas include a 2-1 area and a 2-2 area, the processor 350 determines that the IoU with the 2-1 area is the highest among the plurality of first areas. The 1-1 area can be matched with the 2-1 area, and the 1-2 area with the highest IoU with the 2-2 area among the plurality of first areas can be matched with the 2-2 area.

In one embodiment, the processor 350 compares the first region and the second region to select a region that corresponds to (e.g., matches) at least one second region among the at least one first region included in the mixed frame. A region (hereinafter referred to as “at least one third region”) can be identified.

In one embodiment, the processor 350 selects an object within at least one image based on metadata of the at least one third area and the at least one third area (or at least one first area). The detected area and/or type of the detected object may be obtained. For example, metadata includes identification information of the image from which the third region is obtained among at least one image (image frame) (e.g., identifier of the image including the image and index of the image within the image), Includes the location of a region of interest (e.g., a region of interest enlarged or reduced to obtain the third region) corresponding to the third region in the image in which the third region is acquired, and the position of the third region within the blended frame. can do. The processor 350 determines a region (hereinafter, (referred to as “at least one fourth area”) (eg, coordinates of at least one fourth area) may be obtained. Additionally, the processor 350 may obtain the type of object detected in at least one fourth area.

In one embodiment, the information related to the object may include at least one fourth region (the region in which at least one object is detected within each of the at least one image) and the type of object detected in the at least one fourth region. You can.

In one embodiment, when an object is detected in at least one image as a result of the inference operation, the processor 350 tracks the region of interest in the acquired at least one image without performing some of the operations for detecting the object. (tracking) operations can be performed. For example, when an object is detected within at least one image, processor 350 may, without performing at least a portion of operations 505 to 509, detect the object based on the type of the object and/or the at least one fourth region that was obtained. Therefore, only the operation of tracking the region of interest within at least one image subsequently acquired can be performed.

FIG. 9 is an example diagram 900 illustrating a method of obtaining a frame including at least one first area, according to an embodiment.

Referring to FIG. 9, in one embodiment, the 1-1

patches

911, 912, and 913 are patches acquired based on the 1-1 image acquired at the first time (e.g., the 1-1 patches obtained from regions of interest extracted from image 1), and the 1-2

patches

921, 922, and 923 are based on the 1-2 image acquired at a second time before the first time. These are acquired patches, and the 1-3

patches

931, 932, and 933 may be patches obtained based on the 1-3 image acquired at a third time before the second time. The 1-1st image, the 1-2nd image, and the 1-3rd image may be images that are sequentially acquired from the first external electronic device.

In one embodiment, the 2-1

patches

941 and 942 are patches obtained based on the 2-1 image acquired at the first time, and the 2-2

patches

951 and 952 are, The patches are acquired based on the 2-2 image acquired at the 2nd time before the 1st time, and the 2-3

patches

961 and 962 are the 2nd image acquired at the 3rd time before the 2nd time. -3 These may be patches obtained based on the image. The 2-1st image, the 2-2nd image, and the 2-3rd image may be images sequentially acquired from the second external electronic device.

In one embodiment, the processor 350 may assign priority to patches based on the reverse order (eg, time) in which images (image frames) were acquired. For example, the processor 350 acquires the 1-1

patches

911, 912, and 913 based on the 1-1 image and the 2-1 image acquired at the first time. The highest priority may be given to the 2-1

patches

941 and 942. The processor 350 selects the 1-2

patches

921, 922, and 923 acquired based on the 1-2 image, acquired at a second time, and the

second patch

921, 922, 923 acquired based on the 2-2 image. -2

Patches

951 and 952 can be given medium priority. The processor 350 selects the 1-3

patches

931, 932, and 933 acquired based on the 1-3 image, acquired at a third time, and the 2-3 patches acquired based on the 2-3 image. The lowest priority may be assigned to the 3

patches

961 and 962.

In one embodiment, processor 350 may obtain a mixed frame by including patches in one bin using a bin packing algorithm based on priority. For example, the processor 350 may obtain a mixed frame using a bin packing algorithm so that high-priority patches are preferentially included in the bin.

In one embodiment, the processor 350, as shown in Figure 9, 1-1 patches (911, 912, 913), 1-2 patches (921, 922, 923), 1- A mixture of 3 patches (931, 932, 933), 2-1 patches (951, 952), 2-2 patches (951, 952), and 2-3 patches (961, 962) , a mixed frame 970 can be obtained. In one embodiment, the processor 350 may pad the area that does not contain patches within the blended frame 970 so that the area has a black color.

FIG. 10 is an example diagram 1000 illustrating a method of obtaining information related to an object, according to an embodiment.

Referring to FIG. 10 , in one embodiment, the processor 350 may obtain at least one region of interest (eg, region of interest 1015 ) related to the object 1011 from the image 1010 .

In one embodiment, the processor 350 may acquire at least one first region (at least one patch) corresponding to at least one region of interest. For example, the processor 350 may acquire a plurality of first regions by enlarging or reducing the regions of interest related to the

objects

1011, 1012, and 1013.

In one embodiment, the processor 350 may obtain a mixed frame including at least one first region. For example, the processor 350 may obtain the mixed frame 1020 by mixing a plurality of first areas.

In one embodiment, the processor 350 may obtain information related to the object from the mixed frame using an artificial intelligence model.

In one embodiment, the processor 350 performs an inference operation based on the mixed frame 1020 using an artificial intelligence model, so that the

objects

1011, 1012, and 1013 within the mixed frame 1020 are A plurality of detected second areas may be obtained, and it may be determined that the types of

objects

1011, 1012, and 1013 are people.

In one embodiment, the processor 350 compares the plurality of first areas and the plurality of second areas, thereby selecting the plurality of second areas among the plurality of first areas included in the mixed frame 1020. A plurality of corresponding (eg, matching)

third areas

1021, 1022, and 1023 can be confirmed.

In one embodiment, the processor 350 determines the area in which an object is detected and/or the type of the detected object within the image 1010, based on the metadata of the plurality of

third areas

1021, 1022, and 1023. can be obtained. For example, the processor 350 generates an image based on the metadata of the plurality of third areas, the coordinates of the plurality of second areas, and the coordinates of the plurality of

third areas

1021, 1022, and 1023. A plurality of fourth areas (e.g., fourth area 1031) and an object (e.g., object ( 1011)) types can be obtained.

FIG. 11 is a flowchart 1100 illustrating a method of profiling the relationship between the execution time of an inference operation and the size of a mixed frame for each type of artificial intelligence model, according to an embodiment.

FIG. 12 is an example diagram 1200 illustrating a method of profiling the relationship between the execution time of an inference operation and the size of a mixed frame for each type of artificial intelligence model, according to an embodiment.

Referring to FIGS. 11 and 12 , in one embodiment, the operation of profiling the relationship between the execution time of the inference operation and the size of the mixed frame for each type of artificial intelligence model includes at least one image of operation 501 of FIG. 5. It may be performed before performing the acquisition operation.

In operation 1101, in one embodiment, the processor 350 may check the type of artificial intelligence model and the performance of the electronic device 301. For example, the processor 350 may check the type of artificial intelligence model for performing the operation of detecting an object and the performance of the electronic device that performs the operation for detecting the object.

In operation 1103, in one embodiment, the processor 350 may set the relationship between the performance time of the inference operation and the size of the mixed frame for each artificial intelligence model. For example, as indicated by reference numeral 1201 in FIG. 12, the processor 350 calculates the time and mixing required to detect a face for an artificial intelligence model for detecting a face, considering the performance of the electronic device 301. The relationship between the sizes of the frames (e.g. line 1210) can be set. As shown by reference numeral 1202 in FIG. 12, the processor 350 determines the relationship between the time required to detect an object and the size of the mixed frame for an artificial intelligence model for detecting an object, considering the performance of the electronic device (e.g. : Line (1220)) can be set.

In one embodiment, the processor 350 may store the relationship between the performance time of the inference operation set for each artificial intelligence model and the size of the mixed frame in the memory 340.

In one embodiment, when the processor 350 performs an operation to detect an object after the relationship between the performance time of the inference operation set for each type of artificial intelligence model and the size of the mixed frame is stored in the memory 340, the processor 350 detects the object. The type and number of artificial intelligence models set for detection can be checked, and the size of the mixed frame can be determined based on the type and number of confirmed artificial intelligence models.

For example, referring to reference numeral 1201, the artificial intelligence model set to detect an object includes only the artificial intelligence model for face detection, and at a specified time (e.g., required by an application that performs an operation to detect an object) time) may be 1 second. In order to perform an inference operation to detect a face within 1 second, the processor 350 may determine the size of the mixed frame as f2, which corresponds to 1 second on line 1210.

For example, referring to

reference numerals

1201 and 1202, an artificial intelligence model set to detect an object includes an artificial intelligence model for face detection and an artificial intelligence model for detecting an object, and is activated at a specified time (e.g. The time required by the application to perform an operation to detect an object may be 1 second. In order to perform an inference operation for detecting a face and an inference operation for detecting an object within 1 second, the processor 350 determines the execution time of the inference operation for detecting a face based on the

lines

1210 and 1220, and As the maximum size at which the sum of the performance times of the inference operation for detecting an object is less than 1 second, f1 can be determined as the size of the mixed frame. In one embodiment, at

reference numerals

1201 and 1202 of FIG. 12, the mixed frame When the size of the frame is f1, the execution time of the inference operation for detecting a face and the execution time of the inference operation for detecting an object are exemplified to be the same 500 ms, but are not limited thereto. For example, the execution time of an inference operation for detecting a face and the execution time of an inference operation for detecting an object may be different.

FIG. 13 is an example diagram 1300 illustrating a method of learning the relationship between accuracy and image size, according to an embodiment.

Referring to FIG. 13, in one embodiment, the processor 350 learns the relationship between accuracy and image size (e.g., patch size) for each type of artificial intelligence model and object and stores it in memory 340. You can.

In one embodiment, the processor 350 may obtain a plurality of images having different sizes by enlarging or reducing one image to different sizes. The processor 350 includes an image within a frame of the same size (e.g., a size set to the size of the input image of the inference engine 460) for each of a plurality of images having different sizes, and within the frame. The area that does not contain an image can be padded so that the area has a black color. The processor 350 may input a plurality of frames, each containing a plurality of images having different sizes, into the inference engine 460 using an artificial intelligence model. The processor 350 may obtain (eg, calculate) accuracies corresponding to each size of the plurality of images by comparing the output results of the inference engine 460 and the ground truth.

In one embodiment, in Figure 13, line 1310 may represent the relationship between accuracy and the size of the image (e.g., the size of the patch), learned for a particular artificial intelligence model. In the graph of Figure 13, the X-axis represents the size of the image (e.g., the longer of the horizontal and vertical lengths of the patch), and the Y-axis may represent accuracy.

In one embodiment, in Figure 13, the accuracy of point 1311 on line 1310 is a frame (e.g., 13 pixels wide and 20 pixels tall) where patch 1321 is sized 13*20. When 1331) is used as an input image of the inference engine 460, it may indicate the obtained accuracy. The accuracy of point 1312 on line 1310 may represent the accuracy obtained when a frame 1332 with a size of patch 1322 of 66*100 was used as an input image to inference engine 460. The accuracy of point 1313 on line 1310 may represent the accuracy obtained when a frame 1333 with a size of patch 1323 of 132*200 is used as an input image of inference engine 460. The accuracy of point 1314 on line 1310 may represent the accuracy obtained when frame 1334, where the size of patch 1324 is 276*416, is used as an input image to inference engine 460.

In one embodiment, processor 350 may store in memory 340 a relationship between accuracy and size of an image (e.g., line 1310) learned for a particular artificial intelligence model.

FIG. 14 is an example diagram 1400 illustrating a method for detecting an object according to an embodiment.

In one embodiment, FIG. 14 may represent images acquired while the electronic device 301 performs an operation to detect an object.

Referring to FIG. 14, in one embodiment, Image 1-1 (1411) and Image 1-2 (1412) are images acquired from a first external electronic device, and Image 2-1 (1413) and Image 2- 2 (1414) may be images acquired from a second external electronic device.

In one embodiment, blended frame 1420 includes patches obtained from Image 1-1 (1411), Image 1-2 (1412), Image 2-1 (1413), and Image 2-2 (1414). It can be a mixed frame.

In one embodiment, blended frame 1430 may be the output image of inference engine 460. Areas indicated by indications (eg, indication 1431) (eg, bounding box) within the mixed frame 1430 may be second areas where objects are detected.

In one embodiment, Image 3-1 (1441), Image 3-2 (1442), Image 4-1 (1443), and Image 4-2 (1444) are image 1-1 (1411), respectively. Image 1-2 (1412), Image 2-1 (1413), and Image 2-2 (1414) may be reconstructed images. In one embodiment, in each of the reconstructed images, areas indicated by indications may be fourth areas where objects were detected. For example, in image 4-2 1444, the area indicated by the indication 1444-1 may be the fourth area where the object was detected.

In one embodiment, the processor 350 may perform an operation to detect an object at a faster speed while maintaining accuracy through the above-described operations. For example, [Table 1] below may show the processing speed and accuracy according to Comparative Example 1, the processing speed and accuracy according to Comparative Example 2, and the processing speed and accuracy according to the present disclosure.

	비교예 1Comparative Example 1	비교예 2Comparative Example 2	본 개시This disclosure
처리속도(fps)Processing speed (fps)	9.1709.170	4.2264.226	9.4549.454
정확도(mIoU)Accuracy (mIoU)	0.5690.569	0.5740.574	0.6500.650

In [Table 1], throughput may represent fps (frame per second), and accuracy may represent mIoU (mean intersection over union). Comparative Example 1 may be a case where the patch size is 64*64, and Comparative Example 2 may be a case where the patch size is 160*160.

In one embodiment, as shown in [Table 1], the processing speed (9.454 fps) according to the present disclosure is faster than the processing speed of Comparative Example 1 (9.170 fps) and the processing speed of Comparative Example 2 (4.226 fps). You can. The accuracy (0.650) according to the present disclosure may be higher than the accuracy (0.569) of Comparative Example 1 and the accuracy (0.574) of Comparative Example 2.

In one embodiment, [Table 2] below, as an experiment result, may indicate the average time of operations performed by each component of the processor 350 for each of eight images (eg, in units of eight images).

구성composition	동작 처리 평균 시간(ms)Average motion processing time (ms)
관심 영역 획득 모듈(420)Region of Interest Acquisition Module 420	441.152 (하나의 이미지 당 55.144)441.152 (55.144 per image)
패치 생성 모듈(430) Patch creation module 430	0.0140.014
패치 혼합 모듈(440)Patch Mixing Module (440)	4.4494.449
전처리 모듈Preprocessing module	19.59019.590
추론 엔진(460)Inference Engine (460)	440.487440.487
후처리 모듈Post-processing module	40.73540.735
재구성 모듈(470)Reconfiguration Module (470)	0.2100.210
합산summation	946.637946.637

In one embodiment, in Table 2, the pre-processing performed by the pre-processing module may include an operation of preparing the input image of the inference engine 460 (e.g., normalizing the input image). there is. Post-processing performed by the post-processing module is an operation of post-processing the results of the inference engine 460 (e.g., executing a non-maximum suppression algorithm in second areas (e.g., bounding boxes). action) may be included.

In one embodiment, [Table 1] and [Table 2] use an input image with a size of 640*640, and as experimental images, two images with 1080p resolution and 30fps (e.g., 1 in each of the images) (parts corresponding to minutes) can represent the results of the experiments used.

FIG. 15 is an example diagram 1500 illustrating a method for detecting an object according to an embodiment.

In one embodiment, FIG. 15 may represent screens displayed while the electronic device 301 performs an operation to detect an object.

Referring to FIG. 15, in one embodiment, while performing an operation for detecting an object, the processor 350 displays a currently acquired image (e.g., an image currently being played) and an acquired image through the display module 320. The resulting image of the processed image can be displayed. For example, in FIG. 15, when the first image and the second image are acquired, the processor 350 processes the image 1511 to be processed within the first image at the first time, and the processing result of the image 1511. A screen 1510 including an image 1512, a processing result image 1513 of an image subject to processing in the second image, and a mixed frame 1514 obtained based on the image 1512 and the image 1513, It can be displayed through the display module 320. At a second time following the first time, the processor 350 processes the image 1521 to be processed in the second video, the processing result image 1522 of the image to be processed in the first video, and the image 1521. A screen 1520 including the resulting image 1523 and the image 1522 and the mixed frame 1524 obtained based on the image 1523 may be displayed through the display module 320.

FIG. 16 is an example diagram 1600 illustrating a method of providing information related to an object, according to an embodiment.

Referring to FIG. 16 , in one embodiment, the processor 350 may transmit at least part of the information related to the object to an external electronic device through the communication module 310. For example, when the electronic device operates as an IoT device or a hub device on an IoT network, the processor 350 provides information about the type of object detected in the at least one image along with the acquired at least one image, It can be transmitted to an electronic device that receives IoT services (e.g., an electronic device of a user whose account is registered on the server).

In one embodiment, FIG. 16 may represent a screen 1610 displayed by an external electronic device after receiving, for example, at least part of information related to an object from the electronic device 301 via a server. For example, an external electronic device may include a location and location 1612 registered in the server (e.g., My home and living room), an image 1612, information indicating detection of an object over time 1614, and sound. A screen 1610 including information 1615 indicating the detection of , images 1621, and events and information 1631 that occurred at a location and location registered on the server, including a detected object (e.g., a person). Can be displayed through the display module 320.

A method for detecting an object in an electronic device (eg, the electronic device 301) according to one embodiment may include acquiring at least one image. The method may include obtaining at least one region of interest associated with an object within the at least one image. The method may include obtaining at least one first area corresponding to the at least one area of interest. The method may include obtaining a frame including the at least one first region. The method may include an operation of obtaining information related to the object from the frame using an artificial intelligence model.

In one embodiment, acquiring the at least one first area may include acquiring the at least one first area by enlarging or reducing the at least one area of interest.

In one embodiment, the operation of acquiring the at least one first area may include the operation of confirming the size of the image corresponding to the specified accuracy based on the relationship between the accuracy and the size of the image. The method may include obtaining the at least one first region by enlarging or reducing the size of the at least one region of interest to the confirmed size.

In one embodiment, the operation of acquiring the frame may include determining the size of the frame based on the performance of the electronic device, the type of the artificial intelligence model, and/or the number of the artificial intelligence model. You can.

In one embodiment, acquiring the frame may include obtaining the frame by mixing the at least one first region using a bin packing algorithm based on the size of the frame.

In one embodiment, the operation of acquiring the frame may include an operation of giving priority to the at least one first area based on the reverse order of the order in which the at least one image was acquired. The operation of acquiring the frame may include an operation of mixing the at least one first area based on the priority.

In one embodiment, the operation of obtaining information related to the object includes, using the artificial intelligence model, at least one second area where the object is detected within the frame and the type of the object detected within the frame. It may include an operation to obtain. Obtaining information related to the object may include identifying at least one third area corresponding to the at least one second area among the at least one first area. The operation of obtaining information related to the object may include, based on the at least one second area, the at least one third area, and the metadata of the at least one third area, within the at least one image It may include an operation of acquiring at least one fourth area where an object is detected.

In one embodiment, the electronic device (eg, the electronic device 301) may be an IoT device or a hub device that can provide information related to the acquired object to an external electronic device.

In one embodiment, a non-transitory computer-readable medium recording computer-executable instructions, wherein the computer-executable instructions, when executed, are stored in an electronic device (e.g., processor 350) that includes at least one processor (e.g., processor 350). : The electronic device 301) may be configured to acquire at least one image. The computer-executable instructions, when executed, cause an electronic device (e.g., electronic device 301) including at least one processor (e.g., processor 350) to display at least one image related to an object within the at least one image. It can be configured to obtain a region of interest. The computer-executable instructions, when executed, cause an electronic device (e.g., electronic device 301) including at least one processor (e.g., processor 350) to generate at least one signal corresponding to the at least one region of interest. It may be configured to acquire the first area. The computer-executable instructions, when executed, cause an electronic device (e.g., electronic device 301) including at least one processor (e.g., processor 350) to create a frame including the at least one first area. It can be configured to obtain. The computer-executable instructions are, when executed, an electronic device (e.g., electronic device 301) including at least one processor (e.g., processor 350), using an artificial intelligence model, from the frame, It may be configured to obtain information related to an object.

Additionally, the data structure used in the above-described embodiments of this document can be recorded on a computer-readable recording medium through various means. The computer-readable recording media includes storage media such as magnetic storage media (eg, ROM, floppy disk, hard disk, etc.) and optical read media (eg, CD-ROM, DVD, etc.).

Claims

In electronic devices,

Memory; and

At least one processor operatively coupled to the memory,

The at least one processor,

acquire at least one image,

Obtain at least one region of interest associated with an object within the at least one image,

Obtaining at least one first region corresponding to the at least one region of interest,

Obtaining a frame including the at least one first area, and

An electronic device configured to obtain information related to the object from the frame using an artificial intelligence model.
According to claim 1,

The at least one processor,

An electronic device configured to acquire the at least one first area by enlarging or reducing the at least one area of interest.
According to claim 2,

The at least one processor,

Based on the relationship between the accuracy and the size of the image, determine the size of the image corresponding to the specified accuracy, and

An electronic device configured to obtain the at least one first region by enlarging or reducing the size of the at least one region of interest to the confirmed size.
According to claim 3,

The relationship between the accuracy and the size of the image varies depending on the type of artificial intelligence model and/or the object.
According to claim 1,

The at least one processor,

Confirm the performance of the electronic device, the type of the artificial intelligence model, and/or the number of the artificial intelligence model, and

An electronic device configured to determine the size of the frame based on the performance of the electronic device, the type of the artificial intelligence model, and/or the number of the artificial intelligence model.
According to claim 5,

The at least one processor,

An electronic device configured to obtain the frame by mixing the at least one first region using a bin packing algorithm, based on the size of the frame.
According to claim 6,

The at least one processor,

Priority is given to the at least one first area based on the reverse order of the order in which the at least one image was acquired, and

An electronic device configured to mix the at least one first region based on the priority.
According to claim 1,

The at least one processor,

Using the artificial intelligence model, obtain at least one second area in which the object is detected in the frame and the type of the object detected in the frame,

Among the at least one first area, identify at least one third area corresponding to the at least one second area, and

Based on the at least one second area, the at least one third area, and metadata of the at least one third area, at least one first image in which the object is detected in the at least one image 4 An electronic device configured to acquire areas.
According to claim 1,

The at least one image includes a plurality of images acquired from at least one external electronic device.
According to claim 1,

The electronic device is an IoT (internet of things) device or a hub device that can provide information related to the acquired object to an external electronic device.
In a method for detecting an object in an electronic device,

Acquiring at least one image;

Obtaining at least one region of interest associated with an object within the at least one image;

acquiring at least one first area corresponding to the at least one area of interest;

Obtaining a frame including the at least one first area; and

A method comprising obtaining information related to the object from the frame using an artificial intelligence model.
According to claim 11,

The operation of acquiring the at least one first area includes:

A method comprising acquiring the at least one first area by enlarging or reducing the at least one area of interest.
According to claim 12,

The operation of acquiring the at least one first area includes:

determining the size of the image corresponding to the specified accuracy based on the relationship between the accuracy and the size of the image; and

A method comprising acquiring the at least one first region by enlarging or reducing the size of the at least one region of interest to the confirmed size.
According to claim 13,

The relationship between the accuracy and the size of the image may vary depending on the type of artificial intelligence model and/or the object.
According to claim 11,

The operation of acquiring the frame is:

A method comprising determining the size of the frame based on the performance of the electronic device, the type of the artificial intelligence model, and/or the number of the artificial intelligence model.