KR20180109341A - Method for tracking an object in an image and electronic device thereof - Google Patents

Abstract

The present invention relates to a method for tracking an object in an image and an electronic device thereof which can process to improve the object tracking performance. The electronic device comprises: a camera module; a display; and a processor set to obtain an image including one or more objects by using the camera module, to display a region of interest related to the one or more objects in the obtained image including the one or more objects by using the display, to determine a tracking method for the one or more objects based on the region of interest for the one or more objects included in the displayed image, and to track the one or more objects at least based on the determined tracking method.

Description

≪ Desc / Clms Page number 1 > METHOD FOR TRACKING AN OBJECT IN AN IMAGE AND ELECTRONIC DEVICE THEREOF FIELD OF THE INVENTION [

Various embodiments of the present invention are directed to an apparatus and method for tracking an object contained in an image in an electronic device.

With the development of information communication technology and semiconductor technology, various electronic devices are providing various functions. For example, the electronic device can provide multimedia services such as a voice call service, a message service, a broadcast service, a wireless Internet service, a camera service, and a music playback service.

The electronic device can provide a search service using a camera to enhance the convenience of the user. For example, an electronic device can extract an object to be searched from an image acquired through a camera, recognize the extracted object, and use it as a search target.

The shape, position, etc. of the object included in the image may be changed according to the movement of the electronic device or the movement of the object. Accordingly, the electronic device can continuously track the extracted object to recognize the object. For example, the electronic device can call the tracking engine every time a frame is received through the camera to track the objects contained in the frame. In this case, battery consumption of the electronic device may occur by frequently calling the object tracking engine. In addition, the larger the size of the object, the more time it takes to track the object, which may cause a frame delay to be received by the electronic device through the camera.

Electronic devices and methods in accordance with various embodiments of the present invention can be processed to improve object tracking performance.

The electronic device and method according to various embodiments of the present invention may adjust the frequency of calling of the tracking engine based on the extracted object.

An electronic device according to various embodiments of the present invention includes a camera module, a display, and a processor, wherein the processor is configured to obtain an image comprising one or more objects using the camera module, Displaying a region of interest associated with the one or more objects in an image comprising one or more objects, determining a tracking scheme for the one or more objects based at least in a region of interest for the one or more objects, Lt; RTI ID = 0.0 > at least < / RTI >

An operation method of an electronic device according to various embodiments of the present invention includes: controlling a display to display an image comprising one or more objects obtained via a camera module; Determining a tracking scheme for the one or more objects based at least in a region of interest for the one or more objects; And tracking the one or more objects based at least in part on the determined tracking scheme.

According to various embodiments of the present invention, there is provided a computer readable medium having computer executable instructions for performing the following steps: controlling a display to display an image comprising one or more objects obtained via a camera module; Determining a tracking scheme for the one or more objects based at least in a region of interest for the one or more objects; And tracking the one or more objects based at least in part on the determined tracking scheme.

The electronic device and its operation method according to various embodiments can reduce the battery consumption of the electronic device by variably changing the tracking period based on the extracted object. Further, the electronic device can variably prevent a delay for a frame input through the camera by changing the tracking period.

1 is a diagram illustrating an electronic device in a network environment in accordance with various embodiments.
2 is a block diagram of an electronic device according to various embodiments of the present invention.
3 is a block diagram of a program module in accordance with various embodiments of the present invention.
4A is a diagram illustrating a configuration of an image analysis manager according to various embodiments of the present invention.
4B is a diagram for explaining the processing operation of the image analysis manager.
5 is a flow chart illustrating a procedure for tracking an object included in an image in an electronic device according to various embodiments of the present invention.
6 is a flow diagram illustrating a procedure for determining a region of interest in an electronic device in accordance with various embodiments of the present invention.
7 is an exemplary diagram illustrating a procedure for determining a region of interest.
8A is a flow chart illustrating a procedure for determining a tracking scheme in an electronic device according to various embodiments of the present invention.
8B is an exemplary diagram for explaining a procedure for determining a reference range for determining whether the tracking operation is continued or not.
9 is a flow chart illustrating another procedure for determining a reference range in an electronic device according to various embodiments of the present invention.
10 is a flow diagram illustrating another procedure for determining a tracking scheme in an electronic device according to various embodiments of the present invention.
11A is a flow chart illustrating a procedure for tracking an area of interest in an electronic device according to various embodiments of the present invention.
11B is an exemplary diagram for explaining a procedure for determining whether the tracking operation is continued or not.
12 is a flow chart illustrating a procedure for tracking an area of interest in an electronic device according to various embodiments of the present invention.
13 is a flowchart illustrating a procedure for recognizing an object in an electronic device according to various embodiments of the present invention.
14 is a flow chart illustrating a procedure for tracking an object in an electronic device according to various embodiments of the present invention.
15 is a flow chart illustrating a procedure for tracking an object included in an image in an electronic device according to various embodiments of the present invention.

Hereinafter, various embodiments of the present document will be described with reference to the accompanying drawings. It is to be understood that the embodiments and terminologies used herein are not intended to limit the invention to the particular embodiments described, but to include various modifications, equivalents, and / or alternatives of the embodiments. In connection with the description of the drawings, like reference numerals may be used for similar components. The singular expressions may include plural expressions unless the context clearly dictates otherwise. In this document, the expressions "A or B" or "at least one of A and / or B" and the like may include all possible combinations of the items listed together. Expressions such as "first", "second", "first" or "second" may be used to qualify the components, regardless of order or importance, and to distinguish one component from another And does not limit the constituent elements. When it is mentioned that some (e.g., first) component is "(functionally or communicatively) connected" or "connected" to another (second) component, May be connected directly to the component, or may be connected through another component (e.g., a third component).

In this document, "configured to (or configured) to" as used herein is intended to encompass all types of information, including, for example, hardware or software Quot ;, " made to ", " made to ", " designed to ", or "designed to" In some situations, the expression "a device configured to" may mean that the device can "do " with other devices or components. For example, a processor configured (or configured) to perform the phrases "A, B, and C" may be implemented by executing one or more software programs stored in a memory device or a dedicated processor (e.g., an embedded processor) , And a general purpose processor (e.g., a CPU or an application processor) capable of performing the corresponding operations.

Electronic devices in accordance with various embodiments of the present document may be used in various applications such as, for example, smart phones, tablet PCs, mobile phones, videophones, electronic book readers, desktop PCs, laptop PCs, netbook computers, workstations, a portable multimedia player, an MP3 player, a medical device, a camera, or a wearable device. Wearable devices may be of the type of accessories (eg, watches, rings, bracelets, braces, necklaces, glasses, contact lenses or head-mounted-devices (HMD) (E.g., a skin pad or tattoo), or a bio-implantable circuit. In some embodiments, the electronic device may be, for example, a television, a digital video disk (Such as Samsung HomeSync ^TM , Apple TV ^TM , or Google TV ^TM ), which are used in home appliances such as home appliances, audio, refrigerators, air conditioners, vacuum cleaners, ovens, microwave ovens, washing machines, air cleaners, set top boxes, home automation control panels, , A game console (e.g., Xbox ^TM , PlayStation ^TM ), an electronic dictionary, an electronic key, a camcorder, or an electronic photo frame.

In an alternative embodiment, the electronic device may be any of a variety of medical devices (e.g., various portable medical measurement devices such as a blood glucose meter, a heart rate meter, a blood pressure meter, or a body temperature meter), magnetic resonance angiography (MRA) A navigation system, a global navigation satellite system (GNSS), an event data recorder (EDR), a flight data recorder (FDR), an automobile infotainment device, a marine electronic equipment (For example, marine navigation systems, gyro compasses, etc.), avionics, security devices, head units for vehicles, industrial or domestic robots, drones, ATMs at financial institutions, of sales or at least one of an object Internet device (e.g., a light bulb, various sensors, a sprinkler device, a fire alarm, a thermostat, a streetlight, a toaster, a fitness appliance, a hot water tank, a heater, a boiler, etc.) .

According to some embodiments, the electronic device may be a piece of furniture, a building / structure or part of an automobile, an electronic board, an electronic signature receiving device, a projector, or various measuring devices (e.g., Gas, or radio wave measuring instruments, etc.). In various embodiments, the electronic device is flexible or may be a combination of two or more of the various devices described above. The electronic device according to the embodiment of the present document is not limited to the above-described devices.

In this document, the term user may refer to a person using an electronic device or a device using an electronic device (e.g., an artificial intelligence electronic device).

Figure 1 illustrates an electronic device 101 in a network environment in various embodiments of the present invention.

1, an electronic device 101 includes a bus 110, a processor 120, a memory 130, an input / output interface 150, a display 160, a communication interface 170 and a camera 180 can do. In some embodiments, the electronic device 101 may omit at least one of the components or additionally include other components.

The bus 110 may include circuitry, for example, to connect the components 120-180 to each other and to communicate signals (e.g., control messages and / or data) between the components.

The processor 120 may include one or more of a central processing unit (CPU), an application processor (AP), a communication processor (CP), or an image signal processor (ISP). The processor 120 may perform computations or data processing related to, for example, control and / or communication of at least one other component of the electronic device 101.

According to various embodiments, the processor 120 may display images (e.g., preview images, preview images, etc.) via the display 160. According to one embodiment, the processor 120 may display an image obtained via the camera 180 via the display 160. [

According to various embodiments, the processor 120 may extract at least one object from the image displayed on the display 160. According to one embodiment, the processor 120 may analyze the image displayed on the display 160 and extract feature points contained in the image. In addition, the processor 120 may extract at least one object included in the image based on the extracted feature points of the image. For example, the processor 120 may include at least one object extraction engine (e.g., a barcode extraction engine, a QR code (Quick Response Code) extraction engine, a text extraction engine, a matter extraction engine, a human extraction engine, (Such as a bar code, a QR code, text, objects, people, animals, landscapes, landmarks, etc.) included in the image can be extracted. In another example, the processor 120 provides at least a portion of the image displayed on the display 160 to an external device (e.g., server 106 or electronic device 104) and provides information about the object extracted by the external device .

According to various embodiments, the processor 120 may control the display 160 to display extraction information of the object when the object is extracted from the image displayed on the display 160. According to one embodiment, the processor 120 may control to display extraction information of a corresponding object using graphic parameters corresponding to the extracted object configuration information. For example, the configuration information of the object may include at least one of the shape of the object (e.g., curvature of line, height, edge, shape, etc.) and the color of the object. For example, the processor 120 may control to display various types of graphical variables such as straight lines, curves, polygons, circles, and dots so that the user can recognize the extracted objects. According to another embodiment, the processor 120 may control the display 160 to display extraction information of the object using graphic parameters corresponding to the motion information of the object extracted from the image displayed on the display 160. [ According to various embodiments, the processor 120 may control the display 160 to display extraction information of the object before the object is extracted (e.g., while extracting the object) from the image displayed on the display 160 .

According to various embodiments, the processor 120 may determine a region of interest for the extracted at least one object when the object is extracted from the image. According to one embodiment, the processor 120 may determine at least a portion of the image as a region of interest based on the extracted object. For example, the processor 120 may process to provide a graphical effect for the determined region of interest. For example, the processor 120 may provide a visual effect to the border of the region of interest of the determined object. In addition, the processor 120 may apply different graphic effects so that each of the regions of interest is identified when a plurality of regions of interest are determined.

According to various embodiments, the processor 120 may determine a tracking period for a region of interest (or object) once the region of interest for the object is determined.

According to one embodiment, the processor 120 may determine a tracking period based on the size of the region of interest. For example, the smaller the area of interest, the less time it takes to track. Accordingly, the processor 120 can determine the tracking period so that the tracking engine is called with a relatively high frequency when the size of the region of interest is included in a relatively small first range. For example, the processor 120 may determine the tracking period so that the tracking engine is called each time a frame is input from the camera 180. [ As another example, the larger the size of the region of interest, the more time it takes to track and the delay of the frame input may occur during the tracking operation. Accordingly, the processor 120 may determine the tracking period so that the tracking engine is called at a relatively low frequency when the size of the region of interest is included in a relatively large second range. For example, the processor 120 may determine a tracking period so that the tracking engine is not invoked for at least one frame of frames input from the camera 180 (e.g., the frame in which the delay occurs).

According to another embodiment, the processor 120 may determine a tracking period based on the complexity of the objects contained in the region of interest. The complexity of the object can be related to the minutiae of the object. Further, the complexity of the object may be related to the movement of the electronic device 101 or the movement of the object according to the amount of change in the image. For example, the smaller the number of feature points of an object, the smaller the movement of the object, or the less the movement of the electronic device, the lower the complexity of the object. Accordingly, the processor 120 can determine a tracking cycle to cause the tracking engine to be invoked at a relatively high frequency if the complexity of the objects contained in the region of interest is relatively low (e.g., if it takes less time to track) have. As another example, the more feature points of the extracted object or the more the movement of the object, the higher the complexity of the object. Accordingly, the processor 120 can determine the tracking period so that the tracking engine is called with a relatively low frequency if the complexity of the objects included in the region of interest is relatively high (e.g., if it takes a lot of time to track) .

According to various embodiments, the processor 120 may set a reference range in which movement of the region of interest is allowed, if a tracking period is determined. The reference range may be a reference value for determining whether the tracking operation is continued (or not).

According to one embodiment, the processor 120 may set a reference range based on the size of the region of interest. For example, movement of an object displayed on the display 160 by movement (or tremor) of the electronic device 101 may occur. At this time, the motion amount of the ROI greater than a certain size may be significantly changed. Accordingly, the processor 120 can set the reference range to allow relatively more movement than the region of interest smaller than the predetermined size, for the region of interest larger than a predetermined size.

According to another embodiment, the processor 120 may determine a reference range based on the distance between the object and the electronic device 101 (or the camera 180 of the electronic device). For example, the amount of movement of an object located within a certain distance from an object located a certain distance away from the electronic device 101 can be greatly changed. Accordingly, the processor 120 can set a reference range so as to permit relatively more motion than an object of a certain distance or more to an object of a certain distance or less.

According to various embodiments, the processor 120 may perform a tracking operation on the region of interest based on the determined tracking scheme, once the tracking scheme is determined. According to one embodiment, the processor 120 may determine that a tracking period has arrived when a frame is input corresponding to the tracking period from the camera 180. [ In addition, processor 120 may invoke a tracking engine to track the region of interest.

According to various embodiments, when the tracking engine is called, the processor 120 may determine whether to continue (or not) the tracking operation based on the amount of motion of the ROI. According to one embodiment, the processor 120 may stop the tracking operation for the extracted object if the region of interest is out of the reference range. For example, the processor 120 may extract other objects for tracking. According to another embodiment, the processor 120 may perform a tracking operation on the region of interest using the called tracking engine if the region of interest is included in the reference range.

Memory 130 may include volatile and / or non-volatile memory. Memory 130 may store instructions or data related to at least one other component of electronic device 101, for example. According to one embodiment, the memory 130 may store tracking variables for at least one tracking scheme. For example, the memory 130 may store a period for calling the tracking engine corresponding to the size of the region of interest or the complexity of the object, and a reference range for determining whether or not the tracking is successful. Also, the memory 130 may store graphical variables corresponding to the configuration information of the object.

According to various embodiments, the memory 130 may store software and / or programs 140. For example, the program 140 may include a kernel 141, a middleware 143, an application programming interface (API) 145 or an application program (or "application") 147, At least a portion of the kernel 141, middleware 143, or API 145 may be referred to as an operating system (OS).

The kernel 141 may include system resources used to execute an operation or function implemented in other programs (e.g., middleware 143, API 145, or application program 147) (E.g., bus 110, processor 120, or memory 130). The kernel 141 also provides an interface to control or manage system resources by accessing individual components of the electronic device 101 in the middleware 143, API 145, or application program 147 .

The middleware 143 can perform an intermediary role such that the API 145 or the application program 147 can communicate with the kernel 141 to exchange data. In addition, the middleware 143 may process one or more task requests received from the application program 147 according to the priority order. For example, middleware 143 may use system resources (e.g., bus 110, processor 120, or memory 130, etc.) of electronic device 101 in at least one of application programs 147 Assign priority, and process one or more task requests. The API 145 is an interface for the application program 147 to control the functions provided by the kernel 141 or the middleware 143 and may be an interface for controlling the functions provided by the kernel 141 or the middleware 143 such as for file control, At least one interface or function (e.g., command).

The input / output interface 150 transfers instructions or data entered from a user or other external device to another component (s) of the electronic device 101, or to another component (s) Command or data to a user or other external device. For example, the input / output interface 150 may include at least one physical button such as a home button, a power button, and a volume control button. For example, the input / output interface 150 may include a speaker for outputting an audio signal and a microphone for collecting an audio signal.

Display 160 may display various content (e.g., text, images, video, icons, and / or symbols, etc.) to a user. For example, the display 160 may include a liquid crystal display (LCD), a light emitting diode (LED) display, an organic light emitting diode (OLED) display, or a microelectromechanical system (MEMS) display, or an electronic paper display can do. For example, the display 160 may include a touch screen. For example, the display 160 may receive a touch, gesture, proximity, or hovering input using an electronic pen or a portion of the user's body.

The communication interface 170 may establish communication between the electronic device 101 and an external device (e.g., the first external electronic device 102, the second external electronic device 104, or the server 106). For example, communication interface 170 may be connected to network 172 via wireless or wired communication to communicate with an external device (e.g., second external electronic device 104 or server 106).

According to one embodiment, the wireless communication may be in the form of a wireless communication system such as LTE, LTE Advance, code division multiple access (CDMA), wideband CDMA (WCDMA), universal mobile telecommunications system (UMTS), wireless broadband (WiBro) Global System for Mobile Communications), and the like. According to one embodiment, wireless communication 174 may include wireless fidelity (WiFi), light fidelity (LiFi), Bluetooth, Bluetooth low power (BLE), ZigBee Zigbee, Near Field Communication (NFC), Magnetic Secure Transmission, Radio Frequency (RF), or Body Area Network (BAN). According to one embodiment, the wireless communication may comprise a GNSS. For example, the GNSS may be a Global Positioning System (GPS), a Global Navigation Satellite System (Glonass), a Beidou (Beidou Navigation Satellite System), or a Galileo (the European global satellite-based navigation system). Hereinafter, in this document, "GPS" can be used interchangeably with "GNSS ". According to one embodiment, the wired communication includes at least one of a universal serial bus (USB), a high definition multimedia interface (HDMI), a recommended standard 232 (RS-232), a power line communication, or a plain old telephone service . Network 172 may include at least one of a telecommunications network, e.g., a computer network (e.g., LAN or WAN), the Internet, or a telephone network.

Each of the first and second external electronic devices 102, 104 may be the same or a different kind of device as the electronic device 101. According to various embodiments, all or a portion of the operations performed in the electronic device 101 may be performed in one or more other electronic devices (e.g., electronic devices 102, 104, or server 106). According to the present invention, when electronic device 101 is to perform a function or service automatically or on demand, electronic device 101 may perform at least some functions associated therewith instead of, or in addition to, (E.g., electronic device 102, 104, or server 106) may request the other device (e.g., electronic device 102, 104, or server 106) Perform additional functions, and forward the results to the electronic device 101. The electronic device 101 may process the received results as is or additionally to provide the requested functionality or services. For example, Cloud computing, distributed computing, or client-server computing techniques can be used.

The camera 180 may capture image information of a subject and take a still image (photograph) and a moving image. According to various embodiments, the camera 180 may take a subject under the control of the processor 120. According to one embodiment, the camera 180 may communicate the captured data (e.g., image) to the display 160 and the processor 120. For example, the camera 180 may include at least one of at least one image sensor, a lens, an image signal processor (ISP), or a flash (e.g., LED or xenon lamp, etc.). For example, the image sensor may be implemented as a CCD (charged coupled device) or a CMOS (complementary metal-oxide semiconductor). According to various embodiments, the camera 180 may include a plurality of cameras. According to one embodiment, the camera 180 may include a plurality of cameras disposed on the front or back of the electronic device 101. For example, the camera 180 may include a plurality of cameras disposed on the back surface of the electronic device 101. [

2 shows a block diagram of an electronic device 201 according to various embodiments. The electronic device 201 may include all or part of the electronic device 101 shown in Fig. 1, for example. The electronic device 201 may include one or more processors (e.g., AP) 210, a communication module 220, a subscriber identification module 224, a memory 230, a sensor module 240, an input device 250, a display 260, an interface 270, an audio module 280, a camera module 291, a power management module 295, a battery 296, an indicator 297, and a motor 298.

The processor 210 may control a plurality of hardware or software components connected to the processor 210, for example, by driving an operating system or an application program, and may perform various data processing and calculations. The processor 210 may be implemented with, for example, a system on chip (SoC). According to one embodiment, the processor 210 may further include a graphics processing unit (GPU) and / or an image signal processor (ISP). Processor 210 may include at least some of the components shown in FIG. 2 (e.g., cellular module 221). Processor 210 may load and process instructions or data received from at least one of the other components (e.g., non-volatile memory) into volatile memory and store the resulting data in non-volatile memory.

The communication module 220 may have the same or similar configuration as the communication interface 160 of FIG. The communication module 220 may include, for example, a cellular module 221, a WiFi module 223, a Bluetooth module 225, a GNSS module 227, an NFC module 228 and an RF module 229 have.

The cellular module 221 can provide voice calls, video calls, text services, or Internet services, for example, over a communication network. According to one embodiment, the cellular module 221 may utilize a subscriber identity module (e.g., a SIM card) 224 to perform the identification and authentication of the electronic device 201 within the communication network. According to one embodiment, the cellular module 221 may perform at least some of the functions that the processor 210 may provide. According to one embodiment, the cellular module 221 may comprise a communications processor (CP).

At least some (e.g., two or more) of the cellular module 221, the WiFi module 223, the Bluetooth module 225, the GNSS module 227, or the NFC module 228, according to some embodiments, (IC) or an IC package.

The RF module 229 can, for example, send and receive communication signals (e.g., RF signals). The RF module 229 may include, for example, a transceiver, a power amplifier module (PAM), a frequency filter, a low noise amplifier (LNA), or an antenna. According to another embodiment, at least one of the cellular module 221, the WiFi module 223, the Bluetooth module 225, the GNSS module 227, or the NFC module 228 transmits / receives an RF signal through a separate RF module . The subscriber identification module 224 may include, for example, a card or an embedded SIM containing a subscriber identity module, and may include unique identification information (e.g., ICCID) or subscriber information (e.g., IMSI (international mobile subscriber identity).

Memory 230 (e.g., memory 130 of FIG. 1) may include, for example, internal memory 232 or external memory 234. Volatile memory (e.g., a DRAM, an SRAM, or an SDRAM), a non-volatile memory (e.g., an OTPROM, a PROM, an EPROM, an EEPROM, a mask ROM, a flash ROM , A flash memory, a hard drive, or a solid state drive (SSD). The external memory 234 may be a flash drive, for example, a compact flash (CF) ), Micro-SD, Mini-SD, extreme digital (xD), multi-media card (MMC), or memory stick, etc. External memory 234 may communicate with electronic device 201, Or may be physically connected.

The sensor module 240 may, for example, measure a physical quantity or sense the operating state of the electronic device 201 to convert the measured or sensed information into an electrical signal. The sensor module 240 includes a gesture sensor 240A, a gyro sensor 240B, an air pressure sensor 240C, a magnetic sensor 240D, an acceleration sensor 240E, a grip sensor 240F, A temperature sensor 240G, a UV sensor 240G, a color sensor 240H (e.g., an RGB (red, green, blue) sensor), a living body sensor 240I, And a sensor 240M. Additionally or alternatively, the sensor module 240 may be configured to perform various functions such as, for example, an e-nose sensor, an electromyography (EMG) sensor, an electroencephalograph (EEG) sensor, an electrocardiogram An infrared (IR) sensor, an iris sensor, and / or a fingerprint sensor. The sensor module 240 may further include a control circuit for controlling at least one or more sensors belonging to the sensor module 240. In some embodiments, the electronic device 201 further includes a processor configured to control the sensor module 240, either as part of the processor 210 or separately, so that while the processor 210 is in a sleep state, The sensor module 240 can be controlled.

The input device 250 may include, for example, a touch panel 252, a (digital) pen sensor 254, a key 256, or an ultrasonic input device 258. As the touch panel 252, for example, at least one of an electrostatic type, a pressure sensitive type, an infrared type, and an ultrasonic type can be used. Further, the touch panel 252 may further include a control circuit. The touch panel 252 may further include a tactile layer to provide a tactile response (touch coordinates) to the user. (Digital) pen sensor 254 may be part of, for example, a touch panel or may include a separate recognition sheet. Key 256 may include, for example, a physical button, an optical key, or a keypad. The ultrasonic input device 258 can sense the ultrasonic wave generated from the input tool through the microphone (e.g., the microphone 288) and confirm the data corresponding to the sensed ultrasonic wave.

The display 260 may include a panel 262, a hologram device 264, a projector 266, and / or control circuitry for controlling them. The panel 262 may be embodied, for example, flexibly, transparently, or wearably. The panel 262 may comprise a touch panel 252 and one or more modules. According to one embodiment, the panel 262 may include a pressure sensor (or force sensor) capable of measuring pressure information (e.g., pressure coordinates and pressure intensity) for a user's touch. The pressure sensor may be implemented integrally with the touch panel 252, or may be implemented by one or more sensors separate from the touch panel 252. The hologram device 264 can display a stereoscopic image in the air using interference of light. The projector 266 can display an image by projecting light onto a screen. The screen may be located, for example, inside or outside the electronic device 201. The interface 270 may include, for example, an HDMI 272, a USB 274, an optical interface 276, or a D-sub (D-subminiature) 278. The interface 270 may, for example, be included in the communication interface 160 shown in FIG. Additionally or alternatively, the interface 270 may include, for example, a mobile high-definition link (MHL) interface, an SD card / multi-media card (MMC) interface, or an infrared data association have.

The audio module 280 can, for example, convert sound and electrical signals in both directions. At least some of the components of the audio module 280 may be included, for example, in the input / output interface 150 shown in FIG. The audio module 280 may process sound information input or output through, for example, a speaker 282, a receiver 284, an earphone 286, a microphone 288, or the like. Camera module 291 (e.g., camera 180 of FIG. 1) is, for example, a device capable of capturing still images and moving images, and in accordance with one embodiment, includes one or more image sensors A sensor, a lens, an image signal processor (ISP), or a flash (e.g., an LED or xenon lamp, etc.). The power management module 295 can, for example, manage the power of the electronic device 201. [ According to one embodiment, the power management module 295 may include a power management integrated circuit (PMIC), a charging IC, or a battery or fuel gauge. The PMIC may have a wired and / or wireless charging scheme. The wireless charging scheme may include, for example, a magnetic resonance scheme, a magnetic induction scheme, or an electromagnetic wave scheme, and may further include an additional circuit for wireless charging, for example, a coil loop, a resonant circuit, have. The battery gauge can measure, for example, the remaining amount of the battery 296, the voltage during charging, the current, or the temperature. The battery 296 may include, for example, a rechargeable battery and / or a solar cell.

The indicator 297 may indicate a particular state of the electronic device 201 or a portion thereof (e.g., processor 210), e.g., a boot state, a message state, or a state of charge. The motor 298 can convert the electrical signal to mechanical vibration, and can generate vibration, haptic effects, and the like. Electronic device 201 is, for example, DMB Mobile TV-enabled devices capable of handling media data in accordance with standards such as (digital multimedia broadcasting), DVB (digital video broadcasting), or MediaFLO (mediaFlo ^TM) (for example, : GPU). Each of the components described in this document may be composed of one or more components, and the name of the component may be changed according to the type of the electronic device. In various embodiments, an electronic device (e. G., Electronic device 201) may have some components omitted, further include additional components, or some of the components may be combined into one entity, The functions of the preceding components can be performed in the same manner.

3 shows a block diagram of a program module according to various embodiments. According to one embodiment, program module 310 (e.g., program 140 of FIG. 1) includes an operating system and / or an operating system (E.g., the application program 147 of FIG. 1) running on the computer system. The operating system may include, for example, Android ^TM , iOS ^TM , Windows ^TM , Symbian ^TM , Tizen ^TM , or Bada ^TM .

3, the program module 310 includes a kernel 320 (e.g., the kernel 141 of FIG. 1), middleware 330 (e.g., middleware 143 of FIG. 1), API 360 (E.g., API 145 of Figure 1), and / or an application 370 (e.g., application program 147 of Figure 1). At least a portion of the program module 310 may be pre- Loaded or downloaded from an external electronic device (e.g., electronic device 102, 104, server 106, etc. of FIG. 1).

The kernel 320 may include, for example, a system resource manager 321 and / or a device driver 323. The system resource manager 321 can perform control, allocation, or recovery of system resources. According to one embodiment, the system resource manager 321 may include a process manager, a memory manager, or a file system manager. The device driver 323 may be a display driver, a camera driver, a Bluetooth driver, a shared memory driver, a USB driver, a keypad driver, a WiFi driver, an audio driver, a touch device driver, a pressure device driver, or an inter- ) Driver.

The middleware 330 may provide various functions through the API 360, for example, to provide functions that are commonly needed by the application 370 or allow the application 370 to use limited system resources within the electronic device. Application 370 as shown in FIG. According to one embodiment, the middleware 330 includes a runtime library 335, an application manager 341, a window manager 342, a multimedia manager 343, a resource manager 344, a power manager 345, a database manager At least one of the package manager 346, the package manager 347, the connectivity manager 348, the notification manager 349, the location manager 350, the graphic manager 351, the security manager 352 or the image analysis manager 353 . ≪ / RTI >

The runtime library 335 may include, for example, a library module that the compiler uses to add new functionality via a programming language while the application 370 is executing. The runtime library 335 may perform input / output management, memory management, or arithmetic function processing. The application manager 341 can manage the life cycle of the application 370, for example. The window manager 342 can manage GUI resources used in the screen. The multimedia manager 343 can recognize the format required for reproducing the media files and can perform encoding or decoding of the media file using a codec according to the format. The resource manager 344 can manage the source code of the application 370 or the space of the memory. The power manager 345 manages the capacity, temperature, or power of the battery, for example, and can use this information to determine or provide power information necessary for the operation of the electronic device. According to one embodiment, the power manager 345 may interoperate with a basic input / output system (BIOS). The database manager 346 may create, retrieve, or modify the database to be used in the application 370, for example. The package manager 347 can manage installation or update of an application distributed in the form of a package file.

The connectivity manager 348 may, for example, manage the wireless connection. The notification manager 349 may provide the user with an event such as, for example, an arrival message, an appointment, a proximity notification, and the like. The location manager 350 can manage the location information of the electronic device, for example. The graphic manager 351 may, for example, manage the graphical effects to be presented to the user or a user interface associated therewith. According to one embodiment, when an object is extracted from the image displayed on the display 160, the graphic manager 351 may provide a graphic effect for displaying extraction information of the object.

The security manager 352 may provide, for example, system security or user authentication. The image analysis manager 353 may, for example, track at least one object included in the image displayed on the display 160. [ According to one embodiment, the image analysis manager 353 may extract at least one object included in the image displayed on the display 160. In addition, the image analysis manager 353 may variably change the tracking variable based on the region of interest for the extracted object. For example, the image analysis manager 353 may reduce the tracking frequency or increase the tracking frequency based on the region of interest. According to one embodiment, the middleware 330 may include a telephony manager for managing the voice or video call function of the electronic device, or a middleware module capable of forming a combination of the functions of the above-described components . According to one embodiment, the middleware 330 may provide a module specialized for each type of operating system. Middleware 330 may dynamically delete some existing components or add new ones. The API 360 may be provided in a different configuration depending on the operating system, for example, as a set of API programming functions. For example, for Android or iOS, you can provide a single API set for each platform, and for Tizen, you can provide two or more API sets for each platform.

The application 370 may include a home 371, a dialer 372, an SMS / MMS 373, an instant message 374, a browser 375, a camera 376, an alarm 377, Contact 378, voice dial 379, email 380, calendar 381, media player 382, album 383, watch 384, healthcare (e.g., measuring exercise or blood glucose) , Or environmental information (e.g., air pressure, humidity, or temperature information) application. According to one embodiment, the application 370 may include an information exchange application capable of supporting the exchange of information between the electronic device and the external electronic device. The information exchange application may include, for example, a notification relay application for communicating specific information to an external electronic device, or a device management application for managing an external electronic device. For example, the notification delivery application can transmit notification information generated in another application of the electronic device to the external electronic device, or receive notification information from the external electronic device and provide the notification information to the user. The device management application may, for example, control the turn-on / turn-off or brightness (or resolution) of an external electronic device in communication with the electronic device (e.g., the external electronic device itself Control), or install, delete, or update an application running on an external electronic device. According to one embodiment, the application 370 may include an application (e.g., a healthcare application of a mobile medical device) designated according to the attributes of the external electronic device. According to one embodiment, the application 370 may include an application received from an external electronic device. At least some of the program modules 310 may be implemented (e.g., executed) in software, firmware, hardware (e.g., processor 210), or a combination of at least two of the same, Program, routine, instruction set or process.

4A is a diagram illustrating a configuration of an image analysis manager (e.g., image analysis manager 353) according to various embodiments of the present invention. 4B is a diagram for explaining the processing operation of the image analysis manager.

Referring to FIG. 4A, the image analysis manager 353 may include an object extraction engine 410, a tracking engine 420, and an object recognition engine 430.

According to various embodiments, the object extraction engine 410 may include at least one object (e.g., bar code, QR code, text, object, person, animal, landscape, landmark Etc.) can be extracted. According to one embodiment, the object extraction engine 410 may analyze the image displayed on the display 160 to extract feature points contained in the image. In addition, the object extraction engine 410 may extract at least one object included in the image based on the extracted feature points of the image. For example, the object extraction engine 410 may include a plurality of extraction engines for extracting different objects. 4B, the object extraction engine 410 may operate in parallel the n extraction engines (e.g., the first extraction engine 412 and the second extraction engine 416) At least one object can be extracted. In addition, the object extraction engine 410 may determine a region of interest for the extracted object and provide it to the tracking engine 420.

According to various embodiments, the tracking engine 420 may perform a tracking operation on the region of interest determined by the object extraction engine 410. According to one embodiment, the tracking engine 420 may determine a tracking variable based on the region of interest. For example, the tracking engine 420 may reduce the tracking frequency or increase the tracking frequency based on the region of interest. In addition, the tracking engine 420 may determine a reference value for determining whether the tracking operation is continued (or not). For example, the tracking engine 420 may set a motion tolerated reference range of the region of interest.

According to various embodiments, the object recognition engine 430 may recognize the extracted object. According to one embodiment, the object recognition engine 430 may obtain information of the extracted object. For example, the object recognition engine 430 can obtain barcode information (e.g., product name) by analyzing the barcode object. In addition, the object recognition engine 430 can recognize the text object and acquire the text content.

According to various embodiments, an electronic device includes a camera module, a display, and a processor, wherein the processor is configured to obtain an image comprising one or more objects using the camera module, Displaying a region of interest associated with the one or more objects in an image containing objects, determining a tracking scheme for the one or more objects based at least in an area of interest for the one or more objects, And to track the one or more objects based on the at least one object.

According to one embodiment, the processor may be configured to adjust a paging cycle of the tracking engine for tracking the one or more objects based at least in part on the determined tracking method.

According to one embodiment, the processor may be configured to determine a frame in which a tracking operation is to be performed based at least on the determined tracking scheme.

According to one embodiment, the processor may be configured to determine the tracking method based at least on the size of the region of interest.

According to one embodiment, the processor may be configured to determine the tracking method based at least on feature points included in the region of interest.

According to one embodiment, the processor is further configured to determine a reference range in which movement of the ROI is allowed based at least on the size of the ROI, and determine whether to continue the tracking operation based on the reference range and the movement of the ROI . ≪ / RTI >

According to one embodiment, the processor can be configured to determine the region of interest based on attributes of the extracted objects or user input on the extracted objects.

According to one embodiment, the processor may be configured to determine the tracking method based on a frame rate of the camera module.

According to one embodiment, the processor may be configured to determine the tracking method based on a frame input time.

According to one embodiment, the processor may be configured to extract one or more objects included in the displayed image using one or more object extraction engines.

5 is a flow chart illustrating a procedure for tracking an object included in an image in an electronic device 101 (or electronic device 201) according to various embodiments of the present invention.

Referring to FIG. 5, at operation 501, the electronic device 101 may display an image on the display 160. According to one embodiment, the electronic device 101 may control the display 160 to display an image (e.g., a preview image) obtained via the camera 180 when the camera 180 is activated.

At operation 503, the electronic device 101 may determine a region of interest for at least one object included in the image displayed on the display 160. According to one embodiment, the electronic device 101 may analyze the displayed image to extract at least one object and determine at least a portion of the image containing the extracted object to be a region of interest. For example, the electronic device 101 may determine an area of interest for an object having a feature point that satisfies a specified condition from the image.

At operation 505, the electronic device 101 may determine an object tracking scheme based on the determined region of interest. According to one embodiment, the electronic device 101 may determine the tracking method based on the time spent in tracking the region of interest. For example, when it is determined that a relatively small amount of time is required for the determined region of interest, the electronic device 101 can determine the first tracking method that calls the tracking engine with a high frequency by the object tracking method. In addition, when it is determined that a relatively long time is required for the determined region of interest, the electronic device 101 can determine the second tracking method for calling the tracking engine with a low frequency by the object tracking method.

At operation 507, the electronic device 101 may track an object (or region of interest) determined in a determined object tracking manner. According to one embodiment, when tracking an object in a first tracking manner, the electronic device 101 may invoke a tracking engine based on a period corresponding to the first tracking manner. For example, the electronic device 101 can track an object by calling a tracking engine for each frame input from the camera 180. [ In addition, when tracking the object in the second tracking manner, the electronic device 101 can invoke the tracking engine based on the period corresponding to the second tracking method. For example, the electronic device 101 may process the tracking engine so that at least one of the frames input from the camera 180 is not called.

6 is a flow diagram illustrating a procedure for determining an area of interest in an electronic device 101 (or electronic device 201) in accordance with various embodiments of the present invention. 7 is an exemplary diagram illustrating a procedure for determining a region of interest. According to one embodiment, the procedure for determining the region of interest may be a detailed operation of operation 503 operation shown in FIG.

Referring to Fig. 6, at operation 601, the electronic device 101 may extract an object from the displayed image. According to one embodiment, the electronic device 101 includes at least one object extraction engine (e.g., a barcode extraction engine, a QR code extraction engine, a text extraction engine, a matter extraction engine, a human extraction engine, , A landmark extraction engine, etc.) to extract an object included in the image. For example, when an image including cup 702 printed with text 704 is displayed, as in 700 of FIG. 7, electronic device 101 extracts the feature points of the image and extracts the cup shape It is possible to extract object objects and text objects. At this time, the electronic device 101 may provide a graphic effect corresponding to the feature point extracted from the image. For example, the electronic device 101 may provide 712 various types of graphical effects such as straight lines, curves, polygons, circles, and dots so that the user can recognize the extracted objects, such as 710 of FIG. 7, (714).

At operation 603, the electronic device 101 may determine a region of interest for the extracted object. According to one embodiment, when a plurality of objects are extracted, the electronic device 101 can determine a region of interest for an object satisfying a predetermined condition. For example, the electronic device 101 may determine a region of interest for an object selected by user input among a plurality of extracted objects. As another example, the electronic device 101 may determine a region of interest for an object extracted by an extraction engine designated in priority order among a plurality of extracted objects. As another example, the electronic device 101 may determine a region of interest for an object that includes more than a specified number of feature points among the extracted plurality of objects. For example, the electronic device 101 can determine the size of the region of interest based on the characteristics (e.g., size, type of object) of the object meeting the specified condition. In addition, the electronic device 101 may determine the size of the region of interest for the object that satisfies the specified condition based on the user's input.

At operation 605, the electronic device 101 may output a graphical effect for the region of interest. According to one embodiment, electronic device 101 may provide 722 a visual effect to the border of the region of interest of the determined object, such as 720 of FIG.

In operation 607, the electronic device 101 may display information related to the extracted object processing method. For example, as shown in 720 of FIG. 7, the electronic device 101 may output an icon for selecting an object to be recognized among the extracted objects, an icon 724 for searching for a product related to the extracted object have.

According to various embodiments, the electronic device 101 may omit at least one of the operations associated with Fig. For example, the electronic device 101 may skip operations associated with operation 607. [

8A is a flow diagram illustrating a procedure for determining a tracking scheme in an electronic device 101 (or electronic device 201) according to various embodiments of the present invention. 8B is an exemplary diagram for explaining a procedure for determining a reference range for determining whether the tracking operation is continued or not. According to one embodiment, the procedure for determining the tracking scheme may be a detailed operation for operation 505 shown in FIG.

Referring to FIG. 8A, at operation 801, the electronic device 101 may determine the size of the determined region of interest. According to one embodiment, the electronic device 101 may determine a percentage of the area of interest for the displayed image.

At operation 803, the electronic device 101 may determine a tracking period based on the size of the region of interest. According to one embodiment, the tracking period may comprise a paging period of the tracking engine for tracking the region of interest (or object), a frame period for performing the tracking operation. For example, the smaller the area of interest, the less time it takes to track. Accordingly, the electronic device 101 can determine the tracking period based on the table information defined such that the tracking period is proportional to the size of the ROI, as shown in Table 1.

ROI size Tracking cycle First range 5 frame Second range 1 frame

For example, if the size of the region of interest is included in a first range (e.g., 50% or more of the displayed image), the electronic device 101 may set a tracking period to perform a tracking operation at a 5-frame period. In addition, if the size of the region of interest is included in a second range (e.g., less than 50% of the displayed image), the electronic device 101 may set a tracking period to perform a tracking operation in one frame period.

At operation 805, the electronic device 101 may determine a reference range for determining whether to continue the tracking operation. According to one embodiment, the reference range may be a motion allowed range of the region of interest.

For example, the electronic device 101 may determine a reference range based on the size of the region of interest. For example, when motion (or tremor) of the electronic device 101 occurs, the amount of motion of a region of interest larger than a certain size or larger than a region of interest may be largely changed. Accordingly, the electronic device 101 can determine the reference range based on the table information defined such that the reference range is proportional to the size of the ROI as shown in Table 2.

ROI size Reference range First range ROI size * 150% Second range ROI size * 120%

For example, if the size of the region of interest is included in the first range (e.g., 50% or more of the displayed image), the electronic device 101 may determine that the size of the region of interest is 150% (812). In addition, if the size of the region of interest is included in the second range (e.g., less than 50% of the displayed image), the electronic device 101 may determine that the size of the region of interest is 120% (822).

As another example, for a region of interest having a size greater than a certain level, the user may not consciously move the region of interest. Accordingly, the electronic device 101 may determine the reference range based on the table information defined such that the reference range is not proportional to the size of the ROI, as shown in Table 3.

ROI size Reference range Third Scope ROI size * 130%

For example, if the size of the region of interest is included in a third range (e.g., less than 80% of the displayed image), the electronic device 101 may set a reference range that magnifies the size of the region of interest by 130%.

9 is a flow chart illustrating another procedure for determining a reference range in an electronic device 101 (or electronic device 201) according to various embodiments of the present invention. According to one embodiment, the procedure for determining the tracking scheme may be a detailed operation for operation 805 shown in FIG.

9, at operation 901, the electronic device 101 may obtain distance information between the electronic device 101 and an object. According to one embodiment, the electronic device 101 may acquire distance information using a distance-measurable sensor (e.g., an ultrasonic sensor, an image sensor, a TOF (Time of Flight) sensor, etc.). According to another embodiment, the electronic device 101 may obtain distance information based on focal length, depth of image, and the like.

At operation 903, the electronic device 101 may determine a reference range based on the distance between the object and the electronic device 101. [ According to one embodiment, when the motion (or tremor) of the electronic device 101 occurs, the amount of motion of an object that is not the same as a distance away from the object may be greatly changed. Accordingly, the electronic device 101 can determine the reference range based on the table information defined based on the distance between the object and the electronic device 101, as shown in Table 4 below.

Street Reference range First range ROI size * 120% Second range ROI size * 150%

For example, when the distance between the object and the electronic device 101 is included in the first range (e.g., 5 m or more), the electronic device 101 can set a reference range in which the size of the area of interest is enlarged by 120%. In addition, when the distance between the object and the electronic device 101 is included in the second range (e.g., less than 5 m), the electronic device 101 can set a reference range that enlarges the size of the area of interest by 150%.

10 is a flow chart illustrating another procedure for determining the tracking scheme in electronic device 101 (or electronic device 201) according to various embodiments of the present invention. According to one embodiment, the procedure for determining the tracking scheme may be a detailed operation for operation 505 shown in FIG.

Referring to FIG. 10, at operation 1001, the electronic device 101 may determine the complexity of the objects included in the region of interest. According to one embodiment, the electronic device 101 can ascertain the complexity of the object based on the feature points. For example, the electronic device 101 can determine that the greater the number of extracted minutiae, the higher the complexity of the object. According to another embodiment, the electronic device 101 may determine the complexity of the object based on the motion of the object contained in the region of interest. For example, the electronic device 101 can determine that the more complex the motion of the electronic device 101 or the greater the movement of the object according to the amount of change in the image.

In operation 1003, the electronic device 101 may determine the tracking period based on the complexity of the object. For example, the higher the complexity of an object, the more complicated it can take. Accordingly, as shown in Table 5, the tracking period can be defined based on the complexity of the object.

Object complexity Tracking cycle Complexity above standard 5 frame Complexity below baseline 1 frame

For example, if a complexity of more than a criterion is confirmed, the electronic device 101 can set a tracking period to perform a tracking operation at a 5-frame period. Further, if a complexity lower than the reference is confirmed, the electronic device 101 can set a tracking period to perform a tracking operation in one frame period.

At operation 1005, the electronic device 101 may determine a reference range for determining whether to continue the tracking operation. According to one embodiment, the reference range may be a motion allowed range of the region of interest. For example, the electronic device 101 may determine a reference range based on the size of the region of interest. In another example, the electronic device 101 may determine a reference range based on distance information between the electronic device 101 and the object.

11A is a flow diagram illustrating a procedure for tracking an area of interest in an electronic device 101 (or electronic device 201) in accordance with various embodiments of the present invention. 11B is an exemplary diagram for explaining a procedure for determining whether the tracking operation is continued or not. According to one embodiment, the procedure for tracking the region of interest may be the detailed operation of operation 507 shown in FIG.

Referring to FIG. 11A, at operation 1101, the electronic device 101 can identify an input frame. According to one embodiment, the electronic device 101 can determine the number of frames to be input.

At operation 1103, the electronic device 101 may determine whether a tracking period has arrived. According to one embodiment, the electronic device 101 may determine whether a frame corresponding to a predetermined tracking period is input. For example, the electronic device 101 may determine that a tracking period has arrived when a frame corresponding to a predetermined tracking period is input.

If it is determined that the tracking cycle has not yet arrived, the electronic device 101 can determine whether or not the tracking cycle has arrived. According to one embodiment, electronic device 101 may perform the operations associated with operation 1101. [

If it is determined that a tracking period has arrived, at operation 1105, the electronic device 101 may determine the amount of motion of the region of interest. According to one embodiment, the electronic device 101 may detect a motion vector from a preview image.

At operation 1107, the electronic device 101 may determine whether the amount of motion of the region of interest outside the reference range has been confirmed. According to one embodiment, the electronic device 101 can determine whether the region of interest has moved within a range that can be tracked by the motion of the object of the electronic device 101. For example, electronic device 101 may determine that tracking is possible for a region of interest if region of interest 1124 does not deviate from reference range 1122, such as 1120 in FIG. 11B. In addition, the electronic device 101 may determine that tracking of the region of interest is not possible if the region of interest 1134 is outside the reference range 1132, such as 1130 of Figure 11B.

Once the amount of movement of the region of interest outside the reference range is identified, the electronic device 101 may stop the tracking operation for the region of interest. According to one embodiment, at operation 1115, the electronic device 101 may perform operations to extract other objects contained in the image.

Once the amount of movement of the region of interest is not outside the reference range, at operation 1109, the electronic device 101 may perform tracking on the object. According to one embodiment, the electronic device 101 is able to continuously track moving objects. In addition, the electronic device 101 may update the location of the region of interest based on tracking.

At operation 1111, the electronic device 101 may determine if a tracking stop event is detected. According to one embodiment, the electronic device 101 may determine that a tracking stop event has been detected if the state of the object being tracked satisfies a specified condition. For example, when the specified condition is satisfied, when the tracking object disappears in the displayed area, when the type of the object to be tracked is changed to a certain level or more, if the distance between the object to be tracked and the electronic device 101 is a certain level , And a case where it is distant from the center. According to another embodiment, the electronic device 101 may determine that a tracking stop event has been detected when the electronic device 101 moves beyond a certain level while tracking the object. According to another embodiment, the electronic device 101 may determine that a tracking stop event has been detected when an input of a user changing a region of interest is received.

If a tracking interruption event is not detected, the electronic device 101 may perform an object tracking operation. According to one embodiment, electronic device 101 may perform the operations associated with operation 1101. [

If a tracking interruption event is detected, at operation 1113, the electronic device 101 may stop object tracking.

12 is a flow diagram illustrating a procedure for tracking an area of interest in an electronic device 101 (or electronic device 201) according to various embodiments of the present invention. According to one embodiment, the procedure for tracking the region of interest may be the detailed operation of operation 507 shown in FIG.

Referring to Fig. 12, at operation 1201, the electronic device 101 may perform an object tracking operation. According to one embodiment, the electronic device 101 may continuously track the position of the region of interest, including at least one object or object contained in the image. For example, when 100 ms is required for tracking and a frame is input every 33 ms, the electronic device 101 can perform the tracking operation every four frames.

In operation 1203, the electronic device 101 may determine whether the frame rate for the frame received from the camera 180 is changed. According to one embodiment, the frame rate may be changed by ambient light.

If the frame rate is not changed, at operation 1207, the electronic device 101 may perform a tracking operation by maintaining a predetermined variable. For example, the electronic device 101 can maintain a tracking period for performing a tracking operation every four frames.

If the frame rate is changed, at operation 1205, the electronic device 101 may change the tracking variable based on the frame input time. According to one embodiment, the electronic device 101 may change the tracking period so that the frame input is not delayed. For example, if a frame that takes 100 ms to track and a frame that has been input every 33 ms is input every 66 ms, the electronic device 101 can change the tracking period so that the tracking operation is performed every two frames.

13 is a flow diagram illustrating a procedure for recognizing an object in an electronic device 101 (or electronic device 201) according to various embodiments of the present invention.

Referring to FIG. 13, at operation 1301, the electronic device 101 may output a preview image. According to one embodiment, the electronic device 101 may display a preview image at a specified frame rate (e.g., 30 fps).

In operation 1303, the electronic device 101 may detect motion for the preview image to be output. According to one embodiment, the electronic device 101 may detect a motion vector from a preview image. In addition, the electronic device 101 may determine whether motion above the threshold is detected based on the detected motion vector. For example, electronic device 101 may perform operations associated with operation 1305 if less than a threshold of motion is detected.

In operation 1305, the electronic device 101 may extract at least one object included in the preview image. According to one embodiment, the electronic device 101 includes at least one object extraction engine (e.g., a barcode extraction engine, a QR code extraction engine, a text extraction engine, a matter extraction engine, a human extraction engine, , A landmark extraction engine, etc.) to extract the objects included in the preview image.

At operation 1307, the electronic device 101 may determine a region of interest for the extracted object. According to one embodiment, when a plurality of objects are extracted, the electronic device 101 can determine a region of interest for an object satisfying a predetermined condition. For example, the electronic device 101 is based on the characteristics (e.g., size, object type (e.g., QR code, bar code, object, person, landscape, landmark, text etc.) To determine the size of the region of interest.

At operation 1309, the electronic device 101 may set a tracking variable based on the region of interest. According to one embodiment, the electronic device 101 may set at least one of a period for calling the tracking engine, and a reference range for determining whether the tracking operation is continued or not.

In operation 1311, the electronic device 101 may determine whether the tracking condition is satisfied. According to one embodiment, the electronic device 101 may determine that the tracking condition has been met if a tracking period arrives. According to another embodiment, the electronic device 101 may determine that the tracking condition is satisfied if the region of interest does not deviate from the reference range at the time the tracking cycle arrives.

If the tracking condition is not satisfied, the electronic device 101 can perform the operation of determining the tracking condition. According to one embodiment, electronic device 101 may perform operations associated with operation 1305. [

If the tracking condition is met, at operation 1313, the electronic device 101 may perform an object tracking operation.

In operation 1313, the electronic device 101 may determine whether an object recognition request is received while performing the tracking operation. According to one embodiment, the electronic device 101 may determine whether the user's input to acquire information of the extracted object is sensed.

If an object recognition request is not received, the electronic device 101 can perform an object tracking operation. According to one embodiment, electronic device 101 may perform the operations associated with operation 1313. [

If an object recognition request is received, at operation 1317, it may be determined whether the image correction conditions are satisfied. According to one embodiment, the electronic device 101 may determine whether the image correction conditions are satisfied based on the reliability of the tracking results. For example, the reliability of the tracking results may be related to the focus, brightness, etc. for the object. For example, the electronic device 101 can determine that the image correction condition is satisfied when the reliability of the tracking result is lower than a certain level.

If the image correction conditions are not satisfied, at operation 1321, the electronic device 101 may recognize the object based on the preview image. According to one embodiment, the electronic device 101 may obtain a still image from the preview image to recognize the object.

If the image correction conditions are satisfied, at operation 1319, the electronic device 101 may correct the image. According to one embodiment, the electronic device 101 may obtain a still image after correcting the image based on at least one of focus, brightness, and exposure from the preview image. Further, at operation 1321, the electronic device 101 can recognize the object from the corrected image.

14 is a flow chart illustrating a procedure for tracking an object in an electronic device 101 (or electronic device 201) according to various embodiments of the present invention. According to one embodiment, the procedure for tracking an object may be a detailed operation of operation 1313 shown in FIG.

Referring to Fig. 14, at operation 1401, the electronic device 101 may perform an object tracking operation.

At operation 1403, the electronic device 101 may determine if a tracking stop event is detected. According to one embodiment, the electronic device 101 may determine that a tracking stop event has been detected if the state of the object being tracked satisfies a specified condition. For example, when the specified condition is satisfied, when the tracking object disappears in the displayed area, when the type of the object to be tracked is changed to a certain level or more, if the distance between the object to be tracked and the electronic device 101 is a certain level , And a case where it is distant from the center. According to another embodiment, the electronic device 101 may determine that a tracking stop event has been detected when the electronic device 101 moves beyond a certain level while tracking the object. According to another embodiment, the electronic device 101 may determine that a tracking stop event has been detected when an input of a user changing a region of interest is received.

If a tracking interruption event is detected, at operation 1405, the electronic device 101 may stop the object tracking operation.

If a tracking interruption event is not detected, at operation 1407, the electronic device 101 may determine whether the image correction condition is satisfied. According to one embodiment, the electronic device 101 may determine whether the image correction conditions are satisfied based on the reliability of the tracking results. For example, the reliability of the tracking results may be related to the focus, brightness, etc. for the object. For example, the electronic device 101 can determine that the image correction condition is satisfied when the reliability of the tracking result is lower than a certain level. According to one embodiment, the electronic device 101 may determine whether the image correction conditions are satisfied based on a predefined period. According to another embodiment, the electronic device 101 may change the cycle of determining whether the image correction conditions are satisfied based on the complexity of the object and the size of the area of interest.

If the image correction conditions are not satisfied, the electronic device 101 may track the object in the displayed image.

If the image correction conditions are satisfied, at operation 1409, the electronic device 101 may correct the image. According to one embodiment, the electronic device 101 may obtain a still image after correcting the image based on at least one of focus, brightness, and exposure from the preview image. In addition, the electronic device 101 may track the object in the corrected image

15 is a flow chart illustrating a procedure for tracking an object included in an image in an electronic device 101 (or electronic device 201) according to various various embodiments of the present invention.

Referring to FIG. 15, at operation 1501, the electronic device 101 may display an image on the display 160. According to one embodiment, the electronic device 101 may control the display 160 to display an image (e.g., a preview image) obtained via the camera 180 when the camera 180 is activated.

In operation 1503, the electronic device 101 may determine an object tracking method based on a region of interest for at least one object included in the image displayed on the display 160. According to one embodiment, the electronic device 101 may analyze the displayed image to extract at least one object and determine at least a portion of the image containing the extracted object to be a region of interest. In addition, the electronic device 101 may determine the tracking scheme based on the time spent in tracking the determined region of interest. For example, when it is determined that a relatively small amount of time is required for the determined region of interest, the electronic device 101 can determine the first tracking method that calls the tracking engine with a high frequency by the object tracking method. In addition, when it is determined that a relatively long time is required for the determined region of interest, the electronic device 101 can determine the second tracking method for calling the tracking engine with a low frequency by the object tracking method.

At operation 1505, the electronic device 101 may track an object (or region of interest) determined in a determined object tracking manner. According to one embodiment, when tracking an object in a first tracking manner, the electronic device 101 may invoke a tracking engine based on a period corresponding to the first tracking manner. For example, the electronic device 101 can track an object by calling a tracking engine for each frame input from the camera 180. [ In addition, when tracking the object in the second tracking manner, the electronic device 101 can invoke the tracking engine based on the period corresponding to the second tracking method. For example, the electronic device 101 may process the tracking engine so that at least one of the frames input from the camera 180 is not called.

According to various embodiments, a method of operating an electronic device includes: controlling a display to display an image comprising one or more objects obtained via a camera module; Determining a tracking scheme for the one or more objects based at least in a region of interest for the one or more objects; And tracking the one or more objects based at least in part on the determined tracking scheme.

According to one embodiment, the operation of determining the tracking scheme may include adjusting a paging period of a tracking engine for tracking the one or more objects.

According to one embodiment, the operation of determining the tracking scheme may include determining a frame in which the tracking operation is to be performed.

According to one embodiment, the tracking scheme may be determined based at least on the size of the region of interest.

According to one embodiment, the tracking method may be determined based on at least the feature points included in the region of interest.

According to one embodiment, the determining the tracking scheme comprises: determining a reference range in which movement of the ROI is allowed based at least on the size of the ROI; And determining whether the tracking operation is continued based on the reference range and the motion of the ROI.

According to one embodiment, the region of interest may be determined based on attributes of the extracted objects or user input on the extracted objects.

According to one embodiment, the tracking method may be determined based on a frame rate of the camera module.

According to one embodiment, the tracking method may be determined based on a frame input time.

According to one embodiment, the one or more objects may be extracted by one or more object extraction engines.

And the embodiments disclosed in this document are presented for the purpose of explanation and understanding of the disclosed contents and do not limit the scope of various embodiments of the present invention. Accordingly, the scope of the various embodiments of the present invention should be construed as including all changes or various other embodiments based on the technical idea of various embodiments of the present invention.

Claims (20)

In an electronic device,
A camera module;
display; And
The processor comprising:
Using the camera module to obtain an image comprising one or more objects,
Using the display to display an area of interest associated with the one or more objects in an image containing the acquired one or more objects,
Determine a tracking scheme for the one or more objects based at least in a region of interest for the one or more objects, and
And to track the one or more objects based at least in part on the determined tracking scheme.
The method according to claim 1,
The processor comprising:
And adjust the paging cycle of the tracking engine to track the one or more objects based at least in part on the determined tracking method.
The method according to claim 1,
The processor comprising:
And determine a frame in which a tracking operation is to be performed based at least on the determined tracking method.
The method according to claim 1,
The processor comprising:
And determine the tracking method based at least on the size of the region of interest.
The method according to claim 1,
The processor comprising:
And determine the tracking method based at least on feature points included in the region of interest.
The method according to claim 1,
The processor comprising:
Determine a reference range in which movement of the ROI is allowed based on at least the size of the ROI, and determine whether to continue the tracking operation based on the reference range and the movement of the ROI.
The method according to claim 1,
The processor comprising:
And determine the region of interest based on attributes of the extracted objects or user input on the extracted objects.
The method according to claim 1,
The processor comprising:
And determine the tracking method based on a frame rate of the camera module.
9. The method of claim 8,
The processor comprising:
And determine the tracking method based on a frame input time.
The method according to claim 1,
The processor comprising:
And to extract one or more objects included in the displayed image using one or more object extraction engines.
A method of operating an electronic device,
Controlling the display to display an image comprising one or more objects acquired via a camera module;
Determining a tracking scheme for the one or more objects based at least in a region of interest for the one or more objects; And
And tracking the one or more objects based at least in part on the determined tracking scheme.
12. The method of claim 11,
Wherein the act of determining the tracking scheme comprises adjusting a paging period of a tracking engine for tracking the one or more objects.
12. The method of claim 11,
Wherein the determining the tracking scheme comprises determining a frame in which the tracking operation is to be performed.
12. The method of claim 11,
Wherein the tracking scheme is determined based at least on the size of the region of interest.
12. The method of claim 11,
Wherein the tracking scheme is determined based at least on feature points included in the region of interest.
12. The method of claim 11,
The operation of determining the tracking method includes:
Determining a reference range in which movement of the ROI is allowed based at least on the size of the ROI; And
And determining whether to continue the tracking operation based on the reference range and the movement of the ROI.
12. The method of claim 11,
Wherein the region of interest is determined based on attributes of the extracted objects or user inputs to the extracted objects.
12. The method of claim 11,
Wherein the tracking method is determined based on a frame rate of the camera module.
19. The method of claim 18,
Wherein the tracking method is determined based on a frame input time.
12. The method of claim 11,
Wherein the one or more objects are extracted by one or more object extraction engines.

Images