KR20140103036A - Object tracking device - Google Patents

Abstract

The present invention relates to an electronic device and, more specifically, to an electronic device including a function for tracking an object and an operating method thereof. The object tracking method according to the present invention may comprise a step of separating a barcode from a captured image as an object; a step of extracting data from the barcode; a step of generating an original of the barcode using the data; and a step of acquiring feature points from the original; and a step of tracking the barcode using the feature points.

Description

[0001] OBJECT TRACKING DEVICE [0002]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present disclosure relates to electronic devices, and more particularly to an electronic device including a function of tracking an object and a method of operating the same.

Object tracking is the tracking of objects (eg, images, text, barcodes, etc.) that are recognized by the computer. Electronic devices may thus include techniques for tracking objects. In particular, recent electronic devices can recognize and track square bar codes. For example, technologies for tracking Quick Response (QR) codes have been applied to mobile terminals such as smart phones and tablet PCs. In addition, the bar code tracking technology has been applied to various kinds of electronic devices or appliances.

Conventional bar code tracking technology treats bar codes as images. That is, the conventional bar code tracking technology can detect the bar code (a part of the captured image) in the image captured by the camera and obtain the position of the bar code using information of the detected bar code (e.g., descriptor). However, such detection based tracking may be inefficient in terms of quality and performance, since the detection and decoding of the bar code is sensitive and motionless to the motion of the camera. Barcodes in captured images can also be difficult to recognize due to obstructions, lighting artifacts, moving blur, and the like. If so, the captured image can have a negative impact on the quality of the trace.

At least some of the embodiments of the present disclosure may suggest ways in which, besides other uses, the foregoing can be addressed. The method and apparatus according to the present disclosure can provide a technique for robustly acquiring the position of a barcode with high quality robustness to various obstacles (e.g., camera motion, fuzzy shape, etc.). Here, 'robust' can mean that the bar code tracking technology is less adversely affected from the obstacle. Also, the method and apparatus according to the present disclosure may be used in real-time image processing and computer vision applications.

A method of operating an electronic device in accordance with the present disclosure includes tracking an identified object on a plurality of digital images, wherein the tracing operation is performed on an object recognized from at least one of the images Using at least some of the data to create a reference object; And tracking the object using the generated reference object.

A method of operating an electronic device in accordance with the present disclosure includes: segmenting a bar code in a captured image; Extracting data from the barcode; Generating an original of the barcode using the data; Obtaining characteristic points from the original; And tracking the barcode using the feature points.

A method of operating an electronic device according to the present disclosure includes receiving a captured image; Dividing the barcode into objects in the captured image; Extracting data from the barcode; And displaying the data, a portion of the data, or information related to the data on the barcode.

An electronic device according to the present disclosure includes a memory configured to store a plurality of digital images; And a processor configured to track a recognized object on the plurality of digital images and to track the object recognized from at least one of the images to at least a portion To generate a reference object, and to use the generated reference object to track the object.

An electronic device according to the present disclosure includes a camera for capturing an image; A display unit for displaying the captured image; And a control unit for detecting and tracking a barcode in the captured image, wherein the controller identifies an object of the barcode in the captured image, extracts data from the barcode, and extracts the original of the barcode using the data, Obtain feature points from the original, and track the barcodes using the feature points.

An electronic device according to the present disclosure includes a camera for capturing an image; A display unit for displaying the captured image; And a controller for detecting and tracking a barcode in the captured image, wherein the controller is configured to classify the barcode in the captured image, extract data from the barcode, and store the data and the data on the barcode And to display information related to a part or the data.

As described above, according to the object tracking method and the electronic apparatus according to the present disclosure, the present disclosure provides a technology for robustly and robustly obtaining the position of a barcode in various obstacles (e.g., camera motion, fuzzy shape, etc.) .

Figure 1 is a block diagram illustrating an object processing system in accordance with one embodiment of the present disclosure.
Figure 2 is a block diagram of the main unit of Figure 1 in accordance with one embodiment of the present disclosure.
3 is a diagram illustrating a configuration of a terminal to which the object tracking function of the present disclosure is applied.
4 is a diagram showing an example of a platform to which the object tracking function of the present disclosure is applied.
5 is a general flow diagram of a barcode tracking method in accordance with an embodiment of the present disclosure.
6 is a diagram showing an original image of a QR code.
7 is a diagram showing an original image that is upscaled.
FIG. 8A is a diagram showing a barcode being previewed on the screen, and FIG. 8B is a diagram showing 'related data' displayed on the barcode.
FIG. 9 is a flowchart illustrating a barcode tracking method using characteristic points according to an embodiment of the present disclosure.
FIG. 10 is a diagram for explaining a process of moving a patch along a spiral path to find a corresponding patch in a captured image.
11 is a flowchart illustrating a barcode-related data display method according to an embodiment of the present invention.
12 is a diagram showing finder patterns of a QR code.

In the present disclosure, the electronic device is a device having a camera, for example, a smart phone, a tablet PC, a notebook PC, a digital camera, a computer monitor, a PDA (Personal Digital Assistant), an electronic notebook, a desktop PC, A home appliance (e.g., a refrigerator, a TV, a washing machine) having a touch screen, a media player (e.g., an MP3 player), a sound device, a wrist watch,

Hereinafter, a bar code detecting method and an electronic apparatus according to the present disclosure will be described in detail. It is to be understood that the words or words used herein are not to be construed in a limiting sense, and should be construed in light and descriptive sense consistent with the technical idea of the present disclosure. Therefore, the following description and accompanying drawings are merely exemplary of the present disclosure, and are not intended to represent all of the technical ideas of the present disclosure, so that various equivalents and modifications may be made thereto at the time of the present application have. Also, in the accompanying drawings, some of the elements are exaggerated, omitted or schematically shown, and the size of each element does not entirely reflect the actual size. Accordingly, the present disclosure is not limited by the relative size or spacing depicted in the accompanying drawings. A detailed description of known functions or configurations incorporated herein will be omitted when it may obscure the subject matter of the present disclosure.

Figure 1 is a block diagram illustrating an object processing system in accordance with one embodiment of the present disclosure.

Referring to FIG. 1, an object processing system 100 may include a client 110, a server 120, and a communication network.

The object processing system 100 of the present disclosure including the above-described configuration mounts the client 110 supporting the object tracking function described above, and supports the object tracking function based thereon. At this time, the object processing system 100 of the present disclosure can support the formation of a communication channel between the server 120 and the client 110 through the communication network using the communication unit of the client 110. [ Accordingly, various information required in the process of supporting the object tracking function can be provided to the client 110 through the server 120. Alternatively, the client 110 in the present disclosure may receive and store data stored in the server 120 from the server 120, and support an object tracking function based on the stored data.

In the object processing system 100, the client 110 may be implemented as any one of the above-described electronic devices and performs connection to the server 120 through a communication network. The client 110 may then provide the obtained image information to the server 120. In particular, the client 110 can provide the acquired image information to the server 120 in real time. The server 120 may then perform phase correlation for object tracking based on the received image information and provide the result to the client 110. The client 110 may omit operation for object tracking in the image information based on the values provided by the server 120 and support data processing for easier object tracking. Meanwhile, the client 110 may receive the remote reference object and the content data provided by the server 120. Then, the client 110 can recognize the image information and localize the object using the remote reference object. Also, the client 110 may control the content data to be applied to an Augmented Reality (AR) service.

Meanwhile, the processor of the client 110 can recognize an object from at least one of the digital images, can generate a reference object using at least a part of the recognized object, You can track the recognized object. The processor may also modify an error contained in the recognized object when creating the reference object. The processor may determine whether the recognized object to which the error has been corrected has been modified correctly. If the object includes a bar code, the processor can extract data from the bar code, correct the errors contained in the extracted data, and decode the error corrected data. In addition, when the object includes a bar code, the processor can generate a reference bar code using at least a portion of the recognized object, obtain feature points from the generated reference bar code, track the object using the feature points have.

In the object processing system 100, the client 110 may include an object processing unit 111 that is applied to the above-described electronic apparatus and can be a main unit. The object processing unit 111 can receive camera input data, media input data, audio input data, and sensor input data from a plurality of input units such as a camera, a media unit, an audio unit, and a sensor unit, respectively. The sensor input data may include at least one input data from among, for example, an accelerometer, a gyroscope, a magnetic sensor, a temperature sensor, a gravity sensor, and the like. The object processing unit 111 may use a memory 112, a CPU (Central Processing Unit) 113 and a GPU (Graphic Processing Unit) 114 for necessary processing of the input data. The object processing unit 111 may also identify and recognize a target (e.g., a QR code) (e.g., a reference (e.g., a QR code) to detect a target a reference database may be used as the reference DB 115. The reference DB may include a local reference DB 115 provided in the client 110 and a remote reference DB 121 provided in the server 120. [ The output data output from the object processing unit 111 may include, for example, identification information and localization information. The position measurement information may include two-dimensional (2D) The AR content management unit 116 includes an object processing unit 111 (hereinafter, referred to as " object processing unit 111 "), ) Output And organize the final video / audio output data with the content stored in the data and remote / local content DB 122,117.

In the object processing system 100, the server 120 supports connection of the client 110. The server 120 supports the object tracking function and the augmented reality service function at the request of the client 110. For this purpose, the server 120 may store a remote reference DB 121 for supporting the object tracking function. Also, the server 120 may store the remote content DB 122 to be applied to the augmented reality to support the augmented reality service function. The server 120 may perform at least one of a recognition process of a specific image information, a localization process, and an object tracking process according to a request of the client 110. The server 120 may provide the client 110 with a result of performing the process in response to a request from the client 110.

In the object processing system 100, a communication network may be disposed between the client 110 and the server 120. [ And the communication network can form a communication channel between the two configurations. The communication network may be composed of mobile communication network devices when the client 110 supports the mobile communication function. In addition, the communication network may be composed of devices supporting the Internet network when the server 120 connects the communication device through the Internet network. The communication network may further include a network device for transferring data between the heterogeneous networks. Accordingly, the communication network of the present disclosure is not limited to a specific communication method, a communication unit, and the like, but should be understood as an apparatus to which various apparatuses and methods capable of transmitting and receiving data between the client 110 and the server 120 are applied.

Figure 2 is a block diagram of the main unit of Figure 1 in accordance with one embodiment of the present disclosure.

2, the object processing unit 111 is implemented with at least one processor (e.g., an application processor), and the application processor includes an input control unit 210, a recognition unit 220, a localization unit 230 and a tracking unit 240.

The input control unit 210 can classify the input data provided to the main unit 111. [ The input control unit 210 can determine the transfer route of the input data according to the function execution state of the main unit 111 at present. For example, the input control unit 210 may provide the image information to the recognition unit 220 when the initial image information is acquired. The image information may be obtained from a camera connected to the main unit 111 or a camera disposed in the terminal including the main unit 111.

The input control unit 210 may transmit the image information directly to the tracking unit 240 when the image recognition process by the recognition unit 220 and the object classification unit 230 are completed. Or the input control unit 210 may simultaneously transmit the image information to the recognition unit 220 and the tracking unit 240. [ Accordingly, recognition processing and object tracking processing on image information may be performed in parallel.

The input control unit 210 can control not to provide the image information to the recognition unit 220 when performing the object tracking function by the tracking unit 240. [ And the input control unit 210 can assist in providing the image information to the recognition unit 220 again if the object tracking fails. The input control unit 210 may also provide other input information, such as audio information or sensor information, to the tracking unit 240 when the AR content is applied to the objects being tracked.

When the recognition unit 220 receives the image information from the input control unit 210, it can perform the recognition process. That is, the recognition unit 220 may perform the feature detection 221, the descriptors calculation 222, and the image query process 223 on the received image information.

The feature detection 221 may be a process of detecting feature points in an image. Feature detection 221 may include a binarization process. That is, when the binarization process is performed, the color image is converted into a monochrome image. In addition, the feature detection 221 according to the present disclosure may include a barcode detection process.

The descriptor may be information defining the characteristic of the corresponding image calculated based on the detected feature information. The descriptor may be defined by at least one of the location of the characteristic points, the arrangement pattern between the characteristic points, and the characteristic characteristics of the characteristic points in a predetermined part of the image information. That is, the descriptor may be a value obtained by simplifying the characteristic of a certain point of the image information. Therefore, at least one descriptor may be extracted from one piece of image information.

When the descriptor calculation 222 is completed, the recognition unit 220 performs a comparison with the reference object through an image query process 223. That is, the recognition unit 220 determines whether there is a reference object having the same descriptor as the calculated descriptors or descriptors within a certain error range. The reference object may be provided from an internal memory provided for operation of the main unit 111. [ The reference object may also be provided from an external storage device such as a separate server for the operation of the main unit 111. [ The reference object may be previously stored image information for a specific image. For example, the face recognition process may include a process of recognizing authenticated faces and requesting an external reference face database and identifying differences between the faces. On the other hand, QR codes do not usually need to be updated dynamically. In addition, QR codes require only certain rules to recognize QR codes in the database. Thus, a QR code can generally have an internal reference object. The recognition unit 220 can simplify the operation for the image recognition process through utilization of the reference object. And the recognition unit 220 may utilize the reference object to perform the target object identification.

The object sorting unit 230 is a structure for sorting various objects constituting image information. This object classification unit 230 includes a feature matching 231 and an Initial Pose Estimation step 232. That is, the object classification unit 230 extracts characteristic points of the objects separated from the image information. And the object classification unit 230 matches characteristic points of specific objects with at least a part of the reference object. At this time, the object sorting unit 230 can update the matching information when there is no matching information of the characteristic points. When the feature matching for the object is completed, the object sorting unit 230 performs initial posture estimation of at least one object included in the image information. The object classification unit 230 provides the tracking unit 240 with object related information including at least one of matching information and initial orientation information when the main unit 111 activates the object tracking function.

The tracking unit 240 receives the initial posture estimation of the recognized target objects from the object classification unit 230. [ And the tracking unit 240 can continuously maintain tracking through the posture computation of the target object. The tracking unit 240 may have a basic output of recognition information and localization information included in the object posture.

3 is a diagram illustrating a configuration of a terminal to which the object tracking function of the present disclosure is applied. That is, the terminal 300 shown in FIG. 3 may be an example of an electronic device on which the main unit described above is mounted.

3, the terminal 300 controls a signal flow between the overall operation of the terminal 300 and the internal configurations of the terminal 300, performs a function of processing data, And a control unit 305 for controlling power supply to the configurations. The control unit 305 may include a call processor 310, an application processor 320, and a memory 340.

The call processor 310 performs signal transmission / reception with the RF unit 330, and can perform signal transmission / reception with the memory 340 and the SIM card 311. The call processor 310 may communicate with the application processor 320 to support the processing of the RF unit 330, the memory 340, and the SIM card 311 among the functions processed by the application processor 320 have.

The application processor 320 may operate as the processor of the object processing unit 111 or the client 110 described above. The application processor 320 may also include one or more central processing units (CPUs). In addition, the application processor 320 may further include one or more graphics processing units (GPUs).

The application processor 320 receives power from the power management unit 350 to which the battery 351 is connected. The application processor 320 includes various communication units such as a Wi-Fi unit 321, a Bluetooth (BT) unit 322, a GPS unit 323, an NFC (Near Field Communication) unit 324, Etc., and supports the functions performed by each unit.

The application processor 320 may be connected to a user input unit 325 including a touch panel and a key input unit. Here, the touch panel may include a touch screen provided on the screen of the display unit 326. [ The touch panel generates a touch event in response to a gesture of a touch input mechanism (e.g., a finger, a pen, or the like) with respect to the screen, converts the touch event into an analog to digital (AD) The key input unit may include a touch key. The touch key may be implemented by a capacitance type or a resistive type in order to detect the user's touch. The touch key generates an event in response to a user's touch, and can transmit the event to the AP 320. [ The key input unit may include a key (e.g., a dome key) other than the touch method.

The application processor 320 may perform signal transmission / reception with the memory 340. The memory 340 may include a main memory unit and a secondary memory unit. The auxiliary memory may store a boot program, at least one operating system and applications. The main memory may store various programs loaded from the auxiliary memory, such as boot programs, operating systems, and applications. When the power of the battery 351 is supplied to the application processor 320, the boot program is first loaded into the main memory. These boot programs load the operating system into main memory. The operating system loads applications into main memory. The application processor 320 may access such programs to decrypt the instructions of the program and to execute functions (e.g., object recognition, localization, object tracking) depending on the decoding result.

The application processor 320 may be connected to the display unit 326, the camera 327, the vibration motor 328, the audio processing unit 380, and the like.

The display unit 326 displays data on the screen under the control of the control unit 305, in particular, the application processor 320. [ That is, when the control unit 305 processes (e.g., decodes) the data and stores the data in the buffer, the display unit 326 converts the data stored in the buffer into an analog signal and displays it on the screen. When power is supplied to the display unit 326, the display unit 326 displays a lock image on the screen. If the lock release information is detected while the lock image is being displayed, the control unit 305 releases the lock. The display unit 326 displays, for example, a home image in place of the lock image under the control of the control unit 305. [ The home image includes a background image (e.g., a photo set by the user) and a number of icons displayed thereon. Here, the icons each indicate an application or content (e.g., a photo file, a video file, a recorded file, a document, a message, etc.). When one of the icons, e.g., the object tracking application's icon, is touched by the touch input mechanism, the control unit 305 drives the camera 327 and executes the object tracking application using the image received from the camera 327 have. The display unit 326 receives an image (e.g., preview image and object tracking information) according to the execution of the object tracking application from the control unit 305, and converts the analog signal into an analog signal.

The camera 327 captures an object and outputs the captured object to the application processor 320. The camera 327 includes a lens for collecting light, an image sensor for converting the light into an electrical signal, and an image signal processor (image data processor) 322 for processing electrical signals input from the image sensor into frames (raw data) Signal Processor (ISP). The ISP can resize the frame waiting in the buffer (aka, a queue) of the ISP to a preview image under the control of the AP 320. [ Typically, the ISP shrinks the frame to fit the size of the screen. Then, the ISP outputs the preview image to the AP 320. Then, the AP 320 controls the display unit 326 to display the preview image on the screen. Of course, such resizing may also be performed in the AP 320. [ For example, the frame is transferred from the camera 327 to the buffer of the AP 320, and the AP 320 can process the frame into a preview image and output it to the display unit 326.

The audio processing unit 380 may include a microphone 381, a speaker 382, a receiver 383, an earphone connection unit 384, and the like. The application processor 320 may be coupled to the sensor hub 360. The sensor hub 360 may be connected to a sensor unit 370 including various sensors. The sensor unit 370 includes a magnetic sensor 371, a gyro sensor 372, a barometer 373, an acceleration sensor 374, a grip sensor 375, a temperature / humidity sensor 376, a proximity sensor 377, A sensor 378, an RGB sensor 379a, and a gesture sensor 379b.

4 is a diagram showing an example of a platform to which the object tracking function of the present disclosure is applied.

Referring to FIG. 4, a platform to which the object tracking function of the present disclosure is applied may largely include an application layer 410, an application framework layer 420, a library layer 430, and a kernel layer 440.

The kernel layer 440 may be configured as a Linux kernel, for example. The kernel layer 440 may include a display driver, a camera driver, a BT driver, a shared memory driver, a vendor driver, a USB driver, a keypad driver, a Wi-Fi driver, an audio driver, and a power management unit.

The library layer 430 includes a surface manager, a media framework, SQLite, OpenGL / ES, FreeType, Webkit, SGL, SSL, Libc and the like. Library layer 430 may include an Android runtime configuration. The Android runtime can include core libraries and the Dalvik Virtual Machine. Dalvik Virtual Machine can support widget function of terminal with object tracking function, function requiring real-time execution, function requiring periodical execution according to preset schedule.

The application framework layer 420 may include an activity manager, a window manager, a content provider, a view system, a notification manager, a package manager, a telephony manager, a resource manager, and a location manager. The application layer 410 may include a home application, a dialer application, an SMS / MMS application, an instant messenger (IM) application, a camera application, an alarm application, a calculator application, a content application, a voice dial application, Apps, media player apps, album apps, watch apps, and the like.

Hereinafter, the object tracking function of the present disclosure will be described in more detail with reference to Figs. 5 to 11. Fig.

5 is a general flow diagram of a barcode tracking method in accordance with an embodiment of the present disclosure. 6 is a diagram showing an original image of a QR code. 7 is a diagram showing an original image that is upscaled. FIG. 8A is a diagram showing a barcode being previewed on the screen, and FIG. 8B is a diagram showing 'related data' displayed on the barcode.

3, 5 and 6, the control unit 305, in particular, the application processor 320 receives an execution request of an application related to the tracking of the bar code from the user input unit 325. In response to this, the control unit 305 loads the application from the auxiliary memory to the main memory and operates it as a process. The control unit 305 drives the camera 327 by the instruction of this process and in step 510 the control unit 305 receives the captured image from the camera 327. [ On the other hand, the control unit 305 may receive a captured image from an external device via a wireless communication unit such as a Wi-Fi unit 321, a Bluetooth unit 322, an NFC unit 324, or an RF unit 330. [

In step 520, the control unit 305 determines whether object classification of the bar code is possible in the captured image. There are many ways to classify objects. For example, the method of classifying an object using a find pattern may be as follows. First, the control unit 305 detects candidates that can be a find pattern in the captured image. Here, the find pattern is a symbol for finding the position of the corresponding bar code. For example, in the case of a square-shaped QR code, find patterns are located in three of four corners. Therefore, when these find patterns are retrieved, the area of the bar code is determined, and data can be extracted in the determined area. Next, the control unit 305 detects a contour of the candidates. Here, if the size (e.g., a value indicating the width, height, width, etc.) of the outline is smaller than the first value selected by the control unit 305, the candidate can be discarded from the list. If the outline size is larger than the second value selected by the control unit 305, the candidate can be discarded from the list. Next, the control unit 305 determines the find patterns among the candidates using the detected contours. Next, the control unit 305 determines the version and size of the corresponding bar code using the find patterns. The control unit 305 can localize the corresponding bar code in the image using size information and position information of the determined find patterns (e.g., vertex coordinates). Also, the controller 305 may predict a camera pose (e.g., a viewing angle of the camera) using the find patterns of the object-separated bar code. An example of such camera posture prediction will be described with reference to Fig.

If the barcode in the captured image is object-separated, the control unit 305 extracts data from the barcode in step 530. Specifically, the control unit 305 can warp the object-separated (i.e., separated from the captured image) barcode from the captured image to restore the original correctly. This watermarking process may include a process of correcting the coordinate values of the barcode. This correction process may be referred to as homography calcination. The control unit 305 can extract data from the watermarked bar code. For more accurate object classification and restoration, the controller 305 may detect an alignment pattern of the corresponding bar code using the determined version and size. When the alignment pattern 640 is detected, the control unit 305 can identify the corresponding bar code in the image using the alignment pattern.

The data extracted in step 530 may be encoded data. Where the encoded data may be referred to as encrypted data. In step 540, the control unit 305 may perform error correction on the encrypted data. For example, in the case of QR codes, methods and rules for error correction for encrypted data are described in detail in ISO / IEC 18004: 2006.

In step 550, the control unit 305 decodes the result of the error correction. For example, in the case of a QR code, the control unit 305 decodes the encrypted data according to the decoding rule defined in ISO / IEC 18004: 2006. Then, the control unit 305 determines whether the decoded data is "meaningful data ". For example, when the control unit 305 finds data matching the 'decoded data' in the reference database (for example, the local reference DB 115 or the remote reference DB 121), the control unit 305 converts the 'decoded data' '. That is, if data matching the 'decoded data' is found in the DB, the control unit 305 determines that decoding of the data is successful.

If the decoding is successful, the control unit 305 generates an original image (hereinafter referred to as an original) of the bar code using the decoded data in step 560. In the case of the QR code, the original may be an image as shown in Fig. 6, reference numerals 610, 620, and 630 denote a find pattern, and reference numeral 640 denotes an alignment pattern.

In step 570, the control unit 305 acquires feature points in the original. In particular, the control unit 305 can enlarge the size of the original and obtain characteristic points from the enlarged original.

An example of the feature point acquisition method is as follows. The control unit 305 enlarges the original by m times. For example, one pixel is enlarged to m * m pixels. Where m is a positive integer and can be determined from the type and size of the original (e.g., area). For example, when the original is a QR code, the control unit 305 can determine the value of m to be 4. This enlarged image can be used as an original barcode image for tracking the barcode. That is, the control unit 305 can use the enlarged original as a reference for bar code tracking.

Next, referring to Fig. 7, the control unit 305 divides the enlarged original into grids. Here, the grid consists of n * n pixel areas, called patches. Where the value of n can be determined from the type and size of the original (e.g., area). For example, when the original is a QR code, the control unit 305 can determine the value of n as 15. The control unit 305 can select characteristic points from the center points of each of the patches. For example, the control unit 305 calculates a score for each of the center points. There are many ways to calculate scores. For example, the Shi-Tomashi algorithm can be used as a scoring method in this disclosure. The control unit 305 can determine the center points corresponding to the points exceeding the predetermined threshold among the scores as the original feature points. On the other hand, regardless of the score calculation, the control unit 305 may determine all of the center points as the characteristic points.

In step 580, the control unit 305 tracks the barcode using the feature points of the reference (enlarged original). That is, the control unit 305 tracks the change of the characteristic points. An example of a bar code tracking method using feature points will be described in detail with reference to FIGS. 9 and 10. FIG.

In step 590, the control unit 305 controls the display unit 326 to display data (e.g., decoded data, part of decoded data, and data related to decoded data) related to the bar code on the bar code being tracked. For example, a QR code as shown in Fig. 8A may be tracked and an image as shown in Fig. 8B may be displayed thereon.

FIG. 9 is a flowchart illustrating a barcode tracking method using characteristic points according to an embodiment of the present disclosure. FIG. 10 is a diagram for explaining a process of moving a patch along a spiral path to find a corresponding patch in a captured image.

9, in step 910, the control unit 305 receives the captured image from the camera 327 or the wireless communication unit.

In step 920, the control unit 305 projects each feature point of the reference (e.g., enlarged original) onto the captured image using the homography of the bar code obtained from the camera pose prediction. Where the camera posture prediction may include posture information for a previous frame (e.g., a captured image received at step 510). Also, the camera posture prediction may be obtained using a motion model or rough calculation.

In step 930, the control unit 305 transforms the patches of the reference corresponding to each of the projected characteristic points so as to match the barcode of the captured image. For example, the control unit 305 warps the patches of the reference with rectangular patches on the captured captured image. In this warping operation, inverted homography of the bar code obtained from the camera posture prediction can be used. Referring to FIG. 10, reference numeral 1010 denotes a patch of a warped reference and a center point of the patch 1011, that is, a feature point. During the process of the patch projection process, anti-aliasing (e.g., bilinear interpolation, bicubic interpolation, Multi-Sample Anti-Aliasing (MSAA) Fast Approximate Anti-Aliasing (FXAA), etc.) can be used.

In step 940, the control unit 305 retrieves patches corresponding to the modified patches from the captured image. For example, the control unit 305 moves the warped patch 1010 along the predetermined spiral trajectory 1020 from the feature point 1011. [ While the warped patch 1010 is moved along the spiral trajectory 1020, the control unit 305 calculates a correlation between the captured patch image 1010 and the captured image region. Where the correlation can be calculated using a Normalized Cross Correlation (NCC) method, a Sumof Square Distances (SSD) method, or other methods. The region 1030 having the highest correlation with the patch 1010 is determined as the patch corresponding to the patch 1010 (i.e., the most similar patch) while the patch 1010 is moved along the spiral trace 1020 . Then, the center point 1031 of the area 1030 is determined as a new feature point. That is, the characteristic point 1011 is updated to the searched characteristic point 1031.

In step 950, the control unit 305 estimates (calculates) an updated camera pose based on the search result, that is, the updated feature point 1031. M-estimator, robust hypothesize-to-verify method (e.g., RANSAN, PROSAC, etc.) may be used for the calculation method. The control unit 305 can use a value indicating the interval between the center point 1011 of the patch 1010 and the center point 1031 of the patch 1030 in calculation (estimation).

11 is a flowchart illustrating a barcode-related data display method according to an embodiment of the present invention. In the description of FIG. 11, the description overlapping with the above description may be omitted. 11, in step 1110, the control unit 305 receives the captured image from the camera 327 or the wireless communication unit. In step 1120, the control unit 305 identifies the bar code in the captured image. In step 1130, the control unit 305 extracts data from the object-separated bar code. In step 1140, the control unit 305 controls the display unit 326 to display the extracted data (or a part or the related data) on the object-separated bar code.

12 is a diagram showing finder patterns of a QR code. Referring to FIG. 12, reference numerals 1210, 1220, and 1230 denote the find patterns of QR codes detected in the captured image. The control unit 305 calculates the first length of the first line segment 1240 connecting the first center point 1211 and the second center point 1221. [ The control unit 305 calculates the second length of the second line segment 1250 connecting the second center point 1221 and the third center point 1231. [ Here, the units of length are pixels, blocks, and the like. Next, the control unit 305 calculates a ratio L (= first length / second length or second length / first length) related to the camera posture. For example, if the viewing angle of the camera is ideally right (90 degrees) to the subject (i.e., bar code), the ratio L may be "1 ". As the viewing angle of the camera becomes oblique with respect to the subject, the ratio L may become gradually larger or smaller.

According to some embodiments, a method of operating an electronic device includes the act of tracking a recognized object on a plurality of digital images, wherein the tracking operation is performed on at least one of the images Using at least some of the recognized objects to generate a reference object; And tracking the object using the generated reference object. The act of creating the reference object may include modifying an error contained in the recognized object. Additionally, the act of creating the reference object may include determining whether the recognized object to which the error was corrected has been modified correctly. Wherein the object includes a barcode, the act of creating the reference object comprises: extracting data from the barcode; And correcting errors contained in the extracted data. The act of creating the reference object may further include decoding the error corrected data. Wherein the object includes a barcode, and using the generated reference object, tracking the object comprises: obtaining minutiae points from the generated reference barcode; And tracking the barcode on the plurality of images using the feature points.

According to some embodiments, a method of operating an electronic device includes: segmenting a bar code in a captured image; Extracting data from the barcode; Generating an original of the barcode using the data; Obtaining characteristic points from the original; And tracking the barcode using the feature points. The step of tracking the barcode may include projecting the original feature points onto a newly captured image; Transforming the original patches corresponding to each of the projected characteristic points to match the barcode; Retrieving patches corresponding respectively to the modified patches from the newly captured image; And estimating a posture of the electronic device based on the search result. The step of estimating the attitude of the electronic device may include estimating the attitude of the electronic device using the values representing the intervals of the modified patches and each of the retrieved patches. The intervals may be intervals of each of the center points of each of the modified patches and the center points of each of the retrieved patches. Wherein retrieving the patches corresponding respectively to the modified patches from the newly captured image further comprises moving the modified patches from the projected feature point to a locus selected by the electronic device, And retrieving from the newly captured image. The locus may be spiral. Displaying the data, a portion of the data, or information related to the data on the barcode to be tracked. The step of acquiring minutiae points in the original may include the steps of enlarging the size of the original and dividing the enlarged original into grids composed of a plurality of patches; And acquiring characteristic points in each of the patches.

According to some embodiments, a method of operating an electronic device includes receiving a captured image; Dividing the barcode into objects in the captured image; Extracting data from the barcode; And displaying the data, a portion of the data, or information related to the data on the barcode.

According to some embodiments, the electronic device comprises a memory configured to store a plurality of digital images; Wherein the processor is configured to track a recognized object on the plurality of digital images and to track the object using at least some of the objects recognized from at least one of the images To generate a reference object, and to use the generated reference object to track the object. The processor may be configured to modify an error contained in the recognized object. The processor may be configured to determine whether the recognized object for which the error was corrected has been modified correctly. The object may include a barcode, and the processor may be configured to extract data from the barcode and to correct errors contained in the extracted data. The processor may be configured to decode the error corrected data. The object includes a barcode and the processor is configured to obtain feature points from the generated reference barcode and to use the feature points to track the barcode on the plurality of images.

According to some embodiments, the electronic device comprises a camera for capturing an image; A display unit for displaying the captured image; And a control unit for detecting and tracking a barcode in the captured image, wherein the controller identifies an object of the barcode in the captured image, extracts data from the barcode, and extracts the original of the barcode using the data, Obtain feature points from the original, and track the barcodes using the feature points. Wherein the control unit projects the original feature points onto a newly captured image, transforms the original patches corresponding to each of the projected feature points to match the bar code, Retrieve the patches from the newly captured image, and estimate the posture of the electronic device based on the search results. The control unit may estimate the posture of the electronic device using the values indicating the intervals of the modified patches and the searched patches. The intervals may be intervals of each of the center points of each of the modified patches and the center points of each of the retrieved patches. The control unit may move the deformed patches from the projected feature points, respectively, to a locus selected by the electronic device, and retrieve corresponding patches from the newly captured image. The display unit may display the data, a part of the data, or information related to the data on the barcode to be tracked. The control unit may enlarge the size of the original, divide the enlarged original into grids constituted by a plurality of patches, and obtain characteristic points from each of the patches.

According to some embodiments, the electronic device comprises a camera for capturing an image; A display unit for displaying the captured image; And a controller for detecting and tracking a barcode in the captured image, wherein the controller is configured to classify the barcode in the captured image, extract data from the barcode, and store the data and the data on the barcode And to display information related to a part or the data.

The method according to the present invention as described above can be implemented in a program command that can be executed through various computers and recorded in a computer-readable recording medium. The recording medium may include a program command, a data file, a data structure, and the like. Also, the program instructions may be those specially designed and constructed for the present invention or may be available to those skilled in the computer software. In addition, a recording medium includes a magnetic medium such as a hard disk, a floppy disk and a magnetic tape, an optical medium such as a CD-ROM and a DVD, and a magnetic optical medium such as a floppy disk. A hard disk, a magneto-optical medium, a ROM, a RAM, a flash memory, and the like. The program instructions may also include machine language code such as those generated by the compiler, as well as high-level language code that may be executed by the computer using an interpreter or the like. A hardware device may be configured to operate as one or more software modules for carrying out the invention.

The method and apparatus according to the present invention are not limited to the above-described embodiments, and various modifications may be made within the scope of the technical idea of the present invention.

100: object processing system
110: client 120: server
210: input control unit 220: recognition unit
230: Object sort unit 240: Trace unit
300: Terminal 305:
310: call processor 320: application processor
330: RF unit 340: memory

Claims (30)

A method of operating an electronic device,
The method comprising tracking an object on a plurality of digital images,
The tracking operation may include:
Using at least some of the objects recognized from at least one of the images to generate a reference object; And
And tracing the object using the generated reference object. 2. The method of claim 1, wherein the act of creating the reference object comprises:
And modifying an error contained in the recognized object. 3. The method of claim 2, wherein the act of creating the reference object comprises:
And determining if the recognized object for which the error was corrected has been modified correctly. The method of claim 1, wherein the object comprises a bar code,
The act of creating the reference object may include:
Extracting data from the barcode; And
And correcting errors contained in the extracted data. 5. The method of claim 4, wherein the act of creating the reference object comprises:
Further comprising decoding the error corrected data. The method of claim 1, wherein the object comprises a bar code,
And using the generated reference object to track the object,
Obtaining feature points from the generated reference barcode; And
And using the feature points to track the barcode on the plurality of images. A method of operating an electronic device,
Dividing the barcode into objects in the captured image;
Extracting data from the barcode;
Generating an original of the barcode using the data;
Obtaining characteristic points from the original; And
And tracking the barcode using the feature points. 8. The method of claim 7,
Wherein the step of tracking the barcode comprises:
Projecting the original feature points onto a newly captured image;
Transforming the original patches corresponding to each of the projected characteristic points to match the barcode;
Retrieving patches corresponding respectively to the modified patches from the newly captured image; And
And estimating a posture of the electronic device based on the search result. 9. The method of claim 8,
Wherein the step of estimating the posture of the electronic device includes:
And estimating the attitude of the electronic device using values representing the intervals of the modified patches and each of the retrieved patches. 10. The method of claim 9,
Characterized in that the intervals are the intervals of the center points of each of the deformed patches and the center points of each of the retrieved patches. 9. The method of claim 8,
The step of retrieving, from the newly captured image, patches corresponding to the modified patches, respectively,
Moving the deformed patches from the projected feature points, respectively, to a locus selected by the electronic device, and retrieving corresponding patches from the newly captured image. 12. The method of claim 11,
Characterized in that the locus is helical. 8. The method of claim 7,
Further comprising displaying the data, a portion of the data, or information associated with the data on the tracked bar code. 8. The method of claim 7,
The step of acquiring characteristic points in the original,
Enlarging the size of the original;
Dividing the enlarged original into grids comprising a plurality of patches;
And acquiring characteristic points in each of the patches. A method of operating an electronic device,
Receiving a captured image;
Dividing the barcode into objects in the captured image;
Extracting data from the barcode; And
Displaying the data, a portion of the data, or information related to the data on the barcode. In an electronic device,
A memory configured to store a plurality of digital images; And
≪ / RTI >
The processor comprising:
On the plurality of digital images, to track a recognized object,
For the tracking, at least some of the objects recognized from at least one of the images are used to generate a reference object,
And to trace the object using the generated reference object. 17. The method of claim 16,
The processor comprising:
And to correct an error contained in the recognized object. 18. The method of claim 17,
The processor comprising:
And to determine whether the recognized object for which the error was corrected has been modified correctly. 17. The method of claim 16,
The object including a bar code,
The processor comprising:
And to extract data from the barcode and to correct errors contained in the extracted data. 20. The method of claim 19,
The processor comprising:
And to decode the error corrected data. 17. The method of claim 16,
The object including a bar code,
The processor comprising:
And to obtain feature points from the generated reference barcode,
And use the feature points to track the barcode on the plurality of images. A camera for capturing an image;
A display unit for displaying the captured image; And
And a controller for detecting and tracking a barcode in the captured image,
Wherein,
The method of claim 1, further comprising: extracting data from the barcode; generating an original of the barcode using the data; acquiring characteristic points from the original; And tracking the electronic device. 23. The method of claim 22,
Wherein,
Projecting the original feature points onto a newly captured image, modifying the original patches corresponding to each of the projected feature points to match the barcode, and applying patches corresponding to the modified patches to the new Retrieving from the captured image, and estimating the posture of the electronic device based on the search result. 24. The method of claim 23,
Wherein,
And estimates the attitude of the electronic device by using values indicating the intervals of the modified patches and each of the retrieved patches. 25. The method of claim 24,
Wherein the intervals are intervals of respective center points of each of the modified patches and center points of each of the retrieved patches. 24. The method of claim 23,
Wherein,
Moving the deformed patches from the projected feature points, respectively, to a locus selected by the electronic device, and retrieving corresponding patches from the newly captured image. 27. The method of claim 26,
Wherein the locus is spiral. 23. The method of claim 22,
The display unit includes:
And displays the data, a part of the data, or information related to the data on the barcode to be tracked. 23. The method of claim 22,
Wherein,
Enlarging the size of the original, dividing the enlarged original into grids composed of a plurality of patches, and obtaining characteristic points in each of the patches. A camera for capturing an image;
A display unit for displaying the captured image; And
And a controller for detecting and tracking a barcode in the captured image,
Wherein,
Wherein the controller controls to distinguish a bar code from the captured image, extract data from the bar code, and display the data, a part of the data, or information related to the data on the bar code.

Images