KR20230076048A - Method and system for motion estimation of real-time image target between successive frames - Google Patents

Abstract

According to an embodiment of the present invention, a method of estimating motion of a real-time image target between consecutive frames which is a method in which a motion estimation application executed by at least one processor of a terminal estimates motion of a real-time image target between consecutive frames comprises the steps of: detecting a target object in a first frame image; generating a first frame down image by downscaling the first frame image; setting a plurality of tracking points for the target object within the generated first frame down image; obtaining a second frame image after a predetermined time consecutive to the first frame image; generating a second frame down image by downscaling the obtained second frame image; and tracking the target object in the second frame down image based on the set plurality of tracking points. Accordingly, the image target can be accurately and easily detected and/or tracked.

Description

Method and system for estimating motion of real-time image target between consecutive frames

The present invention relates to a method and system for estimating motion of a real-time image target between successive frames. More particularly, it relates to a method and system for estimating motion of an image target in consecutive frames by assuming homography between consecutive down-scaled frames.

As information and communication technology (ICT) develops, technologies for identifying objects included in an image including a plurality of frames are being developed.

In particular, by applying a recognition method possessed by humans to an electronic device, technologies are being developed that allow the electronic device to identify an object within an image or to identify a predetermined object in an image by itself.

Recently, technologies for tracking an object in an image and processing the image for the tracked object in various forms (eg, augmented reality content and / or SLAM (simultaneous localization and mapping)-based data, etc.) have been actively studied. Devices and software that provide real-time content for processed images are being released.

However, general object tracking or processing technology in an image has limitations in accurately determining the location of an object in real time, and there is a problem that considerable device resources are consumed in tracking the location of an object or editing an image.

In addition, in the related art, various noises that may occur during image shifting including a predetermined motion, such as when a target object in a corresponding image moves too fast (eg, motion blur, light smearing, and/or the like) Alternatively, there is a problem in that the performance of object motion estimation is deteriorated due to a change in scale and/or a change in viewpoint of a corresponding object, or a rolling shutter effect.

KR 10-1890612 B1

The present invention has been made to solve the above-described problems, and a method and system for estimating the motion of an image target in consecutive frames by assuming homography between consecutive down-scaled frames Its purpose is to provide

However, the technical problems to be achieved by the present invention and the embodiments of the present invention are not limited to the technical problems described above, and other technical problems may exist.

A motion estimation method of a real-time image target between successive frames according to an embodiment of the present invention is a method in which a motion estimation application executed by at least one processor of a terminal estimates motion of a real-time image target between successive frames, wherein a first frame detecting a target object in an image; Generating a first frame-down image by down-scaling the first frame image; setting a plurality of tracking points for the target object in the generated first frame-down image; obtaining a second frame image after a predetermined time consecutive to the first frame image; generating a second frame-down image by down-scaling the acquired second frame image; and tracking the target object in the second frame-down image based on the set plurality of tracking points.

At this time, the step of tracking the target object in the second frame-down image based on the set plurality of tracking points includes generating a tracking point set based on the set plurality of tracking points, and among the set tracking points. determining a point group having the highest matching score with the second frame-down image as a tracking point main group; and tracking the target object in continuous frame images including the first frame image and the second frame image based on the determined tracking point main group.

In addition, the setting of the plurality of tracking points may include detecting edges of the target object in the first frame-down image, and setting the plurality of tracking points at points located on the detected edges It includes steps to

In addition, setting the plurality of tracking points on the edge includes setting the plurality of tracking points along a predetermined interval based on a predetermined position within the detected edge.

In addition, generating a tracking point set based on the plurality of set tracking points may include converting a tracking point group including the plurality of tracking points based on preset translation parameters; Generating a tracking conversion point group corresponding to each of the preset translation parameters through conversion, and generating a tracking point set including the generated at least one tracking conversion point group and the tracking point group. include

Also, the tracking point main group is a point group having the highest matching score with the second frame-down image among a plurality of point groups in the tracking point set.

Also, the matching score is a parameter value representing a matching rate between one point group in the tracking point set and a target edge, which is an edge in the second frame-down image.

In addition, the determining of the tracking point main group may include detecting a target edge in the second frame-down image, and a plurality of point groups in the tracking point set on a target edge region including the detected target edge Projecting each point, detecting a matching point located on the target edge among a plurality of points included in each of the projected point groups, and based on the detected matching point, each of the points and calculating the matching score for the point group.

The determining of the tracking point main group may further include determining a point group having the highest matching score among a plurality of matching scores calculated for each point group as the tracking point main group.

In addition, the tracking of the target object in the continuous frame image may include performing a dense image alignment operation on the continuous frame image based on a translation parameter corresponding to the tracking point main group; , estimating homography of the continuous frame image based on the performed operation, and tracking the target object based on the estimated homography.

A method and system for estimating the motion of an image target in real time between consecutive frames according to an embodiment of the present invention track the motion of an image target using down-scaled consecutive frame images, so that the corresponding image in the down-scaled image The movement of the image target in the continuous frame image (i.e., image shifting Even if there is noise (e.g., motion blur, light blur, and/or rolling shutter effect) due to (shifting) or scale change and/or viewpoint change for the image target, it is offset and the image target is accurately and easily It has an effect that can be detected and/or tracked appropriately.

In addition, a method and system for estimating motion of a real-time image target between consecutive frames according to an embodiment of the present invention assumes homography between corresponding consecutive frame images based on the down-scaled consecutive frame images, By estimating the motion of the target, it is possible to improve the performance of the estimation algorithm for the motion of the image target, such as reducing the amount of data processing required for the homography operation, thereby improving operation speed and/or efficiency.

In addition, the method and system for estimating the motion of a real-time image target between successive frames according to an embodiment of the present invention support various object detection and/or tracking services based on the estimation algorithm as described above, thereby providing the various object detection and / or tracking service (eg, augmented reality-based SLAM (simultaneous localization and mapping) service, etc.) has an effect that can further enhance the quality and utility.

However, the effects obtainable in the present invention are not limited to the effects mentioned above, and other effects not mentioned can be clearly understood from the description below.

1 is a conceptual diagram of a motion estimation system of a real-time image target between successive frames according to an embodiment of the present invention.
2 is an internal block diagram of a terminal according to an embodiment of the present invention.
3 and 4 are flowcharts illustrating a method for estimating motion of a real-time image target between successive frames according to an embodiment of the present invention.
5 and 6 are examples of drawings for explaining a method of setting a tracking point for a target object in a first frame image according to an embodiment of the present invention.
7 is an example of a diagram for explaining a method of determining a tracking point main group according to an embodiment of the present invention.

Since the present invention can apply various transformations and have various embodiments, specific embodiments will be illustrated in the drawings and described in detail in the detailed description. Effects and features of the present invention, and methods for achieving them will become clear with reference to the embodiments described later in detail together with the drawings. However, the present invention is not limited to the embodiments disclosed below and may be implemented in various forms. In the following embodiments, terms such as first and second are used for the purpose of distinguishing one component from another component without limiting meaning. Also, expressions in the singular number include plural expressions unless the context clearly dictates otherwise. In addition, terms such as include or have mean that features or elements described in the specification exist, and do not preclude the possibility that one or more other features or elements may be added. In addition, in the drawings, the size of components may be exaggerated or reduced for convenience of description. For example, since the size and thickness of each component shown in the drawings are arbitrarily shown for convenience of description, the present invention is not necessarily limited to the illustrated bar.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings, and when describing with reference to the drawings, the same or corresponding components are assigned the same reference numerals, and overlapping descriptions thereof will be omitted. .

1 is a conceptual diagram of a motion estimation system of a real-time image target between successive frames according to an embodiment of the present invention.

Referring to FIG. 1 , a real-time image target motion estimation system 1000 (hereinafter referred to as a real-time image target motion estimation system) between consecutive frames according to an embodiment of the present invention includes homography between down-scaled consecutive frames ( Homography) , a real-time image target motion estimation service (hereinafter, a target motion estimation service) between successive frames for estimating the motion of the image target within the successive frames may be provided.

In an embodiment, the real-time image target motion estimation system 1000 providing the above target motion estimation service may include a terminal 100, a database server 200, and a network 300.

At this time, the terminal 100 and the database server 200 may be connected through the network 300 .

Here, the network 300 according to the embodiment means a connection structure capable of exchanging information between nodes such as the terminal 100 and/or the database server 200. Examples include 3rd Generation Partnership Project (3GPP) networks, Long Term Evolution (LTE) networks, World Interoperability for Microwave Access (WIMAX) networks, the Internet, Local Area Networks (LANs), Wireless Local Area Networks (Wireless Local Area Networks), A wide area network (WAN), a personal area network (PAN), a Bluetooth network, a satellite broadcasting network, an analog broadcasting network, a digital multimedia broadcasting (DMB) network, and the like are included, but are not limited thereto.

Hereinafter, the terminal 100 and the database server 200 implementing the real-time image target motion estimation system 1000 will be described in detail with reference to the accompanying drawings.

- Terminal (100: Terminal)

The terminal 100 according to an embodiment of the present invention may be a predetermined computing device in which a motion estimation application (hereinafter, application) providing a target motion estimation service is installed.

In detail, from a hardware point of view, the terminal 100 may include a mobile type computing device 100 - 1 and/or a desktop type computing device 100 - 2 in which applications are installed.

Here, the mobile type computing device 100 - 1 may be a mobile device such as a smart phone or a tablet PC in which applications are installed.

For example, the mobile type computing device 100 - 1 includes a smart phone, a mobile phone, a digital broadcasting device, a PDA (personal digital assistants), a PMP (portable multimedia player), a tablet PC, and the like. can be included

In addition, the desktop type computing device 100 - 2 provides a target motion estimation service based on wired/wireless communication, such as a fixed desktop PC, a laptop computer, and a personal computer such as an ultrabook with applications installed thereon. It may include a device in which a program for execution is installed.

Also, according to embodiments, the terminal 100 may further include a predetermined server computing device providing a target motion estimation service environment.

2 is an internal block diagram of a terminal 100 according to an embodiment of the present invention.

Meanwhile, referring to FIG. 2 , from a functional point of view, the terminal 100 includes a memory 110, a processor assembly 120, a communication processor 130, an interface unit 140, an input system 150, and a sensor system 160. ) and a display system 170 . These components may be configured to be included in the housing of the terminal 100 .

In detail, an application 111 is stored in the memory 110 , and the application 111 may store any one or more of various application programs, data, and commands for providing a target motion estimation service environment.

That is, the memory 110 may store commands and data that may be used to create a target motion estimation service environment.

Also, the memory 110 may include a program area and a data area.

Here, the program area according to the embodiment may be linked between an operating system (OS) that boots the terminal 100 and functional elements, and the data area may be data generated according to the use of the terminal 100. can be stored.

In addition, the memory 110 may include at least one or more non-transitory computer-readable storage media and temporary computer-readable storage media.

For example, the memory 110 may be various storage devices such as ROM, EPROM, flash drive, hard drive, etc., and web storage that performs the storage function of the memory 110 on the Internet can include

The processor assembly 120 may include at least one processor capable of executing instructions of the application 111 stored in the memory 110 to perform various tasks for generating a target motion estimation service environment.

In an embodiment, the processor assembly 120 may control overall operations of components through the application 111 of the memory 110 to provide a target motion estimation service.

The processor assembly 120 may be a system on chip (SOC) suitable for the terminal 100 including a central processing unit (CPU) and/or a graphics processing unit (GPU), and an operating system stored in the memory 110. (OS) and/or application programs may be executed, and components mounted on the terminal 100 may be controlled.

In addition, the processor assembly 120 may communicate internally with each component through a system bus, and may include one or more predetermined bus structures including a local bus.

In addition, the processor assembly 120 includes application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), and controllers. ), micro-controllers, microprocessors, and electrical units for performing other functions.

The communication processor 130 may include one or more devices for communicating with external devices. Such a communication processor 130 may communicate through a wireless network.

In detail, the communication processor 130 may communicate with the terminal 100 storing a content source for implementing a target motion estimation service environment, and may communicate with various user input components such as a controller that receives a user input.

In an embodiment, the communication processor 130 may transmit and receive various types of data related to the target motion estimation service to and from other terminals 100 and/or external servers.

This communication processor 130, technical standards or communication schemes for mobile communication (eg, LTE (Long Term Evolution), LTE-A (Long Term Evolution-Advanced), 5G NR (New Radio), WIFI) Alternatively, data may be transmitted and received wirelessly with at least one of a base station, an external terminal 100, and an arbitrary server on a mobile communication network constructed through a communication device capable of performing a short-distance communication method.

The sensor system 160 may include various sensors such as an image sensor 161, a position sensor (IMU) 163, an audio sensor 165, a distance sensor, a proximity sensor, and a contact sensor.

Here, the image sensor 161 may capture an image and/or video of a physical space around the terminal 100 .

In an embodiment, the image sensor 161 may capture and obtain an image (eg, a first frame image and/or a second frame image) related to the target motion estimation service.

In addition, the image sensor 161 may be disposed on the front or/or rear side of the terminal 100 to acquire an image by photographing the direction side of the terminal 100, and may acquire a physical image through a camera disposed toward the outside of the terminal 100. space can be photographed.

The image sensor 161 may include an image sensor device and an image processing module. In detail, the image sensor 161 may process still images or moving images obtained by an image sensor device (eg, CMOS or CCD).

In addition, the image sensor 161 may extract necessary information by processing a still image or moving image obtained through an image sensor device using an image processing module, and transmit the extracted information to a processor.

The image sensor 161 may be a camera assembly including one or more cameras. The camera assembly may include a general camera that captures a visible light band, and may further include a special camera such as an infrared camera and a stereo camera.

In addition, the image sensor 161 as described above may be included in the terminal 100 and operated according to embodiments, or may be included in an external device (eg, an external server, etc.) to include the above-described communication processor 130 and/or Alternatively, it may operate through interworking based on the interface unit 140 .

The position sensor (IMU) 163 may detect at least one of motion and acceleration of the terminal 100 . For example, it may be made of a combination of various position sensors such as an accelerometer, a gyroscope, and a magnetometer.

In addition, the location sensor (IMU) 163 may recognize spatial information about a physical space around the terminal 100 by interworking with the location communication processor 130 such as a GPS of the communication processor 130 .

The audio sensor 165 may recognize sounds around the terminal 100 .

In detail, the audio sensor 165 may include a microphone capable of detecting a user's voice input using the terminal 100 .

In an embodiment, the audio sensor 165 may receive voice data necessary for a target motion estimation service from a user.

The interface unit 140 may communicatively connect the terminal 100 with one or more other devices. In detail, the interface unit 140 may include wired and/or wireless communication devices compatible with one or more different communication protocols.

Through this interface unit 140, the terminal 100 may be connected to various input/output devices.

For example, the interface unit 140 may output audio by being connected to an audio output device such as a headset port or a speaker.

Although it has been described that the audio output device is exemplarily connected through the interface unit 140, an embodiment installed inside the terminal 100 may also be included.

Also, for example, the interface unit 140 may obtain a user input by being connected to an input device such as a keyboard and/or a mouse.

Although it has been described that the keyboard and/or mouse are exemplarily connected through the interface unit 140, an embodiment installed inside the terminal 100 may also be included.

The interface unit 140 connects a device equipped with a wired/wireless headset port, an external charger port, a wired/wireless data port, a memory card port, and an identification module. Ports, audio I/O (Input/Output) ports, video I/O (Input/Output) ports, earphone ports, power amplifiers, RF circuits, transceivers and other communication circuits It may be configured to include at least one of.

Input system 150 can sense a user's input (eg, a gesture, voice command, button actuation, or other type of input) related to the target motion estimation service.

In detail, the input system 150 may include a predetermined button, a touch sensor, and/or an image sensor 161 that receives a user motion input.

Also, the input system 150 may be connected to an external controller through the interface unit 140 to receive a user's input.

The display system 170 may output various information related to the target motion estimation service as a graphic image.

As an example, the display system 170 may display an image including a predetermined target object, a first frame image, a second frame image, and/or various user interfaces.

Such displays include a liquid crystal display (LCD), a thin film transistor-liquid crystal display (TFT LCD), an organic light-emitting diode (OLED), and a flexible display. , a 3D display, and an e-ink display.

The components may be disposed in the housing of the terminal 100, and the user interface may include a touch sensor 173 on the display 171 configured to receive a user touch input.

In detail, the display system 170 may include a display 171 that outputs an image and a touch sensor 173 that detects a user's touch input.

For example, the display 171 may be implemented as a touch screen by forming a mutual layer structure or integrally with the touch sensor 173 . Such a touch screen may function as a user input unit providing an input interface between the terminal 100 and the user, and may provide an output interface between the terminal 100 and the user.

Meanwhile, the terminal 100 according to an embodiment of the present invention may perform various functional operations required for a target motion estimation service using at least one disclosed algorithm.

As an embodiment, the terminal 100 performs object detection, image segmentation, image down scaling, feature point detection, and/or homography estimation. Various functional operations necessary for the target motion estimation service can be performed based on various algorithms that perform the following.

Also, according to embodiments, the terminal 100 may further perform at least a part of functional operations performed by the database server 200 to be described later.

- Database server (200: Database server)

Meanwhile, the database server 200 according to an embodiment of the present invention may perform a series of processes for providing a target motion estimation service.

In detail, in the embodiment, the database server 200 exchanges data necessary for driving a motion estimation process of a real-time image target between successive frames in an external device such as the terminal 100 with the external device, thereby exchanging the target A motion estimation service may be provided.

More specifically, in the embodiment, the database server 200, the application 111 in the external device (in the embodiment, the mobile type computing device 100-1 and / or desktop type computing device 100-2, etc.) An operating environment can be provided.

To this end, the database server 200 may include an application program, data, and/or commands for the application 111 to operate, and may transmit/receive data based thereon with the external device.

Also, in an embodiment, the database server 200 may detect a target object within a predetermined first frame image.

In detail, the database server 200 may obtain the first frame image from a predetermined basic image implemented based on a plurality of consecutive frames.

Also, the database server 200 may detect a target object in the first frame image by performing a predetermined image processing process based on the first frame image.

Also, in an embodiment, the database server 200 may downscale the first frame image in which the target object is detected.

Also, in an embodiment, the database server 200 may set a tracking point for a target object in the down-scaled first frame image.

Here, the tracking point according to the embodiment may be a keypoint indicating a characteristic point of a target object in order to detect and/or track a predetermined target object.

In addition, in an embodiment, the database server 200 may obtain, as a second frame image, a predetermined other frame image consecutive to the first frame image from the base image.

Also, in an embodiment, the database server 200 may down-scale the obtained second frame image.

In addition, in an embodiment, the database server 200 may determine a tracking point main group based on the down-scaled second frame image and the set tracking point.

Here, the tracking point main group according to the embodiment may mean a group of tracking points having the highest matching score with the down-scaled second frame image among the set tracking points.

Also, in an embodiment, the database server 200 may perform target object tracking based on the determined main group of tracking points.

That is, the database server 200 may implement a target object tracking service capable of detecting and/or tracking a predetermined target object based on the main group of tracking points.

Also, in an embodiment, the database server 200 may perform a predetermined function operation required for a target motion estimation service using at least one disclosed algorithm.

In an embodiment, the database server 200 performs object detection, image segmentation, image down scaling, feature point detection, and/or homography estimation. ), etc., various functional operations necessary for the target motion estimation service can be performed.

In more detail, in the embodiment, the database server 200 reads a predetermined algorithm driving program built to perform the above functional operation from the memory module 230, and corresponds to the predetermined algorithm system according to the read predetermined algorithm system. function can be performed.

At this time, depending on the embodiment, the above predetermined algorithm is directly included in the database server 200 or implemented in a device and/or server separate from the database server 200 to perform functional operations for the target motion estimation service. can be done

In the following description, it is described that the predetermined algorithm is included in the database server 200 and implemented, but is not limited thereto.

Also, in an embodiment, the database server 200 may store and manage various application programs, instructions, and/or data for implementing a target motion estimation service.

As an embodiment, the database server 200 may store and manage at least one basic image, a first frame image, a second frame image, a tracking point, and/or various algorithms required for a target motion estimation service.

Meanwhile, further referring to FIG. 1 , in the embodiment, the database server 200 as described above includes at least one processor module 210 for data processing and at least one or more processor modules for exchanging data with an external device. Implemented as a predetermined computing device including a communication module (220) and at least one memory module (230) storing various application programs, data and/or instructions for providing a target motion estimation service. It can be.

Here, the memory module 230 may store any one or more of an operating system (OS), various application programs, data, and instructions for providing a target motion estimation service.

Also, the memory module 230 may include a program area and a data area.

Here, the program area according to the embodiment may be linked between an Operating System (OS) and functional elements for booting the server, and the data area may store data generated according to the use of the server.

In an embodiment, the memory module 230 may be a variety of storage devices such as ROM, RAM, EPROM, flash drive, hard drive, etc., and a web device that performs the storage function of the memory module 230 on the Internet. It may also be web storage.

Also, the memory module 230 may be a recording medium detachable from the server.

Meanwhile, the processor module 210 may control the overall operation of each unit described above in order to implement the target motion estimation service.

The processor module 210 may be a system-on-a-chip (SOC) suitable for a server including a central processing unit (CPU) and/or a graphic processing unit (GPU), and an operating system (OS) stored in the memory module 230. ) and/or application programs, etc., and can control each component mounted on the server.

In addition, the processor module 210 may communicate internally with each component through a system bus, and may include one or more predetermined bus structures including a local bus.

In addition, the processor module 210 includes application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), and controllers. ), micro-controllers, microprocessors, and electrical units for performing other functions.

In the above description, it has been described that the database server 200 according to an embodiment of the present invention performs the functional operation as described above, but according to the embodiment, at least a part of the functional operation performed by the database server 200 is performed by an external device. (eg, the terminal 100, etc.), or at least a part of the functional operation performed in the external device may be further performed in the database server 200. Various embodiments may be possible.

- Motion estimation method of real-time image target between consecutive frames

Hereinafter, a method of estimating motion of a real-time image target between successive frames by an application 111 executed by at least one processor of the terminal 100 according to an embodiment of the present invention will be described with reference to FIGS. 3 to 7. Explain in detail.

In an embodiment of the present invention, at least one processor of the terminal 100 may execute at least one application 111 stored in at least one memory 110 or operate in a background state.

Hereinafter, the execution of the above-described method of providing the target motion estimation service by operating the at least one or more processors to execute commands of the application 111 will be briefly described as being performed by the application 111 .

3 and 4 are flowcharts for explaining a motion estimation method of a real-time image target between successive frames according to an embodiment of the present invention.

Referring to FIGS. 3 and 4 , in an embodiment, an application 111 executed by at least one processor of the terminal 100 or operating in a background state may detect a target object in a first frame image. (S101)

In detail, in an embodiment, the application 111 may acquire a predetermined first frame image from a predetermined basic image implemented based on a plurality of consecutive frames.

In addition, in an embodiment, the application 111 may detect a target object, which is a target object to be detected in the base image, from the first frame image.

In an embodiment, the application 111 may perform a predetermined image processing process (eg, object detection, image segmentation, and/or feature point detection) based on the first frame image. ) to detect the target object in the first frame image. However, it is not limited thereto.

In addition, in an embodiment, the application 111 may down-scale the first frame image in which the target object is detected . (S103)

That is, in an embodiment, the application 111 may perform downscaling to adjust the resolution and aspect ratio of the first frame image by reducing the size of the first frame image in which the target object is detected.

Illustratively, the application 111 may perform downscaling to adjust the resolution of the first frame image from '4K (4096x2160)' to 'VGA (640x480)'.

Therefore, the application 111, in the down-scaled image, is insensitive to the positional movement of the desired characteristic or pattern (in the embodiment, the target object) in the image (ie, positional independence (translation 　invariance) is improved) and , Noise (e.g., motion blur, light) according to the movement of the target object in a plurality of frame images later, using the characteristic that the presence or absence of a desired characteristic or pattern (in the embodiment, the target object) can be easily detected. Even if blurring and/or rolling shutter effects, etc.) exist, it is possible to easily detect and/or track a target object in a corresponding plurality of frame images by offsetting such noise.

Also, in an embodiment, the application 111 may set a tracking point for a target object in the down-scaled first frame image. (S105)

Here, the tracking point according to the embodiment may be a keypoint indicating a characteristic point of a target object in order to detect and/or track a predetermined target object.

5 and 6 are examples of drawings for explaining a method of setting a tracking point for a target object in a first frame image according to an embodiment of the present invention.

In detail, referring to FIG. 5, in the embodiment, the application 111 determines the shape of the target object in the down-scaled first frame image (DI-1: hereinafter, the first frame-down image), that is, the edge (edges) can be detected.

In an embodiment, the application 111 performs a predetermined image processing process (eg, edge detection) based on the first frame-down image DI-1 to obtain the first frame-down image DI-1. -1) I can detect the edge of my target object. However, it is not limited thereto.

Also, in an embodiment, the application 111 may set a plurality of tracking points (TP) on the detected edge.

In detail, the application 111 may set the plurality of tracking points TP to be located on the detected edge at predetermined intervals.

In this case, the application 111 may set a plurality of tracking points TP to be positioned along the preset interval based on preset positions (eg, corners and/or corners) on the edge.

Here, the plurality of tracking points (TP) set as above may have mutual positional relationships based on coordinate information for each tracking point (TP), and a translation parameter matched to the plurality of tracking points (TP). may have already been set.

Meanwhile, referring to FIG. 6, according to an embodiment, the application 111 converts the plurality of tracking points (TP) (ie, the tracking point group (TPG)) based on preset translation parameters. By doing so, a plurality of tracking conversion points (ie, tracking conversion point groups (TTG)) can be generated.

In detail, in an embodiment, the application 111 may generate at least one tracking conversion point group (TTG) by transforming the tracking point group (TPG) based on each of at least one preset translation parameter.

As an embodiment, the application 111 may generate a first tracking conversion point group (TTG) by converting the tracking point group (TPG) based on a first translation parameter, and second to second to second tracking point groups (TTG) may be generated in the same manner. Second to N th tracking conversion point groups (TTG) may be generated by transforming the tracking point group (TPG) using N translation parameters.

Also, in an embodiment, the application 111 may generate a tracking point set (TS) including the generated at least one tracking conversion point group (TTG) and the tracking point group (TPG).

Through this, the application 111, when detecting and / or tracking the target object in the frames through comparison between the first frame image and predetermined other frame images consecutive to the first frame image later, more rich It is possible to further improve accuracy and reliability by performing detection and/or tracking of the corresponding target object through comparison using the base data.

Also, in an embodiment, the application 111 may obtain a second frame image. (S107)

In detail, in the embodiment, the application 111 converts a predetermined other frame image (eg, a frame image after a predetermined frame from the first frame image) consecutive to the first frame image of the above-described basic image to the first frame image. It can be obtained as a 2-frame image.

Also, in an embodiment, the application 111 may down-scale the obtained second frame image. (S109)

That is, in an embodiment, the application 111 may perform downscaling to adjust the resolution and aspect ratio of the second frame image by reducing the size of the obtained second frame image.

Illustratively, the application 111 may perform downscaling to adjust the resolution of the second frame image from '4K (4096x2160)' to 'VGA (640x480)'.

Also, in an embodiment, the application 111 may determine a tracking point (TP) main group based on the down-scaled second frame image and the set tracking point (TP). (S111)

Here, the tracking point (TP) main group according to the embodiment refers to a plurality of point groups included in the aforementioned tracking point set (TS) (in the embodiment, a tracking point group (TPG) and/or at least one tracking transformation). It may mean a point group having the highest matching score with the down-scaled second frame image among the point groups (TTG).

In this case, the matching score according to the embodiment may be a parameter value representing a matching rate between a point group in the tracking point set (TS) and an edge existing in the down-scaled second frame image.

7 is an example of a diagram for explaining a method of determining a tracking point (TP) main group according to an embodiment of the present invention.

In detail, referring to FIG. 7, in the embodiment, the application 111 detects boundaries, that is, edges, existing in the down-scaled second frame image (DI-2: hereinafter referred to as a second frame-down image) can do.

In an embodiment, the application 111 performs a predetermined image processing process (eg, edge detection) based on the second frame-down image DI-2 to obtain the second frame-down image DI-2. -2) It is possible to detect my edge, but it is not limited thereto.

Also, in an embodiment, the application 111 may calculate a matching score between an edge in the detected second frame-down image DI-2 and each point group in the tracking point set TS.

In detail, in an embodiment, the application 111 sets a plurality of points (hereinafter, a plurality of reference points) included in a first point group in the tracking point set (TS) in the second frame-down image (DI-2). Projection may be performed on an edge (hereinafter, a target edge) area EA.

Here, the target edge area EA according to the embodiment may be a predetermined bounding box area including the target edge.

In addition, at this time, the plurality of reference points are in a state in which mutual positional relationships based on coordinate information for each reference point are all set, and the set mutual positional relations can be maintained and projected onto the target edge.

Also, the application 111 may detect a reference point (hereinafter, a matching point) located on the target edge from among a plurality of reference points projected on the target edge area EA.

Also, the application 111 may calculate a matching score for the first point group based on the number of the detected matching points.

Subsequently, in the embodiment, the application 111 may calculate matching scores for each of the second to Nth point groups in the tracking point set (TS) in the same manner as above.

Also, in an embodiment, the application 111 may determine a point group having the highest matching score among the calculated matching scores for each point group as the tracking point main group (TMG).

In this way, the application 111 detects a point group having the highest matching rate with an edge in the second frame-down image DI-2 from among a plurality of point groups according to various translation parameters, so that the high matching rate can be obtained later. It is possible to detect and/or track a target object in the above-described frame images based on a plurality of points included in a point group having a rate (ie, a point group having good target object detection and/or tracking performance), through which Accuracy and reliability of the target object detection and/or tracking result may be improved.

Also, in an embodiment, the application 111 may perform target object tracking based on the determined tracking point main group (TMG). (S113)

That is, in an embodiment, the application 111 may implement a target object tracking service capable of detecting and/or tracking a predetermined target object based on the tracking point main group (TMG).

In detail, in the embodiment, the application 111, based on the translation parameter corresponding to the determined tracking point main group (TMG), the above-described first frame-down image (DI-1) and the second frame-down image (DI-1) Target object tracking according to 2) can be performed.

In more detail, in the embodiment, the application 111 uses a translation parameter (hereinafter referred to as a main translation parameter) corresponding to the determined tracking point main group (TMG) to transmit the first frame-down image DI-1 and A dense image alignment operation may be performed on the second frame-down image DI-2 (hereinafter referred to as a continuous frame image).

Also, in an embodiment, the application 111 may estimate homography of the continuous frame image through the performed dense image alignment operation.

For reference, the homography may mean a constant conversion relationship established between projected corresponding points when one plane is projected onto another plane.

In addition, in the embodiment, the application 111 performs target object tracking for the second frame-down image D-2 based on the first frame-down image DI-1 based on the estimated homography. can be done

That is, in an embodiment, the application 111 performs a dense image alignment operation on the continuous frame images, which are down-scaled frame images, and through this, assumes a homography of the continuous frame image, and performs the assumed homography. By using it for target tracking based on the continuous frame image, various noises (e.g., motion blur, light blur, and/or rolling shutter effect, etc.) that may be generated by image shifting of the continuous frame image or It is possible to minimize the degradation in accuracy of target object motion estimation due to a change in scale and/or a change in viewpoint, and at the same time, significantly improve the performance of an estimation algorithm for the motion of a target object in a continuous frame image.

As described above, the method and system for estimating motion of a real-time image target between consecutive frames according to an embodiment of the present invention track the motion of an image target using down-scaled consecutive frame images, so that in the down-scaled image, The movement of the image target in the continuous frame image (i.e., the image target) by utilizing the characteristic of being insensitive to the positional movement of the desired characteristic or pattern in the image and the characteristic of being able to easily detect the presence or absence of the desired characteristic or pattern Even if noise (e.g., motion blur, light blur, and/or rolling shutter effect, etc.) due to shifting) or scale change and/or viewpoint change for the image target exist, they are offset and the image target can be accurately and easily captured. There is an effect that can be detected and/or tracked.

In addition, a method and system for estimating motion of a real-time image target between consecutive frames according to an embodiment of the present invention assumes homography between corresponding consecutive frame images based on the down-scaled consecutive frame images, By estimating the motion of the target, it is possible to improve the performance of the estimation algorithm for the motion of the image target, such as reducing the amount of data processing required for the homography operation, thereby improving operation speed and/or efficiency.

In addition, the method and system for estimating the motion of a real-time image target between successive frames according to an embodiment of the present invention support various object detection and/or tracking services based on the estimation algorithm as described above, thereby providing the various object detection and / or tracking service (eg, augmented reality-based SLAM (simultaneous localization and mapping) service, etc.) has an effect that can further enhance the quality and utility.

On the other hand, the embodiments according to the present invention described above may be implemented in the form of program instructions that can be executed through various computer components and recorded on a computer-readable recording medium. The computer readable recording medium may include program instructions, data files, data structures, etc. alone or in combination. Program instructions recorded on the computer-readable recording medium may be specially designed and configured for the present invention, or may be known and usable to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical recording media such as CD-ROMs and DVDs, and magneto-optical media such as floptical disks. medium), and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include high-level language codes that can be executed by a computer using an interpreter or the like as well as machine language codes generated by a compiler. A hardware device may be modified with one or more software modules to perform processing according to the present invention and vice versa.

Specific implementations described in the present invention are examples and do not limit the scope of the present invention in any way. For brevity of the specification, description of conventional electronic components, control systems, software, and other functional aspects of the systems may be omitted. In addition, the connection of lines or connecting members between the components shown in the drawings are examples of functional connections and / or physical or circuit connections, which can be replaced in actual devices or additional various functional connections, physical connection, or circuit connections. In addition, if there is no specific reference such as “essential” or “important”, it may not be a component necessarily required for the application of the present invention.

In addition, the detailed description of the present invention described has been described with reference to preferred embodiments of the present invention, but those skilled in the art or those having ordinary knowledge in the art will find the spirit of the present invention described in the claims to be described later. And it will be understood that the present invention can be variously modified and changed without departing from the technical scope. Therefore, the technical scope of the present invention is not limited to the contents described in the detailed description of the specification, but should be defined by the claims.

Claims (10)

A method in which a motion estimation application executed by at least one processor of a terminal estimates motion of a real-time image target between successive frames, the method comprising:
detecting a target object in a first frame image;
Generating a first frame-down image by down-scaling the first frame image;
setting a plurality of tracking points for the target object in the generated first frame-down image;
obtaining a second frame image after a predetermined time consecutive to the first frame image;
generating a second frame-down image by down-scaling the acquired second frame image; and
Tracking the target object in the second frame-down image based on the set plurality of tracking points
A method for estimating the motion of a real-time image target between successive frames. According to claim 1,
Tracking the target object in the second frame-down image based on the set plurality of tracking points,
generating a tracking point set based on the set plurality of tracking points;
determining a point group having the highest matching score with the second frame-down image among the tracking point set as a tracking point main group; and
Tracking the target object in consecutive frame images including the first frame image and the second frame image based on the determined tracking point main group
A method for estimating the motion of a real-time image target between successive frames. According to claim 2,
The step of setting the plurality of tracking points,
Detecting edges of the target object in the first frame-down image;
Setting the plurality of tracking points at points located on the detected edge.
A method for estimating the motion of a real-time image target between successive frames. According to claim 3,
The step of setting a plurality of tracking points on the edge,
Setting the plurality of tracking points along a preset interval based on a preset position within the detected edge
A method for estimating the motion of a real-time image target between successive frames. According to claim 2,
Generating a tracking point set based on the set plurality of tracking points,
converting a tracking point group including the plurality of tracking points based on preset translation parameters;
generating a tracking conversion point group corresponding to each of the preset translation parameters through the conversion;
Further comprising generating a tracking point set including the generated at least one tracking conversion point group and the tracking point group
A method for estimating the motion of a real-time image target between successive frames. According to claim 5,
The tracking point main group,
A point group having the highest matching score with the second frame-down image among a plurality of point groups in the tracking point set
A method for estimating the motion of a real-time image target between successive frames. According to claim 6,
The matching score is
A parameter value representing a matching rate between one point group in the tracking point set and a target edge, which is an edge in the second frame down image
A method for estimating the motion of a real-time image target between successive frames. According to claim 7,
The step of determining the tracking point as the main group,
detecting a target edge in the second frame down image;
Projecting each of a plurality of point groups in the tracking point set onto a target edge region including the detected target edge;
detecting a matching point located on the target edge among a plurality of points included in each of the projected point groups;
Calculating the matching score for each point group based on the detected matching points.
A method for estimating the motion of a real-time image target between successive frames. According to claim 8,
The step of determining the tracking point as the main group,
Further comprising determining a point group having the highest matching score among a plurality of matching scores calculated for each point group as the tracking point main group.
A method for estimating the motion of a real-time image target between successive frames. According to claim 2,
The step of tracking the target object in the continuous frame image,
performing a dense image alignment operation on the continuous frame image based on a translation parameter corresponding to the tracking point main group;
estimating homography of the continuous frame image based on the performed operation;
Tracking the target object based on the estimated homography
A method for estimating the motion of a real-time image target between successive frames.

Images