WO2024005279A1

WO2024005279A1 - Software-based object tracking provision method, and computing device therefor

Info

Publication number: WO2024005279A1
Application number: PCT/KR2022/018565
Authority: WO
Inventors: 김켄; 정지욱
Original assignee: 주식회사 쓰리아이
Priority date: 2022-06-29
Filing date: 2022-11-23
Publication date: 2024-01-04
Also published as: KR102617213B1

Abstract

A software-based tracking provision method, according to one technical aspect of the present invention is an object tracking provision method performed by a computing device comprising a camera module, the method comprising the steps of: receiving a search frame image captured at a first resolution from the camera module; setting a second resolution for a viewing window; identifying whether an object being tracked exists in the search frame image; and on the basis of the position of the object being tracked within the search frame image, setting an area of the search frame image including the object being tracked as the viewing window. The second resolution of the viewing window may be lower than the first resolution of the frame image.

Description

Method for providing software-based object tracking and computing device therefor

The present invention relates to a method for providing software-based object tracking and a computing device therefor.

With the development of computing devices, the miniaturization and portability of computing devices are increasing, and a more user-friendly computing environment is being developed.

In this computing environment, the main interest of users is the tracking function for objects of interest in the image being shot.

In the conventional case, for object tracking, it is necessary to use multiple photographic equipment or physically drive the photographic equipment.

However, this has the limitation of being difficult to apply in a miniaturized and portable computing device environment and requiring separate equipment.

One technical aspect of the present application is to solve the problems of the prior art described above. According to an embodiment disclosed in the present application, tracking of an object is effectively performed based on software for an image captured while fixed in a certain direction. The purpose is to provide

According to an embodiment disclosed in the present application, a tracking object is identified within a frame image using a first deep learning model learned with a large amount of training data for identifying the tracking object, and a large amount of information associated with the external characteristics of the tracking object is used. The purpose is to quickly and accurately perform object identification and identity judgment by determining the identity of the tracked object using the second deep learning model learned with the learning data.

According to an embodiment disclosed in the present application, the viewing window is reset based on the positional criticality of the viewing window in consecutive frame images, thereby preventing errors in the viewing window setting due to errors or misrecognition of other objects. The purpose is to provide higher tracking performance.

The tasks of this application are not limited to the tasks mentioned above, and other tasks not mentioned will be clearly understood by those skilled in the art from the description below.

One technical aspect of the present application proposes a method for providing software-based tracking. The software-based tracking providing method is a method of providing object tracking performed in a computing device including a camera module, comprising: receiving a search frame image captured at a first resolution from the camera module; a second resolution for a viewing window; , identifying whether a tracking object exists in the search frame image, and based on the location of the tracking object in the search frame image, selecting a partial region of the search frame image containing the tracking object. It includes the step of setting it as a viewing window. The second resolution of the viewing window may be a lower resolution than the first resolution of the frame image.

One technical aspect of the present application proposes another example of a method for providing software-based tracking. Another example of the software-based tracking providing method is a method of providing object tracking performed in a computing device including a camera module that is fixed in a preset forward direction and generates a capture frame image, wherein continuous tracking is performed at a first resolution from the camera module. receiving a plurality of captured frame images, selecting at least a portion of the plurality of captured frame images to select at least one search frame image, and identifying whether a tracking object exists in the at least one search frame image. and setting a partial region of the at least one search frame image including the tracking object as the viewing window, based on the location of the tracking object in the at least one search frame image. You can.

Another technical aspect of the present application proposes a storage medium. The storage medium is a storage medium that stores computer-readable instructions. The instructions, when executed by a computing device, cause the computing device to: receive a search frame image captured at a first resolution, set a second resolution for a viewing window, and track an object within the search frame image. Based on the operation of identifying whether exists and the location of the tracking object in the search frame image, an operation of setting a partial area of the search frame image including the tracking object as the viewing window is performed. The second resolution of the viewing window may be a lower resolution than the first resolution of the frame image.

Another technical aspect of the present application proposes a computing device. The computing device includes a camera module, a memory that stores one or more instructions, and at least one processor that executes the one or more instructions stored in the memory. The at least one processor, by executing the one or more instructions, receives a search frame image captured at a first resolution from the camera module, sets a second resolution for a viewing window, and tracks an object in the search frame image. , and, based on the location of the tracking object within the search frame image, set a partial area of the search frame image including the tracking object as the viewing window. The second resolution of the viewing window may be a lower resolution than the first resolution of the frame image.

The means for solving the above problems do not enumerate all the features of the present application. Various means for solving the problems of this application can be understood in more detail by referring to specific embodiments in the detailed description below.

According to an embodiment disclosed in the present application, there is an effect of effectively providing tracking of an object based on software for an image captured while fixed in a certain direction.

According to an embodiment disclosed in the present application, a tracking object is identified within a frame image using a first deep learning model learned with a large amount of training data for identifying the tracking object, and a large amount of information associated with the external characteristics of the tracking object is used. By determining the identity of the tracked object using the second deep learning model learned with the learning data, there is an effect of quickly and accurately performing object identification and identity judgment.

According to an embodiment disclosed in the present application, the viewing window is reset based on the positional criticality of the viewing window in consecutive frame images, thereby preventing errors in the viewing window setting due to errors or misrecognition of other objects. It has the effect of providing higher tracking performance.

FIG. 1 is a diagram illustrating an example of a computing device that performs software-based object tracking according to an embodiment of the present application.

FIG. 2 is a diagram illustrating an exemplary computing operating environment of a computing device according to an embodiment of the present application.

Figure 3 is a flowchart explaining a method of providing software-based object tracking according to an embodiment of the present application.

Figures 4 to 6 are diagrams for explaining the software-based object tracking method shown in Figure 3.

Figure 7 is a block diagram illustrating a control function block of a computing device according to an embodiment of the present application.

FIG. 8 is a flowchart explaining an embodiment of a method for providing object tracking performed in the search frame selection module shown in FIG. 7, and FIG. 9 is a diagram explaining an embodiment shown in FIG. 8.

FIG. 10 is a flowchart explaining another embodiment of a method for providing object tracking performed in the search frame selection module shown in FIG. 7, and FIG. 11 is a diagram explaining another embodiment shown in FIG. 10.

FIG. 12 is a flowchart explaining an embodiment of a method for providing object tracking performed in the object detection module shown in FIG. 7.

FIG. 13 is a flowchart explaining another embodiment of a method for providing object tracking performed in the object detection module shown in FIG. 7, and FIGS. 14 and 15 are diagrams for explaining another embodiment shown in FIG. 13. .

FIG. 16 is a flowchart explaining an embodiment of a method for providing object tracking performed in the window setting module shown in FIG. 7.

FIG. 17 is a flowchart explaining another embodiment of a method for providing object tracking performed in the window setting module shown in FIG. 7, and FIGS. 18 to 20 are diagrams for explaining another embodiment shown in FIG. 17. .

Hereinafter, preferred embodiments of the present invention will be described with reference to the attached drawings.

However, the embodiments of the present invention may be modified into various other forms, and the scope of the present invention is not limited to the embodiments described below. Additionally, the embodiments of the present invention are provided to more completely explain the present invention to those with average knowledge in the relevant technical field.

That is, the above-described objects, features, and advantages will be described in detail later with reference to the attached drawings, so that those skilled in the art will be able to easily implement the technical idea of the present invention. In describing the present invention, if it is determined that a detailed description of known technologies related to the present invention may unnecessarily obscure the gist of the present invention, the detailed description will be omitted. Hereinafter, preferred embodiments according to the present invention will be described in detail with reference to the attached drawings. In the drawings, identical reference numerals are used to indicate identical or similar components.

Additionally, as used herein, singular expressions include plural expressions, unless the context clearly dictates otherwise. In the present application, terms such as “consists of” or “comprises” should not be construed as necessarily including all of the various components or steps described in the specification, and some of the components or steps may include It may not be included, or it should be interpreted as including additional components or steps.

In addition, in order to explain the system according to the present invention, various components and their sub-components are described below. These components and their sub-components may be implemented in various forms such as hardware, software, or a combination thereof. For example, each element may be implemented as an electronic configuration to perform the corresponding function, or may be implemented as software itself that can be run in an electronic system, or as a functional element of such software. Alternatively, it may be implemented with an electronic configuration and corresponding driving software.

The various techniques described herein may be implemented with hardware or software, or a combination of both as appropriate. As used herein, terms such as “Unit,” “Server,” and “System” likewise refer to a computer-related entity, i.e., hardware, a combination of hardware and software, software or It can be treated as equivalent to software at the time of execution. Additionally, each function executed in the system of the present invention may be configured in module units and may be recorded in one physical memory, or may be distributed and recorded between two or more memories and recording media.

Various embodiments of the present application are software (e.g., machine) including one or more instructions stored in a storage medium that can be read by a machine (e.g., user terminal 100 or computing device 300). For example, it may be implemented as a program). For example, the processor 301 may call at least one instruction among one or more instructions stored from a storage medium and execute it. This allows the device to be operated to perform at least one function according to the at least one instruction called. The one or more instructions may include code generated by a compiler or code that can be executed by an interpreter. A storage medium that can be read by a device may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' only means that the storage medium is a tangible device and does not contain signals (e.g., electromagnetic waves), and this term refers to data being semi-permanently stored in the storage medium. There is no distinction between temporary storage and temporary storage.

Various flowcharts are disclosed to explain embodiments of the present invention, but these are for convenience of explanation of each step, and each step is not necessarily performed in accordance with the order of the flowchart. That is, each step in the flowchart may be performed simultaneously, in an order according to the flowchart, or in an order opposite to the order in the flowchart.

Referring to FIG. 1, the computing device 100 is fixed in the front direction and performs imaging. The computing device 100 identifies an object among the captured images for the front, and creates a display window area 11 displayed on the user terminal centered on the object among the entire captured images 10 - hereinafter referred to as the viewing window 11. Referred to as - is extracted and displayed on the user terminal (101).

The computing device 100 provides a software-based object tracking function to the user by changing the viewing window 11 in response to the object moving in each frame of the front captured image (hereinafter referred to as a captured image frame). You can. That is, in the present application, the computing device 100 sets the resolution of the viewing window 11 to be smaller than the resolution of the preset captured image frame, and as the object moves within the captured image frame captured in a fixed forward direction, the viewing window 11 By setting to change, a software-based object tracking function can be provided to the user without physically rotating or changing the camera unit of the computing device 100.

The computing device 100 includes a camera and may be an electronic device that the user can carry. For example, the computing device 100 may be a smart phone, a mobile phone, a laptop computer, a digital broadcasting terminal, a personal digital assistant (PDA), a portable multimedia player (PMP), a navigation device, or a slate PC. PC), tablet PC, ultrabook, wearable device (e.g., smartwatch, smart glass, HMD (head mounted display)), etc. You can.

Referring to FIG. 2, the computing device 100 includes a communication unit 110, a camera unit 120, an output unit 130, a memory 140, a power supply unit 150, and a processor 160. The components shown in FIG. 2 are not essential for implementing a computing device, so the computing device described herein may have more or fewer components than those listed above.

The communication unit 110 may include one or more modules that enable communication, such as between the computing device 100 and a wireless communication system or between the computing device 100 and another computing device. This communication unit 110 may include a mobile communication module 211, a wireless Internet module 212, and a short-distance communication module 213. The short-distance communication module 213 can perform a communication connection with the terminal holder 100 by wire or wirelessly. For example, the short-range communication module 213 may include a short-range wireless communication module such as Bluetooth or a wired communication module such as RS232.

The camera unit 120 or camera module may include at least one camera. Camera unit 120 may include one or more lenses, image sensors, image signal processors, or flashes.

As an example, the camera unit 120 may include a first camera 221 to a second camera 222. The first camera 221 or the second camera 222 may capture a front image of the computing device 100.

The output unit 130 is intended to generate output related to vision, hearing, or tactile senses, and may include a display 131 and a speaker 132. The display 131 can implement a touch screen by forming a mutual layer structure or being integrated with the touch sensor. Such a touch screen may function as a user input unit that provides an input interface between the computing device 100 and the user, and may also provide an output interface between the computing device 100 and the user.

The power supply unit 150 receives external or internal power under the control of the processor 160 and supplies power to each component included in the computing device 100. This power supply unit 150 includes a battery, and the battery may be a built-in battery or a replaceable battery.

The processor 160 may control at least some of the components examined with FIG. 2 to run an application program stored in the memory 140, that is, an application. Furthermore, the processor 160 may operate in combination with at least two or more of the components included in the computing device 100 to run an application program.

The processor 160 may run an application by executing instructions stored in the memory 140. Hereinafter, the processor 160 is expressed as the subject of control, instruction, or function by driving an application. However, this means that the processor 160 operates by driving instructions or applications stored in the memory 140.

At least some of the above-described components may operate in cooperation with each other to implement the operation, control, or control method of the computing device 100 according to various embodiments described below. Additionally, the operation, control, or control method of the computing device 100 may be implemented on the computing device by running at least one application program stored in the memory 140.

In addition to operations related to application programs, the processor 160 typically controls the overall operation of the computing device 100. The processor 260 can provide or process appropriate information or functions to the user by processing signals, data, information, etc. input or output through the components discussed above, or by running an application program stored in the memory 240. The processor 160 can be implemented as one processor or multiple processors.

The components of FIG. 7 described below may be functions or software modules implemented in the processor 260 according to instructions stored in the memory 240.

Meanwhile, the control method performed in the computing device 100 according to the above-described embodiment may be implemented as a program and provided to the computing device 100. For example, a program including a control method for the computing device 100 may be stored and provided in a non-transitory computer readable medium.

FIG. 3 is a flowchart illustrating a method of providing software-based object tracking according to an embodiment of the present application, and FIGS. 4 to 6 are diagrams illustrating the method of providing software-based object tracking shown in FIG. 3 . The method for providing software-based object tracking shown in FIG. 3 is explained in terms of each step performed by driving the processor 160 of the computing device 100 shown in FIG. 2.

Referring to FIG. 3, the processor 160 controls the camera module to generate and receive a frame image for the front direction (S310). The camera module is fixed in a preset forward direction regardless of the presence and movement of the tracking object and captures the image at a first resolution to generate a frame image. Figure 4 shows this example, showing an object 402 being imaged so that it exists within a frame image 401.

The processor 160 may set the second resolution of the viewing window to have a lower resolution than the first resolution captured by the camera module (S320). As an example, the resolution of the viewing window may be determined based on user input. As another example, the processor 160 may dynamically change the resolution of the viewing window while providing the object tracking function according to the size of the tracked object in the frame image.

The processor 160 may identify whether a tracking object exists in the frame image (S330) and, based on the position of the tracking object in the frame image, set a partial area of the frame image including the tracking object as a viewing window. There is (S340). Figure 5 shows this example, and after identifying the tracking object 502 in the frame image 501, a viewing window 503 can be set up around the search object. The processor 160 may display a viewing window using a user display interface (S350). That is, only the viewing window 503, not the entire captured frame image, is displayed on the user display interface, and the remaining area 505 excluding the viewing window may not be displayed on the user display interface.

The processor 160 may repeatedly perform the above-described process of setting a viewing window for all or at least a portion of consecutive frame images captured by the camera module, which are referred to as captured frame images. FIG. 6 shows a captured frame image 601 taken after a certain period of time in FIG. 5 . Comparing FIGS. 5 and 6 , it can be seen that the tracking object 602 has moved from location A to location B. The processor 160 may reset the position of the viewing window 603 in response to the movement of the tracking object 602, and accordingly, the viewing window 503 in FIG. 5 and the viewing window 603 in FIG. 6 may be displayed differently. You can see that it has been set.

Hereinafter, various control features of the processor 260 will be described with reference to FIGS. 7 to 20.

FIG. 7 is a block diagram for explaining functions performed in the processor 160. The components shown in FIG. 7, that is, each module, are implemented in the processor 260 according to instructions stored in the memory 240. It may be a function or a software module. However, hereinafter, each module of the processor 160 is expressed as the subject of control, instruction, or function, but this means that the processor 160 operates by driving instructions or applications stored in the memory 140.

Referring to FIG. 7 , the processor 160 may include a search frame selection module 161, an object detection module 162, a window setting module 613, and an interface module 164.

The search frame selection module 161 can determine a frame image for setting the viewing window - this is referred to as a search frame image. The camera module generates a captured frame image by shooting at a preset frame rate and provides it to the search frame selection module 161 (S810). The search frame selection module 161 may select at least some of the captured frame images and determine them as search frame images for setting a viewing window.

For example, the search frame selection module 161 may set the entire captured frame image as a search frame image. This example is suitable when computing device 100 has sufficient resources.

As another example, the search frame selection module 161 may select some of the captured frame images and set them as search frame images. These other examples are appropriate because computing resources may be limited, such as in mobile computing environments.

As an embodiment, Figure 8 discloses an example of a search frame selection method performed by the search frame selection module 161. Referring to FIG. 8, the search frame selection module 161 receives a captured frame image from the camera module by shooting at a preset frame rate (S810). The search frame selection module 161 may set a search frame image at a time interval with a frequency lower than the frame rate. That is, the search frame selection module 161 may select frame images at preset time intervals from a plurality of consecutive captured frame images (S820) and set the selected frame images as search frame images (S830). Figure 9 is a diagram illustrating this embodiment. Figure (a) shows capture frame images 1 to 12 continuously captured by a camera module, and Figure (b) shows images selected by the search frame selection module 161 among them.

Frame images

1, 4, 7, and 10 selected as search frame images are displayed. In one embodiment of FIGS. 8 and 9, search frame images are selected at equal time intervals.

As another embodiment, Figure 10 discloses another example of a search frame selection method performed by the search frame selection module 161. Referring to FIG. 10, the search frame selection module 161 receives a captured frame image from the camera module by shooting at a preset frame rate (S1010). The search frame selection module 161 confirms the first position of the tracking object on the previous first search frame image (S1020) and confirms the second position of the tracking object on the current second search frame image (S1030) ). The search frame selection module 161 determines the next search frame image according to the difference between the first position of the tracked object on the previous first search frame image and the second position of the tracked object on the current second search frame image. You can. That is, the search frame selection module 161 can select the next search frame image more quickly when the tracking object moves quickly. Figure 11 is a diagram illustrating this embodiment. Figure (a) shows capture frame images 1 to 12 continuously captured by a camera module, and Figure (b) shows images selected by the search frame selection module 161 among them. It displays

frame images

1, 4, 6, 9, 12, and 14 selected as search frame images, and Figure (c) shows the distance between the object position in the previous search frame image and the object in the current search frame image. After selecting search frame image 4, the search frame selection module 161 determines the movement distance between the objects in the previous search frame image 1 and the current search frame image 4 - for example, the unit pixel through which the tracking object moves (ex. 10 pixels, etc.) As a result of calculating the number, etc., it was decided to be 60, and since this exceeded the standard for moving distance (ex. 40), the selection frequency of the search frame image was increased and the 6th shooting frame image was selected as the search frame image. Meanwhile, in the 6th and 9th frame images, the moving distances of the tracking objects are 40 and 30, which are below the standard, so the 9th and 12th frame images are selected. Meanwhile, in the 12th frame image, the moving distance of the tracking object is 70, which exceeds the moving distance standard (ex. 40), so the selection frequency of the search frame image is increased and the 14th shooting frame image is selected as the search frame image. . In this way, software-based tracking can be performed more smoothly by adjusting the selection frequency of the viewing window according to the moving distance of the tracking object on the viewing window.

Referring again to FIG. 7, the object detection module 162 can identify whether a tracking object exists in this search frame image.

In one embodiment, as in the example shown in FIG. 12, the object detection module 162 may perform deep learning-based object detection. Referring to FIG. 12, the object detection module 162 may be equipped with a first deep learning model learned with a large amount of training data associated with the tracked object (S1210). The first deep learning model may be an artificial neural network model learned from a large amount of learning data showing tracked objects, and various models such as CNN and RNN may be applied to the structure of this artificial neural network. The object detection module 162 may identify a tracking object existing in the search frame image using the first deep learning model (S1220). The object detection module 162 can display a bounding box on the tracked object identified in the search frame image (S1230), and the window setting module 613 can set a viewing window based on this bounding box.

Examples of tracking objects include those related to people, such as a person's face, a person's torso, and the overall shape of a person, as well as various objects such as a horseback riding, a dog, etc., can be set as tracking objects. This is because the tracking object is set according to the training data of the first deep learning model. The first deep learning model is capable of learning and tracking in various ways, such as being trained to detect at least some of one or several objects depending on the settings.

In one embodiment, as in the example shown in FIG. 13, the object detection module 162 may determine the identity of the previous tracking object and the current tracking object using a separate deep learning model. Since the first deep learning model identifies and classifies objects, all non-identical objects, for example, people, are identified as tracking objects. Therefore, in this embodiment, a separate second deep learning model is used to identify the same objects. , For example, only the same person can be set as a tracking object. Referring to FIG. 13, the object detection module 162 may be equipped with a second deep learning model learned with a large amount of learning data associated with the external characteristics of the tracked object (S1310). The second deep learning model may be an artificial neural network model learned with a large amount of learning data to determine similarity based on the external characteristics of the tracked object. The object detection module 162 uses a second deep learning model to generate first feature data associated with the external characteristics of the first tracking object identified in the first search frame image (S1320), and creates a second search frame image Second characteristic data associated with the external characteristics of the identified second tracking object may be generated (S1330). The object detection module 162 may determine whether the first tracking object and the second tracking object are the same object based on the similarity between the first feature data and the second feature data (S1340). Depending on the embodiment, the object detection module 162 may determine whether the object is the same by directly generating data for determining similarity - for example, a feature vector - without generating feature data. Figures 14 and 15 show these examples. In the example of FIG. 14 , when the tracking object 1402 is detected in the first search frame image 1401, the viewing window 1403 is set based on it. FIG. 15 is a second search frame image 1401 after FIG. 14. The object detection module 162 may detect two

objects

1502 and 1504 in the second search frame image 1501. The object detection module 162 determines whether the first tracking object 1402 in the first search frame image 1401 and the two

objects

1502 and 1504 in the second search frame image 1501 are the same object. 2 Tracking object 1502 can be determined, and it can be seen that the viewing window 1503 is set based on the second tracking object 1502.

Referring again to FIG. 7 , the window setting module 613 may set at least a portion of the area in the search frame image as a viewing window based on information provided by the object detection module 162 - for example, a bounding box.

Figure 16 is a flowchart explaining the operation of the window setting module 613. Referring to Figure 16, the window setting module 613 is based on information provided by the object detection module 162 - for example, a bounding box, The location of the tracking object within the search frame image can be confirmed (S1610). The window setting module 613 may extract a part of the search frame image corresponding to the second resolution based on the position of the tracking object (S1620) and set the part of the extracted search frame image as a viewing window (S1630). .

The window setting module 613 may set the viewing window determined in the latest search frame image to be the same for a capture frame image that is not a search frame image. This corresponds to the case where some, but not all captured frame images, are set as search frame images. For example, in at least one non-search frame image that is continuously displayed after the first search frame image, the viewing window determined in the first search frame image may be set to be the same, that is, set to the same position.

In one embodiment, as shown in the example shown in FIG. 17, the window setting module 613 corrects the viewing window in the previous search frame image when the viewing window in the current search frame image is separated by a certain amount or more. can do. This will be described with reference to FIGS. 17 to 20. The window setting module 613 sets the first viewing window 1803 for the first search frame image (FIG. 18, 1801) and sets the second viewing window 1803 for the second search frame image (FIG. 19, 1901) (S1710). The window 1903 can be set (S1720). The window setting module 613 may determine the positional criticality between the first viewing window and the second viewing window (S1730) and determine whether the positional criticality is satisfied (S1740). The positional criticality may be set as a variable distance of the viewing window that is set in proportion to the time interval between search frames (eg, number of frame rates). In the example of FIG. 19, the window setting module 613 sets the position of the first viewing window in the first search frame image (FIG. 19, 1803) based on the upper left corner and the second viewing window selected in the second search frame image. (1904) Calculate the distance △Lt1 between the two and determine the positional criticality based on this. The example of FIG. 19 is an example that deviates from the positional criticality, and the window setting module 613 uses the position of the first viewing window 1803 in the previous first search frame image as the example shown in FIG. 20. Thus, the second viewing window 1903 can be reset. If the positional criticality is satisfied, the window setting module 613 maintains the second viewing window (S1750). In this embodiment, when an error occurs because externally similar objects are detected at the same time, tracking errors can be prevented by correcting them using only the viewing window itself.

In one embodiment, the window setting module 613 may adjust the size of the viewing window in response to the size of the tracking object. For example, a case may occur where the size of the second tracking object in the second search frame image is reduced by a certain amount or more compared to the first tracking object in the first search frame image. The window setting module 613 may reflect the reduced ratio and set the size of the second viewing window in the second search frame image to be smaller than the size of the first viewing window in the first search frame image. For example, this may occur when a human object moves away from the computing device. In this case, the size of the viewing window can be reduced to maintain the size of the human object relative to the viewing window.

The interface module 164 can display a user display interface based on the viewing window provided by the window setting module 613.

For example, the resolution of the user display interface and the resolution of the viewing window may be different, and the interface module 164 may enlarge or reduce the resolution of the viewing window to correspond to the resolution of the user display interface. Since the resolution of the viewing window is variable, it is not limited to the absolute size of the viewing window, and the resolution of the viewing window is enlarged or reduced to match the resolution of the user display interface, providing the user with an effect such as zooming in or zooming out. can be provided.

The present invention described above is not limited by the above-described embodiments and the accompanying drawings, but is limited by the scope of the patent claims described later, and the configuration of the present invention can be varied within the scope without departing from the technical spirit of the present invention. Those skilled in the art can easily see that changes and modifications can be made.

[Explanation of symbols]

100: computing device

200: Cradle

110: Department of Communications

120: camera unit

130: output unit

140: memory

150: power supply unit

160: processor

161: Search frame selection module

162: Object detection module

163: Windows settings module

164: interface module

[acknowledgment]

This invention was applied overseas with support from the following research project supported by the government of the Republic of Korea.

Research project information

Name of Ministry: Korea Tourism Organization

Research project name: Follow-up support for global leading companies in tourism

Project name: Smartphone-linked automatic person/object recognition and tracking recording device

Host organization: 3I Co., Ltd.

Research period: 2022.03.04~2022.12.31

The present invention has high industrial applicability because it has the effect of effectively providing tracking of objects based on software for images captured while fixed in a certain direction.

In addition, a first deep learning model learned with a large amount of learning data to identify the tracking object is used to identify the tracking object within the frame image, and a second deep learning model learned with a large amount of learning data related to the external characteristics of the tracking object is used. By using a learning model to determine the identity of a tracked object, it has the effect of quickly and accurately performing object identification and identity judgment, so it has high industrial applicability.

In addition, by resetting the viewing window based on the positional criticality of the viewing window in consecutive frame images, it is possible to provide higher tracking performance by preventing errors in viewing window settings due to errors or misrecognition of other objects. It is effective and has high industrial applicability.

Claims

A method for providing object tracking performed on a computing device including a camera module, comprising:

Receiving a search frame image captured at a first resolution from the camera module;

setting a second resolution for the viewing window;

identifying whether a tracked object exists within the search frame image; and

Based on the position of the tracking object in the search frame image, setting a partial area of the search frame image including the tracking object as the viewing window; Including,

The second resolution of the viewing window is a lower resolution than the first resolution of the frame image,

Method for providing software-based object tracking.
The method of claim 1, wherein the software-based object tracking method includes:

displaying the viewing window using a user display interface; Containing more,

Method for providing software-based object tracking.
The method of claim 2, wherein the camera module:

Generating the search frame image by shooting at the first resolution while being fixed in a preset forward direction regardless of the presence and movement of the tracking object.

Method for providing software-based object tracking.
According to paragraph 3,

The first resolution of the search frame image is preset and fixed,

The second resolution of the viewing window is changeable while providing the object tracking function,

Method for providing software-based object tracking.
The method of claim 3, wherein the step of receiving a search frame image captured by the camera module at a first resolution comprises:

Receiving a plurality of consecutive captured frame images at a preset frame rate from the camera module; and

selecting some frame images from among the plurality of consecutive captured frame images and setting them as the search frame images; Including,

Method for providing software-based object tracking.
The method of claim 1, wherein identifying whether an object exists in the frame image comprises:

Identifying the tracking object within the search frame image using a first deep learning model learned with a large amount of training data associated with the tracking object; Including,

Method for providing software-based object tracking.
The method of claim 6, wherein identifying whether an object exists in the frame image comprises:

determining whether a first tracking object identified in the first search frame image and a second tracking object identified in the second search frame image are the same object; Containing more,

Method for providing software-based object tracking.
The method of claim 7, wherein determining whether the second tracking object identified in the second search frame image is the same object comprises:

Using a second deep learning model learned with a large amount of learning data associated with the external features of the tracking object, generate first feature data related to the external features of the first tracking object identified in the first search frame image. steps;

generating second feature data associated with external features of the second tracked object identified in the second search frame image; and

determining whether the first tracking object and the second tracking object are the same object based on similarity between the first characteristic data of the first tracking object and the second characteristic data of the second tracking object; Including,

Method for providing software-based object tracking.
The method of claim 2, wherein setting a partial area of the search frame image including the tracking object as the viewing window comprises:

confirming the location of the tracked object within the search frame image;

extracting a portion of the search frame image corresponding to the second resolution based on the location of the tracking object; and

setting a part of the extracted search frame image as the viewing window; Including,

Method for providing software-based object tracking.
The method of claim 9, wherein setting a partial area of the search frame image including the tracking object as the viewing window comprises:

determining positional criticality between a first viewing window for a first search frame image and a second viewing window for a second search frame image; and

If the positional criticality is not satisfied, resetting the second viewing window based on the first viewing window; Including,

Method for providing software-based object tracking.
A method for providing object tracking performed in a computing device including a camera module that is fixed in a preset forward direction and generates a shooting frame image, comprising:

Receiving a plurality of captured frame images sequentially captured at a first resolution from the camera module;

selecting at least one search frame image by selecting at least a portion of the plurality of captured frame images;

identifying whether a tracked object exists within the at least one search frame image; and

Based on the position of the tracking object in the at least one search frame image, respectively setting a partial region of the at least one search frame image including the tracking object as the viewing window; Including,

Method for providing software-based object tracking.
A storage medium storing computer-readable instructions,

The instructions, when executed by a computing device, cause the computing device to:

An operation of receiving a search frame image captured at a first resolution;

setting a second resolution for the viewing window;

Identifying whether a tracking object exists within the search frame image; and

Setting a partial area of the search frame image including the tracking object as the viewing window based on the location of the tracking object in the search frame image; to perform,

The second resolution of the viewing window is a lower resolution than the first resolution of the frame image,

storage media.
As a computing device,

A memory that stores one or more instructions; and

At least one processor executing the one or more instructions stored in the memory,

The at least one processor executes the one or more instructions,

Receive a search frame image captured at a first resolution from the camera module,

set a second resolution for the viewing window;

Identify whether a tracking object exists within the search frame image,

Based on the position of the tracking object in the search frame image, set a partial area of the search frame image including the tracking object as the viewing window,

The second resolution of the viewing window is a lower resolution than the first resolution of the frame image,

Computing device.
14. The method of claim 13, wherein the at least one processor:

By executing one or more of the instructions above,

Providing a user display interface to display the viewing window,

Computing device.
The method of claim 13, wherein the camera module:

Generating the search frame image by shooting at the first resolution while being fixed in a preset forward direction regardless of the presence and movement of the tracking object.

Computing device.
According to clause 15,

The first resolution of the search frame image is preset and fixed,

The second resolution of the viewing window is changeable while providing the object tracking function,

Computing device.
16. The method of claim 15, wherein the at least one processor:

By executing one or more of the instructions above,

Receiving a plurality of consecutive captured frame images at a preset frame rate from the camera module,

Selecting some frame images from the plurality of consecutive captured frame images and setting them as the search frame images,

Computing device.
14. The method of claim 13, wherein the at least one processor:

By executing one or more of the instructions above,

Identifying the tracking object within the search frame image using a first deep learning model learned with a large amount of training data associated with the tracking object,

Determining whether the first tracking object identified in the first search frame image and the second tracking object identified in the second search frame image are the same object,

Computing device.
19. The method of claim 18, wherein the at least one processor:

By executing one or more of the instructions above,

Using a second deep learning model learned with a large amount of learning data associated with the external features of the tracking object, generate first feature data related to the external features of the first tracking object identified in the first search frame image. do,

generate second feature data associated with external features of the second tracked object identified in the second search frame image;

Based on the similarity between the first characteristic data of the first tracking object and the second characteristic data of the second tracking object, determining whether the first tracking object and the second tracking object are the same object,

Computing device.
14. The method of claim 13, wherein the at least one processor:

By executing one or more of the instructions above,

Determine positional criticality between a first viewing window for a first search frame image and a second viewing window for a second search frame image;

If the positional criticality is not satisfied, resetting the second viewing window based on the first viewing window,

Computing device.