KR20240045743A - Software-based object tracking method and computing device therefor - Google Patents

Abstract

A method of providing an image synthesis function according to a technical aspect of the present invention is a method of providing object tracking performed in a computing device including a camera module, comprising: receiving a frame image captured by the camera module at a first resolution; setting a second resolution of the viewing window to have a lower resolution than the first resolution, identifying whether an object exists within the frame image, and setting a portion of the frame image containing the identified object as the viewing window. and displaying the viewing window using a user display interface.

Description

Method for providing video synthesis function and computing device therefor {Software-based object tracking method and computing device therefor}

The present invention relates to a method for providing an image synthesis function and a computing device therefor.

With the development of computing devices, the miniaturization and portability of computing devices are increasing, and a more user-friendly computing environment is being developed.

In this computing environment, the main interest of users is the tracking function for objects of interest in the image being shot.

In the conventional case, for object tracking, it is necessary to use multiple photographic equipment or physically drive the photographic equipment.

However, this has the limitation of being difficult to apply in a miniaturized and portable computing device environment and requiring separate equipment.

One technical aspect of the present application is to solve the problems of the prior art described above, and according to an embodiment disclosed in the present application, the purpose is to effectively provide tracking of objects based on software.

The tasks of this application are not limited to the tasks mentioned above, and other tasks not mentioned will be clearly understood by those skilled in the art from the description below.

One technical aspect of the present application proposes a method for providing an image synthesis function. The method of providing the image synthesis function is a method of providing object tracking performed in a computing device including a camera module, comprising the steps of receiving a frame image captured by the camera module at a first resolution, and providing a frame image with a resolution lower than the first resolution. setting a second resolution of the viewing window to have, identifying whether an object exists within the frame image, setting a portion of the frame image containing the identified object as the viewing window, and using a user display interface. Thus, it includes the step of displaying the viewing window.

Another technical aspect of the present application proposes a storage medium. The storage medium is a storage medium that stores computer-readable instructions. The instructions, when executed by a computing device, cause the computing device to: receive a frame image captured by the camera module at a first resolution; and set a second frame image of the viewing window to have a resolution lower than the first resolution. Setting a resolution, identifying whether an object exists in the frame image, setting a portion of the frame image including the identified object as the viewing window, and displaying the viewing window using a user display interface. Make sure to perform the action.

The means for solving the above problems do not enumerate all the features of the present application. Various means for solving the problems of this application can be understood in more detail by referring to specific embodiments in the detailed description below.

According to one embodiment of the present invention, by constructing a 3D model corresponding to the construction site and setting a plurality of camera positions that can cover the management area of the 3D space, a 3D model that can cover the risk area of the construction site as a whole is provided. It has the effect of providing a method for creating a virtual model and a system for it.

In addition, according to an embodiment of the present invention, objects are identified based on image recognition for a plurality of images captured by a plurality of cameras, and a time-series tracking data set is set for the identified objects to more accurately track the objects based on this. It has the effect of providing a method and system for creating a 3D virtual model that can determine risk situations and index risk factors.

In addition, according to an embodiment of the present invention, 3D space is differentiated according to the degree of risk according to the determined risk situation, and camera function enhancement is provided to the high risk space, so that the space where the risk situation occurs is provided. It has the effect of providing closer and strengthened risk management.

1 is a diagram illustrating a terminal holder and a portable terminal interoperating with the terminal holder according to an embodiment disclosed in this document.
Figure 2 is a block diagram illustrating a terminal holder according to an embodiment disclosed in this document.
Figure 3 is a block diagram illustrating a portable terminal according to an embodiment disclosed in this document.
FIG. 4 is a block diagram illustrating an embodiment of the processor shown in FIG. 3.
FIG. 5 is a block diagram illustrating an embodiment of the control unit shown in FIG. 4.
Figure 6 is a flow chart explaining an image generation method using a terminal holder according to an embodiment disclosed in this document.
Figures 7 and 8 are diagrams to explain examples of an image generation method using a terminal holder.
9 is a flow chart illustrating an example of a rephotographing process according to an embodiment disclosed in this document.
10 is a flowchart illustrating another example of a rephotographing process according to an embodiment disclosed in this document.
11 is a flowchart explaining another example of a rephotographing process according to an embodiment disclosed in this document.
12 and 13 are diagrams showing examples of a rephotographed image according to a rephotograph process and a 360-degree image registered based on the rephotographed image.

Hereinafter, preferred embodiments of the present invention will be described with reference to the attached drawings.

However, the embodiments of the present invention may be modified into various other forms, and the scope of the present invention is not limited to the embodiments described below. Additionally, the embodiments of the present invention are provided to more completely explain the present invention to those with average knowledge in the relevant technical field.

That is, the above-described objects, features, and advantages will be described in detail later with reference to the attached drawings, so that those skilled in the art will be able to easily implement the technical idea of the present invention. In describing the present invention, if it is determined that a detailed description of known technologies related to the present invention may unnecessarily obscure the gist of the present invention, the detailed description will be omitted. Hereinafter, preferred embodiments according to the present invention will be described in detail with reference to the attached drawings. In the drawings, identical reference numerals are used to indicate identical or similar components.

Additionally, as used herein, singular expressions include plural expressions, unless the context clearly dictates otherwise. In the present application, terms such as “consists of” or “comprises” should not be construed as necessarily including all of the various components or steps described in the specification, and some of the components or steps may include It may not be included, or it should be interpreted as including additional components or steps.

In addition, in order to explain the system according to the present invention, various components and their sub-components are described below. These components and their sub-components may be implemented in various forms such as hardware, software, or a combination thereof. For example, each element may be implemented as an electronic configuration to perform the corresponding function, or may be implemented as software itself that can be run in an electronic system, or as a functional element of such software. Alternatively, it may be implemented with an electronic configuration and corresponding driving software.

The various techniques described herein may be implemented with hardware or software, or a combination of both as appropriate. As used herein, terms such as “Unit,” “Server,” and “System” likewise refer to a computer-related entity, i.e., hardware, a combination of hardware and software, software or It can be treated as equivalent to software at the time of execution. Additionally, each function executed in the system of the present invention may be configured in module units and may be recorded in one physical memory, or may be distributed and recorded between two or more memories and recording media.

Various embodiments of the present application are software (e.g., machine) including one or more instructions stored in a storage medium that can be read by a machine (e.g., user terminal 100 or computing device 300). For example, it may be implemented as a program). For example, the processor 301 may call at least one instruction among one or more instructions stored from a storage medium and execute it. This allows the device to be operated to perform at least one function according to the at least one instruction called. The one or more instructions may include code generated by a compiler or code that can be executed by an interpreter. A storage medium that can be read by a device may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' only means that the storage medium is a tangible device and does not contain signals (e.g., electromagnetic waves), and this term refers to data being semi-permanently stored in the storage medium. There is no distinction between temporary storage and temporary storage.

Various flowcharts are disclosed to explain embodiments of the present invention, but these are for convenience of explanation of each step, and each step is not necessarily performed in accordance with the order of the flowchart. That is, each step in the flowchart may be performed simultaneously, in an order according to the flowchart, or in an order opposite to the order in the flowchart.

1 is a diagram illustrating a terminal holder and a portable terminal interoperating with the terminal holder according to an embodiment disclosed in this document.

Referring to FIG. 1, a terminal holder 100 and a mobile terminal 200 interoperating with the terminal holder are shown.

The terminal holder 100 can hold the mobile terminal 200, and can be rotated under the control of the mobile terminal 200 to rotate the mounting direction, that is, the shooting direction, of the mobile terminal 200.

Hereinafter, it will be described as an example that the terminal holder 100 rotates in the horizontal direction to rotate the shooting direction of the portable terminal 200 in the horizontal direction (or left and right direction) based on the shooting direction. However, this is for convenience of explanation. It is not limited to this. For example, the terminal holder 100 may tilt the mounted mobile terminal 200 in the vertical direction, or may perform both horizontal rotation (panning) and vertical rotation (tilting).

The mobile terminal 200 can perform photography in a photographing direction. The mobile terminal 200 can generate a 360-degree image in conjunction with the terminal holder 100. For example, the mobile terminal 200 can be mounted on the terminal holder 100 and generate a 360-degree captured image of the shooting point by providing a driving command to the terminal holder 100.

The generated 360-degree captured image is a 360-degree image of a real space, which can be used to create a virtual space corresponding to the space. For example, 360-degree captured images are used to provide a virtual 360-degree space corresponding to real space based on 360-degree images at each point in real space, or 360-degree images taken at various points in real space - depth It can be used to create a three-dimensional virtual space corresponding to a real space based on a map image.

The portable terminal 200 is an electronic device that a user can carry and is a computing device that includes a camera. For example, the mobile terminal 200 may be a smart phone, a mobile phone, a laptop computer, a digital broadcasting terminal, a personal digital assistant (PDA), a portable multimedia player (PMP), a navigation device, or a slate PC. PC), tablet PC, ultrabook, wearable devices (e.g. smartwatch, smart glass, HMD (head mounted display)), etc. Comprehensive.

Figure 2 is a block diagram illustrating a portable terminal according to an embodiment disclosed in this document.

Referring to FIG. 2, the portable terminal 200 includes a communication unit 210, a camera unit 220, an output unit 230, a memory 240, a power supply unit 250, and a processor 260. The components shown in FIG. 2 are not essential for implementing a portable terminal, so the portable terminal described in this specification may have more or fewer components than the components listed above.

The communication unit 210 enables communication between the portable terminal 200 and the wireless communication system, between the portable terminal 200 and another portable terminal 200, or between the portable terminal 200 and the terminal holder 100. It may contain one or more modules. This communication unit 210 may include a mobile communication module 211, a wireless Internet module 212, and a short-range communication module 213. The short-distance communication module 213 can perform a communication connection with the terminal holder 100 by wire or wirelessly. For example, the short-range communication module 213 may include a short-range wireless communication module such as Bluetooth or a wired communication module such as RS232.

The camera unit 220 may include at least one camera. Camera unit 220 may include one or more lenses, image sensors, image signal processors, or flashes.

The camera unit 220 may include a first camera 221 to a second camera 222. The first camera 221 or the second camera 222 may capture a front image of the portable terminal 200.

The output unit 230 is intended to generate output related to vision, hearing, or tactile senses, and may include a display unit 231 and an audio output unit 152. The display unit 231 can implement a touch screen by forming a mutual layer structure or being integrated with the touch sensor. This touch screen can function as a user input unit that provides an input interface between the portable terminal 200 and the user, and at the same time, can provide an output interface between the portable terminal 200 and the user.

The power supply unit 250 receives external power and internal power under the control of the processor 260 and supplies power to each component included in the portable terminal 200. This power supply unit 190 includes a battery, and the battery may be a built-in battery or a replaceable battery.

In addition to operations related to the application program, the processor 260 typically controls the overall operation of the mobile terminal 200. The processor 260 can provide or process appropriate information or functions to the user by processing signals, data, information, etc. input or output through the components discussed above, or by running an application program stored in the memory 240. The processor 260 can be implemented as one processor or multiple processors.

The processor 260 may control at least some of the components examined with FIG. 3 to run an application program stored in the memory 240, that is, an application. Furthermore, the processor 260 can operate at least two of the components included in the mobile terminal 100 in combination with each other in order to run the application program.

The processor 260 may run an application by executing instructions stored in the memory 240. Hereinafter, the processor 260 is expressed as the subject of control, instruction, or function by driving an application. However, this means that the processor 260 operates by driving instructions or applications stored in the memory 240.

At least some of the above components may cooperate with each other to implement operation, control, or a control method of a portable terminal according to various embodiments described below. Additionally, the operation, control, or control method of the portable terminal may be implemented on the portable terminal by running at least one application program stored in the memory 170.

In addition to operations related to application programs, the processor 260 typically controls the overall operation of the mobile terminal 200. The processor 260 can provide or process appropriate information or functions to the user by processing signals, data, information, etc. input or output through the components discussed above, or by running an application program stored in the memory 240.

The processor 260 may control at least some of the components examined with FIG. 3 in order to run an application program stored in the memory 240, that is, an application. Furthermore, the processor 260 may operate at least two of the components included in the mobile terminal 100 in combination with each other in order to run an application program.

The processor 260 may run an application by executing instructions stored in the memory 240. Hereinafter, the processor 260 is expressed as the subject of control, instruction, or function by driving an application. However, the processor 260 may operate by driving instructions or applications stored in the memory 240. The components of FIG. 4 or FIG. 5 described below may be functions or software modules implemented in the processor 260 according to instructions stored in the memory 240.

Meanwhile, the control method performed in the mobile terminal 200 according to the above-described embodiment may be implemented as a program and provided to the mobile terminal 200. In particular, a program including a control method for the portable terminal 200 may be stored and provided in a non-transitory computer readable medium.

In the above, the control method of the electronic device 100 and the computer-readable recording medium including the program for executing the control method of the electronic device 100 have been briefly described. However, this is only to omit redundant description, and the electronic device 100 control method has been briefly described. Of course, various embodiments of the device 100 can also be applied to a control method of the electronic device 100 and a computer-readable recording medium including a program that executes the control method of the electronic device 100.

At least some of the above components may cooperate with each other to implement operation, control, or a control method of a mobile terminal according to various embodiments described below. This operation, control, or control method of the mobile terminal can be implemented on the mobile terminal 200 by running at least one application program stored in the memory 240.

Hereinafter, various controls of the processor 260 will be described with reference to FIGS. 4 to 14.

A method for providing object tracking performed in a computing device including a camera module, the processor 260 comprising: receiving a frame image captured by a camera module at a first resolution; setting a viewing window to have a resolution lower than the first resolution; setting a second resolution of the frame image, identifying whether an object exists within the frame image, setting a portion of the frame image containing the identified object as the viewing window, and using a user display interface to display the viewing window. It may include the step of displaying a window.

In one embodiment, the camera module may be fixed in a preset forward direction and capture images at the first resolution to generate the frame image, regardless of whether the object exists or moves.

In one embodiment, in the step of identifying whether an object exists in the frame image, the processor 260 includes: identifying whether an object to be tracked exists in the frame image; extracting a first external feature point for the identified first object; , It may include generating an object identifier for the first object based on the extracted first external feature point.

In one embodiment, identifying whether an object exists in the frame image includes extracting a second appearance feature point for a second object identified in the previous frame image and a first appearance feature point of the first object. and determining whether the first object and the second object are the same object based on the second external feature point of the second object; may include.

In one embodiment, setting a portion of the frame image as the viewing window includes confirming the location of an identified object, and selecting a portion of the frame image having a second resolution based on the location of the identified object. It may include extracting and setting a part of the extracted frame image as the viewing window.

In one embodiment, setting a portion of the frame image as the viewing window includes determining positional criticality between a first viewing window for a current frame image and a second viewing window for a previous frame image, and If the positional criticality is not satisfied, the method may include resetting the first viewing window based on the second viewing window.

In one embodiment, the method of providing the image synthesis function may further include displaying the viewing window through the display module.

Figure 14 is a block diagram illustrating a terminal holder according to an embodiment disclosed in this document. Referring to FIG. 14, an embodiment of a terminal holder 100 according to an embodiment of the present invention will be described.

The terminal holder 100 may include a rotating part 110 and a main body 120.

The mobile terminal 200 is mounted on the rotating unit 110 and can be rotated by the operation of the motor unit 121. The shooting direction of the portable terminal 200 may change according to the rotation of the rotating unit 110. Additionally, the rotation direction and rotation speed of the rotation unit 110 may change depending on the driving of the motor unit 121.

As an example, the rotating unit 110 may include a fixture, a clamp, and a rotating table. The fixture and fastener may be placed on the rotating table. The fixture and clamp can secure the mobile terminal 200. The user can position the mobile terminal 200 between the fixture and the clamp and tighten the clamp to ensure that the mobile terminal 200 is fixed without shaking. The rotary table can rotate according to the operation of the motor unit 121, and for this purpose, the rotary table can be mechanically connected to the motor unit 121.

The main body 120 may include a motor unit 121, a control unit 122, and a communication unit 123. The control unit 122 can control the operation of the terminal holder 100 by controlling the components of the main body 120.

The communication unit 123 may perform a communication connection with the mobile terminal 200 and receive a control signal for driving the terminal holder 100 from the mobile terminal 200. As an example, the communication unit 123 may establish a communication connection with the mobile terminal 200 using at least one of a short-range communication module or wired communication.

The control unit 122 may control the driving of the rotary unit 110 by driving the motor unit 121 in response to a control signal received through the communication unit 123.

Hereinafter, the control signal sent from the mobile terminal 200 to the terminal holder 100 will be described as a 'drive command' as a control signal for driving the terminal holder 100 - for example, a horizontal rotation motion. , This is for convenience of explanation and is not limited thereto. That is, depending on the embodiment, the control signal sent from the mobile terminal 200 to the terminal holder 100 not only controls the rotational drive of the rotating part 110 of the terminal holder 100, but also controls the power on of the terminal holder 100. Controls for on/off, wireless link formation request, authentication, and remaining battery level display are also possible.

The present invention described above is not limited by the above-described embodiments and the accompanying drawings, but is limited by the scope of the patent claims described later, and the configuration of the present invention can be varied within the scope without departing from the technical spirit of the present invention. Those skilled in the art can easily see that changes and modifications can be made.

Claims (11)

A method of providing image synthesis performed in a portable terminal operating in conjunction with a terminal cradle, comprising:
Identifying an object present in each of a plurality of consecutive frame images included in a captured image using a deep learning module;
selecting some frame images among the plurality of frame images as display frame images; and
setting an object identified in the display frame image as a display object and displaying the display object by adding an image for the display object to a plurality of consecutive frame images preceding the display frame image; Including,
How to provide video synthesis function.
The method of claim 1, wherein the step of selecting some frame images from among the plurality of frame images as display frame images comprises:
selecting the partial frame images such that the distance between the display objects falls within a preset range; Including,
How to provide video synthesis function.
The method of claim 1, wherein the step of selecting some frame images from among the plurality of frame images as display frame images comprises:
determining some frame images selected based on a preset time interval as the display frame images; Including,
How to provide video synthesis function.
A method for providing object tracking performed on a computing device including a camera module, comprising:
Receiving a frame image captured by the camera module at a first resolution;
setting a second resolution of the viewing window to have a lower resolution than the first resolution;
identifying whether an object exists within the frame image;
setting a portion of the frame image containing an identified object as the viewing window; and
Including, displaying the viewing window using a user display interface.
How to provide video synthesis function.
The method of claim 1, wherein the camera module:
Generating the frame image by shooting at the first resolution while being fixed in a preset forward direction regardless of whether the object exists or moves,
How to provide video synthesis function.
The method of claim 2, wherein identifying whether an object exists in the frame image comprises:
Identifying whether a tracking target object exists in the frame image;
Extracting a first external feature point for an identified first object and generating an object identifier for the first object based on the extracted first external feature point; Including,
How to provide video synthesis function.
The method of claim 3, wherein identifying whether an object exists in the frame image comprises:
extracting a second external feature point for a second object identified in the previous frame image; and
determining whether the first object and the second object are the same object based on the first appearance feature point of the first object and the second appearance feature point of the second object; Including,
How to provide video synthesis function.
The method of claim 2, wherein setting a portion of the frame image as the viewing window comprises:
Confirming the location of the identified object;
extracting a portion of the frame image having a second resolution based on the location of the identified object; and
setting a part of the extracted frame image as the viewing window; Including,
How to provide video synthesis function.
The method of claim 5, wherein setting a portion of the frame image as the viewing window comprises:
determining positional criticality between a first viewing window for the current frame image and a second viewing window for the previous frame image; and
If the positional criticality is not satisfied, resetting the first viewing window based on the second viewing window; Including,
How to provide video synthesis function.
The method of claim 2, wherein the method of providing the image synthesis function includes:
displaying the viewing window through the display module; Containing more,
How to provide video synthesis function.
A storage medium storing computer-readable instructions,
The instructions, when executed by a computing device, cause the computing device to:
An operation of identifying an object present in each of a plurality of consecutive frame images included in a captured image using a deep learning module;
An operation of selecting some frame images among the plurality of frame images as display frame images; and
An operation of setting an object identified in the display frame image as a display object and displaying the image by adding an image for the display object to a plurality of frame images consecutive before the display frame image; to perform,
storage media.