KR102576587B1 - A method and apparatus for generating a moving object trajectory using multiple cameras - Google Patents

Abstract

According to one aspect, a method of generating a moving object trajectory using multiple cameras includes a first camera and a second camera that photograph the same moving object at different positions, and a first image captured by the first camera and a second image captured by the second camera. calculating homography matrices for each second image; By perspective converting the first image and the second image using the homography matrices on a map image, the first unit trajectory and the second image of the moving object traveling in the first area corresponding to the first image acquiring a second unit trajectory of the moving object traveling in a second area corresponding to; and generating a combined trajectory for the same moving object by combining at least a part of the first unit trajectory and at least a part of the second unit trajectory included in an overlapping area where the first area and the second area overlap. Includes.

Description

Method and device for generating moving object trajectories using multiple cameras {A METHOD AND APPARATUS FOR GENERATING A MOVING OBJECT TRAJECTORY USING MULTIPLE CAMERAS}

This relates to a method and device for generating moving object trajectories using multiple cameras.

Recently, the demand for intelligent systems that detect and track moving objects by analyzing images input from cameras in real time and automatically detect meaningful events based on this is increasing day by day. In addition, there is a growing movement to utilize intelligent systems not only in the field of security and monitoring, but also in the field of BI (Business Intelligence). Representative examples include counting passing visitors by analyzing images from cameras installed at the entrance of a store or at the top of the passageway, or measuring the time visitors stay in a specific place.

However, when an important space is monitored with multiple cameras, it is not easy to accurately identify individual objects, and it is not easy to comprehensively track and manage objects within the space based on space. In addition, the parallax of each camera determines whether the object is the same or not. Not only is it difficult to determine whether objects are clustered, but it is even more difficult to maintain object identification when multiple objects are clustered and then differentiated.

The above-mentioned background technology is technical information that the inventor possessed for deriving the present invention or acquired in the process of deriving the present invention, and cannot necessarily be said to be known art disclosed to the general public before filing the application for the present invention.

Domestic Patent Publication No. 10-2022-0062857 (May 17, 2022)

The purpose is to provide a method and device for generating moving object trajectories using multiple cameras. Additionally, the object is to provide a computer-readable recording medium on which a program for executing the method on a computer is recorded. The technical challenges to be solved are not limited to those described above, and other technical challenges may exist.

According to one aspect, a method of generating a moving object trajectory using multiple cameras includes a first camera and a second camera that photograph the same moving object at different positions, and a first image captured by the first camera and a second image captured by the second camera. calculating homography matrices for each second image; By perspective converting the first image and the second image using the homography matrices on a map image, the first unit trajectory and the second image of the moving object traveling in the first area corresponding to the first image acquiring a second unit trajectory of the moving object traveling in a second area corresponding to; and generating a combined trajectory for the same moving object by combining at least a part of the first unit trajectory and at least a part of the second unit trajectory included in an overlapping area where the first area and the second area overlap. Includes.

A computer-readable recording medium according to another aspect includes a recording medium recording a method for executing the above-described method on a computer.

According to another aspect, a computing device includes at least one memory; and at least one processor, wherein the at least one processor is configured to select the first image and the second image captured from the first camera with respect to the first camera and the second camera that photograph the same moving object from different positions, respectively. By calculating homography matrices for each second image captured from a camera and performing perspective conversion on the first image and the second image using the homography matrices on a map image, the first image Obtain a first unit trajectory of the moving object traveling in a first area corresponding to and a second unit trajectory of the moving object traveling in a second area corresponding to the second image, and obtain the first area and the second area. At least a part of the first unit trajectory and at least a part of the second unit trajectory included in this overlapping area are combined to generate a combined trajectory for the same moving object.

In addition, another method for implementing the present invention, another system, and a computer-readable recording medium storing a computer program for executing the method may be further provided.

Other aspects, features and advantages in addition to those described above will become apparent from the following drawings, claims and detailed description of the invention.

According to the present invention, it is possible to comprehensively track and provide the trajectory of a moving object existing in a single space captured by multiple cameras.

Additionally, by using multiple cameras in a space that cannot be tracked with a single camera, the trajectory of a single moving object can be provided without spatial constraints.

Additionally, by providing the trajectory of a single moving object, more precise simulation of traffic flow is possible and dangerous situations can be predicted.

The effects of the present invention are not limited to those mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description below.

FIG. 1 is a diagram illustrating an example of a system for generating a moving object trajectory using multiple cameras according to an embodiment.
Figure 2 is a configuration diagram illustrating an example of a user terminal according to an embodiment.
FIG. 3 is a flowchart illustrating an example of a method for generating a moving object trajectory using a plurality of images captured from multiple cameras according to an embodiment.
FIG. 4 is a diagram illustrating an example in which a processor according to an embodiment generates a map image including images acquired from a camera and the ground surface represented by the images.
FIG. 5 is a diagram illustrating an example in which a processor designates a plurality of pairs of corresponding points using location information on the ground surface, according to an embodiment.
FIG. 6A is a diagram illustrating an example in which a processor calculates homography matrices when a plurality of pairs of corresponding points share corresponding points on a map image, according to an embodiment.
FIG. 6B is a diagram illustrating an example in which a processor calculates homography matrices when a plurality of pairs of corresponding points do not share corresponding points on the map image, according to an embodiment.
FIG. 7 is a diagram illustrating an example in which a processor according to an embodiment acquires a unit trajectory in an area corresponding to an image using a homography matrix.
FIG. 8 is a diagram illustrating an example in which a processor acquires a first unit trajectory and a second unit trajectory existing in a first area and a second area, according to an embodiment.
FIG. 9 is a diagram illustrating an example in which a processor generates a combined trajectory by combining a part of a first unit trajectory and a part of a second unit trajectory in an overlapping area, according to an embodiment.
FIG. 10 is a diagram illustrating an example in which a processor selects a first unit trajectory and a second unit trajectory from an area that is not an overlapping area, according to an embodiment.
FIG. 11 is a diagram illustrating an example in which a processor according to an embodiment determines a moving object trajectory based on the combined trajectory generated in FIG. 9 and the first and second unit trajectories selected in FIG. 10 .
FIG. 12 is a diagram illustrating another example in which a processor according to an embodiment generates a combined trajectory by combining a part of a first unit trajectory and a part of a second unit trajectory in an overlapping area.
FIG. 13 illustrates an example in which a processor according to an embodiment acquires a first unit trajectory, a second unit trajectory, and a third unit trajectory included in an overlapping area where the first area, the second area, and the third area overlap. This is a drawing for this purpose.
FIG. 14 is a diagram illustrating an example in which a processor generates a combination trajectory in an area where a first area and a third area overlap, according to an embodiment.
FIG. 15 is a diagram illustrating an example in which a processor generates a combination trajectory in an area where a first area, a second area, and a third area overlap, according to an embodiment.
FIG. 16 is a diagram illustrating an example in which a processor generates a combination trajectory in an area where a second area and a third area overlap, according to an embodiment.
FIG. 17 is a diagram illustrating an example in which a processor selects a first unit trajectory and a second unit trajectory from a non-overlapping area, according to an embodiment.
FIG. 18 is a diagram illustrating an example in which a processor according to an embodiment determines a moving object trajectory based on the combined trajectory generated in FIGS. 14 to 16 and the first and second unit trajectories selected in FIG. 17. .

The terms used in the embodiments are general terms that are currently widely used as much as possible, but may vary depending on the intention or precedent of a technician working in the art, the emergence of new technology, etc. In addition, in certain cases, there are terms arbitrarily selected by the applicant, and in this case, the meaning will be described in detail in the relevant description. Therefore, terms used in the specification should be defined based on the meaning of the term and the overall content of the specification, not just the name of the term.

When it is said that a part "includes" a certain element throughout the specification, this means that, unless specifically stated to the contrary, it does not exclude other elements but may further include other elements. In addition, terms such as “??unit” and “??module” used in the specification refer to a unit that processes at least one function or operation, which may be implemented as hardware or software, or as a combination of hardware and software. there is.

Additionally, terms including ordinal numbers, such as “first” or “second,” used in the specification may be used to describe various components, but the components should not be limited by the terms. The above terms may be used for the purpose of distinguishing one component from another component.

Below, the embodiment will be described in detail with reference to the attached drawings. However, the embodiments may be implemented in various different forms and are not limited to the examples described herein.

FIG. 1 is a diagram illustrating an example of a system for generating a moving object trajectory using multiple cameras according to an embodiment.

Referring to Figure 1, system 1 includes a user terminal 10 and a server 20. For example, the user terminal 10 and the server 20 may be connected through wired or wireless communication to transmit and receive data (for example, images taken from a camera, etc.) between them.

For convenience of explanation, FIG. 1 shows that the system 1 includes a user terminal 10 and a server 20, but the system 1 is not limited thereto. For example, the system 1 may include other external devices (not shown), and the operations of the user terminal 10 and the server 20, which will be described below, may be performed using a single device (e.g., the user terminal 10). Alternatively, it may be implemented by the server 20).

The user terminal 10 may be a computing device that includes a display device and a device that receives user input (eg, a keyboard, a mouse, etc.), and includes memory and a processor. For example, the user terminal 10 may include a laptop PC, desktop PC, laptop, tablet computer, smart phone, etc., but is not limited thereto.

The server 20 may be a device that communicates with external devices (not shown), including the user terminal 10. For example, the server 20 may be a device that stores various data, including at least one image taken from at least one camera and location information on the earth's surface, and in some cases, may be a device with its own computing capabilities. there is. For example, the server 20 may be a cloud server, but is not limited thereto.

The user terminal 10 obtains a moving object unit trajectory corresponding to each image of the same object included in a plurality of images taken from multiple cameras, and combines the moving object unit trajectories to generate a single moving object trajectory.

A moving object trajectory refers to a single moving object trajectory created by obtaining unit trajectories of the area corresponding to each image from multiple images captured by multiple cameras that capture the same moving object at different locations, and combining the generated unit objects. . As an example, the moving object trajectory is a single image generated by acquiring unit trajectories of four areas corresponding to each image from four images captured by four cameras placed at an intersection and combining the four obtained unit trajectories. It may be a moving object trajectory.

The system 1 according to an embodiment acquires a moving object unit trajectory corresponding to each image of the same object included in a plurality of images taken from multiple cameras, and combines the moving object unit trajectories to generate a single moving object trajectory. do. Specifically, the user terminal 10 records the first camera and the second camera that capture the same moving object at different positions, respectively, the first image captured by the first camera and the second image captured by the second camera. By calculating homography matrices and perspective converting the first image and the second image using the homography matrices on the map image, the first unit of the moving object traveled in the first area corresponding to the first image. Acquire a second unit trajectory of a moving object traveling in a second area corresponding to the trajectory and the second image, and at least a portion of the first unit trajectory and a second unit included in an overlapping area where the first area and the second area overlap. At least part of the trajectories are combined to generate a combined trajectory for the same moving object. The user 30 can check information on a single trajectory for the same moving object shown in a plurality of images captured from multiple cameras.

Hereinafter, with reference to FIGS. 2 to 18, the user terminal 10 acquires unit trajectories of moving objects corresponding to each image for the same object included in a plurality of images captured from multiple cameras, and determines the moving object unit trajectories. Examples of combining to determine a single moving object trajectory are explained. Meanwhile, as described above with reference to FIG. 1, operations to be described later with reference to FIGS. 2 to 18 may be performed in the server 20.

Figure 2 is a configuration diagram illustrating an example of a user terminal according to an embodiment.

Referring to FIG. 2, the user terminal 100 includes a processor 110 and a memory 120. For convenience of explanation, only components related to the present invention are shown in Figure 2. In addition to the components shown in FIG. 2, other general-purpose components may be further included in the user terminal 100. As an example, the user terminal 100 may include an input/output interface (not shown) and/or a communication module (not shown). Additionally, it is obvious to those skilled in the art that the processor 110 and memory 120 shown in FIG. 2 may be implemented as independent devices.

The processor 110 can process computer program instructions by performing basic arithmetic, logic, and input/output operations. Here, the command may be provided from the memory 120 or an external device (eg, server 20, etc.). Additionally, the processor 110 may generally control the operations of other components included in the user terminal 100.

In particular, the processor 110 generates a moving object trajectory using a plurality of images captured from multiple cameras. Specifically, the processor 110 acquires a first image captured from the first camera and a second image captured from the second camera with respect to the first camera and the second camera that capture the same moving object from different positions. Then, the processor 110 generates a map image including the earth surface represented by the images. Then, the processor 110 uses location information on the ground surface to match coordinates on the images with coordinates on the map image to designate a plurality of pairs of corresponding points. Then, the processor 110 calculates homography matrices for each of the first image and the second image using a plurality of corresponding point pairs. Then, the processor 110 designates a first area corresponding to the first image and a second area corresponding to the second image on the map image. Then, the processor 110 performs perspective conversion on the first image and the second image using homography matrices on the map image, thereby converting the first unit trajectory of the moving object traveling in the first area and the moving object traveling in the second area. Obtain the second unit trajectory. Then, the processor 110 combines a part of the first unit trajectory and a part of the second unit trajectory included in an overlapping area where the first area and the second area overlap to generate a combined trajectory for the same moving object. Then, the processor 110 determines a moving object trajectory based on the first unit trajectory, the second unit trajectory, and the combined trajectory.

Specific examples of how the processor 110 according to an embodiment operates will be described with reference to FIGS. 3 to 18.

The processor 110 may be implemented as an array of multiple logic gates, or may be implemented as a combination of a general-purpose microprocessor and a memory storing a program that can be executed on the microprocessor. For example, processor 110 may include a general-purpose processor, central processing unit (CPU), microprocessor, digital signal processor (DSP), controller, microcontroller, state machine, etc. In some circumstances, processor 110 may include an application specific integrated circuit (ASIC), a programmable logic device (PLD), a field programmable gate array (FPGA), etc. For example, processor 110 may be a combination of a digital signal processor (DSP) and a microprocessor, a combination of a plurality of microprocessors, a combination of one or more microprocessors combined with a digital signal processor (DSP) core, or any other such. It may also refer to a combination of processing devices, such as a combination of configurations.

Memory 120 may include any non-transitory computer-readable recording medium. As an example, the memory 120 is a non-permanent mass storage device such as random access memory (RAM), read only memory (ROM), disk drive, solid state drive (SSD), flash memory, etc. device). As another example, non-perishable mass storage devices such as ROM, SSD, flash memory, disk drives, etc. may be a separate persistent storage device distinct from memory. Additionally, the memory 210 may store an operating system (OS) and at least one program code (e.g., code for the processor 110 to perform operations to be described later with reference to FIGS. 3 to 18).

Such software components may be loaded from a computer-readable recording medium separate from the memory 120. The recording medium readable by such a separate computer may be a recording medium that can be directly connected to the user terminal 100, for example, a floppy drive, disk, tape, DVD/CD-ROM drive, memory card, etc. It may include a readable recording medium. Alternatively, software components may be loaded into the memory 120 through a communication module (not shown) rather than a computer-readable recording medium. For example, at least one program is a computer program installed by files provided by developers or a file distribution system that distributes the installation file of the application through a communication module (not shown) (e.g., FIGS. 3 to 3). It may be loaded into the memory 120 based on a computer program (such as a computer program for the processor 110 to perform an operation to be described later with reference to 18).

The input/output interface (not shown) may be a means for interfacing with an input or output device (eg, keyboard, mouse, etc.) that may be connected to or included in the user terminal 100. The input/output interface (not shown) may be configured separately from the processor 110, but is not limited to this, and the input/output interface (not shown) may be configured to be included in the processor 110.

A communication module (not shown) may provide a configuration or function for the server 20 and the user terminal 100 to communicate with each other through a network. Additionally, a communication module (not shown) may provide a configuration or function for the user terminal 100 to communicate with other external devices. For example, control signals, commands, data, etc. provided under control of the processor 110 may be transmitted to the server 20 and/or an external device through a communication module (not shown) and a network.

Meanwhile, although not shown in FIG. 2, the user terminal 100 may further include a display device. Alternatively, the user terminal 100 may be connected to an independent display device through wired or wireless communication to transmit and receive data between them.

FIG. 3 is a flowchart illustrating an example of a method for generating a moving object trajectory using a plurality of images captured from multiple cameras according to an embodiment.

Referring to FIG. 3, the method for generating a moving object trajectory consists of steps processed in time series in the user terminals 10 and 100 or the processor 110 shown in FIGS. 1 and 2. Therefore, even if the content is omitted below, the content described above regarding the user terminals 10 and 100 or the processor 110 shown in FIGS. 1 and 2 can also be applied to the method of generating the moving object trajectory of FIG. 3. .

In step 310, the processor 110 processes the first camera and the second camera that capture the same moving object at different positions, respectively, for the first image captured from the first camera and the second image captured from the second camera. Compute homography matrices.

First, the processor 110 acquires a first image captured from the first camera and a second image captured from the second camera with respect to the first camera and the second camera that capture the same moving object from different positions. However, the processor 110 is not limited to the first camera and the second camera and may acquire a plurality of images from a plurality of cameras. As an example, the processor 110 may acquire at least one additional image taken from at least one additional camera.

Camera refers to any camera that can take images. Single-lens reflex camera, twin-lens reflex camera, range finder camera, digital camera, video camera, movie camera ( Movie camera, view camera, panoramic camera, infrared camera, CCTV, etc. may be included, but are not limited thereto.

An image is taken from a camera and refers to something that includes a moving object. Image formats may include JPEG, JFIF, EXIF, IFF, GIF, BMP, PNG, PPM, PGM, PBM, PNM HEIF, BPG IMG, LIFF, NRRD, PAM, PCS, PGF, SGI, SID, TGA, etc. , but is not limited to this. Some of the images may be of the same land surface. The image may be a video image or a serial image, but is not limited thereto. As an example, part of the first image and part of the second image may be images of the same ground surface. As another example, the image may be a serial image containing the same moving object.

Multiple cameras may photograph the same moving object from different locations. The positions of multiple cameras are not limited. As an example, the multiple cameras may be a first camera and a second camera that photograph the same moving object from different positions.

Multiple cameras may be positioned to capture images of the same ground surface overlapping each other. As an example, a portion of the first image captured by the first camera and a portion of the second image captured by the second camera may be captured so that they overlap the same ground surface. As another example, a portion of a first image captured by a first camera, a portion of a second image captured by a second camera, and a portion of a third image captured by a third camera each touch the same ground surface as the portion of the other image. It may have been filmed so that they overlap.

Then, the processor 110 generates a map image including the earth surface represented by the images. As an example, the processor 110 generates a map image that includes both the ground surface represented by the first image captured by the first camera and the ground surface represented by the second image captured by the second camera.

The earth's surface refers to the surface of the Earth shown in the image and all surfaces that can be represented in the map image.

A map image refers to an image containing a predetermined land surface represented by images taken from multiple cameras. The map image may be generated by the processor 110, acquired from an external device (not shown), or obtained from memory. There are no restrictions on the format or creation method of the map image. As an example, the map image may be a bird's eye view image. A bird's eye view image refers to a graphic representation of an image looking down at the ground from a high place, like a bird looking down from the sky. Specifically, it generally refers to all images drawn from a high point looking down. As an example, a map image may represent roads, crosswalks, walks, pedestrians, cars, etc. that appear in the image, but is not limited thereto.

Hereinafter, with reference to FIG. 4 , an example in which the processor 110 acquires an image captured from a camera and generates a map image including the ground surface represented by the image will be described.

FIG. 4 is a diagram illustrating an example in which a processor according to an embodiment generates a map image including images acquired from a camera and the ground surface represented by the images.

Referring to FIG. 4, the processor 110 acquires an image of the ABC intersection captured by a camera and generates a map image including the ABC intersection. Specifically, the processor 110 acquires an image showing a road in one direction among the intersections, a crosswalk, a road line, etc. Then, the processor 110 generates a bird's eye view image, which is a map image expressing roads, crosswalks, road lines, etc. shown in the image. Here, roads, crosswalks, road lines, etc. expressed in images and map images all correspond to different compositions of the same object.

Referring again to FIG. 3, in step 310, the processor 110 specifies a plurality of pairs of corresponding points by matching coordinates on the images with coordinates on the map image using location information on the ground surface. The corresponding point pair consists of a corresponding point on the image side corresponding to coordinates on the image and a corresponding point on the map image side corresponding to coordinates on the map image. As an example, the processor 110 retrieves location information on the ground surface corresponding to a road line and specifies pairs of corresponding points by matching coordinates on each image with coordinates on the map image. Corresponding points corresponding to coordinates on the image and coordinates on the map image indicate the location of the same ground surface corresponding to the road line.

Location information on the earth's surface refers to information on the actual location of objects shown in images and map images. Images and map images express different compositions for the same location, but each can be expressed by specifying pairs of corresponding points using location information on the same ground surface.

Hereinafter, with reference to FIG. 5, an example will be described in which the processor 110 designates a plurality of pairs of corresponding points by matching coordinates on images with coordinates on a map image using location information on the ground surface.

FIG. 5 is a diagram illustrating an example in which the processor 110 designates a plurality of pairs of corresponding points using location information on the ground surface, according to an embodiment.

Referring to FIG. 5, the processor 110 specifies a plurality of pairs of corresponding points by matching coordinates on the image with coordinates on the map image using location information of the crosswalk, road line, and center line. Specifically, the processor 110 uses the location information of the center line to match the coordinates (A1) on the image for the same place and the coordinates (A2) on the map image to designate corresponding point pairs. Then, the processor 110 uses the location information of the crosswalk to designate corresponding points by matching the coordinates (B1) on the image for the same place with the coordinates (B2) on the map image. Likewise, the processor 110 uses the location information of the road line to designate corresponding points by matching the coordinates (C1, D1) on the image for the same place with the coordinates (C2, D2) on the map image.

The processor 110 designates at least one corresponding point pair to each image and map image. As an example, the processor 110 uses the location information of the center line to create a corresponding point pair (A1, A2) consisting of corresponding points that correspond to the coordinates (A1) on the image for the same place and the coordinates (A2) on the map image. Specify. As another example, the processor 110 uses location information on the earth's surface to calculate coordinates on the image (A1) and coordinates on the map image (A2), coordinates on the image (B1), and coordinates on the map image (B2) for the same four places. , Corresponding point pairs ((A1, A2), (B1) consisting of corresponding points that correspond to the coordinates (C1) on the image and the coordinates (C2) on the map image, and the coordinates (D1) on the image and the coordinates (D2) on the map image. , B2), (C1, C2), (D1, D2)).

Referring again to FIG. 3, in step 310, the processor 110 calculates homography matrices for each of the first image and the second image using a plurality of corresponding point pairs. However, the processor 110 is not limited to the first image and the second image, and may calculate homography matrices for each of a plurality of images. As an example, processor 110 may calculate a homography matrix for additional images.

A homography matrix refers to a matrix that represents the relationship between images obtained when looking at a plane in three-dimensional space from different viewpoints. As an example, the processor 110 may calculate the first homography matrix using the coordinates on the first image captured by the first camera and the coordinates on the map image.

As an example, when the first corresponding point pairs and the second corresponding point pairs share a corresponding point on the map image, the processor 110 generates the first image and the second corresponding point pair based on the first corresponding point pairs and the second corresponding point pairs. Homography matrices for each image can be calculated. Here, the first image is associated with first pairs of corresponding points, and the second image is associated with pairs of second corresponding points.

As another example, when the first corresponding point pair and the second corresponding point pair do not share the same corresponding point on the map image side, the processor 110 generates the first image and the second image based on the first corresponding point pair and the second corresponding point pair. Homography matrices for each can be calculated. Here, the first image is associated with a first corresponding point pair, and the second image is associated with a second corresponding point pair.

Hereinafter, with reference to FIGS. 6A and 6B, an example in which the processor 110 calculates a homography matrix for an image using a plurality of pairs of corresponding points will be described.

FIG. 6A is a diagram illustrating an example in which a processor calculates homography matrices when a plurality of pairs of corresponding points share corresponding points on a map image, according to an embodiment.

FIG. 6B is a diagram illustrating an example in which a processor calculates homography matrices when a plurality of pairs of corresponding points do not share corresponding points on the map image, according to an embodiment.

Referring to FIG. 6A, the processor 110 stores coordinates (A1, A3, B1, B3, C1, C3, D1, D3) on the image for the ABC intersection and coordinates (A2, B2, C2, D2) on the map image. ), each corresponding point pair is designated based on, and the homography matrices (H1, H2) are calculated using the corresponding point pairs. Specifically, the processor 110 uses corresponding point pairs of (A1, A2), (B1, B2), (C1, C2), and (D1, D2) when calculating the homography matrix H1 for the first image. . In addition, the processor 110 uses corresponding point pairs of (A3, A2), (B3, B2), (C3, C2), and (D3, D2) when calculating the homography matrix H2 for the second image. Here, corresponding points A2, B2, C2, and D2 correspond to corresponding points on the map image side shared by a plurality of corresponding point pairs. Sharing corresponding points on the map image side means that multiple pairs of corresponding points consist of corresponding points on the same map image side. That is, the corresponding point pair (A1, A2) associated with the first image and the corresponding point pair (A3, A2) associated with the second image share the corresponding point A2 on the map image side. The processor 110 calculates homography matrices (H1, H2) for each image by using a plurality of pairs of corresponding points that share corresponding points (A2, B2, C2, D2) on the map image side. Therefore, the corresponding points of (A1, A2, A3), (B1, B2, B3), (C1, C2, C3), and (D1, D2, D3) express the same location on the earth's surface.

Referring to FIG. 6B, the processor 110 stores coordinates (A1, A3, B1, B3, C1, C3, D1, D3) on the image for the ABC intersection and coordinates (A2, A4, B2, B4) on the map image. , C2, C4, D2, D4) are designated as corresponding point pairs, respectively, and homography matrices (H3, H4) are calculated using the corresponding point pairs. Specifically, the processor 110 uses corresponding point pairs of (A1, A2), (B1, B2), (C1, C2), and (D1, D2) when calculating the homography matrix H3 for the first image. . In addition, the processor 110 uses corresponding point pairs of (A3, A4), (B3, B4), (C3, C4), and (D3, D4) when calculating the homography matrix H4 for the second image. That is, the processor 110 calculates homography matrices (H3, H4) for each image by using a plurality of pairs of corresponding points that do not share the same corresponding points (A2, B2, C2, D2) on the map image side. . Unlike Figure 6a, each of (A1, A3), (B1, B3), (C1, C3), and (D1, D3) does not represent a location on the same ground surface.

The homography matrix calculated here is used when obtaining the area and unit trajectory corresponding to the image on the map image, and the specific details will be described below.

Referring again to FIG. 3, in step 320, the processor 110 performs perspective conversion on the first image and the second image using homography matrices on the map image, thereby driving the first area corresponding to the first image. A first unit trajectory of the moving object and a second unit trajectory of the moving object traveling in the second area corresponding to the second image are acquired.

First, the processor 110 designates a first area corresponding to the first image and a second area corresponding to the second image on the map image. The method of designating a plurality of areas corresponding to a plurality of images on a map image can be performed by a normal method and is not limited. As an example, the processor 110 converts the first image taken on the north side of the intersection with the first camera into perspective using a homography matrix for the first image, thereby creating a first area corresponding to the first image on the map image. Specify . Specifically, the processor 110 specifies a first area corresponding to the first image on the map image by performing perspective transformation on the four corner coordinates of the first image using a homography matrix for the first image. The first area designated on the map image represents the area north of the intersection. The processor 110 is not limited to the first image and the second image, and can designate a plurality of areas on the map image by performing perspective conversion on the plurality of images. The processor 110 may designate an additional area corresponding to the additional image on the map image by performing perspective conversion on the additional image using a homography matrix.

Perspective transformation refers to converting coordinates on an arbitrary image to coordinates on a map image using a homography matrix calculated using coordinates on the image and coordinates on the map image. As an example, the processor 110 may convert coordinates present in the first image into coordinates on the map image using a homography matrix calculated using pairs of corresponding points specified in the first image and the map image.

Then, the processor 110 performs perspective conversion on the first image and the second image using homography matrices, thereby creating a first unit trajectory of the moving object traveling in the first area and a second unit trajectory of the moving object traveling in the second area. obtain. As an example, the processor 110 may obtain the first unit trajectory of the moving object traveling in the first area by perspective converting the coordinates of the moving object included in the first image using the homography matrix for the first image. there is. Additionally, the processor 110 can obtain a second unit trajectory of the moving object traveling in the second area by performing perspective transformation on the coordinates of the moving object included in the second image using the homography matrix for the second image. However, the processor 110 is not limited to the first unit trajectory and the second unit trajectory, and may acquire a plurality of unit trajectories. As an example, the processor 110 may obtain an additional unit trajectory of an additional moving object.

A unit trajectory refers to a trajectory created by perspective converting the coordinates of a moving object shown in each image into coordinates on a map image. The processor 110 combines unit trajectories to generate a moving object trajectory on the map image. The term unit trajectory is used to distinguish it from the ultimately generated moving object trajectory. As an example, the processor 110 may acquire the unit trajectory of the moving object in a time sequence on the map image as the moving object moves in real time. As another example, the processor 110 may obtain an image or image containing the movement of a moving object for a certain period of time and obtain a unit trajectory of the moving object according to the movement of the moving object on a map image.

Since the processor 110 generates a unit trajectory for a moving object starting from one coordinate on the map image and moving to another coordinate as a destination, the acquisition order of the unit trajectory varies depending on the moving direction of the moving object. In response to the moving direction of the moving object, there is a unit trajectory that is generated first, and there is a unit trajectory that is generated later with a time difference than the unit trajectory that is generated earlier. Specifically, when the moving object moves from the first area to the second area, the first unit trajectory is first generated in the first area that does not overlap with the second area. Then, a second unit trajectory is generated from an overlapping area where the first area and the second area overlap. That is, when the moving object moves from the first area to the second area, the first unit trajectory is acquired before the second unit trajectory.

Hereinafter, with reference to FIG. 7, an example in which the processor 110 performs perspective conversion on an image using a homography matrix to designate an area corresponding to the image on a map image and obtain a unit trajectory of a moving object traveling in the area. Explain.

FIG. 7 is a diagram illustrating an example in which the processor 110 according to an embodiment acquires a unit trajectory in an area corresponding to an image using a homography matrix (H).

Referring to FIG. 7, the processor 110 designates an area M corresponding to the image on the map image. Then, the processor 110 uses the coordinates (J1, J2, J3, J4, J5, J6) on the image of the moving object using the homography matrix (H) for the image to coordinates (K1, K2, By performing perspective transformation with K3, K4, K5, K6), the unit trajectory of the moving object is obtained on the map image. The processor 110 may obtain coordinates (J1, J2, J3, J4, J5, J6) on the image of the moving object from a plurality of images, but is not limited thereto. In Figure 7, for convenience, 6 coordinates on the image of the moving object and 6 coordinates on the map image are expressed, but the number of coordinates that can be perspective converted is not limited.

Referring again to FIG. 3, in step 330, the processor 110 combines a portion of the first unit trajectory and a portion of the second unit trajectory included in an overlapping region where the first region and the second region overlap to create a combined trajectory. Create. Here, the combined trajectory refers to a trajectory that combines at least a portion of each unit trajectory included in an overlapping region where each region corresponding to a plurality of images overlaps into one.

As an example, when pairs of corresponding points share corresponding points on the map image side, the processor 110 combines unit trajectories by using a line segment created by connecting a plurality of shared corresponding points on the map image side. Specifically, unit trajectories are combined based on intersection points created by the intersection of line segments created by connecting a plurality of corresponding points and unit trajectories. The processor 110 creates line segments by connecting a plurality of shared corresponding points on the map image, and creates intersection points by intersecting the line segments with unit trajectories. Then, the processor 110 combines unit trajectories by using line segments created by connecting a plurality of shared corresponding points on the map image. As an example, the processor 110 combines unit trajectories based on intersection points created by intersecting unit trajectories with line segments created by connecting a plurality of shared corresponding points on the map image. As another example, the processor 110 combines unit trajectories by including line segments created by connecting intersections as part of a combined trajectory or using them as a baseline for generating a combined trajectory. As another example, the processor 110 generates a combined trajectory through a smoothing operation based on intersection points. The smoothing operation can be performed in a conventional manner and is not limited. To combine unit trajectories, the processor 110 may designate a line segment created by connecting a plurality of corresponding points to intersect the unit trajectories. When obtaining unit trajectories by performing perspective conversion on the same moving object from different images, even if the unit trajectories are for the same moving object, they do not perfectly match on the map image and may have certain errors. In the step of acquiring a unit trajectory, the present invention calculates a homography matrix using pairs of corresponding points sharing the same corresponding point on the map image, obtains a unit trajectory using the calculated homography matrix, and obtains a unit trajectory using a plurality of shared Since a combined trajectory is created using line segments created by connecting corresponding points, it has the effect of reducing errors in unit trajectories.

As another example, the processor 110 may consider the order in which unit trajectories are acquired on a map image and combine unit trajectories based on the unit trajectory obtained first. Specifically, the processor 110 may generate a combined trajectory for a plurality of unit trajectories existing in an overlapping area by maintaining the unit trajectory created first and removing the remaining unit trajectories from the overlapping area.

As another example, the processor 110 may combine unit trajectories of three or more unit trajectories existing in an overlapping area based on the unit trajectory closest to the midpoint of the outermost trajectories. As an example, for three or more unit trajectories existing in an overlapping area, the processor 110 maintains the unit trajectory closest to the coordinate having the average value of the coordinates of each moving object at the same point in time, and stores the remaining unit trajectories. By removing overlapping regions, a combined trajectory can be generated. As another example, for three or more unit trajectories existing in an overlapping area, the processor 110 maintains the unit trajectory closest to the coordinate having the average value of the coordinates of each moving object of the outermost trajectories at the same point in time, , a combined trajectory can be created by removing the remaining unit trajectories from the overlapping area.

However, the processor 110 is not limited to the first unit trajectory and the second unit trajectory, and may generate a combined trajectory by combining three or more unit trajectories. As an example, the processor 110 may generate a combined trajectory by combining a part of the first unit trajectory, a part of the second unit trajectory, and a part of the third unit trajectory.

The processor 110 selects a plurality of candidate unit trajectories for generating a combined trajectory among the unit trajectories using the Euclidean distance for three or more unit trajectories. As an example, the processor 110 may select a plurality of unit trajectories whose Euclidean distance is within a certain distance from three or more unit trajectories as candidate unit trajectories, but is not limited to this. A certain distance may be determined by considering the speed of the moving object, the number of cameras, etc.

When there are three or more candidate unit trajectories, the processor 110 combines the candidate unit trajectories based on the candidate unit trajectory closest to the midpoint of the outermost trajectories among the candidate unit trajectories. As an example, if there are three or more candidate unit trajectories, the processor 110 may combine the candidate unit trajectories based on the candidate unit trajectory closest to the midpoint of the outermost trajectories at a certain point in time, but is not limited to this. No.

Then, the processor 110 determines a moving object trajectory based on the first unit trajectory, the second unit trajectory, and the combined trajectory. As an example, the processor 110 determines the moving object trajectory by smoothing trajectories based on the first unit trajectory, the second unit trajectory, and the combined trajectory. The smoothing operation can be performed by conventional methods and is not limited. Details of generating a combined trajectory for the same moving object and determining the moving object trajectory based on a plurality of unit trajectories and the combined trajectory will be described below with reference to FIGS. 8 to 18.

The moving object trajectory refers to a single moving object trajectory determined based on the combined trajectory in the overlapping area and each unit trajectory included in the non-overlapping area. It has the effect of providing a single moving object trajectory that comprehensively tracks the trajectory of a moving object existing in a single space captured by multiple cameras. Additionally, for spaces that cannot be tracked with a single camera, the use of multiple cameras has the effect of providing the trajectory of a single moving object without spatial constraints.

Hereinafter, with reference to FIGS. 8 to 18, the processor 110 generates a combined trajectory by combining parts of a plurality of unit trajectories included in an overlapping region where a plurality of regions overlap, and adds a combined trajectory to the plurality of unit trajectories and the combined trajectory. An example of determining the based moving object trajectory will be explained. FIGS. 8 to 18 are diagrams for explaining an example of a map image generated by the processor 110 according to an embodiment of the present invention by the method described above.

First, with reference to FIGS. 8 to 11, the processor 110 combines a portion of the first unit trajectory and a portion of the second unit trajectory included in an overlapping region where the first region and the second region overlap to create a combined trajectory. An example of generating and determining a moving object trajectory based on the first unit trajectory, the second unit trajectory, and the combined trajectory will be described.

FIG. 8 is a diagram illustrating an example in which the processor 110 acquires a first unit trajectory and a second unit trajectory existing in a first area and a second area, according to an embodiment.

The processor 110 designates a first area corresponding to the first image capturing the western part of the intersection and a second area corresponding to the second image capturing the southern part of the intersection on the map image. The first area is composed of an area M1 in which only a portion of the first unit locus L1 exists and an overlapping area M3 in which a portion of the first unit locus L1 and a portion of the second unit locus L2 exist. The second area is composed of an area M2 in which only a portion of the second unit locus L2 exists and an overlapping area M3 in which a portion of the first unit locus L1 and a portion of the second unit locus L2 exist.

Then, the processor 110 acquires the first unit trajectory L1 existing in the first area and the second unit trajectory L2 existing in the second area. At this time, a part of the first unit trajectory L1 and a part of the second unit trajectory L2 coexist in the overlapping area M3 where the first area and the second area overlap. For this reason, in order to generate a single moving object trajectory, a step of combining the first unit trajectory (L1) and the second unit trajectory (L2) in the overlapping area (M3) is required. The specific details of the combining step will be explained in Figure 9 below.

FIG. 9 is a diagram illustrating an example in which the processor 110 according to an embodiment generates a combined trajectory by combining a part of the first unit trajectory and a part of the second unit trajectory in the overlapping area M3.

Referring to FIG. 9, the processor 110 combines a portion of the first unit trajectory L1 and a portion of the second unit trajectory L2 existing in the overlapping area M3. Specifically, the processor 110 maintains the first unit trajectory (L1) generated earlier on the map image in the overlap area (M3), and the second unit trajectory (L2) generated later on the map image is maintained in the overlap area (M3). Create a combined trajectory by removing it from M3).

FIG. 10 is a diagram illustrating an example in which the processor 110 selects a first unit trajectory and a second unit trajectory from a non-overlapping area, according to an embodiment.

Referring to FIG. 10, the processor 110 selects the first unit trajectory L1 in the first area M1, which is not an overlapping area, and selects the second unit trajectory L2 in the second area M2, which is not an overlapping area. ).

FIG. 11 is a diagram illustrating an example in which the processor 110 according to an embodiment determines a moving object trajectory based on the combined trajectory generated in FIG. 9 and the first and second unit trajectories selected in FIG. 10. .

The processor 110 synthesizes the first unit trajectory in the first non-overlapping area (M1), the second unit trajectory in the second non-overlapping area (M2), and the combined trajectory in the overlapping area (M3). This creates a moving object trajectory. The processor 110 generates a trajectory of a single moving object existing in a single space captured by multiple cameras.

FIG. 12 is a diagram illustrating another example in which a processor according to an embodiment generates a combined trajectory by combining a part of a first unit trajectory and a part of a second unit trajectory in an overlapping area.

The processor 110 uses a line segment created by connecting a plurality of shared corresponding points (D1, D2) and intersection points (E1, E2) created by intersecting the first unit trajectory (L1) and the second unit trajectory (L2). A part of the first unit trajectory (L1) and a part of the second unit trajectory (L2) can be combined. The present invention has the effect of reducing the error range of a plurality of unit trajectories obtained by performing perspective conversion from images taken from multiple cameras.

And, with reference to FIGS. 13 to 18, the processor 110 processes a portion of the first unit trajectory, a portion of the second unit trajectory, and the first unit trajectory included in an overlapping region where the first region, the second region, and the third region overlap. An example of combining parts of three unit trajectories to generate a combined trajectory and determining a moving object trajectory based on the first unit trajectory, second unit trajectory, third unit trajectory, and combined trajectory will be described.

FIG. 13 shows the processor 110 according to an embodiment acquiring a first unit trajectory, a second unit trajectory, and a third unit trajectory included in an overlapping area where the first area, the second area, and the third area overlap. This is a drawing to explain an example.

The processor 110 includes a first area corresponding to the first image capturing the western portion of the intersection, a second area corresponding to the second image capturing the southern portion of the intersection, and a third image capturing the northeastern portion of the intersection. A corresponding third area is designated on the map image.

And, the processor 110 acquires the first unit trajectory (L1) existing in the first area, the second unit trajectory (L2) existing in the second area, and the third unit trajectory (L3) existing in the third area. do. At this time, a part of the first unit trajectory L1 and a part of the third unit trajectory L3 coexist in the overlapping area M4 where the first area and the third area overlap. And, in the overlapping area M3 where the first area, the second area, and the third area overlap, a portion of the first unit locus L1, a portion of the second unit locus L2, and a third unit locus L3 are included. Some coexist. Likewise, a part of the second unit locus L2 and a part of the third unit locus L3 coexist in the overlapping area M5 where the second area and the third area overlap. In order to generate a single moving object trajectory, a step of combining the first unit trajectory, the second unit trajectory, and the third unit trajectory in the overlapping areas M3, M4, and M5 is required. The specific details of the combining step will be explained in Figures 14 to 16 below.

FIG. 14 is a diagram illustrating an example in which the processor 110 according to an embodiment generates a combination trajectory in an area where the first area and the third area overlap.

The processor 110 generates a combined trajectory by combining a part of the first unit trajectory and a part of the third unit trajectory in the overlapping area M4 where the first area and the third area overlap. Specifically, the processor 110 maintains a part of the first unit trajectory (L1) generated first on the map image among the first unit trajectory (L1) and the third unit trajectory (L3) in the overlap area (M4), A combined trajectory is generated by removing part of the third unit trajectory (L3) from the overlapping area (M4).

FIG. 15 is a diagram illustrating an example in which the processor 110 according to an embodiment generates a combination trajectory in an area M3 where the first area, the second area, and the third area overlap.

Referring to FIG. 15, the processor 110 stores a portion of the first unit locus L1 and a portion of the second unit locus L2 in the overlapping area M3 where the first area, the second area, and the third area overlap. and combining a part of the third unit trajectory (L3) to generate a combined trajectory. Specifically, the processor 110 maintains a part of the first unit trajectory L1, which is the unit trajectory closest to the coordinate having the average value of the coordinates of each moving object, with respect to the unit trajectories existing in the overlapping area M3, , A combined trajectory is created by removing some of the remaining unit trajectories from the overlapping area (M3).

FIG. 16 is a diagram illustrating an example in which the processor 110 according to an embodiment generates a combination trajectory in the area M5 where the second area and the third area overlap.

Referring to FIG. 16, the processor 110 combines a part of the second unit locus L2 and a part of the third unit locus L3 in the overlapping area M5 where the second area and the third area overlap. Create a trajectory. Specifically, the processor 110 selects a part of the third unit trajectory (L3) generated first on the map image among a part of the second unit trajectory (L2) and a part of the third unit trajectory (L3) as an overlap area (M5). and a part of the second unit trajectory (L2) is removed from the overlapping area (M5) to create a combined trajectory.

FIG. 17 is a diagram illustrating an example in which the processor 110 selects a first unit trajectory and a second unit trajectory from a non-overlapping area, according to an embodiment.

Referring to FIG. 17, the processor 110 selects a part of the first unit trajectory L1 in the first area M1, which is not an overlapping area, and selects a part of the second unit trajectory L1 in the second area M2, which is not an overlapping area. Select part of (L2).

FIG. 18 illustrates an example in which the processor 110 according to an embodiment determines a moving object trajectory based on the combined trajectory generated in FIGS. 14 to 16 and the first and second unit trajectories selected in FIG. 17. This is a drawing for

Referring to FIG. 18, the processor 110 operates on a first unit trajectory in the first area M1 that is not an overlapping area, a second unit trajectory in a second area M2 that is not an overlapping area, and overlapping areas M3. , M4, and M5), the combined trajectories are synthesized to generate the moving object trajectory. The processor 110 generates a trajectory of a single moving object existing in a single space captured by multiple cameras.

As described above, the processor 110 generates a moving object trajectory using multiple cameras. The user 30 can check the trajectory of a moving object without space constraints by using multiple cameras in a space that cannot be tracked with a single camera.

Meanwhile, the above-described method can be written as a program that can be executed on a computer, and can be implemented in a general-purpose digital computer that operates the program using a computer-readable recording medium. Additionally, the data structure used in the above-described method can be recorded on a computer-readable recording medium through various means. The computer-readable recording media includes storage media such as magnetic storage media (e.g., ROM, RAM, USB, floppy disk, hard disk, etc.) and optical read media (e.g., CD-ROM, DVD, etc.) do.

Those skilled in the art related to the present embodiment will understand that the above-described substrate can be implemented in a modified form without departing from the essential characteristics. Therefore, the disclosed methods should be considered from an explanatory rather than a limiting perspective, and the scope of rights is indicated in the claims, not the foregoing description, and should be interpreted to include all differences within the equivalent scope.

Claims (10)

In a method of generating a moving object trajectory using multiple cameras,
For a first camera and a second camera that capture the same moving object at different positions, a homography matrix for each of the first image captured by the first camera and the second image captured by the second camera. calculating them;
By perspective converting the first image and the second image using the homography matrices on a map image, the first unit trajectory and the second image of the moving object traveling in the first area corresponding to the first image acquiring a second unit trajectory of the moving object traveling in a second area corresponding to; and
Generating a combined trajectory for the same moving object by combining at least a part of the first unit trajectory and at least a part of the second unit trajectory included in an overlapping area where the first area and the second area overlap; comprising: do,
The step of generating the binding trajectory is,
When each of the calculated homography matrices is calculated by sharing the same corresponding points on the map image, the unit trajectories intersect with a line segment created by connecting a plurality of shared corresponding points on the map image. A method of generating moving object trajectories, including the step of combining unit trajectories. According to claim 1,
A method for generating a moving object trajectory, further comprising: determining a moving object trajectory based on the first unit trajectory, the second unit trajectory, and the combined trajectory. According to claim 1,
A method of generating a moving object trajectory, characterized in that some of the images are of the same ground surface. delete delete delete delete delete A computer-readable recording medium recording a program for executing the method according to claim 1 on a computer. at least one memory; and
At least one processor;
The at least one processor,
For a first camera and a second camera that capture the same moving object at different positions, a homography matrix for each of the first image captured by the first camera and the second image captured by the second camera. calculate them,
By perspective converting the first image and the second image using the homography matrices on a map image, the first unit trajectory and the second image of the moving object traveling in the first area corresponding to the first image Obtaining a second unit trajectory of the moving object traveling in the second area corresponding to
Generating a combined trajectory for the same moving object by combining at least a part of the first unit trajectory and at least a part of the second unit trajectory included in an overlapping area where the first area and the second area overlap,
Creating the binding trajectory is,
When each of the calculated homography matrices is calculated by sharing the same corresponding points on the map image, the unit trajectories intersect with a line segment created by connecting a plurality of shared corresponding points on the map image. A computing device that combines unit trajectories.

Images