KR102227562B1 - System and method for counting floating population based on image - Google Patents

Abstract

An image-based floating population measurement system is disclosed. The measurement system includes an image capturing unit including a plurality of cameras each collecting an image for a corresponding region, a panoramic image generating unit and an object recognition method for generating a single panoramic image from the images collected by the image capturing unit. And a floating population counting unit that counts the floating population from the panoramic image generated by the panoramic image generating unit by using.

Description

Image-based floating population measurement system and method {SYSTEM AND METHOD FOR COUNTING FLOATING POPULATION BASED ON IMAGE}

An embodiment according to the concept of the present invention relates to an image-based floating population measurement technology, and more particularly, it is possible to measure the floating population by generating a panoramic image from a plurality of images. It relates to an image-based floating population measurement technology capable of accurately measuring the floating population by preventing duplicate population counts by matching with each other.

Outdoor advertising refers to advertisements posted and installed outdoors for an unspecified number of people. According to the recent development of electronic and communication technologies, various information about products or services to consumers, such as digital signage. And digital outdoor advertisements capable of outputting advertisement data are receiving great attention. Accordingly, information on how much the floating population is in the point or area where the outdoor billboard will be installed is required, but the conventional floating population measurement was measured by a hand counting method by the National Statistical Office, a national agency, every four years. The floating population data was data from the past, and there was a problem that it did not accurately reflect the current situation. Recently, a technology that can acquire floating population information by detecting the location of sample targets in real time using the GPS of a smartphone has appeared, but the GPS error of about 30m is the floating population information for a specific area where the outdoor billboard will be installed. For example, it may be inaccurate information, and there is a problem that the floating population with mobile phones that do not have a GPS function is not reflected at all.

Patent Publication 10-2015-0121437

The technical problem to be solved by the present invention is an image-based floating population capable of counting the floating population by generating a panoramic image and counting the floating population through image matching when it is difficult to generate a panoramic image. It is to provide a counting system.

Another technical problem to be solved by the present invention is to create a panoramic image to count the floating population, and when it is difficult to generate a panoramic image, image-based flow that can accurately count the floating population by removing duplicate populations through image matching. It provides a population counting method.

An image-based floating population counting system according to an embodiment of the present invention includes an image photographing unit composed of a plurality of cameras each collecting an image for a corresponding region, and a single panoramic image from images collected by the image photographing unit. And a floating population counting unit that counts a floating population from the panoramic image generated by the panoramic image generation unit by using a panorama image generation unit for generating a and an object recognition method.

The panoramic image generation unit extracts feature points having the same characteristics from each of the images collected by the image capturing unit using a local feature detection method, and uses an affine transform or a perspective transform. The panoramic image may be generated by arranging the images so that the extracted feature points coincide by using.

According to an embodiment, the image-based floating population counting system according to an embodiment of the present invention further includes a feature point display unit implemented as a structure installed in the corresponding area or implemented as a pattern coated on a structure existing in the corresponding area. can do.

According to an embodiment, the image-based floating population counting system according to an embodiment of the present invention may further include an image matching processor configured to match the collected images to generate a corrected image, and the image matching processor When the panorama image generator fails to generate the panorama image, the correction image is generated, and the floating population counting unit counts the floating population from the correction image generated by the image matching processing unit.

At this time, the image matching processor calculates the Euclidean distance between the columns of the collected images to determine the similarity between the columns, and matches the images based on the determined columns with the largest similarity. Thus, it may be characterized in that the correction image is generated.

In addition, the image matching processor may be characterized in that the collected images are pre-processed by max pooling, and the pre-processed images are image-matched to generate the corrected image.

The floating population counting unit counts, as the floating population, a sum of the number of floating populations of each of the images included in the corrected image minus the number of floating populations present in an area where the images included in the corrected image overlap each other. I can.

In an image-based floating population counting method according to an embodiment of the present invention, an image capture unit transmits images of a corresponding region collected by a plurality of cameras to a panorama image generation unit, and the panorama image generation unit from the transmitted images. Generating a single panoramic image and transmitting it to a floating population counting unit; transmitting the transmitted images to an image matching processing unit by the panoramic image generation unit; and generating a correction image from the transmitted images by the image matching processing unit; And transmitting to the floating population counting unit and counting the floating population from the transmitted panoramic image or the corrected image by using an object recognition method by the floating population counting unit.

In this case, the step of generating the panoramic image by the panoramic image generation unit and transmitting it to the floating population counting unit includes extracting feature points having the same characteristics from the transmitted images using a local feature detection method, and affine transformation or projection. Aligning the transmitted images to match the extracted feature points using a transformation method, generating the aligned images as the panoramic image, and transmitting the generated panoramic image to the floating population counting unit. Can include.

In addition, the step of generating the corrected image by the image matching processing unit and transmitting it to the floating population counting unit includes pre-processing the transmitted images by max-pooling and calculating a Euclidean distance between each column of the pre-processed images. Determining the similarity between the columns according to the calculated Euclidean distance between the columns, and generating the corrected image by matching the preprocessed images based on the determined columns with the largest similarity, and the generation And transmitting the corrected image to the population counting unit.

In addition, the step of counting the floating population from the corrected image by the floating population counting unit includes counting the number of floating populations of each of the images included in the corrected image, and in an area overlapping each other with images included in the corrected image. Counting the number of floating populations present, and counting a value obtained by subtracting the number of floating populations existing in the counted overlapping area from the sum of the number of floating populations of each of the counted images as the floating population.

As described above, the image-based floating population counting system and method according to an embodiment of the present invention generates a panoramic image through a plurality of cameras, thereby counting the floating population over a wide area far exceeding the coverage of a single camera. It works.

In addition, it is possible to extract the feature points through the installation of the feature point display unit even for topography and regional characteristics in which it is difficult to generate a panoramic image, so that it is possible to easily generate a panoramic image.

Furthermore, even when a panoramic image cannot be generated, it is possible to accurately count the floating population through matching between images captured by a plurality of cameras.

A detailed description of each drawing is provided in order to more fully understand the drawings cited in the detailed description of the present invention.
1 is a block diagram showing the configuration of an image-based floating population counting system according to an embodiment of the present invention.
FIG. 2 is an exemplary diagram illustrating an image of a specific area captured by the image photographing unit illustrated in FIG. 1.
3 is an exemplary diagram illustrating a panorama image generated by the panorama image generator shown in FIG. 1.
4 is an exemplary diagram illustrating feature points extracted from an image collected by an image capture unit according to an embodiment of the present invention.
4 is a diagram illustrating an example of extracting a feature point from an image collected by a camera of an image capturing unit according to an embodiment of the present invention.
5 is a diagram for explaining an example in which a panorama image generator according to an embodiment of the present invention generates a panorama image using extracted feature points.
6 is a diagram illustrating an example in which it is difficult to extract feature points from a corresponding image.
7 is a diagram illustrating an example in which a panoramic image generator according to an embodiment of the present invention extracts a feature point through a feature point display unit.
8 is a diagram illustrating an example in which a panorama image generator according to an embodiment of the present invention generates a panorama image using feature points extracted through a feature point display unit.
9 is an exemplary diagram showing an example in which images are horizontally, vertically, and rotated.
10 is an exemplary diagram illustrating a method of matching images by an image matching processor according to an embodiment of the present invention.
11 is an exemplary diagram illustrating a result of matching images according to similarity by an image matching processor according to an embodiment of the present invention.
12 is a flowchart illustrating an image-based floating population counting method according to an embodiment of the present invention.
13 is a flowchart illustrating a process of generating a panorama image by a panorama image generator according to an embodiment of the present invention.
14 is a flowchart illustrating a process of generating a corrected image by an image matching processor according to an embodiment of the present invention.

Specific structural or functional descriptions of the embodiments according to the concept of the present invention disclosed in the present specification are exemplified only for the purpose of describing the embodiments according to the concept of the present invention, and the embodiments according to the concept of the present invention are It may be implemented in various forms and is not limited to the embodiments described herein.

Since the embodiments according to the concept of the present invention can apply various changes and have various forms, the embodiments will be illustrated in the drawings and described in detail in the present specification. However, this is not intended to limit the embodiments according to the concept of the present invention to specific disclosed forms, and includes all changes, equivalents, or substitutes included in the spirit and scope of the present invention.

Terms such as first or second may be used to describe various constituent elements, but the constituent elements should not be limited by the terms. The terms are only for the purpose of distinguishing one component from other components, for example, without departing from the scope of the rights according to the concept of the present invention, the first component may be referred to as the second component, and similarly The second component may also be referred to as a first component.

When a component is referred to as being "connected" or "connected" to another component, it is understood that it may be directly connected or connected to the other component, but other components may exist in the middle. It should be. On the other hand, when a component is referred to as being "directly connected" or "directly connected" to another component, it should be understood that there is no other component in the middle. Other expressions describing the relationship between components, such as "between" and "directly between" or "adjacent to" and "directly adjacent to" should be interpreted as well.

The terms used in the present specification are only used to describe specific embodiments, and are not intended to limit the present invention. Singular expressions include plural expressions unless the context clearly indicates otherwise.

In this specification, terms such as "comprises" or "have" are intended to designate the presence of the described feature, number, step, action, component, part, or combination thereof, but one or more other features or numbers. It is to be understood that the possibility of addition or presence of, steps, actions, components, parts, or combinations thereof is not preliminarily excluded.

Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which the present invention belongs.

Terms such as those defined in commonly used dictionaries should be interpreted as including meanings consistent with the meanings in the context of the related technology, and should be interpreted as ideal or excessively formal meanings unless explicitly defined in the present specification. It doesn't work.

Hereinafter, the present invention will be described in detail by describing a preferred embodiment of the present invention with reference to the accompanying drawings.

1 is a block diagram showing the configuration of an image-based floating population counting system (hereinafter referred to as a'floating population counting system') according to an embodiment of the present invention.

Referring to FIG. 1, the floating population counting system 10 can count a floating population in real time in a specific area, such as an area where an outdoor billboard is installed or a specific store, and the image photographing unit 100, It includes a panoramic image generator 200, a feature point display unit 300, a floating population counting unit 400, and an image matching processing unit 500.

First, the image capturing unit 100 includes a plurality of cameras 130-1, 130-2, and 130-3, and each of the plurality of cameras 130-1, 130-2, and 130-3 is outdoor In a specific area where the billboard is installed, an image (ca_img) for the area is collected and transmitted to the panorama image generator 200.

FIG. 2 is an exemplary view showing an image (ca_img) for a specific area captured by the image capturing unit 100 illustrated in FIG. 1.

In this case, ca_img1 in FIG. 2 is a first image in which the first camera 130-1 photographs the first region, ca_img2 is a second image in which the second camera 130-2 photographs the second region, and ca_img3 Is a third image captured by the third camera 130-3.

1 and 2, the first camera 130-1 collects an image (ca_img1) of the first area among a specific area, and the second camera 130-2 collects the image (ca_img1) of the second area. The image ca_img2 is collected, and the third camera 130-3 collects the image ca_img3 for the third area in real time.

In the present specification, the image capture unit 100 is illustrated as including only three cameras 130-1, 130-2, and 130-3, but this is for convenience of description and is not limited thereto, and a single camera is It goes without saying that a larger number of cameras may be applied according to the user's design in order to overcome the range of images that can be collected.

On the other hand, the panoramic image generating unit 200 includes an image (ca_img) transmitted from the image capturing unit 100, that is, a plurality of cameras 130-1, 130-2 and 130-included in the image capturing unit 100. 3) A single panoramic image PA_img is generated from the images ca_img1, ca_img2 and ca_img3 captured and transmitted by each and transmitted to the floating population counting unit 400.

FIG. 3 is an exemplary diagram illustrating a panorama image PA_img1 generated by the panorama image generator 200 illustrated in FIG. 1.

Referring to FIG. 3, the panorama image generator 200 includes a first image (ca_img1) generated by the first camera 130-1, a second image (ca_img2) generated by the second camera 130-2, and A single panoramic image PA_img1 is generated from the third image ca_img3 generated by the third camera 130-3.

That is, the panorama image generator 200 constructs a panorama image PA_img1 from a plurality of camera images ca_img1, ca_img2, and ca_img3, and in this case, the panorama image refers to each of the cameras 130-1, 130-2, and 130- 3) may mean that the individually photographed images (ca_img1, ca_img2, and ca_img3) are configured as one image (PA_img1).

Hereinafter, a specific method for the panoramic image generator 200 to construct a panoramic image will be described in detail.

4 is a view for explaining an example of extracting a keypoint from images (ca_img4, ca_img5) collected by the cameras 130-1 and 130-2 of the image capturing unit 100 according to an embodiment of the present invention. 5 is a diagram for explaining an example in which the panorama image generator 200 generates a panorama image PA_img2 using the extracted feature points according to an embodiment of the present invention.

1, 4, and 5, the panoramic image generator 200 includes feature points (e.g., stars) having the same characteristics in two adjacent images (e.g., ca_img4 and ca_img5) in order to construct a panoramic image (PA_img2). Display), and two images (eg, ca_img4 and ca_img5) are aligned based on this.

At this time, the above feature points are known local features such as Scale-Invariant Feature Transform (SIFT), Speeded Up Robust Feature (SURF), Binary Robust Independent Elementary Features (BRISK), and Fast Retina Keypoint (FREAK) detection. It can be extracted through the method.

In addition, the two images from which the feature points are extracted (ca_img4 and ca_img5) may be aligned through a known method such as an affine transform or a perspective transform, which is a method of matching the extracted feature points.

Eventually, the panorama image generator 200 extracts the feature points from the fourth image (ca_img4) and the fifth image (ca_img5), and based on the extracted feature points, the fourth image (ca_img4) and the fifth image (ca_img5) are A panorama image PA_img2 is formed by arranging the fifth image ca_img5 to the fourth image ca_img4 so that the feature points coincide.

Meanwhile, in relation to the extraction of the feature points, it may be assumed that it is difficult to extract the feature points from the corresponding image.

6 shows an example in which it is difficult to extract feature points from a corresponding image. In this case, ca_img6 in FIG. 6 is a sixth image of a first area collected by the first camera 130-1, and ca_img7 is a second camera 130- This is the 7th image of the second area collected by 2).

6, the sixth image (ca_img6) collected by the first camera (130-1) and the seventh image (ca_img7) collected by the second camera respectively correspond to a sidewalk. If there are no pedestrians on the sidewalk, it becomes very difficult to extract feature points in the image.

In this case, the panorama image generator 200 may fail to extract the feature points from the sixth image ca_img6 and the seventh image ca_img7, and thus, the construction of the panorama image may also fail.

The floating population counting system 10 according to an embodiment of the present invention extracts the feature points through the feature point display unit 300 arranged in a specific area to be photographed, thereby preventing the failure of generating a panorama image due to a failure to extract the feature points. Can be prevented.

7 illustrates an example in which the panoramic image generator 200 extracts a feature point through the feature point display unit 300 according to an embodiment of the present invention.

In this case, (a) of FIG. 7 is a diagram showing an example in which the feature point (star mark) is extracted from the sixth image (ca_img6) taken by the first camera 130-1 through the feature point display unit 300, and FIG. 7B is a diagram showing an example in which the feature point (star mark) is extracted from the seventh image ca_img7 captured by the second camera 130-2 through the feature point display unit 300.

7 illustrates an example in which two feature point display units 300 are installed on an image (a sixth image and a seventh image) for convenience of explanation, but if necessary, such as when the number of images increases or the measurement range of the image increases. It goes without saying that a larger number of feature point display units 300 may be installed in the corresponding area.

Depending on the embodiment, the feature point display unit 300 may be implemented as a structure separately installed in a specific area, and may also be implemented as a pattern coated on a floor or structure in a specific area.

FIG. 8 illustrates an example in which the panorama image generator 200 generates a panorama image PA_img3 using feature points (eg, star marks) extracted through the feature point display unit 300 according to an embodiment of the present invention.

7 and 8, in the floating population counting system 10 according to an embodiment of the present invention, specific areas to be photographed by the image photographing unit 100 are composed of very similar terrain and very similar structures. Even when extraction is difficult, there is an effect of generating a panoramic image PA_img3 by extracting the feature points through the feature point display unit 300.

Referring back to FIG. 1, the population counting unit 400 performs a role of counting the floating population in real time from the panoramic image PA_img transmitted from the panoramic image generating unit 300.

The population counting unit 400 uses a known object recognition method such as Region-based Convolutional Neural Networks (R-CNN), Fast R-CNN, Faster R-CNN, R-FCN, YOLO, SSD, etc. ), you can count the floating population in real time.

According to an embodiment, the floating population counting system 10 is implemented to count the floating population even when the panoramic image generator 300 fails to generate the panoramic image PA_img.

Referring back to FIG. 1, even when the panorama image generation unit 300 fails to generate the panorama image PA_img, the population counting unit 400 is an image other than the panorama image PA_img transmitted from the panorama image generation unit 300. The floating population is counted from the calibration image (calib_img) transmitted from the matching processing unit 500.

As described above, when the population counting unit 400 receives the panoramic image PA_img generated from the panoramic image generating unit 300, the floating population can be counted without the problem of overlapping the floating population in the corresponding panoramic image PA_img. I can.

However, when the panorama image generator 300 fails to generate the panorama image (PA_img), when the population counting unit 400 directly counts the floating population from the images (ca_img) of the image photographing unit 100, the images ( ca_img) There may be a problem of counting due to overlapping floating populations in overlapping areas.

Accordingly, the floating population counting system 10 according to an embodiment of the present invention solves such a problem of overlapping the floating population through image matching of the image matching processing unit 500.

In more detail, the image matching processing unit 500 includes the population counting unit 400 in the sum of the number of floating populations included in one image and the number of floating populations included in another image. Images are matched with each other so that the number of subtracting the number of floating populations existing in the overlapping area in the image can be processed as the total number of floating populations.

In this case, the one image and the other image may be moved only in a horizontal direction to each other, and movement and rotation in not only the horizontal direction but also the vertical direction may occur.

FIG. 9 is an exemplary diagram showing an example in which horizontal, vertical, and rotational images occur, FIG. 9A is an exemplary diagram illustrating a case in which only horizontal movement between images occurs, and FIG. 9B is This is an exemplary diagram showing a case in which the images (Left_img and Right_img) occur horizontally, vertically, and rotated with each other.

The case in which the horizontal movement shown in (a) of FIG. 9 occurs is the simplest case, but in an actual environment, as shown in (b) of FIG. 9, a slight shift in the vertical direction as well as the horizontal direction is performed according to the camera setting. May occur, and the right image (Right_img) may be rotated to some extent compared to the left image (Left_img).

Therefore, the image matching processing unit 500 not only moves horizontally between images, but also moves vertically and rotates images so that the population counting unit 400 accurately counts the floating population in an area where the images (Left_img and Right_img) overlap. The images (Left_img and Right_img) are matched in consideration of.

A method in which the image matching processing unit 500 matches the images Left_img and Right_img in consideration of such a horizontal direction, a vertical direction, and rotation will be described in detail below.

10 is an exemplary diagram illustrating a method of matching images by an image matching processor according to an embodiment of the present invention.

Referring to FIG. 10, the image matching processing unit 500 sequentially starts from the rightmost column CL_1 of the left image (Left_img) to the left column (CL_2 to CL_n), and the similarity with the leftmost column (CR_PL) of the right image (Right_img). Compare

At this time, each column is assumed to have n elements as a vector having a dimension equal to the image height, and each of the columns CL_1 to CL_n of the left image (Left_img) and the leftmost column of the right image (Right_img) The similarity between the columns is compared by calculating the distance of (CR_PL).

According to an embodiment, the image matching processor 500 may obtain a distance between each of the columns by calculating an Euclidean distance or L2 distance.

For example, the image matching processor 500 may calculate the distance D between the fourth column CL_4 of the left image Left_img and the leftmost column CR_PL of the right image Right_img as shown in the following equation. .

At this time, CL_41 refers to the first element of the fourth column (CL_4) of the left image (Left_img), CL_4n refers to the nth element, and CR_PL1 refers to the first element of the leftmost column (CR_PL) of the right image (Right_img). Means, and CR_PLn means the nth element.

The image matching processing unit 500 determines that the similarity between the columns is the largest when the Euclidean distance calculated as described above is the smallest.

11 is an exemplary view showing a result of matching images Left_img and Right_img according to similarity by the image matching processing unit 500 according to an embodiment of the present invention.

1, 10, and 11, the image matching processing unit 500 displays images (Left_img) based on the two columns CL_4 and CR_PL having the highest similarity according to the similarity between the columns obtained as described above. And Right_img) perform mutual matching.

In addition, when the images (Left_img and Right_img) are matched from the image matching processing unit 500, an overlapping area between the images (Left_img and Right_img) appears.

The image matching processing unit 500 transmits the matched images to the population counting unit 400 as a correction image (calib_img) through the above process.

However, as previously mentioned in the description of (b) of FIG. 9, the image matching processing unit 500 not only moves in the horizontal direction but also moves in the vertical direction and rotates the image when searching for an overlapping area between images (Left_img and Right_img). Should be considered.

This is because even slight changes, such as vertical movement and image rotation, may actually show similar columns with low similarity in the similarity calculation.

Accordingly, the image matching processor 500 may perform a predetermined pre-processing process before matching according to the similarity between images, and even if the image (ca_img) is slightly vertically shifted or rotated through the above pre-processing process, Do not make a big impact on the similarity.

According to an embodiment, the predetermined preprocessing process may be performed according to max pooling of a convolutional neural network (CNN).

In general, a method of separating each layer of an input image, resizing it, and then stacking it again is called pooling, and max pooling is a pooling that converts the size based on the concept that the maximum value represents a different value. That's the way.

Since the image matching processing unit 500 preprocesses images (eg, Left_img and Right_img in FIG. 9) using max pooling, even if the image (Right_img in FIG. 9) is slightly vertically shifted or rotated, images (eg, FIG. 9 The similarity between the columns of Left_img and Right_img) does not have a significant influence on the judgment.

Thereafter, the image matching processing unit 500 is based on the two columns having the highest similarity as described in FIGS. 10 and 11 according to the similarity between the columns of the pre-processed images. Perform matching.

Referring back to FIG. 1, the population counting unit 400 counts the correct floating population by excluding the duplicated population from the corrected image (calib_img) transmitted from the image matching processing unit 500.

That is, the population counting unit 400 calculates the number of the total floating population from the sum of the total number of floating populations in a certain image (eg, Left_img in FIG. 11) and the total number of populations in another image (eg, Right_img in FIG. 11). It is processed by subtracting the number of populations existing in the overlapping area (eg, the overlapping area of FIG. 11).

In more detail, when the panorama image generation unit 200 fails to generate the panorama image PA_img, the population counting unit 400 receives a correction image calib_img from the image matching processing unit 500.

In the calibrated image (calib_img), an area where each image (for example, Left_img and Right_img in FIG. 11) overlaps is displayed, and the population counting unit 400 counts the number of floating populations in each image, and flows existing in the area where these images overlap. Count the number of people.

Subsequently, the population counting unit 400 calculates the number of redundantly removed floating populations by subtracting the number of floating populations existing in an area where these images overlap from the sum of the number of floating populations counted in each image (eg, Left_img and Right_img in FIG. 11).

At this time, the population counting unit 400 uses a known object recognition method such as R-CNN, Fast R-CNN, Faster R-CNN, R-FCN, YOLO, SSD, etc. to determine the floating population in real time from the calibrated image (calib_img). You can count.

12 is a flowchart illustrating an image-based floating population counting method (hereinafter referred to as a'floating population counting method') according to an embodiment of the present invention.

1 to 12, the image capturing unit 100 panoramas an image (ca_img) for a corresponding region collected (S100) by each of a plurality of cameras 130-1, 130-2, and 130-3. It is transmitted to the image generating unit 200 (S130).

At this time, the first camera 130-1 collects the image (ca_img1) for the first area, the second camera 130-2 collects the image (ca_img2) for the second area, and the third camera ( 130-3) collects an image (ca_img3) for the third area.

Sequentially, the panoramic image generation unit 200 generates a single panoramic image PA_img from the image ca_img transmitted from the image capturing unit 100 (S200), and calculates the generated panoramic image PA_img. It is transmitted to the unit 400 (S230).

Hereinafter, a method of generating a single panoramic image PA_img by the panoramic image generator 200 will be described in detail.

13 is a flowchart illustrating a process of generating a panorama image PA_img by the panorama image generator 200 according to an embodiment of the present invention.

1 to 13, the panoramic image generation unit 200 extracts keypoints having the same characteristics from images transmitted from the image capturing unit 100 (S150 ).

In this case, the panorama image generator 200 may extract the above feature points using a known local feature detection method such as SIFT, SURF, BRISK, and FREAK.

Sequentially, the panorama image generator 200 arranges the images (eg, ca_img4 and ca_img5 in FIG. 4 or ca_img6 and ca_img7 in FIG. 7) by matching the extracted feature points (S180).

In this case, the panorama image generator 200 may align the images using an affine transformation or a projection transformation method based on the extracted feature points (S180).

Eventually, the panoramic image generation unit 200 aligns the images from which the feature points are extracted as described above through a method of matching the feature points, and generates the aligned images as a panoramic image (eg, PA_img2 in FIG. 5 or PA_img3 in FIG. 8). Do (S200).

If the panorama image generation unit 200 fails to extract the feature points, this is the same as the failure to generate the panorama image, and when the panorama image generation fails, the panorama image generation unit 200 performs the image capture unit 100 The image (ca_img) transmitted from is transmitted to the image matching processing unit 500 (S280).

Depending on the embodiment, the panorama image generator 200 may transmit the image (ca_img) transmitted from the image capturing unit 100 to the image matching processor 500 even if the panorama image generation has not failed (S280). .

The image matching processing unit 500 generates a correction image (calib_img) from the images (ca_img) transmitted from the panorama image generation unit 200 (S400), and transmits the generated corrected image to the population counting unit (S430).

Hereinafter, a process (S400) of generating the corrected image (calib_img) by the image matching processing unit 500 will be described in more detail.

14 is a flowchart illustrating a process of generating a corrected image by an image matching processor according to an embodiment of the present invention.

1 to 14, the image matching processing unit 500 performs a pre-processing process to minimize errors in determining similarity due to vertical and rotation between images (ca_img) transmitted from the panoramic image generation unit 200 ( S300).

According to an embodiment, the predetermined preprocessing process may be performed according to max pooling of a convolutional neural network (CNN) (S300).

Since the image matching processing unit 500 preprocesses images (eg, Left_img and Right_img in FIG. 9) using max pooling, even if the image (Right_img in FIG. 9) is slightly vertically shifted or rotated, images (eg, FIG. The similarity between the columns of (Left_img and Right_img of 9) does not have a significant influence on the determination.

Based on the preprocessed left image (Left_img) and right image (Right_img), the image matching processing unit 500 sequentially proceeds from the rightmost column (CL_1) of the left image (Left_img) to the right image (CL_2 to CL_n). Right_img) is compared with the leftmost column CR_PL (S350).

At this time, each column is assumed to have n elements as a vector having a dimension equal to the image height, and each of the columns CL_1 to CL_n of the left image (Left_img) and the leftmost column of the right image (Right_img) The similarity between the columns is compared by calculating the distance of (CR_PL) (S320) (S350).

According to an embodiment, the image matching processor 500 may calculate the distance between the columns by calculating the Euclidean distance or L2 distance as in Equation 1 (S320).

The image matching processing unit 500 determines that the similarity between the columns is the largest when the Euclidean distance calculated as described above is the smallest (S350).

The image matching processing unit 500 matches the images (Left_img and Right_img) based on the two columns having the highest similarity (eg, CL_4 and CR_PL in FIG. 11) according to the similarity between the columns obtained as described above. matching) is performed (S390).

That is, the image matching processing unit 500 finds an overlapping area between the images (Left_img and Right_img) by performing (S390) matching (S390), and generates the matched image as a corrected image (calib_img) ( S400).

Thereafter, the image matching processing unit 500 transmits the generated correction image (calib_img) to the population counting unit 400 (S430).

Meanwhile, the population counting unit 400 counts the floating population in real time from the panoramic image PA_img transmitted from the panoramic image generating unit 300 or the corrected image calib_img transmitted from the image matching unit 500 (S500).

At this time, the population counting unit 400 uses a known object recognition method such as R-CNN, Fast R-CNN, Faster R-CNN, R-FCN, YOLO, SSD, etc. to determine the floating population in real time from the panoramic image (PA_img). Count (S500-1).

In addition, the population counting unit 400 uses a known object recognition method such as R-CNN, Fast R-CNN, Faster R-CNN, R-FCN, YOLO, SSD, etc. from the corrected image (calib_img) to each image ( The number of floating population existing in Left_img and Right_img) is counted in real time, and the total floating population is calculated by subtracting the floating population existing in the overlapping area (overlapping area) from these (Left_img and Right_img) (S500-2).

The above description is merely illustrative of the technical idea of the present invention, and those of ordinary skill in the technical field to which the present invention pertains can make various modifications and variations without departing from the essential characteristics of the present invention.

Accordingly, the embodiments disclosed in the present invention are not intended to limit the technical idea of the present invention, but to explain the technical idea, and the scope of the technical idea of the present invention is not limited by these embodiments. The scope of protection of the present invention should be construed by the following claims, and all technical ideas within the scope equivalent thereto should be construed as being included in the scope of the present invention.

10: Floating population counting system
100: video recording unit
130-1: first camera
130-2: second camera
130-3: third camera
200: Panorama image generation unit
300: feature point display
400: floating population counting unit
500: image matching processing unit

Claims (11)

An image capture unit comprising a plurality of cameras each collecting an image for a corresponding region;
A panoramic image generating unit generating a single panoramic image from the images collected by the image capturing unit;
An image matching processor configured to generate a corrected image by matching the collected images when the panoramic image generator fails to generate the panoramic image;
A floating population counting unit for counting a floating population from the panoramic image generated by the panoramic image generating unit or the corrected image generated by the image matching processing unit using an object recognition method,
The image matching processing unit,
The images collected by the image capturing unit are preprocessed by performing max pooling of a convolutional neural network (CNN), and the Euclidean distance between columns of each image in the preprocessed images ( Euclidean Distance) is calculated, the similarity between the columns is determined, and the images are matched based on the columns with the largest similarity to generate the corrected image. The method of claim 1, wherein the panorama image generating unit,
Using a local feature detection method, feature points having the same characteristics are extracted from each of the images collected by the image capturing unit,
An image-based floating population counting system for generating the panoramic image by arranging the images so that the extracted feature points coincide using an affine transform or a perspective transform. The method of claim 1,
An image-based floating population counting system further comprising a feature point display unit implemented as a structure installed in the corresponding area or implemented as a pattern coated on a structure existing in the corresponding area. delete The method of claim 1, wherein the image matching processing unit,
Similarity is obtained by calculating the Euclidean Distance with the leftmost column (CR_PL) of the second image on the right for each of the sequential left columns (CL_2 to CL_n) from the rightmost column (CL_1) of the first image on the left. And, as a result of comparing the similarity, overlapping columns that are most similar to each other to calculate an overlapping area between the first image and the second image. The method of claim 5, wherein the image matching processing unit,
Each of the columns is a vector having a dimension equal to the height of the image and is divided into n elements, and elements corresponding to each other between the columns to be compared between the first image on the left and the second image on the right are used. Image-based floating population counting system, characterized in that configured to calculate the Euclidean distance. The method of claim 1, wherein the floating population counting unit,
An image-based floating population counting as the floating population the number obtained by subtracting the number of floating populations present in regions where images included in the corrected image overlap each other from the sum of the number of floating populations of each of the images included in the correction image Counting system. Transmitting, by an image capturing unit, images of a corresponding region collected by a plurality of cameras to a panorama image generating unit;
Generating a single panoramic image from the transmitted images by the panoramic image generating unit and transmitting the generated panoramic image to the floating population counting unit;
If the panorama image generator fails to generate the panorama image, transmitting, by the panorama image generator, the transmitted images to an image matching processor;
Generating a correction image from the transmitted images by the image matching processing unit and transmitting the generated correction image to the floating population counting unit;
Counting the floating population from the transmitted panoramic image or the corrected image using an object recognition method by the floating population counting unit,
The step of generating a correction image from the transmitted images by the image matching processing unit and transmitting it to the floating population counting unit,
The images collected by the image capturing unit are preprocessed by performing max pooling of a convolutional neural network (CNN), and the Euclidean distance between columns of each image in the preprocessed images ( Euclidean Distance) is calculated to determine the similarity between the columns, and the image-based floating population counting method is configured to generate the corrected image by matching the images based on the columns with the largest similarity. The method of claim 8, wherein the step of generating the panoramic image by the panoramic image generating unit and transmitting it to the floating population counting unit,
Extracting feature points having the same characteristics from the transmitted images using a local feature detection method;
Aligning the transmitted images to match the extracted feature points using an affine transformation or projection transformation method;
Generating the aligned images as the panoramic image; And
Transmitting the generated panoramic image to the floating population counting unit; Image-based floating population counting method comprising a. The method of claim 8, wherein the image matching processing unit generates the corrected image and transmits it to the floating population counting unit,
Similarity is compared by calculating the Euclidean Distance with the leftmost column (CR_PL) of the second image on the right for each of the sequential left columns (CL_2 to CL_n) from the rightmost column (CL_1) of the first image on the left. And, as a result of the similarity comparison, overlapping the most similar columns to calculate an overlapping area between the first image and the second image,
When calculating the Euclidean distance, each column is divided into n elements as a vector having a dimension equal to the height of the image, and between the first image on the left and the second image on the right are compared with each other. An image-based floating population counting method, characterized in that the Euclidean distance is calculated using elements corresponding to each other. The method of claim 8, wherein the step of counting the floating population from the correction image by the floating population counting unit,
Counting the number of floating populations of each of the images included in the corrected image;
Counting the number of floating populations existing in an area overlapping each other of the images included in the corrected image; And
Counting as the floating population a value obtained by subtracting the number of floating populations present in the counted overlapping area from the sum of the number of floating populations of each of the counted images as the floating population.

Images