KR102096926B1 - Method and system for detecting change point of interest - Google Patents

Abstract

A method and system for detecting a region of interest change is disclosed. POI changes can be automatically detected by detecting the signboard part of the store that can best detect changes such as new opening, closing, and renaming in pairs of images shot at the same location at a time interval.

Description

METHOD AND SYSTEM FOR DETECTING CHANGE POINT OF INTEREST}

The description below relates to a technique for automatically detecting a change in a region of interest existing in a real space.

In the real space, there are various types of points of interest (POIs) such as restaurants, bookstores, and shops. In order to display information about the region of interest (hereinafter, 'POI information') on a map or provide it to a user, corresponding POI information must be collected.

Conventionally, a person visits an actual space by walking or using a vehicle, etc., and checks the location, type, name, etc. of the POI in the system, or records an image of the actual space, road, etc. using a vehicle equipped with a camera. A method was used to record images in a system by recognizing the type, name, etc. of a POI by analyzing the image captured by a person later.

For example, Korean Patent Publication No. 10-2011-0037045 is a technology for a video collection system using a camera of a vehicle and a control method thereof, depending on whether photographing is required by using a camera mounted on the vehicle, or When building a database by shooting a distance and installing a camera on a vehicle that moves in various areas such as taxis, courier trucks, or buses, more areas or distances are taken to provide users with more video information on the map at a lower cost. It is disclosed that it can be done.

However, in the prior art, in order to check the POI information in the real space, a person visits the location where the POI is located and checks and records the information directly, or visits the location to shoot the location and a person is photographed later. Since a person must intervene throughout the processing process, such as confirming and recording information by checking the image, there is a problem in that data collection and analysis and processing are expensive and time-consuming.

In addition, even if the POI information for any real space area is already secured, the POI information may be changed from time to time due to new opening, closing, mutual change, and start-up. Therefore, in order to immediately recognize the change in POI, it is necessary to quickly update the POI information through frequent monitoring of the corresponding spatial area, but as a method of human intervention, this also consumes cost and effort, and in effect changes the POI. It is virtually impossible to obtain / provide relevant and up-to-date information. In particular, when the range of the real space is wide, there is a problem that the process of checking the latest POI information according to the change of the POI becomes more difficult.

Provided is a method and system for automatically detecting a POI change existing in a real space for location-based services such as a map in a real space environment such as a city street or an indoor shopping mall.

Provides a method and system that can detect POI changes effectively and reliably by detecting the signboard part of the store that can best detect changes such as new opening, closing, and renaming.

A method performed in a computer system, the computer system comprising at least one processor configured to execute computer readable instructions contained in memory, the method comprising, by the at least one processor, a target at predetermined time intervals Selecting a pair of images photographed from a location; Detecting, by the at least one processor, a signage area in each image included in the image pair; And comparing, by the at least one processor, the signage area between the images to recognize a change in a point of interest with respect to the target location.

According to one aspect, the detecting of the signage area may detect the signage area from the image through a machine learning model that has learned a set of signage images.

According to another aspect, the detecting of the signage area includes detecting a plurality of signage areas in the panoramic image when the pair of images is a panoramic image, and recognizing a change in the region of interest, The method may include classifying a pair of signage areas having a similarity of image matching among a plurality of signage areas and recognizing the remaining signage area as a change area.

According to another aspect, the step of recognizing a change in the region of interest may include extracting each descriptor from the signage area and comparing the extracted descriptors between the signage areas to determine whether or not a match exists.

According to another aspect, the step of detecting the signage area may include detecting the signage area through scene layout detection of the panoramic image when the pair of images is a panoramic image.

According to another aspect, the method further includes training, by the at least one processor, the machine learning model using a signboard image in which an obstacle object independent of the signboard is inserted, wherein the training step comprises: And, when the disturbing object is detected, assigning a negative cost.

According to another aspect, the detecting of the signage area may include detecting a final signage area from the main map after merging the signage areas detected at multiple scales of the image to generate a saliency map. It can contain.

According to another aspect, in the step of selecting the image pair, before and after images of different photographing points are selected for the target location, but after and after the images are selected from the images photographed at the latest point of view. The previous image may be selected from images captured before the image.

According to another aspect, the step of selecting the pair of images may include pairing of images having a direction similarity to a preset ratio or higher based on photographing information of the image, or a pair of images having a photographing direction within a preset angle difference. And it can be selected as the image after.

According to another aspect, the method may further include visualizing a region of interest in which the change is recognized by the at least one processor.

According to another aspect, the step of visualizing may include generating a region of interest change information including the image pair for an region of interest in which the change is recognized and providing it to an operator.

It provides a computer-readable recording medium, characterized in that a program for executing the method on a computer is recorded.

It includes at least one processor that is implemented to execute a computer-readable instruction, and by the at least one processor, a pair of images photographing a target location at predetermined time intervals is selected, and each image included in the image pair Provided is a computer system characterized by detecting a signage area and comparing the signage area between the images to recognize a change in a region of interest with respect to the target location.

According to embodiments of the present invention, a POI change existing in a real space may be automatically detected for a location-based service such as a map in a real space environment such as a city street or an indoor shopping mall.

According to embodiments of the present invention, a POI change can be effectively and reliably detected by detecting a signboard portion of a store that can best detect changes such as new opening, closing, trade name change.

1 is a diagram illustrating an example of an information collection and update system according to an embodiment of the present invention.
2 is a flowchart illustrating an example of a method for collecting and updating information according to an embodiment of the present invention.
3 is a flowchart illustrating an example of a basic information acquisition process in an embodiment of the present invention.
4 is a diagram showing an example of data collected through a mapping robot in one embodiment of the present invention.
5 is a flowchart illustrating an example of a process of acquiring information at any time in an embodiment of the present invention.
6 is a flowchart illustrating an example of a POI change detection process in an embodiment of the present invention.
7 to 9 are exemplary views for explaining a process of detecting a POI change area from an image photographed with a panoramic cut in an embodiment of the present invention.
10 to 12 are exemplary views for explaining a process of detecting a POI change area from an image photographed with a general single cut in one embodiment of the present invention.
13 is a block diagram showing an example of a computer system according to an embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

Embodiments of the present invention provide a method and system capable of efficiently performing point of interest (POI) change detection while minimizing human intervention.

In a real environment, POIs are frequently changed in various forms, such as opening, closing, expanding, or changing to a different store. Maintaining the latest POI information by identifying these POI changes from time to time is very important for location-related services such as maps.

The POI change detection technology according to embodiments of the present invention automatically detects a POI change from an image captured by using a vehicle or a robot, or provides the operator with relevant information to record the POI change in the system or change When is very clear, POI information can be automatically recorded in the system to efficiently keep the POI information up to date.

In one embodiment, an example of a large-scale indoor shopping mall as a real space will be described to describe POI change detection technology. This is for convenience of description, and the actual space in the embodiments of the present invention is not limited to a space such as an indoor shopping mall. In addition, in this embodiment, an example of using a robot capable of autonomous driving as a means for acquiring images and related data for detecting POI changes will be described. Acquisition of information may be achieved through various means, such as a vehicle, a person, a tray, and CCTV, depending on the type of environment in the real space, and is not limited to information acquisition means such as a robot presented in this embodiment.

1 is a diagram illustrating an example of an information collection and update system according to an embodiment of the present invention, and FIG. 2 is a flowchart illustrating an example of an information collection and update method according to an embodiment of the present invention.

The information collection and update system 100 according to the present embodiment may include a cloud server 110, a mapping robot 120, and a service robot 130. The cloud server 110 and the mapping robot 120, and the cloud server 110 and the service robot 130 are implemented to enable data communication through a network for transmission of collected data, location information, and map information. Can be. In particular, the service robot 130 may be implemented to include a wireless network interface to give real-time data communication with the cloud server 110.

In addition, the information collection and update method according to the present embodiment may include a basic information acquisition step 210, an occasional information acquisition step 220, and an occasional POI information processing step 230, as shown in FIG. 2. have. The basic information acquiring step 210 may be performed once for the first time (two or more times if necessary) for acquiring basic information, and the occasional information acquiring step 220 may be repeatedly performed at any time or whenever necessary. In addition, the POI information processing step 230 may be repeatedly performed at regular intervals, such as a day unit or a week unit.

At this time, the mapping robot 120 may be implemented to collect data about the target place 140 and drive it to the cloud server 110 while driving the target place 140 in the basic information acquisition step 210. The 110 generates basic information about the target place 140 based on the data collected and provided by the mapping robot 120, and uses the generated basic information in the target place 140 of the service robot 130. It can be implemented to support autonomous driving and service provision.

In addition, the service robot 130 collects the occasional information while autonomously driving the target place 140 based on the information provided from the cloud server 110 in the step 220 of acquiring the occasional information so that it can be delivered to the cloud server 110. Can be implemented.

At this time, the cloud server 110 may update the information about the target place 140, such as recognizing and updating the POI change based on the comparison between the basic information and the occasional information in the POI information processing step 230 at any time.

In the basic information acquisition step 210, the information collection and update system 100 may acquire basic information. The POI change detection technology basically detects whether the POI is changed by comparing the current image and the previous image using various techniques. Therefore, a previous image to be compared is required, and thereafter, an indoor map configuration for the robot is required for autonomous driving of the robot (eg, the service robot 130). In order to acquire the previous image and the indoor map, the basic information acquisition step 110 may be performed for the first time (two or more times if necessary). Detailed steps for obtaining the basic information 110 will be described with reference to FIG. 3.

3 is a flowchart illustrating an example of a basic information acquisition process in an embodiment of the present invention. Acquisition of the basic information can be achieved by the cloud server 110 and the mapping robot 120 included in the information collection and update system 100, and steps 310 to 340 of FIG. 3 are steps 210 of FIG. ).

In step 310, the mapping robot 120 may collect data while autonomously driving the selected target place 140 when the target place 140 is selected, such as an indoor shopping mall. At this time, the collected data may include data for generating a previous image to be utilized for detecting POI change, and data for constructing an indoor map for autonomous driving of the service robot 130. To this end, the mapping robot 120 may be implemented to include a rider (Lidar), a wheel encoder (Wheel Encoder), an IMU (Inertial Measurement Unit), a camera, a communication interface, etc., and the service robot 130 is already a mapping robot Since autonomous driving is performed by using the indoor map constructed based on the data collected by the 120, there is no need to mount an expensive high-precision sensor like the mapping robot 120, and thus, compared to the mapping robot 120. It can be implemented to mount a relatively low-cost sensor. The data collected by the mapping robot 120 will be described in more detail through FIG. 4 later.

In step 320, the mapping robot 120 may transmit the collected data to the cloud server 110. The collected data is transmitted to the cloud server 110 at the same time as the collection according to the embodiment, or is transmitted to the cloud server 110 by being bound by the zone unit of the target place 140, or the entire zones of the target place 140 After the collection of data for is completed, it may be transmitted to the cloud server 110 at a time.

In step 330, the cloud server 110 may store data received from the mapping robot 120. For example, the cloud server 110 collects and transmits data that the mapping robot 120 collects and transmits from the mapping robot 120 to the entire areas of the target location 140 in a database (POI database) ) To manage them continuously.

In step 340, the cloud server 110 may generate a 3D map using data stored in the database. The generated 3D map may be used to help the service robot 130 autonomously drive the target place 140 while providing the desired service.

4 is a diagram showing an example of data collected through a mapping robot in one embodiment of the present invention. The mapping robot 120 may collect mapping data 410 for generating a 3D map of the target place 140 and POI data 420 for use in detecting POI changes. For example, the mapping data 410 may include measurement values (rider data, wheel encoder data, IMU data, etc.) measured through a rider, a wheel encoder, an IMU, etc. that the mapping robot 120 may include, , POI data 420 is a camera that can be included in the mapping robot 120, data obtained through communication interfaces (Wi-Fi interface, Bluetooth interface, etc.) (camera data such as a captured image, signal strength of Wi-Fi or Bluetooth Beacon (beacon, etc.). In the embodiment of FIG. 4, for convenience of description, the type of the mapping data 410 and the type of the POI data 420 are divided, but the collected data is overlapped for both generation of a 3D map and detection of POI change. It may be utilized. For example, an image captured through a camera or Wi-Fi signal strength may be further utilized to generate a 3D map. In order to collect the mapping data 410 and / or POI data 420, various types of sensors, such as a stereo camera or an infrared sensor, may be further utilized in the mapping robot 120 in addition to the sensors described through FIG. 4.

The mapping robot 120 may, for example, photograph a surrounding area at regular intervals (for example, 1 second interval while moving at 1 m / sec) with a camera mounted on the mapping robot 120 while moving the indoor space. A 360-degree camera, a wide-angle camera, and / or multiple cameras may be utilized so that the shape of the front of the store and the front of the store, which are mainly used for POI change detection, are effectively included in the captured image, and the target space 140 An image may be taken so that the entire region of the image is at least partially included in the image. At this time, in order to check the location of the POI change, it is necessary to know where the captured image is obtained from the target location 140, so the obtained image can be stored together in connection with the shooting information of the mapping robot 120 at the time of shooting. have. The photographing information stored with the image may include location information (shooting location) and / or direction information (shooting direction) of the mapping robot 120. At this time, information about the point in time at which the image was captured (shooting point in time) may also be stored together with the image. For example, to obtain location information, the mapping robot 120 may further collect Wi-Fi finger printing data for Bluetooth beacon information or Wi-Fi-based location identification, and the mapping robot 120 for obtaining direction information ), The measured value of the rider or IMU may be used. The mapping robot 120 may transmit the collected data to the cloud server 110, and the cloud server 110 generates a 3D map by using the data received from the mapping robot 120, and the generated 3D map Based on, it may process localization, path planning, and the like for the service robot 130. Also, the cloud server 110 may compare data received from the mapping robot 120 with data collected by the service robot 130 to update information on the target place 140.

When the target place is indoor, the location included in the map data generated by the mapping robot 120 (for example, image according to location information, Wi-Fi signal strength, Bluetooth beacon information, sensor measurement value, etc.) is based on the starting location. It has to be relatively determined. This is because accurate global positioning data cannot be obtained in an indoor space. In addition, if the same space is divided and scanned several times, it is difficult to obtain consistent position data because the starting position is different every time. Therefore, for a consistent location data surface and utilization, a process of converting location data obtained through the mapping robot 120 into a form capable of global positioning is required. To this end, the cloud server 110 confirms the exact location indicated by the actual latitude / longitude of the indoor space, and then converts and stores the location data included in the map data according to a geodetic reference system such as WGS84, ITRF, PZ, etc. Later, it can be used in a later process.

Referring back to FIGS. 1 and 2, the information collection and update system 100 in the information acquisition step 220 may acquire the information about the target place 140 at any time. In the occasional information acquisition step 220, the 3D map obtained in the previous step, the basic information acquisition step 210, the previous image, location information, and the like may be continuously utilized.

Since the cloud server 110 already collects, processes, and stores information about the entire space of the target place 140 in the basic information acquisition step 210, in the information acquisition step 220, only partially changed information is acquired, By processing and storing, it becomes possible to efficiently keep information on the target place 140 such as map data up to date. Therefore, there is no need to collect data for the entire spatial area of the target place 140 each time.

In addition, as already described, in the step 220 of acquiring information at any time, the cloud server 110 is required for autonomous driving of the service robot 130 using data of various expensive and high-precision sensors mounted on the mapping robot 120. Since the high-precision map data is generated and stored, there is no need for the service robot 130 to mount an expensive high-precision sensor. Accordingly, in the step 220 of acquiring information at any time, the service robot 130 may be implemented by using a low-cost robot that operates according to a purpose of using a natural service such as security, guidance, and cleaning of the target place 140.

5 is a flowchart illustrating an example of a process of acquiring information at any time in an embodiment of the present invention. The service robot 130 may be disposed inside the target place 140 for natural service purposes such as security, guidance, cleaning, and the like. Depending on the target location 140 and the service purpose, two or more service robots may be disposed in the target location 140 and may be designated to operate in different areas. The acquisition of the information at any time may be achieved by the cloud server 110 and the service robot 130 included in the information collection and update system 100, and the steps 510 to 580 of FIG. 5 are steps 220 of FIG. ).

In step 510, the service robot 130 may take a picture of the surroundings in the target place. To this end, the service robot 130 may be implemented to include a camera for photographing surrounding images at a target location. The captured image can be used for two purposes. First, the photographed image may be utilized for the purpose of assisting autonomous driving of the service robot 130 by checking the current position (shooting position) and / or direction (shooting direction) of the service robot 130. Second, the photographed image may be used as an occasional image for confirming a change in POI as a purpose of comparison with a previous image obtained in step 210 of obtaining basic information. The image captured for both purposes may request the location and / or direction information of the service robot 130 at the time when the corresponding image was captured (shooting time). According to an embodiment, the imaging cycle of the image for the first purpose and the imaging cycle of the image for the second purpose may be different, or may be dynamically determined based at least on the moving speed of the service robot 130. If the service robot 130 confirms the location and / or direction by using a Wi-Fi signal strength or a Bluetooth beacon, etc., rather than an image, the imaging of the image may be used only for the second purpose. If the Wi-Fi signal strength or Bluetooth beacon is utilized, the service robot 130 transmits the Wi-Fi signal strength or Bluetooth beacon obtained to confirm the location or direction to the cloud server 110 to determine the location and / or direction. You can also request information. On the other hand, in this case, it is also required to obtain position and / or direction information related to the image for the second purpose. Subsequently, steps 520 to 540 may describe an example of a process of obtaining position and / or direction information related to an image. When the service robot 130 is moved, steps 510 to 540 may be performed periodically and / or repeatedly to continuously acquire the location of the service robot 130.

In step 520, the service robot 130 may transmit the captured image to the cloud server 130. At this time, the service robot 130 may request location and / or direction information corresponding to the transmitted image while transmitting the image.

In step 530, the cloud server 130 may analyze the image received from the service robot 130 to generate location and / or direction information of the service robot 130. At this time, the position and / or direction information may be generated through the information obtained in the basic information acquisition step 210. For example, the cloud server 130 compares the image collected from the mapping robot 120 with the image received from the service robot 130 to find a matching image, and then stores and / or the direction associated with the image. Based on the information, location and / or direction information according to the request of the service robot 130 may be generated. The direction information may be direction information of the camera.

In step 540, the cloud server 130 may transmit the generated location and / or direction information to the service robot 130.

In step 550, the service robot 130 may store the received location and / or direction information as occasional information in association with the captured image. Occasional information may refer to information used for the purpose of confirming the POI change described above. In this case, the occasional information may further include information on a time point of capturing the image.

In step 560, the service robot 130 may transmit the stored occasional information to the cloud server 130. As the service robot 130 moves, the amount of information at any time may also increase, and the service robot 130 may transmit the stored information at any time, periodically or whenever necessary, to the cloud server 130.

In step 570, the cloud server 130 may store the received occasional information in a database (POI database). The stored occasional information may be used to detect a change in POI through comparison with information obtained at step 210 of obtaining basic information.

In step 580, the service robot 130 may perform a service mission based on the received location and / or direction information. In the embodiment of FIG. 5, step 580 is described as being performed after step 570, but step 580 of performing the service mission is the location of the service robot 130 received at step 540 and / or Alternatively, it may be performed in parallel with steps 550 to 570 using direction information. Localization and path planning for performing service missions may be performed by the service robot 130 or cloud server 130 according to an embodiment.

In the above embodiments, it is described that data is collected for the target place 140 using the mapping robot 120 and the service robot 130, but the present invention is not premised on the use of the robot, and is equivalent. Various methods of level can be used. For example, in the basic information acquisition step 210, a device such as a trolley that can be dragged by a person instead of using an expensive mapping robot 120 capable of autonomous driving for the first time to collect basic information It is also possible to mount the sensor to collect the data of the space, and in the step of acquiring information at any time, the image captured by the smartphone of ordinary users visiting the target place 140 is collected or utilized, or the target place 140 It is also possible to collect and utilize the video of CCTV (Closed Circuit Television) installed on the screen. In other words, the cloud server 110 is a base image obtained through a camera and a sensor included in at least one of a mapping robot 120 autonomously driving the target place 140 and a trolley moving the target place 140. And it is possible to build a POI database by receiving the shooting location and the shooting time point of the basic video through the network. In addition, the cloud server 110 is a service robot 130 that performs a predetermined service mission while autonomously driving the target place 140, a terminal and a target place 140 including cameras of users located in the target place 140 ), The POI database may be updated by receiving the occasional image captured for the target place 140 from the at least one of the closed circuit televisions (CCTVs) and the location and time of the image capture.

Referring back to FIGS. 1 and 2, in the occasional POI information processing step 230, the cloud server 110 collects the basic information collected in the basic information acquisition step 210 and the occasional information collected in the occasional information acquisition step 220. It may be a process for obtaining POI-related information by utilizing.

For example, in the POI change detection technology, the cloud server 110 analyzes and compares one basic image and a plurality of occasional images by using machine learning technology in the POI information processing step 230 at any time. It may be a process for detecting a POI from images, determining whether there is a change in the POI, and updating the POI change in the information collection and update system 100 when there is a change. In one example, the cloud server 110 may visualize the POI change detection result to inform the operator of the information collection and update system 100. For example, the cloud server 110 may notify the operator of the information collection and update system 100 before and after the change to the POI. Even if the information collection and update system 100 determines and selects whether the POI is changed in advance and provides it to the operator, the amount of video to be reviewed by the operator is significantly reduced and further added to the unit time. It is possible to analyze, review, and update POI information for a wide area. As another example, the cloud server 110 may directly update the name, type, and changed image of the changed POI to the information collection and update system 100.

6 is a flowchart illustrating an example of a POI change detection process in an embodiment of the present invention. As already described, steps 610 to 640 of FIG. 6 may be performed by the cloud server 110.

In step 610, the cloud server 110 may select an image pair photographed at a predetermined time interval from the target location 140. To select a pair of images, the cloud server 110 may select a target location in the target place 140. For example, the cloud server 110 may determine a plurality of locations in the target location 140 in advance, and select a location from which the POI change is to be detected. For example, the cloud server 110 may determine a plurality of locations by dividing the target place 140 in a grid form having a predetermined interval, and select one of the determined plurality of locations as the target location. . In addition, the cloud server 110 may select an image pair consisting of a previous image and a later image captured at a time interval among images stored in the POI database in association with a shooting location located within a predetermined distance from the target location. have.

In this case, selection of the previous image and the subsequent image separately may be based on a photographing time point of the images. In order to detect the change in POI, each of the previous images and the subsequent images to be compared are basically required. Initially, a previous image may be selected from the images collected through the basic information acquisition step 210, and a subsequent image may be selected from the images collected through the frequent information acquisition step 220. As another example, a previous image and a later image of different viewpoints may be selected from among images collected through the occasional information acquisition step 220. When the images are collected through the frequent information acquisition step 220, a subsequent image may be selected from the most recent images taken at the most recent time point, and later images or previous time points (for example, one day before, which were used for the previous comparison) Previous images may be selected from among the occasional images photographed a week ago.

The cloud server 110 may select the same direction image based on the shooting information stored with the corresponding image in the process of selecting the previous image and the subsequent image. Selecting the same direction image is for comparing the previous image and the subsequent image taken in a similar direction at a similar location, and the two images of the previous image and the subsequent image taken at a similar location are equal to or greater than a predetermined ratio. If it has a degree of direction similarity that is expected to have been captured, it may be selected as a pair of images in the same direction. As another example, when the photographing directions of two images form within a predetermined angle difference, the cloud server 110 may select the two images as a pair of the same direction image.

In step 620, the cloud server 110 may detect the signage area from the previous image and the subsequent image formed by the image pair selected in step 610. In the present invention, a POI change can be detected by detecting a signboard portion of a store that can best detect changes such as a new opening, closing, or changing a company name. The cloud server 110 may detect a signage area from a previous image and a subsequent image through a machine learning model that learns a set of signboard images collected in advance based on deep learning.

The cloud server 110 can build a machine learning model by learning images of various types or types of signboards as training data, thereby detecting a signboard area in the image. When a general object detection algorithm is used in the learning process, a false positive occurs as a disruptor such as a signboard independent of a signboard, an emergency exit pictogram, an event phrase, etc. is detected. To solve this problem, the cloud server 110 generates synthetic data by randomly inserting an obstruction object into an image where a sign is taken, that is, a learning target image, and detects a sign during detection training using the synthetic data. When a positive value is given, and a negative object is detected, a negative error can be reduced. In addition, the cloud server 110 acquires detection results at multiple scales of the image to detect signboards of various sizes existing in the image, merges the detection results, generates a saliency map, and finally detects the result from the main map. The signage area can be detected.

In step 630, the cloud server 110 may detect the POI change area by comparing the signage area detected in each image between the previous image and the subsequent image. The cloud server 110 may compare the signage area between the previous image and the subsequent image to determine that there is no POI change if the match is successful, and that there is a POI change if the match is unsuccessful. The cloud server 110 may compare the signage area between the previous image and the subsequent image to recognize the unmatched region as a POI change region and store the recognized POI change region in association with information on the target location.

The cloud server 110 may determine whether processing for the entire location in the target location 140 has been completed. At this time, if the processing for the entire location is not completed, the cloud server 110 selects the next location in the target location 140 as the target location to detect the POI change, step 610 to step 630 It can be done repeatedly. When the processing for the entire location is completed, the cloud server 110 may perform step 640.

In step 640, the cloud server 110 may visualize the POI change area within the target place 140. The cloud server 110 visualizes a target location where the POI change region is detected using a previous image and a subsequent image related to the recognition of the POI change, or a target where the POI change region is detected on a 3D map of the target place 140 The location can be visualized to request a review of the POI change area. The request for such a review may be communicated to the operator of the information collection and update system 100. In other words, the previous image and the subsequent image corresponding to the POI change area may be transmitted to the operator together with location information (target location). This information is displayed on a map on software that allows the operator to input change information according to the change in POI, so that the operator can once again review and confirm information about the change in POI and then enter the information. In other words, the cloud server 110 generates POI change information including a previous image and a subsequent image related to the recognition of the POI change, so that the operator updates information on the corresponding POI through the information on the recognized POI change. It can be provided to the operator.

7 to 9 are exemplary views for explaining a process of detecting a POI change area from an image photographed with a panoramic cut in an embodiment of the present invention.

Referring to FIG. 7, when an image photographed to detect a change in POI is a panoramic image, the cloud server 110 consists of a previous panoramic image 710 and a subsequent panoramic image 720 photographing the same location at a time interval. You can take a pair of images as input. The panoramic image has a layout characteristic in which an arch-shaped frame is created between several vanishing points and vanishing points. The cloud server 110 uses the algorithm to detect the characteristic scene layout of the panoramic image, so that the ceiling region 701, the floor region 702, and the shop area () in the previous panoramic image 710 and the subsequent panoramic image 720, respectively. 703). In other words, the cloud server 110 stores actual stores except for the ceiling area 701 and the floor area 702 through semantic segmentation in the previous panoramic image 710 and the subsequent panoramic image 720. It is possible to distinguish the store area 703 which is an area to be located.

Referring to FIG. 8, the cloud server 110 uses the signage detection algorithm described above to signage area 805 in a portion corresponding to each of the shop area 703 of the previous panoramic image 710 and the subsequent panoramic image 720. Can be detected. In addition, the cloud server 110 may extract the POI area 806 representing the individual store centering on the signage area 805 for each of the previous panoramic image 710 and the subsequent panoramic image 720.

Referring to FIG. 9, the cloud server 110 may calculate the similarity between POI regions 806 using an image matching algorithm. The cloud server 110 uses the Deep Image Retrieval (DIR) algorithm for the POI region 806 extracted from the previous panoramic image 710 and the POI region 806 extracted from the subsequent panoramic image 720 through an image descriptor (image) descriptor) and compares the extracted descriptors with each other to find a matching POI pair. For example, the DIR algorithm may refer to a paper disclosed in <https://arxiv.org/abs/1604.01325>. Descriptors are descriptors that express image features as vectors, and are widely used in image search and image analysis techniques. For deep learning-based image search, global image representation is learned. In the case of a global descriptor, a 2048-dimensional feature is extracted through a feature extraction process (convolution, pooling, etc.) when an input image comes in. can do. In this case, a comparison between the extracted 2048-dimensional descriptors may use cosine similarity. In addition, it is also possible to find pairs of POIs that match each other using other known algorithms such as Scale Invariant Feature Transform (SIFT) and Speeded Up Robust Features (SURF).

When the cloud server 110 has the POI region 806 extracted from the previous panoramic image 710 and the POI region 806 extracted from the subsequent panoramic image 720 in the same location, the similarity is higher than a certain level It can be classified into POI pairs representing the same store. The positional relationship between the POI regions 806 may be obtained using coordinate information on the previous panoramic image 710 and the subsequent panoramic image 720.

The cloud server 110 may detect a POI change based on the matching result between the POI areas 806. In other words, the cloud server 110 may recognize the remaining POI region as a POI change region 907 without finding a pair between the previous panoramic image 710 and the subsequent panoramic image 720. The cloud server 110 may visualize the area 907 when there is an area 907 in which a change in the POI is detected in the previous panoramic image 710 and the subsequent panoramic image 720 using the POI pair matching information.

In the above description, the POI change area 907 is detected by comparing the POI area 806 between the previous panorama image 710 and the subsequent panorama image 720, but the signage area (without extracting the POI area 806) is described. It is also possible to detect the POI change region 907 by comparing only 805).

10 to 12 are exemplary views for explaining a process of detecting a POI change area from an image photographed with a general single cut in one embodiment of the present invention.

The cloud server 110 may receive, as input, a pair of images photographing the same location at a time interval using a 3D map, location information (shooting location), and direction information (shooting direction) of the target place 140. . Referring to FIG. 10, the cloud server 110 may generate a 3D map 1000 by utilizing data collected for the target place 140, and the most recent target location of the 3D map 1000 A subsequent image 1020 may be selected from the occasional images photographed at a time point, and the images collected through the basic information acquisition step 210 or later images used for the previous comparison or the previous time point (for example, a day ago, a week ago) Previous images 1010 may be selected from among the occasional images photographed on the front and the like.

The previous image 1010 and the subsequent image 1020 are images shot with a normal cut rather than a panoramic cut, and may be selected as a pair of images in the same direction. The screening of the same direction image is for comparing the previous image 1010 and the subsequent image 1020 photographed in a similar location at similar locations, and the previous image 1010 and the subsequent image 1020 photographed at a similar location are compared. If two images have a degree of direction similarity that is expected to have taken the same portion of each other over a predetermined ratio, it may be selected as a pair of images in the same direction. As another example, when the photographing directions of the two images form within a predetermined angle difference, the two images may be selected as a pair of the same direction image.

Referring to FIG. 11, the cloud server 110 may respectively detect the signage area 1005 in the previous image 1010 and the subsequent image 1020 using the signage detection algorithm described above. For example, the cloud server 110 may detect at least one candidate region through a learning model for signage detection for each of the previous image 1010 and the subsequent image 1020, and the candidate region having the highest signage prediction score Can be selected as the final signage area 1005.

The cloud server 110 may calculate the similarity between the signboard area 1005 detected in the previous image 1010 and the subsequent image 1020 using an image matching algorithm. The cloud server 110 extracts natural feature descriptors through algorithms such as SIFT and SURF for the signboard region 1005 detected in the previous image 1010 and the subsequent image 1020, compares the extracted descriptors, and compares them. Can judge.

The cloud server 110 compares the signage area 1005 detected in the previous image 1010 and the subsequent image 1020 with each other, and if the match is successful, there is no POI change, and if the match fails, there is a POI change. Can decide. Then, when the signage area 1005 is not detected in the previous image 1010 and the signage area 1005 is detected in the subsequent image 1020, or the signage area 1005 is detected in the previous image 1010 If the signboard area 1005 is not detected at 1020, it may be determined that there is a POI change. In addition, when the signboard area 1005 is not detected in both the previous image 1010 and the subsequent image 1020, it may be determined that there is no POI change.

According to an embodiment, the POI change of the target location may be verified by referring to the previous and subsequent images of the location adjacent to the target location. In other words, it is possible to detect the POI change of the target location by comparing the signs of nearby stores together.

Referring to FIG. 12, the cloud server 110 changes the POI of the target positions of the previous image 1010 and the subsequent image 1020 determined to have a POI change in the 3D map 1000 of the target place 140 It can be visualized as a region 1207.

Accordingly, the cloud server 110 may use a pair of images photographed at the same location at a time interval to detect a change in the POI, and detect the signage area 805 (1005) in each image to detect the POI change area (907) ( 1207).

13 is a block diagram showing an example of a computer system according to an embodiment of the present invention. The cloud server 110 described above may be implemented by one computer system 1300 or a plurality of computer systems shown in FIG. 13. For example, a computer program according to an embodiment may be installed and driven in the computer system 1300, and the computer system 1300 may collect information according to embodiments of the present invention under the control of the driven computer program and Update method can be performed.

13, the computer system 1300 may include a memory 1310, a processor 1320, a communication interface 1330, and an input / output interface 1340. The memory 1310 is a computer-readable recording medium, and may include a non-permanent mass storage device such as random access memory (RAM), read only memory (ROM), and a disk drive. Here, a non-destructive large-capacity recording device such as a ROM and a disk drive may be included in the computer system 1300 as a separate permanent storage device separate from the memory 1310. In addition, an operating system and at least one program code may be stored in the memory 1310. These software components may be loaded into the memory 1310 from a computer-readable recording medium separate from the memory 1310. Such a separate computer-readable recording medium may include a computer-readable recording medium such as a floppy drive, disk, tape, DVD / CD-ROM drive, and memory card. In other embodiments, software components may be loaded into memory 1310 through communication interface 1330 rather than a computer-readable recording medium. For example, software components may be loaded into memory 1310 of computer system 1300 based on a computer program installed by files received over network 1360.

The processor 1320 may be configured to process instructions of a computer program by performing basic arithmetic, logic, and input / output operations. Instructions may be provided to the processor 1320 by memory 1310 or communication interface 1330. For example, the processor 1320 may be configured to execute instructions received according to program codes stored in a recording device such as the memory 1310.

The communication interface 1330 may provide a function for the computer system 1300 to communicate with other devices (eg, the above-described storage devices) through the network 1360. For example, requests, commands, data, files, etc. generated by the processor 1320 of the computer system 1300 according to program codes stored in a recording device such as the memory 1310 are controlled by the communication interface 1330. 1360). Conversely, signals, commands, data, files, and the like from other devices may be received via the network 1360 to the computer system 1300 through the communication interface 1330 of the computer system 1300. Signals, instructions, data, etc. received through the communication interface 1330 may be transferred to the processor 1320 or the memory 1310, and files, etc., may be a storage medium (described above) that the computer system 1300 may further include. Permanent storage device).

The input / output interface 1340 may be a means for interfacing with the input / output device 1350. For example, the input device may include a device such as a microphone, keyboard, or mouse, and the output device may include a device such as a display or speaker. As another example, the input / output interface 1340 may be a means for an interface with a device in which functions for input and output are integrated into one, such as a touch screen. The input / output device 1350 may be configured with the computer system 1300 and one device.

Further, in other embodiments, the computer system 1300 may include fewer or more components than those in FIG. 13. However, there is no need to clearly show most prior art components. For example, the computer system 1300 may be implemented to include at least some of the input / output devices 1350 described above, or may further include other components such as a transceiver, database, and the like.

As described above, according to embodiments of the present invention, a POI change existing in a real space can be automatically detected for a location-based service such as a map in a real space environment such as a city street or an indoor shopping mall. In particular, it is possible to detect POI changes effectively and reliably by detecting the signboard part of the store that can best detect changes such as new opening, closing, and mutual change.

The system or device described above may be implemented as a hardware component, or a combination of hardware components and software components. For example, the devices and components described in the embodiments include, for example, processors, controllers, arithmetic logic units (ALUs), digital signal processors (micro signal processors), microcomputers, field programmable gate arrays (FPGAs). , A programmable logic unit (PLU), microprocessor, or any other device capable of executing and responding to instructions, may be implemented using one or more general purpose computers or special purpose computers. The processing device may perform an operating system (OS) and one or more software applications running on the operating system. In addition, the processing device may access, store, manipulate, process, and generate data in response to the execution of the software. For convenience of understanding, a processing device may be described as one being used, but a person having ordinary skill in the art, the processing device may include a plurality of processing elements and / or a plurality of types of processing elements. It can be seen that may include. For example, the processing device may include a plurality of processors or a processor and a controller. In addition, other processing configurations, such as parallel processors, are possible.

The software may include a computer program, code, instruction, or a combination of one or more of these, and configure the processing device to operate as desired, or process independently or collectively You can command the device. Software and / or data may be interpreted by a processing device, or to provide instructions or data to a processing device, of any type of machine, component, physical device, virtual equipment, computer storage medium or device. Can be embodied in The software may be distributed on networked computer systems, and stored or executed in a distributed manner. Software and data may be stored on one or more computer-readable recording media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer readable medium. The computer-readable medium may include program instructions, data files, data structures, or the like alone or in combination. The medium may be a computer that continuously stores executable programs or may be temporarily stored for execution or download. In addition, the medium may be various recording means or storage means in the form of a single or several hardware combinations, and is not limited to a medium directly connected to a computer system, but may be distributed on a network. Examples of the medium include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical recording media such as CD-ROMs and DVDs, and magneto-optical media such as floptical disks, And program instructions including ROM, RAM, flash memory, and the like. In addition, examples of other media include an application store for distributing applications, a site for distributing or distributing various software, and a recording medium or storage medium managed by a server. Examples of program instructions include high-level language code that can be executed by a computer using an interpreter, etc., as well as machine language codes produced by a compiler.

As described above, although the embodiments have been described by a limited embodiment and drawings, those skilled in the art can make various modifications and variations from the above description. For example, the described techniques are performed in a different order than the described method, and / or the components of the described system, structure, device, circuit, etc. are combined or combined in a different form from the described method, or other components Alternatively, even if replaced or substituted by equivalents, appropriate results can be achieved.

Therefore, other implementations, other embodiments, and equivalents to the claims are also within the scope of the following claims.

Claims (13)

In a method performed in a computer system,
The computer system includes at least one processor configured to execute computer readable instructions contained in memory,
The above method,
Selecting, by the at least one processor, an image pair photographing a target location at predetermined time intervals;
Detecting, by the at least one processor, a signage area in each image included in the image pair; And
Recognizing a change of a point of interest with respect to the target location by comparing the signage area between the images by the at least one processor;
Including,
The step of detecting the signage area,
Detecting a plurality of signage areas in the previous image and the subsequent image, respectively, when the image pair is composed of a previous image and a subsequent image photographing the target location at the time interval;
Including,
Recognizing the change in the region of interest,
Comparing the signage area detected in the previous image and the signage area detected in the subsequent image, classifying a pair of signage areas with a similar image matching level and recognizing the remaining signage area as a change area.
How to include. According to claim 1,
The step of detecting the signage area,
Detecting the signage area from the image through a machine learning model that has learned a set of signage images
Method characterized by. According to claim 1,
The pair of images is composed of a previous panoramic image and a subsequent panoramic image in which the target position is photographed with a panoramic cut
Method characterized by. According to claim 1,
Recognizing the change in the region of interest,
Extracting each descriptor from the signage area and comparing the extracted descriptors between the signage areas to determine whether or not a match exists
How to include. According to claim 3,
The detecting of the plurality of signage areas may include:
Detecting the plurality of signage areas through scene layout detection of the previous panoramic image and the subsequent panoramic image
How to include. According to claim 2,
The above method,
Training, by the at least one processor, the machine learning model using a signboard image in which an obstruction-independent object is inserted.
Further comprising,
The training step,
Assigning a negative cost when the object is detected
How to include. According to claim 1,
The step of detecting the signage area,
Generating a main map (saliency map) by merging the signage areas detected at multiple scales of the image, and then detecting a final signage area from the main map
How to include. According to claim 1,
The step of selecting the pair of images,
Selecting a previous image and a subsequent image at different shooting points for the target location, but selecting the subsequent image from among the images captured at the latest time point and selecting the previous image from the images taken before the subsequent image
Method characterized by. The method of claim 8,
The step of selecting the pair of images,
Selecting a pair of images having a similarity in a direction equal to or greater than a preset ratio based on the photographing information of the image or a pair of images in which a photographing direction is within a preset angle difference as the previous image and the subsequent image
Method characterized by. According to claim 1,
The above method,
Visualizing the region of interest in which the change is recognized by the at least one processor
How to include more. The method of claim 10,
The step of visualizing,
Generating interest region change information including the image pair for the region of interest in which the change is recognized and providing it to an operator
How to include. A computer-readable recording medium, characterized in that a program for executing the method of any one of claims 1 to 11 is recorded on a computer. In the computer system,
At least one processor implemented to execute computer readable instructions
Including,
By the at least one processor,
Selecting a pair of images photographing the target location at predetermined time intervals;
Detecting a signage area in each image included in the image pair; And,
A process of recognizing a change in a region of interest with respect to the target location by comparing the signage region between the images
And,
The process of detecting the signage area,
When the pair of images is composed of a previous image and a subsequent image photographing the target location at the time interval, a plurality of signage regions are respectively detected from the previous image and the subsequent image,
The process of recognizing a change in the region of interest may include:
By comparing the signage area detected in the previous image and the signage area detected in the subsequent image, classifying a pair of signage areas with a similar image matching level or higher and recognizing the remaining signage area as a change area
Computer system characterized by.

Images