KR102546077B1 - Apparatus and method for classifying travel images using panoramic image conversion - Google Patents

Abstract

The present invention relates to an apparatus and method for classifying travel images using panoramic image conversion of images. To this end, it includes a CNN module that takes a representative frame as input data and outputs class classification information, which is classification information about objects included in the representative frame, and object size information, which is information about the ratio of objects in the representative frame, as output data. It includes an artificial neural network module that takes classification information and object size information as input data and zone classification information, which is classification information on the theme of a zone including location information, as output data, and provides zone classification information for travel images and image information. A representative frame classification module to output may be provided. According to this, it is possible to classify travel images into zones according to user needs.

Description

Apparatus and method for classifying travel images using panoramic image conversion {Apparatus and method for classifying travel images using panoramic image conversion}

The present invention relates to an apparatus and method for classifying travel images using panoramic image conversion of images.

The travel industry has changed rapidly with the advent of smartphones, and is one of the industries that is growing steadily every year. The travel industry can be largely divided into air travel, lodging, rental cars, activity tickets, travel guides, or package tours. Among them, the lodging reservation industry is an essential element in the travel industry, with the largest demand and supply and the most intense competition. Representative players in the accommodation reservation industry include Hotels.com, Expedia, Booking.com, Agoda, Hotel Enjoy, Daily Hotel, Sale Tonight, Hotel Time (How about here), Hotel Now ( Yanolja), Yanolja for motel reservation, How about here, Pension reservation for Yanolja Pension, Woori Pension, Go away.com, Guest house or B&B reservation for Airbnb, Kozaza, Allstay, Hotel meta search for TripAdvisor, Hotels Combine, Skyscanner, Trivago, etc.

With the advent of these various travel services, travelers have begun to develop a tendency to design their own travel experience and itinerary, unlike the way they simply purchased package tours in the past. In addition, travel services are focusing on providing a new consumption experience of travel design by providing products in a variety of ways, away from pushing products and promotions that customers do not want.

Republic of Korea Registered Patent No. 10-1979764, Hotel reservation method and system that can compensate for the difference according to the lowest price hotel reservation, Tripbitoz Co., Ltd. US registered patent US 10346402 B2, Optimized system and method for finding best fares, Expedia, Inc. US registered patent US 7783506 B2, System and method for managing reservation requests for one or more inventory items, Expedia, Inc. US registered patent US 6826543 B1, System and method for conducting transactions involving generically identified items, Hotels.com US Patent Publication US 2016-0078374 A1, GRAPHICAL USER INTERFACE FOR HOTEL SEARCH SYSTEMS, GOOGLE INC. US Patent Publication US 2013-0031506 A1, HOTEL RESULTS INTERFACE, GOOGLE INC.

As social networks such as Instagram, Facebook, TikTok, Snap, and Snow developed based on video, the need for the method of providing travel destination experiences in the travel industry to be converted to video-based was increasing. In terms of providing such an image-based travel experience, one of the methods for providing a new consumption experience of travel design is a method in which travel creators provide travel images to users on various social networks or travel services. Existing revenue that travel creators who upload travel videos (videos or images) on social networks/travel services can generate is limited to advertising revenue (banner advertisements, PPL advertisements, planned video advertisements, pre-video advertisements, etc.) However, there is no travel service system that promotes the production of travel images that efficiently induce the purchase of travel products by providing compensation when sales of travel products occur from uploaded travel images. In order to effectively configure such a travel service system, it was necessary to classify travel images according to user needs.

Accordingly, an object of the present invention is to provide an apparatus and method for classifying travel images using panoramic image conversion of images that classify travel images uploaded by creators of travel images into zones classified according to the needs of travelers.

Hereinafter, specific means for achieving the object of the present invention will be described.

An object of the present invention is to provide a travel image receiving module configured to receive a travel image generated by a creator client and composed of a plurality of travel image frames in time series and image information including location information of the travel image; A plurality of consecutive travel image frames are used as input data, and motion information including the magnitude of the acceleration of the starting point of the shooting direction vector or the angular speed of the shooting direction vector in the travel image frames is pre-learned to be output as output data. a motion information output module including an artificial neural network and generating the motion information for each of the plurality of travel image frames; When the motion information of the travel image frame is arranged in frame order, at least one frame section having relatively low motion information is selected, and the difference between the motion information and reference motion information of each frame is determined among the frame sections. a representative section selection module that selects a frame section having the largest sum value as a representative section representing the travel video; a representative frame selection module for selecting a frame having the lowest motion information within the representative section as a representative frame; And a CNN module that takes the representative frame as input data and outputs Class classification information, which is classification information about an object included in the representative frame, and object size information, which is information about a ratio of the object in the representative frame, as output data, , an artificial neural network module having the class classification information and the object size information as input data and zone classification information, which is classification information on the theme of the zone including the location information, as output data, and the travel image and the image It can be achieved by providing an artificial intelligence-based travel image classification device including a; representative frame classification module for outputting the zone classification information for information.

Another object of the present invention is a travel image reception step in which a travel image receiving module receives a travel image generated by a creator client and composed of a plurality of travel image frames in time series and image information including location information of the travel image; A plurality of consecutive travel image frames are used as input data, and motion information including the magnitude of the acceleration of the starting point of the shooting direction vector or the angular speed of the shooting direction vector in the travel image frames is pre-learned to be output as output data. a motion information outputting step of generating, by a motion information output module including an artificial neural network, the motion information for each of the plurality of travel image frames; When the representative section selecting module arranges the motion information of the travel video frame in frame order, at least one frame section having the relatively low motion information is selected, and the motion information and the motion information of each frame are selected from among the frame sections. a representative section selecting step of selecting a frame section having the largest sum of differences in reference motion information as a representative section representing the travel video; a representative frame selection step of selecting, by a representative frame selection module, a frame having the lowest motion information within the representative section as a representative frame; And a CNN module that takes the representative frame as input data and outputs Class classification information, which is classification information about an object included in the representative frame, and object size information, which is information about a ratio of the object in the representative frame, as output data, , A representative frame classification module including an artificial neural network module having the class classification information and the object size information as input data and zone classification information, which is classification information on the theme of the zone including the location information, as output data, It can be achieved by providing an artificial intelligence-based travel image classification method, including a representative frame classification step of outputting the travel image and the zone classification information for the image information.

Another object of the present invention is a memory module including a travel image classification program code; and a processing module that executes the travel image classification program code. The travel image classification program code includes a travel image receiving step of receiving a travel image generated by a creator client and composed of a plurality of travel image frames in time series and image information including location information of the travel image; A plurality of consecutive travel image frames are used as input data, and motion information including the magnitude of the acceleration of the starting point of the shooting direction vector or the angular speed of the shooting direction vector in the travel image frames is pre-learned to be output as output data. a motion information output step of inputting the travel image frames to an artificial neural network and generating the motion information for each of the plurality of travel image frames; When the motion information of the travel image frame is arranged in frame order, at least one frame section having relatively low motion information is selected, and the difference between the motion information and reference motion information of each frame is determined among the frame sections. a representative section selecting step of selecting a frame section having the largest sum value as a representative section representing the travel video; a representative frame selecting step of selecting a frame having the lowest motion information within the representative section as a representative frame; and Class classification information, which is classification information for an object included in the representative frame, and object size information, which is information about a ratio of the object in the representative frame, to a CNN module that takes the representative frame as input data and outputs the object size information as output data. A frame is input to output the class classification information and the object size information, and the class classification information and the object size information are used as input data and zone classification information, which is classification information on the theme of the zone including the location information, is output. A representative frame classification step of inputting the class classification information and the object size information to an artificial neural network module as data and outputting the zone classification information for the travel image and the image information; configured to perform steps including on a computer Characterized in that, it can be achieved by providing an artificial intelligence-based travel image classification device.

Another object of the present invention is a travel image receiving module configured to receive a travel image generated by a creator client and composed of a plurality of travel image frames in time series and image information including location information of the travel image; a representative frame selection module for selecting a specific frame among the frames of the travel video as a representative frame; A representative frame classification artificial neural network module having the representative frame as input data and first zone classification information, which is classification information on the theme of the zone including the location information, as output data, a representative frame classification module outputting classification information of the first zone for the first zone; A feature extraction module including a CNN-based feature extraction artificial neural network that uses input data as a representative frame and includes feature class information (class probability) and feature coordinate information (coordinate data) as output data; It is connected to a specific convolution layer of the representative frame classification module to receive class activation information, which means an activation map for each class, and uses the class activation information and the feature coordinate information to generate information about each class. a core feature selection module that generates feature importance information and selects a class having the largest generated feature importance information as a core feature; a candidate frame selection module for selecting a plurality of frames including the core feature among frames before and after the representative frame selected by the representative frame selection module as candidate frames; and a frame matching module generating a representative panorama frame by stitching the representative frame and the candidate frame based on the core feature, wherein the representative panorama frame generated by the frame matching module is classified as the representative frame. Characterized in that it is input to the module as input data and configured to output the second zone classification information as output data. This can be achieved by providing an apparatus for classifying travel images using panoramic image conversion of images.

In addition, the feature importance information may include a ratio in which a hitmap of the class activation information is included within a bounding box range according to the feature coordinate information of each class or the class activation information within the bounding box range of each class. It may be characterized in that it includes the sum of the activation values of.

Another object of the present invention is a travel image reception step in which a travel image receiving module receives a travel image generated by a creator client and composed of a plurality of travel image frames in time series and image information including location information of the travel image; a representative frame selection step in which a representative frame selection module selects a specific frame among the frames of the travel video as a representative frame; A representative frame classification module including a representative frame classification artificial neural network module that takes the representative frame as input data and first zone classification information, which is classification information on the theme of the zone including the location information, as output data, and a representative frame classification step of outputting the first zone classification information for the image information. A feature extraction module including a CNN-based feature extraction artificial neural network using input data as a representative frame and including feature class information (class probability) and feature coordinate information (coordinate data) as output data, the feature class information and the feature a feature extraction step of outputting coordinate information; A core feature selection module is connected to a specific convolution layer of the representative frame classification module to receive class activation information meaning an activation map for each class, and the class activation information and the feature coordinate information a core feature selection step of generating feature importance information of each class by using and selecting a class having the largest generated feature importance information as a core feature; a candidate frame selection step of selecting, by a candidate frame selection module, a plurality of frames including the core feature among frames before and after the representative frame as candidate frames; and a frame matching step of stitching, by a frame matching module, the representative frame and the candidate frame based on the core feature to generate a representative panorama frame, wherein the representative frame of the panorama is sent to the representative frame classification module. Characterized in that it is input as input data and configured to output the second zone classification information as output data. This can be achieved by providing a method for classifying travel images using panoramic image conversion of images.

Another object of the present invention is a memory module including a travel image classification program code; and a processing module that executes the travel image classification program code, wherein the travel image classification program code includes a travel image generated by a creator client and composed of a plurality of travel image frames in time series and location information of the travel image. A travel image reception step of receiving image information to be performed; a representative frame selection step of selecting a specific frame among the frames of the travel video as a representative frame; A representative frame classification artificial neural network module having the representative frame as input data and first zone classification information, which is classification information on the theme of the zone including the location information, as output data Representative frame classification step of outputting zone 1 classification information; CNN-based feature extraction using input data as a representative frame and including feature class information (class probability) and feature coordinate information (coordinate data) as output data Feature extraction in which an artificial neural network outputs the feature class information and the feature coordinate information step; It is connected to a specific convolution layer of the representative frame classification artificial neural network module to receive class activation information, which means an activation map for each class, and uses the class activation information and the feature coordinate information to a core feature selection step of generating feature importance information of a class and selecting a class having the largest generated feature importance information as a core feature; a candidate frame selection step of selecting a plurality of frames including the core feature among frames before and after the representative frame as candidate frames; and a frame stitching step of generating a panorama representative frame by stitching the representative frame and the candidate frame based on the core feature, wherein the panorama representative frame is It can be achieved by providing a travel image classification device using panoramic image conversion of an image, characterized in that it is input as input data to the representative frame classification module and configured to output second zone classification information as output data.

Another object of the present invention is a travel image reception step in which a travel image receiving module receives a travel image generated by a creator client and composed of a plurality of travel image frames in time series and image information including location information of the travel image; a representative frame selection step in which a representative frame selection module selects a specific frame among the frames of the travel video as a representative frame; A representative frame classification module including a representative frame classification artificial neural network module that takes the representative frame as input data and first zone classification information, which is classification information on the theme of the zone including the location information, as output data, and a representative frame classification step of outputting the first zone classification information for the image information. A feature extraction module including a CNN-based feature extraction artificial neural network using input data as a representative frame and including feature class information (class probability) and feature coordinate information (coordinate data) as output data, the feature class information and the feature a feature extraction step of outputting coordinate information; A core feature selection module is connected to a specific convolution layer of the representative frame classification module to receive class activation information meaning an activation map for each class, and the class activation information and the feature coordinate information a core feature selection step of generating feature importance information of each class by using and selecting a class having the largest generated feature importance information as a core feature; a candidate frame selection step of selecting, by a candidate frame selection module, a plurality of frames including the core feature among frames before and after the representative frame as candidate frames; and a frame stitching step in which a frame matching module generates a panorama representative frame by stitching the representative frame and the candidate frame based on the core feature, on a computer. Characterized in that the panoramic representative frame is input as input data to the representative frame classification module and configured to output second zone classification information as output data. It can be achieved by providing a program stored in a recording medium configured to perform a travel image classification method using a panoramic image conversion of an image on a computer.

As described above, the present invention has the following effects.

First, according to an embodiment of the present invention, it is possible to classify travel images into zones according to user needs.

Second, according to one embodiment of the present invention, the creator who uploaded the travel video has an effect of being able to give a different reward according to the zone classification information of the travel video.

The following drawings attached to this specification illustrate preferred embodiments of the present invention, and together with the detailed description of the invention serve to further understand the technical idea of the present invention, the present invention is limited only to those described in the drawings. and should not be interpreted.
1 is a schematic diagram showing an artificial intelligence-based travel image classification device according to an embodiment of the present invention;
2 is a schematic diagram showing the operational relationship of an artificial intelligence-based travel image classification device according to an embodiment of the present invention;
3 is a schematic diagram showing the operational relationship of the motion information output module 11 according to an embodiment of the present invention;
4 is a schematic diagram showing a motion information output module 11 including a reinforcement learning module according to another embodiment of the present invention;
5 is a schematic diagram showing the representative section selection of the representative section selection module 12;
6 is a schematic diagram showing the configuration of a representative frame classification module 14 according to an embodiment of the present invention;
7 is a schematic diagram showing a th nCNN module according to an embodiment of the present invention;
8 is a schematic diagram showing a classification artificial neural network module according to an embodiment of the present invention;
9 is a schematic diagram of mapping the area classification information output by the area classification artificial neural network module to a map of a travel area according to an embodiment of the present invention;
10 is a schematic diagram showing a travel image classification device using panoramic image conversion, which is a travel image classification device according to a modified example of the present invention;
11 is a schematic diagram showing the operational relationship of a travel image classification device using panoramic image conversion, which is a travel image classification device according to a modified example of the present invention;
12 is a schematic diagram showing the structure of a feature extraction module 21, which is a component of a travel image classification device according to a modified example of the present invention;
13 is a schematic diagram showing the structure when a feature extraction artificial neural network module according to a modified example of the present invention is composed of YOLO v1 (CVPR 2016);
14 is a schematic diagram showing the operational relationship of the core feature selection module 22 according to a modified example of the present invention;
15 is a schematic diagram showing the operational relationship of the core feature selection module 22 according to the second modified example of the present invention;
16 is a schematic diagram showing a candidate frame selection module 23 according to a modified example of the present invention.

Hereinafter, an embodiment in which a person skilled in the art can easily practice the present invention will be described in detail with reference to the accompanying drawings. However, in the detailed description of the operating principle of the preferred embodiment of the present invention, if it is determined that a detailed description of a related known function or configuration may unnecessarily obscure the subject matter of the present invention, the detailed description will be omitted.

In addition, the same reference numerals are used for parts having similar functions and actions throughout the drawings. Throughout the specification, when a specific part is said to be connected to another part, this includes not only the case where it is directly connected but also the case where it is indirectly connected with another element interposed therebetween. In addition, including a specific component does not exclude other components unless otherwise stated, but means that other components may be further included.

In the following description of the invention, the convolutional neural network, which is a neural network using convolutional products, may be described as CNN, ConvNet, and the like.

In the following description of the invention, the description is made based on hotel reservations for convenience of explanation, but the scope of the present invention is not limited to hotel reservations, but it is not limited to bed and breakfasts, hostels, motels, time commerce of hotels, activities, package tours, guides, rental cars, etc. You can include coverage for all travel products.

Hereinafter, travel images may refer to audiovisual information of various compression methods and various codecs captured in a specific travel region.

AI-based travel image classification device

1 is a schematic diagram showing an artificial intelligence-based travel image classification device according to an embodiment of the present invention. As shown in FIG. 1, the artificial intelligence-based travel image classification device 1 according to an embodiment of the present invention includes a travel image reception module 10, a motion information output module 11, and a representative section selection module 12. ), a representative frame selection module 13, and a representative frame classification module 14. 2 is a schematic diagram showing the operational relationship of an artificial intelligence-based travel image classification device according to an embodiment of the present invention. As shown in FIG. 2, the application module configured in the client (creator client, 100) of the user (travel creator) who creates the travel image generates the travel image, and the AI-based travel image classification device 1 The travel image reception module 10 transmits each frame of the received travel image, that is, a travel image frame, to the motion information output module 11. The motion information output module 11 generates motion information based on a plurality of consecutive travel video frames, and the representative section selection module 12 divides the consecutive travel video frames into a plurality of sections based on the motion information for each frame. And, representative section information is generated by selecting a representative section among a plurality of sections. The representative frame selection module 13 selects a specific travel video frame within the representative section information as a representative frame to generate representative frame information, and the representative frame classification module 14 receives a representative frame corresponding to the representative frame information It can be configured to output zone classification information. Detailed descriptions and examples of each configuration are as follows.

The travel video reception module 10 is a module that receives the travel video and image information transmitted from the creator client 100 in the application module of the creator client 100 . A travel video according to an embodiment of the present invention is composed of a plurality of travel image frames configured in time series, and the image information includes creator information, location information, time information, audio format information, audio sample rate (for example, , hertz units), audio stream channel number information, file name information, size information, total frame number information, frames per second information, video height information (eg, pixel units), video width information (eg, pixel units) , image type information, compression codec information, and the like. The travel video according to an embodiment of the present invention may include an image taken by a travel creator, who is a creator of the travel video, through a camera of the creator client 100 or an image pre-stored in the creator client 100. According to another embodiment of the present invention, the creator client 100 may be configured to transmit the travel image and image information to the travel image receiving module 10 from a server or database that has received the travel image.

The motion information output module 11 is a module including a pretrained artificial neural network to receive travel image frames in frame time order and to output motion information using two consecutive travel image frames as input data. 3 is a schematic diagram showing an operating relationship of the motion information output module 11 according to an embodiment of the present invention. As shown in FIG. 3, the motion information output module 11 may be configured to include an artificial neural network module that takes two consecutive travel image frames as input data and motion information as output data, and in frame order. It may be configured to output motion information according to the At this time, inputting two travel image frames as input data is an embodiment of the present invention, and the scope of the present invention may include inputting a plurality of travel image frames as input data. As shown in FIG. 3, in the motion information output module 11 according to an embodiment of the present invention, travel image frames t-3 and t-2 are input as input data, and motion information t-2 is output. Image frames t-2 and t-1 are input as input data to output motion information t-1, and travel image frames t-1 and t are input as input data and motion information t is output.

Motion information is the magnitude of acceleration (scalar value, magnitude of acceleration) of the shooting camera (the camera configured in the creator client according to one embodiment, the starting point of the shooting direction vector), the angular speed of the shooting direction vector (scalar value, angular speed ) or a value obtained by combining the magnitude of the camera's acceleration and the angular velocity of the shooting direction vector. In the learning session of the motion information output module 11 , comparison data serving as reference information for motion information for calculating loss may be calculated as sensing values of an acceleration sensor and a gyro sensor of the creator client 100 . For example, the magnitude of the acceleration of the camera can be calculated by the motion information output module 11 as the magnitude of an acceleration vector output from the acceleration sensor of the creator client 100, and specifically, the x-axis output value of the acceleration sensor (d ² x/dt ² , where x is the x-axis displacement vector, y-axis output (d ² y/dt ² , y is the y-axis displacement vector), and z-axis output (d ² z/dt ² , where z is the z-axis displacement vector) ) can be calculated by adding the squares of each, then taking the square root. In addition, the angular speed of the photographing direction vector may be calculated by the motion information output module 11 as the magnitude of the angular velocity vector output from the gyro sensor of the creator client 100, and specifically, the x-axis output value of the gyro sensor (dθ _x /dt, θ _x is the x-axis rotation angle), the y-axis output value (dθ _y /dt, θ _y is the y-axis rotation angle), and the z-axis output value (dθ _z /dt, θz is the z-axis rotation angle) by squaring each It can be calculated by adding and then taking the square root.

In an artificial neural network training session of the motion information output module 11, the motion information output module 11 receives a plurality of consecutive travel image frames as input data, and selects a specific frame among the input travel image frames. After outputting the motion information as output data, the loss of the comparison data of the output data is reduced through calculation of a loss function based on the difference between the comparison data and the output data, which are motion information calculated for the specific frame. It may be configured to update parameters such as weight of the motion information output module 11. At this time, as the loss function of the motion information output module 11 according to an embodiment of the present invention, a cross-entropy loss function, a contrast loss function, a center loss function, and the like may be used.

In the artificial neural network inference session of the motion information output module 11, the motion information output module 11 receives a plurality of consecutive travel image frames as input data, and selects a specific frame among the input travel image frames. It may be configured to output motion information as output data.

According to this, the creator client 100 does not have to receive the output value of the acceleration sensor and the output value of the gyro sensor every frame, thereby simplifying the application module and reducing the size of information to be transmitted and received and the size of information to be processed.

According to another embodiment of the present invention, the motion information output module 11 may include a reinforcement learning module. 4 is a schematic diagram showing a motion information output module 11 including a reinforcement learning module according to another embodiment of the present invention. As shown in FIG. 4, the reinforcement learning module of the motion information output module 11 according to another embodiment of the present invention uses a travel image as an environment and the motion information output module 11 as an agent, and input data A plurality of contiguous travel video frames are taken as a state, and in this state, the motion information (output data) output for a specific frame by the motion information output module 11, which is an agent, is taken as an action, and the difference between the output data and the comparison data The smaller the is, the higher the Reward is generated and can be configured to update the motion information output module 11, which is an agent.

The representative section selection module 12 selects at least one frame section composed of consecutive travel image frames showing a relatively low camera movement based on the motion information generated for each travel image frame in the motion information output module 11. and a module for selecting a representative section, which is a section representing the travel video, among the corresponding sections. 5 is a schematic diagram showing selection of a representative section by the representative section selection module 12 . As shown in FIG. 5, when the motion information generated for each travel image frame in the motion information output module 11 is arranged according to the frame order, a plurality of frame sections are selected, and a representative section is selected from among the corresponding frame sections. It can be configured to select. A specific representative section selection method of the representative section selection module 12 according to an embodiment of the present invention is as follows.

(1) Selecting at least one frame section composed of a plurality of consecutive frames having motion information equal to or less than specific motion information (reference motion information)

(2) If there is only one selected frame section, the frame section is selected as a representative section

(2) If there are a plurality of selected frame sections, the frame section having the largest sum of the difference between the motion information of each frame and the reference motion information among the selected plurality of frame sections is selected as a representative section.

According to this, an effect of being able to automatically extract a frame section including a main context in a travel image created by a travel creator is generated by an algorithm with a low computational load.

The representative frame selection module 13 is a module for selecting a representative frame within the representative section selected by the representative section selection module 12 . According to an embodiment of the present invention, a frame having the lowest motion information within a representative section may be selected as a representative frame and representative frame information may be generated.

According to the representative section selection module 12 and the representative frame selection module 13, an effect of being able to prevent a case in which a representative frame is selected when a camera is momentarily stopped occurs. In addition, an effect of being able to automatically extract a representative frame including a main context from a travel image created by a travel creator is generated by an algorithm with a low computational load. In the past, the creator had no choice but to directly perform the task of extracting the frame representing the context of the travel video.

The representative frame classification module 14 may refer to an artificial neural network module that uses the representative frame selected by the representative frame selection module 13 as input data and receives zone classification information as output data. Zone classification information refers to classification information on a theme of a zone in which corresponding image information is photographed. For example, this may indicate zone classification such as a play zone, a drinking zone, an eating zone, a viewing zone, and a shopping zone. According to one embodiment of the present invention, the playing zone is an area including a club, beach, swimming pool, etc., the drinking zone is an area including a wine bar, cocktail bar, beer pub, bartle shop, etc., and the eating zone is an area including various restaurants, the corresponding area Zones that include restaurants or street food, viewing zones are zones that include various tourist zones or natural scenery zones, and shopping zones are zones that include luxury stores, department stores, outlets, duty-free shops, souvenir shops, and street shopping. can

Regarding the configuration of the representative frame classification module 14, FIG. 6 is a schematic diagram showing the configuration of the representative frame classification module 14 according to an embodiment of the present invention. As shown in FIG. 6 , the representative frame classification module 14 according to an embodiment of the present invention may include a plurality of CNN modules and a region classification artificial neural network module.

A plurality of CNN modules of the representative frame classification module 14 are trained with different training data or configured as networks having different parameters. 7 is a schematic diagram showing a th nCNN module according to an embodiment of the present invention. As shown in FIG. 7, a representative frame is input as source data to the input layer of the nCNN module according to an embodiment of the present invention, for example, a channel of 32 width, 32 length, and height n. With , the size of the input can be composed of a matrix of [32x32xn]. The CONV layer (Conv. layer) is Conv. It is calculated by being connected to some area of the representative frame, which is the source data, by Filter, and the dot product of this connected area and weight is calculated. For example, Conv. The layer's volume will have dimensions equal to [32x32x12]. After that, an activation function such as a RELU layer is calculated, and RELU is an activation function applied to each element, such as max(0,x), and does not change the size of the volume (for example, still [32x32x12] ) to generate an activation map. The POOL layer (pooling layer) performs downsampling on the “horizontal, vertical” dimension and outputs a reduced volume (activation map), for example [16x16x12]. Class scores are calculated after a fully-connected (FC) layer connected to the n-th activation map n), and an output layer containing n-th classification information having a size of [m x m x 1] is output. Classification information according to an embodiment of the present invention may mean classification information on an object included in a representative frame, and may be configured to output at least one or more from each CNN module.

Classification information of a plurality of CNN modules according to an embodiment of the present invention includes information on the classified class of the detected object (class classification information) and information on the ratio of the corresponding object in the input representative frame (object size information) ), and the loss function may include a class loss function and an object size loss function. The loss function of the CNN module is used to update the weight of the artificial neural network in the training session of the CNN module, and the weight of the CNN module can be updated so that the loss function is the minimum (global minimum or local minimum). The class loss function may mean a difference between class classification information included in classification information output from a CNN module inputting a representative frame and ground truth (comparison data), which is class classification information of a real object included in the representative frame. The object size loss function means the difference between the object size information included in the classification information output from the CNN module with the representative frame input and the ground truth (comparison data), which is the ratio of the actual object included in the representative frame to the representative frame. can Class classification information according to an embodiment of the present invention may refer to information for classifying objects such as sky, sea, swimming pool, food, drink, table, chair, car, bicycle, and person, for example. there is. Object size information according to an embodiment of the present invention may mean ratios such as 30% and 4%.

In addition, a plurality of CNN modules according to an embodiment of the present invention may be trained to classify different objects through different learning data. According to the configuration of the learning data and the loss function, a plurality of CNN modules may be configured to output a class for a specific object that is different from each other and a size of the object.

According to another embodiment of the present invention, a CNN architecture for multiple object detection such as YOLO or R-CNN can be configured instead of a plurality of CNN modules, and the output data and loss function are multiple The output data tensor (class classification information and object size information) and loss function (class loss function and object size loss function) of the CNN module of can be applied.

The zone classification artificial neural network module of the representative frame classification module 14 takes as input data a plurality of classification information (class classification information and object size information) for a plurality of objects output from a plurality of CNN modules, and outputs zone classification information. It can be composed of an artificial neural network with data. 8 is a schematic diagram showing a classification artificial neural network module according to an embodiment of the present invention. As shown in FIG. 8, a plurality of classification information (first classification information, second classification information, ..., n-th classification information) output from a plurality of CNN modules is used as input data, and zone classification information is output information. It can be composed of an artificial neural network.

In a training session of the zone classification artificial neural network module according to an embodiment of the present invention, classification information output from a plurality of CNN modules inputted with representative frames is input to the zone classification artificial neural network module as input data, and Based on the difference between the zone classification information output as output data from the classification artificial neural network module and the actual zone classification information (comparison data, ground truth) of the corresponding travel image, back propagation is used to learn the hidden layer of the zone classification artificial neural network module. It can be configured to update weights.

In an inference session of the zone classification artificial neural network module according to an embodiment of the present invention, classification information output from a plurality of CNN modules inputted with representative frames is input to the zone classification artificial neural network module as input data, and It may be configured to utilize zone classification information output as output data from the classification artificial neural network module as zone classification information of the corresponding travel image.

For example, in multiple CNN modules, classification information is output as (food, 12%), (table, 30%), (person, 20%), (spoon, 3%), etc., and the classification information is classified into zones. When input is input to the artificial neural network module, the zone classification artificial neural network module may be configured to output zone classification information as 'eating zone'.

According to this, compared to a structure composed of one CNN module that receives a representative frame and outputs zone classification information, an effect of facilitating optimization occurs even with small training data or learning data with a severe imbalance in the amount of data for each class. In addition, since the images or videos that can be classified into each zone are very diverse, it is very difficult to achieve high classification performance with the CNN architecture. According to one embodiment of the present invention, classification information output by the CNN module is transferred to the artificial neural network module. Reclassification has the effect of solving these problems.

Regarding the use of zone classification information, FIG. 9 is a schematic diagram of mapping zone classification information output by the zone classification artificial neural network module to a map of a travel area according to an embodiment of the present invention. As shown in FIG. 9, zone classification information according to an embodiment of the present invention is mapped to a map of a specific travel area based on location information included in image information, and zones are displayed on a display of a user client configured with a corresponding application module. Map information of the travel area including classification information is output, and based on the map information, the user can move to the location to enjoy the desired theme or obtain information about the area related to the desired theme online through the application module. do.

[Modified Example - Travel Image Classification Device Using Panoramic Image Conversion of Images]

In relation to a travel image classification device using panoramic image conversion of an image, which is a travel image classification device according to a modified example of the present invention, FIG. 10 is a travel image classification device according to a modified example of the present invention, travel using panoramic image conversion 11 is a schematic diagram showing an image classification device, which is a travel image classification device according to a modified example of the present invention, showing the operational relationship of the travel image classification device using panoramic image conversion. As shown in FIGS. 10 and 11, according to the travel image classification device using panoramic image conversion, which is a travel image classification device according to a modified example of the present invention, the artificial intelligence-based travel image according to an embodiment of the present invention The classification device 1 further includes a feature extraction module 21, a core feature selection module 22, a candidate frame selection module 23, and a frame matching module 24, and the representative frame of the representative frame selection module 13 After the selection, the representative frame is converted into a panoramic image by the feature extraction module 21, the core feature selection module 22, the candidate frame selection module 23, and the frame matching module 24 to generate a panoramic representative frame, It may be characterized by inputting the generated representative frame of the panorama to the representative frame classification module 14 .

Regarding the feature extraction module 21, FIG. 12 is a schematic diagram showing the structure of the feature extraction module 21, which is a component of a travel image classification device according to a modified example of the present invention. As shown in FIG. 12, the feature extraction module 21 includes a feature extraction artificial neural network module for extracting object features of the representative frame selected by the representative frame selection module 13, and the feature extraction artificial neural network module It may be composed of a ConvNet-based artificial neural network that includes feature class information (class probability) and feature coordinate information (coordinate data) as output data and input data as a representative frame or candidate frame.

The feature extraction artificial neural network module according to an embodiment of the present invention may include a multi-object detection artificial neural network module that performs classification and localization, and this multi-object detection artificial neural network module includes RCNN as a 2-stage detector ( 2013), OverFeat (ICLR 2014), Fast RCNN (ICCV 2015), Faster RCNN (NIPS 2015), Mask RCNN (ICCV 2017), etc. can be used, and as a 1-stage detector, anchor-based YOLO v1 (CVPR 2016) , YOLO v2 (CVPR 2017), YOLO v3 (arXiv 2018), SSD (ECCV 2016), RetinaNet (ICCV 2017), etc. can be used, and as a 1-stage detector, non-anchor based CornerNet (ECCV 2018), ExtreamNet (2019), CenterNet (2019), etc. can be utilized.

Output data of the feature extraction artificial neural network module according to an embodiment of the present invention includes feature class information (confidence score or class probability) indicating a specific class of at least one object and its reliability, feature coordinate information ( coordinate data), and the feature coordinate information of the object includes the coordinates of the top-left corner and bottom-right corner of the bounding box, the coordinates of the centeral region of the bounding box, and the width and height of the bounding box according to the configuration of the artificial neural network. can be configured to include

13 is a schematic diagram showing the structure of a feature extraction artificial neural network module according to a modified example of the present invention composed of YOLO v1 (CVPR 2016). For example, as shown in FIG. 13, when the feature extraction artificial neural network module according to the modified example of the present invention is configured with YOLO v1 (CVPR 2016), DarkNet Architecture is used, and feature maps are extracted through convolution layers It is configured to infer the coordinate data (feature coordinate information) and class probability (class confidence, feature class information) of the bounding box directly through the fully connected layer and output them as output data. In YOLO, a representative frame or candidate frame, which is an input image, is divided into SxS grids, and bounding boxes (SxSxB) corresponding to each grid area, class confidence (Probability (object) × IoU (prediction, ground truth)), Class probability map (Probability (Class_i|object)). For example, a specific network structure can be configured to output 5 bounding box coordinates (feature coordinate information) and confidence score (feature class information) per grid area, and a representative frame or candidate frame is input in a size of 448x448x3 Activation map of DarkNet Architecture can be configured in size of 7x7x1024, and Fully Connected Layer of 4096 and 7x7x30 can be configured after DarkNet Architecture.

Regarding the core feature selection module 22, FIG. 14 is a schematic diagram showing the operational relationship of the core feature selection module 22 according to a modified example of the present invention. As shown in FIG. 14, the core feature selection module 22 is connected to the last Conv.layer (Conv.layer before the fully connected layer) of the nCNN module of the representative frame classification module 14, and is connected to the representative frame classification module ( When the input data of 14) is a representative frame, activation information (including activation values for each coordinate) including at least one-dimensional activation map output from Conv.layer is input, and the representative frame is input to the feature extraction module 21 It receives the coordinate data (feature coordinate information) of each class that is input as data and output as output data, and the coordinate data (feature coordinate information) of each class input from the feature extraction module 21 or the coordinates of activation information is converted into a representative frame. It is adjusted according to the size of the activation map and the size of the activation map, and the rate (activation rate) or activation that the hitmap of the activation map of the representative frame classification module (14) is included within the bounding box range according to the coordinate data (feature coordinate information) of each class. The sum of weights (activation values) is output as feature importance information for each class. The class with the largest output feature importance information is selected as the core feature.

Regarding the second modified example of the core feature selection module 22, FIG. 15 is a schematic diagram showing the operational relationship of the core feature selection module 22 according to the second modified example of the present invention. As shown in FIG. 15, according to the second modified example of the present invention, the representative frame classification module 14 is composed of an artificial neural network composed of a single convolutional network that outputs zone classification information as output data, and selects core features. A class activation generation module 220 for dimension reduction is further included between the module 22 and the representative frame classification module 14, and the class activation generation module 220 of the second modified example comprises the representative frame classification module 14 Activation information including at least one-dimensional activation map output from the Conv.layer when the input data of the representative frame classification module 14 is a representative frame when it is connected to the last Conv.layer (Conv.layer prior to the fully connected layer) of (including activation values by coordinates), outputs class activation information (including activation values by coordinates according to each class) including activation maps corresponding to each class, and outputs zone classification information through the sigmoid function It can be learned together with the representative frame classification module 14. In addition, the core feature selection module 22 receives the class activation information and the coordinate data (feature coordinate information) of each class that is output as output data after the representative frame is input to the feature extraction module 21 as input data. The class of the representative frame classification module 14 within the bounding box range according to the coordinate data (feature coordinate information) of each class by adjusting the coordinates of the feature coordinate information or class activation information to be used according to the size of the representative frame and the size of the class activation map. The ratio (activation rate) or the sum of activation weights (activation values) included in the hitmap of the activation map is output as feature importance information for each class. The class with the largest output feature importance information is selected as the core feature.

At this time, generation of class activation information by the class activation generation module 220 may be performed as follows.

In Equation 1 above, M _c (x, y) is the activation value located at (x, y) that affects classification into class c, w _k ^c is the weight of the k-th channel for class c in the activation map, f _k (x, y) means an activation value located at (x, y) of the k-th channel of the activation map.

According to the second modified example of the present invention, since feature importance information for each class is output using a class activation map generated differently for each class instead of an activation map for all classes, inference of zone classification information There is an effect of being able to select a more important object as core feature information. In addition, since the representative frame classification module 14 to the class activation generation module 220 can be configured as a single artificial neural network module, the efficiency of artificial neural network learning and reasoning is improved.

Regarding the candidate frame selection module 23, FIG. 16 is a schematic diagram showing the candidate frame selection module 23 according to a modified example of the present invention. As shown in FIG. 16, the candidate frame selection module 23 selects a plurality of frames including the core feature among the frames before and after the representative frame selected by the representative frame selection module 13 as candidate frames. am. The candidate frame selection module 23 inputs the front and back frames to the feature extraction module 21 and sequentially selects frames having a confidence score (class probability) of a certain level or higher for the same class as the core feature among the output feature class information. module to select. For example, in FIG. 15, when a frame having a confidence score of 0.7 or higher for the same class as a core feature is selected as a candidate frame, a frame having a confidence score of 0.65 and a representative frame may be sequentially selected as candidate frames. there is.

The frame matching module 24 is a module that creates a panorama representative frame by stitching a representative frame and a candidate frame based on a core feature. As a matching method between a representative frame and a candidate frame, an image of a candidate frame is divided based on a core feature to generate a divided image, and among the divided images, a divided image that does not include a core feature is matched to the representative frame. It can be configured to create a frame. The panoramic representative frame generated by the frame matching module 24 may be input to the representative frame classification module 14 as input data to output second zone classification information, which is zone classification information having better performance, as output data.

According to the modified example of the present invention, it is possible to infer zone classification by converting a representative frame into a panoramic image, so that the disadvantages of the method of classifying zones of the entire travel image through the representative frame can be offset. . In addition, since it is configured to generate a panoramic image in the direction of improving the inference performance of the artificial neural network module that infers the region classification, the accuracy of the region classification of the travel image is improved compared to the existing computer vision panoramic image generation algorithm. occurs.

Conventionally, in order to generate a panoramic image, feature points are extracted with computer vision algorithms such as SIFT (Scale-invariant Feature Transform) algorithm, SURF (Speed Up Robust Features) algorithm, CDVS (Compact Descriptor for Visual Search) algorithm, and RANSAC algorithm, PROSAC algorithm, etc. are applied to remove outliers among feature points, select matching points among inlier feature points of two images, and generate a homography matrix using matching points selected as true feature points (inliers) of each image. A panorama image is generated by calculating and performing image matching (stiching) using the calculated homography matrix. When this conventional method is applied to the generation of a representative panorama frame according to an embodiment of the present invention, significant computing resources are required to compute feature point extraction and outlier removal for the entire frame of the travel image, and computer vision-based existing Due to the nature of the algorithm, various types of problems arise because outliers cannot be removed reliably, and a problem arises in that a panoramic image is generated that is not at all related to the region classification of the travel video.

As described above, those skilled in the art to which the present invention pertains will be able to understand that the present invention can be embodied in other specific forms without changing its technical spirit or essential features. Therefore, the above-described embodiments should be understood as illustrative in all respects and not restrictive. The scope of the present invention is indicated by the claims to be described later rather than the detailed description, and all changes or modifications derived from the meaning and scope of the claims and equivalent concepts should be construed as being included in the scope of the present invention.

The features and advantages described in this specification are not all inclusive, and many additional features and advantages will become apparent to those skilled in the art, particularly from consideration of the drawings, specification, and claims. Moreover, it should be noted that the language used herein has been chosen primarily for readability and instructional purposes, and may not have been chosen to delineate or limit the subject matter of the invention.

The foregoing description of embodiments of the present invention has been presented for purposes of illustration. It is not intended to limit the invention to the precise form disclosed or to make it without omission. Those skilled in the art can appreciate that many modifications and variations are possible in light of the above disclosure.

Therefore, the scope of the present invention is not limited by the detailed description, but by any claims of the application based thereon. Accordingly, the disclosure of embodiments of the invention is illustrative and not limiting of the scope of the invention set forth in the claims below.

1: AI-based travel image classification device
10: travel video receiving module
11: motion information output module
12: Representative section selection module
13: Representative frame selection module
14: representative frame classification module
21: feature extraction module
22: Core feature selection module
23: candidate frame selection module
24: frame matching module
220: class activation generation module
100: constructor client

Claims (5)

a travel image receiving module configured to receive a travel image generated by a creator client and composed of a plurality of travel image frames in time series and image information including location information of the travel image;
a representative frame selection module for selecting a specific frame among the frames of the travel video as a representative frame;
A representative frame classification artificial neural network module having the representative frame as input data and first zone classification information, which is classification information on the theme of the zone including the location information, as output data, a representative frame classification module outputting classification information of the first zone for the first zone;
Characteristics comprising a CNN-based feature extraction artificial neural network including feature class information (class probability) and feature coordinate information (coordinate data) of one or more object features extracted from the representative frame as input data and output data as output data extraction module;
It is connected to a specific convolution layer of the representative frame classification module and receives class activation information including an activation map for each class, and uses the class activation information and the feature coordinate information to determine the classification of each class. a core feature selection module that generates feature importance information and selects a class having the largest generated feature importance information as a core feature;
a candidate frame selection module for selecting a plurality of frames including the core feature among frames before and after the representative frame selected by the representative frame selection module as candidate frames; and
a frame stitching module generating a panoramic representative frame by stitching the representative frame and the candidate frame based on the core feature;
including,
Characterized in that the panorama representative frame generated by the frame matching module is input as input data to the representative frame classification module to output second zone classification information as output data.
An apparatus for classifying travel images using panoramic image conversion of images.
According to claim 1,
The feature importance information,
The rate at which the hitmap of the class activation information is included within the bounding box range according to the feature coordinate information of each class or the sum of the activation values of the class activation information within the bounding box range of each class characterized in that,
An apparatus for classifying travel images using panoramic image conversion of images.
a travel image reception step in which a travel image reception module receives a travel image generated by a creator client and composed of a plurality of travel image frames in time series and image information including location information of the travel image;
a representative frame selection step in which a representative frame selection module selects a specific frame among the frames of the travel video as a representative frame;
A representative frame classification module including a representative frame classification artificial neural network module that takes the representative frame as input data and first zone classification information, which is classification information on the theme of the zone including the location information, as output data, and a representative frame classification step of outputting the first zone classification information for the image information.
Characteristics comprising a CNN-based feature extraction artificial neural network including feature class information (class probability) and feature coordinate information (coordinate data) of one or more object features extracted from the representative frame as input data and output data as output data a feature extraction step of outputting, by an extraction module, the feature class information and the feature coordinate information;
A core feature selection module is connected to a specific convolution layer of the representative frame classification module to receive class activation information including an activation map for each class, and the class activation information and the feature coordinate information a core feature selection step of generating feature importance information of each class by using and selecting a class having the largest generated feature importance information as a core feature;
a candidate frame selection step of selecting, by a candidate frame selection module, a plurality of frames including the core feature among frames before and after the representative frame as candidate frames; and
a frame matching step of generating, by a frame matching module, a panoramic representative frame by stitching the representative frame and the candidate frame based on the core feature;
including,
Characterized in that the panoramic representative frame is input as input data to the representative frame classification module and configured to output second zone classification information as output data.
A travel image classification method using panoramic image conversion of images.
a memory module containing travel image classification program code; and
a processing module that executes the travel image classification program code;
including,
The travel image classification program code,
A travel image reception step of receiving a travel image generated by a generator client and composed of a plurality of travel image frames in time series and image information including location information of the travel image;
a representative frame selection step of selecting a specific frame among the frames of the travel video as a representative frame;
A representative frame classification artificial neural network module having the representative frame as input data and first zone classification information, which is classification information on the theme of the zone including the location information, as output data Representative frame classification step of outputting zone 1 classification information;
A CNN-based feature extraction artificial neural network that uses input data as a representative frame and includes feature class information (class probability) and feature coordinate information (coordinate data) of one or more object features extracted from the representative frame as output data, the feature class a feature extraction step of outputting information and the feature coordinate information;
It is connected to a specific convolution layer of the representative frame classification artificial neural network module, receives class activation information including an activation map for each class, and uses the class activation information and the feature coordinate information to a core feature selection step of generating feature importance information of a class and selecting a class having the largest generated feature importance information as a core feature;
a candidate frame selection step of selecting a plurality of frames including the core feature among frames before and after the representative frame as candidate frames; and
a frame matching step of generating a panorama representative frame by stitching the representative frame and the candidate frame based on the core feature;
Characterized in that it is configured to perform steps including on a computer,
Characterized in that the panoramic representative frame is input as input data to the representative frame classification module and configured to output second zone classification information as output data.
An apparatus for classifying travel images using panoramic image conversion of images.
a travel image reception step in which a travel image reception module receives a travel image generated by a creator client and composed of a plurality of travel image frames in time series and image information including location information of the travel image;
a representative frame selection step in which a representative frame selection module selects a specific frame among the frames of the travel video as a representative frame;
A representative frame classification module including a representative frame classification artificial neural network module that takes the representative frame as input data and first zone classification information, which is classification information on the theme of the zone including the location information, as output data, and a representative frame classification step of outputting the first zone classification information for the image information.
Characteristics comprising a CNN-based feature extraction artificial neural network including feature class information (class probability) and feature coordinate information (coordinate data) of one or more object features extracted from the representative frame as input data and output data as output data a feature extraction step of outputting, by an extraction module, the feature class information and the feature coordinate information;
A core feature selection module is connected to a specific convolution layer of the representative frame classification module to receive class activation information including an activation map for each class, and the class activation information and the feature coordinate information a core feature selection step of generating feature importance information of each class by using and selecting a class having the largest generated feature importance information as a core feature;
a candidate frame selection step of selecting, by a candidate frame selection module, a plurality of frames including the core feature among frames before and after the representative frame as candidate frames; and
a frame matching step of generating, by a frame matching module, a panoramic representative frame by stitching the representative frame and the candidate frame based on the core feature;
Characterized in that it is configured to perform steps including on a computer,
Characterized in that the panoramic representative frame is input as input data to the representative frame classification module and configured to output second zone classification information as output data.
A program stored in a recording medium configured to perform a travel image classification method using a panoramic image conversion of an image on a computer.

Images