KR20210060252A - Method and system for searching missing person - Google Patents

Abstract

According to various embodiments, disclosed are a method and a system for searching a missing person. The method for searching a missing person performed by a server comprises the steps of: receiving missing person information including the color of the top and bottom of a missing person; receiving an input image; searching for an object matching the color of the top and bottom in the input image using a previously trained learning model; and outputting a portion including the searched object from the input image.

Description

{Method and system for searching missing person}

The present disclosure relates to a method and system for searching for a missing person, and more particularly, to a system, a method, and a computer program for quickly searching for a missing person based on the clothes of the missing person using a drone.

The disappearance of the elderly or infants with dementia is continuing. Currently, various methods are being used to quickly find the missing person. For example, it is using technology to match the faces of missing persons in the images of millions of CCTV installed across the country. However, when searching for a large amount of data as a whole, it takes too long, and when the image quality is not good, the effect is remarkably reduced.

The problem to be solved by the present invention is to overcome the fact that it is impossible to quickly search by matching the face of the missing person in a fixed CCTV image. It is to provide a system, method, and computer program for searching for missing persons.

As a technical means for achieving the above-described technical problems, a method for searching a missing person performed by a server according to an aspect of the present invention includes the steps of receiving information on the missing person including the colors of the top and bottom of the missing person, and receiving an input image. And searching for an object matching the colors of the top and bottom in the input image using a pre-trained learning model, and outputting a portion including the searched object in the input image.

According to an example, the learning model may be trained based on images of a person labeled with a color of a top and a bottom worn.

According to another example, the learning model may include a first learning model for classifying a person from an image, and a second learning model for classifying colors of a top and a bottom worn by the person from the image of the person.

According to another example, the input image may be received from an unmanned aerial vehicle. The processor of the unmanned aerial vehicle may be configured to generate the input image based on a color image captured by a general camera and a thermal image captured by a thermal imaging camera.

According to another example, the processor of the unmanned aerial vehicle may be configured to generate the input image by masking a region of the color image below a preset reference temperature of the thermal image.

According to another example, the object may be searched by inputting an unmasked color portion of the color image into the learning model.

According to another example, the missing person information may be received from a terminal, and a portion including the searched object may be transmitted to the terminal.

According to another aspect of the present invention, a computer program stored in a medium is provided in order to execute the above-described missing person search method using a server.

A missing person search system according to another aspect of the present invention includes a server and an unmanned aerial vehicle that communicate with each other. The server receives missing person information including the color of the top and bottom of the missing person, receives an input image from the unmanned aerial vehicle, and uses a pre-trained learning model to locate an object matching the color of the top and bottom in the input image. And outputs a portion including the searched object in the input image.

According to an example, the learning model may be trained based on images of a person labeled with a color of a top and a bottom worn.

According to another example, the learning model may include a first learning model for classifying a person from an image, and a second learning model for classifying colors of a top and a bottom worn by the person from the image of the person.

According to another example, the UAV includes a general camera that generates a color image, a thermal image camera that generates a thermal image, and a processor configured to generate the input image based on the color image and the thermal image. can do.

According to another example, the general camera and the thermal imaging camera may be configured to capture the same area.

According to another example, the processor may be configured to generate the input image by masking an area of the color image below a preset reference temperature of the thermal image.

According to another example, the object may be searched by inputting an unmasked color portion of the color image into the learning model.

According to another example, the unmanned aerial vehicle may further include a location measurement module that generates location information from a satellite signal. The processor may transmit the location information together with the input image to the server.

According to another example, the missing person search system may further include at least one terminal that transmits the missing person information to the server and receives a portion including the searched object.

According to the present invention, not only can the missing person be searched more quickly and quickly by using a drone, but also a high amount of computation is not required because the missing person is searched based on the color of the missing person's clothing rather than the missing person's face. Since it is possible to extract only the region corresponding to the body temperature of the human body from the color image using the thermal imaging camera as well as the camera, the amount of computation can be further reduced. Therefore, according to the present invention, the search time of the missing person can be drastically reduced.

1 shows a configuration of a missing person search system according to an embodiment.
2 shows a partial internal configuration of the unmanned aerial vehicle according to an embodiment.
3 is a diagram illustrating an internal configuration of a server and a terminal according to an embodiment.
4 is a diagram illustrating an internal configuration of a processor of an unmanned aerial vehicle according to an embodiment.
5 is a diagram illustrating an internal configuration of a processor of a server according to an embodiment.
6 is a diagram for explaining a method of searching for a missing person in a missing person search system including an unmanned aerial vehicle, a server, and a terminal according to an embodiment.

Hereinafter, various embodiments will be described in detail with reference to the accompanying drawings so that those of ordinary skill in the art may easily implement the present disclosure. However, the technical idea of the present disclosure is not limited to the embodiments described in the present specification because it may be modified and implemented in various forms. In describing the embodiments disclosed in the present specification, when it is determined that a detailed description of a related known technology may obscure the gist of the technical idea of the present disclosure, a detailed description of the known technology will be omitted. The same or similar components are given the same reference numerals, and redundant descriptions thereof will be omitted.

In the present specification, when an element is described as being "connected" with another element, this includes not only the case of being "directly connected" but also the case of being "indirectly connected" with another element interposed therebetween. When a certain element "includes" another element, it means that another element may be included in addition to the other element, not excluded, unless specifically stated to the contrary.

Some embodiments may be described with functional block configurations and various processing steps. Some or all of these functional blocks may be implemented with various numbers of hardware and/or software components that perform a specific function. For example, the functional blocks of the present disclosure may be implemented by one or more microprocessors, or may be implemented by circuit configurations for a predetermined function. Functional blocks of the present disclosure may be implemented in various programming or scripting languages. The functional blocks of the present disclosure may be implemented as an algorithm executed on one or more processors. Functions performed by a function block of the present disclosure may be performed by a plurality of function blocks, or functions performed by a plurality of function blocks in the present disclosure may be performed by a single function block. In addition, the present disclosure may employ conventional techniques for electronic environment setting, signal processing, and/or data processing.

1 shows a configuration of a missing person search system according to an embodiment.

Referring to FIG. 1, the missing person search system 1000 includes an unmanned aerial vehicle 100, a server 200, a terminal 300, and a network 10 connecting them.

The unmanned aerial vehicle 100 is an unmanned aerial vehicle referred to as a so-called drone, and includes an airplane or helicopter-shaped unmanned aerial vehicle capable of flying and manipulating by radio wave guidance without a pilot on board, and an air vehicle controller that wirelessly controls the unmanned aerial vehicle. The unmanned aerial vehicle 100 may perform various functions such as taking high-altitude images or pictures, delivering goods, collecting weather information, spraying pesticides, and so on.

Communicating with the unmanned aerial vehicle 100 includes communicating with an unmanned aerial vehicle or an aircraft controller, and communicating with an unmanned aerial vehicle includes communicating through an unmanned aerial vehicle controller. For example, when the unmanned aerial vehicle 100 transmits the video to the network 10, the unmanned aerial vehicle may directly connect to the network 10 and transmit the video, or the unmanned aerial vehicle can transmit the video to the network 10 through the vehicle controller. Can also be transmitted.

The server 200 manages missing person information and performs a function of searching for a missing person from an image. The server 200 communicates with the unmanned aerial vehicle 100 and/or the terminal 300 through the network 10, and provides commands, codes, files, contents, services, etc. through a single computer device or network 10. It may be implemented with a plurality of computer devices that are connected to each other.

The server 200 receives the missing person information from the terminal 300, receives the input image from the unmanned aerial vehicle 100, searches for an object matching the missing person information from the input image using a learning model, and the object retrieved from the input image. An object image, which is a part including is, can be output.

The terminal 300 is a user terminal that performs a computing function, and may communicate with the server 200 by accessing the network 10 through wired communication and/or wireless communication. The terminal 300 may include, for example, a personal computer, a notebook computer, a smart phone, a tablet PC, and the like. The terminal 300 may transmit missing person information to the server 200 and receive an object image from the server 200. The terminal 300 may be a personal computer device used by a family member or friend of the missing person. The terminal 300 may be a computer device of a police station, a fire station, or an organization that receives a report of a missing person.

The network 10 may connect the unmanned aerial vehicle 100, the server 200, and the terminal 300 to each other so that communication is possible. For example, the network 10 may provide an access path through which the terminal 300 can access the server 200 to transmit missing person information and receive an image of an object corresponding to the missing person information. The network 10 may provide an access path through which the drone 100 transmits an input image to the server 200 and receives a command from the server 200.

The network 10 may include a wired network and/or a wireless network. The unmanned aerial vehicle 100 and the server 200 may communicate through a wireless network 10. For example, the network 10 may include various networks such as a local area network (LAN), a metropolitan area network (MAN), and a wide area network (WAN). The network 10 may include the World Wide Web (WWW). However, the network 10 according to the present embodiment is not limited to the networks listed above, and includes at least one of a known mobile communication network, a known wireless data network, a known telephone network, and a known wired/wireless television network. can do. Network 10 may include any one or more of network topologies including bus networks, star networks, ring networks, mesh networks, star-bus networks, tree or hierarchical networks, and the like.

2 shows a partial internal configuration of the unmanned aerial vehicle according to an embodiment.

Referring to FIG. 2, the drone 100 may include a processor 110, a memory 120, a communication module 130, a first camera 150, and a second camera 160. The unmanned aerial vehicle 100 may further include a position measurement module 170. The processor 110, the memory 120, the communication module 130, the first camera 150, the second camera 160, and the position measurement module 170 can exchange data with each other through the bus 101. have. Although not shown in FIG. 2, the unmanned aerial vehicle 100 may further include a sensor unit such as a flying body, a thruster including a motor and a propeller, an inertial sensor, an infrared sensor, and the like.

The processor 110 performs basic arithmetic, logic, and input/output operations, and may execute, for example, program code stored in the memory 120. The processor 110 may basically control the thruster for the flight of the unmanned aerial vehicle 100. The processor 110 may execute a program code for autonomous flight and/or a program code for flight according to a control command received from the vehicle controller.

The processor 110 may include a first processor mounted on an unmanned aerial vehicle of the unmanned aerial vehicle 100 and a second processor mounted on an air vehicle controller of the unmanned aerial vehicle 100.

According to an embodiment, the processor 110 may generate an input image by processing a color image from the first camera 150 and a thermal image from the second camera 150. The input image is an image to be input to the server 200 and refers to an image generated by the processor 110 based on a color image and a thermal image.

The memory 120 is a recording medium that can be read by the processor 110 and may include random access memory (RAM) and/or read only memory (ROM). The memory 120 may store an operating system and at least one program code. The memory 120 may store a program code for image processing a color image and a thermal image according to various embodiments.

The communication module 130 may exchange data with the server 200 through the wireless network 10. For example, the communication module 130 may transmit an input image generated by the processor 110 to the server 200 and receive a control command from the server 200. The communication module 130 may include a first communication module mounted on an unmanned aerial vehicle of the unmanned aerial vehicle 100 and a second communication module mounted on an aircraft controller of the unmanned aerial vehicle 100. The unmanned aerial vehicle and the vehicle controller may communicate with each other through a first communication module and a second communication module.

The first camera 150 is a general visible light camera and may generate a color image. The color image may be provided to the processor 110. The first camera 150 may have a zoom function, and may generate an enlarged image according to the control of the processor 110 or the vehicle controller.

The second camera 160 is a thermal imaging camera and may generate a thermal image. The thermal image is an image in which the temperature distribution of a surface obtained by measuring radiated infrared rays is displayed in black and white shades or colors. Using thermal imaging, it is easy to detect a forest fire in the mountains or a calm animal such as a person who can maintain a constant body temperature.

The second camera 160 may be referred to as a solar camera, and the first camera 150 may be referred to as a general camera.

The first camera 150 and the second camera 160 may be configured to photograph the same area. An image captured by the first camera 150 at a basic magnification and an image captured by the second camera 160 at a basic magnification may be substantially the same object or area. Accordingly, a process of matching the color image generated by the first camera 150 and the thermal image captured by the second camera 160 may be omitted.

According to an example, the first camera 150 and the second camera 160 may be disposed adjacent to each other so as to have a photographing direction parallel to each other. According to another example, the first camera 150 and the second camera 160 may be mounted on a driving unit controlled by the processor 110. The driving unit may change the photographing direction by rotating the first camera 150 and the second camera 160. The first camera 150 and the second camera 160 are mounted on one driving unit and may rotate identically to each other.

The processor 110 receives a color image captured by the first camera 150, receives a thermal image captured by the second camera 160, and generates an input image based on the color image and the thermal image. I can.

According to an example, the processor 110 may be configured to generate an input image by masking a region below a preset reference temperature of a thermal image in a color image. For example, the processor 110 may generate a filter image by dividing a black area and a white area based on a preset reference temperature in the thermal image. The black region may be a region in which the temperature corresponding to the pixel is equal to or lower than the reference temperature, and the white region may be a region in which the temperature corresponding to the pixel is higher than the reference temperature.

The reference temperature may be set in advance based on the ambient temperature. For example, when the air temperature is about 20 degrees, the reference temperature may be set to, for example, 22 degrees. Due to the human body temperature, a region corresponding to a person in the thermal image may be included in the white region.

The processor 110 may extract only a part of the color image by combining the filter image and the color image. In the filter image, a part of the color image corresponding to the black region may be masked as a black region, and only the rest of the color image corresponding to the white region may remain in the filter image. The processor 110 may perform an AND operation on the filter image and the color image.

The processor 110 may prevent a part corresponding to a person in the color image from being set too tight by expanding the white area of the filter image by a preset margin and then combining the color image with the color image.

The processor 110 may transmit only a part of the color image extracted through the above process to the server 200. Since the processor 110 extracts only an area in which a person is likely to exist from the color image and transmits it to the server 200, the range in which the server 200 searches for a missing person is reduced, so that a quick search is possible. In addition, since only an area in which a person is likely to exist is extracted and transmitted to the server 200, the amount of data to be transmitted through the wireless network 10 may also be reduced. Accordingly, the unmanned aerial vehicle 100 may further increase the resolution of an image captured by the first camera 150. If the resolution of the first camera 150 is too high, the data size of the generated color image increases, so it may be difficult to transmit in real time through the wireless network 10. However, according to the present invention, since only a part of data is extracted and transmitted, even if the resolution of a color image is high, it can be transmitted to the server 200 in real time within the bandwidth of the wireless network 10.

The location measurement module 170 is a module that measures a location on the earth using radio waves transmitted from a Global Navigation Satellite System (GNSS) satellite. The location measurement module 170 may measure the location of the unmanned aerial vehicle 100 to generate location information and provide location information to the processor 110. The processor 110 may be utilized for autonomous flight or the like based on location information. The processor 110 may transmit location information to the server 200 together with an input image generated based on a color image and a thermal image.

3 is a diagram illustrating an internal configuration of a server and a terminal according to an embodiment.

Referring to FIG. 3, the missing person search system may include, for example, a server 200 and a terminal 300 connected to the network 10.

The server 200 includes a processor 210, a memory 220, a communication module 230, and an input/output interface 240. The processor 210, the memory 220, the communication module 230, and the input/output interface 240 may exchange data with each other through the bus 201.

The processor 210 may perform basic arithmetic, logic, and input/output operations, and may execute, for example, program code stored in the memory 220, for example, a learning model.

The memory 220 is a recording medium that can be read by the processor 210 and may include a permanent mass storage device such as RAM, ROM, and a disk drive. The memory 220 may store an operating system and at least one program or application code. A program code for performing a method of detecting an object matching the missing person information from an input image using a learning model may be stored in the memory 220. In addition, the memory 220 may store images of a person labeled with colors of a top and a bottom worn, and a program code for training a learning model based on the labeled images of the people may be stored.

The communication module 230 may wirelessly access the network 10 to receive data from the terminal 300 and transmit data to the terminal 300. The input/output interface 540 may provide an interface means with an input/output device.

The terminal 300 includes a processor 310, a memory 320, a communication module 330, and an input/output device 340. The processor 310, the memory 320, the communication module 330, and the input/output device 340 may exchange data with each other through the bus 301.

The processor 310 may perform basic arithmetic, logic, and input/output operations, and may execute, for example, program code stored in the memory 320.

The memory 320 is a recording medium that can be read by the processor 310 and may include a non-destructive large-capacity recording device such as a RAM, a ROM, and a disk drive. A program or application code such as an operating system and an Internet browser may be stored in the memory 120.

The communication module 330 may connect to the network 10 to exchange data with the server 200. The input/output device 340 may receive an input from a user, transmit it to the processor 310, and output information received from the processor 310 to the user. The input/output device 340 is an input device and may include a keyboard, a mouse, a touch screen, and a microphone. The input/output device 340 is an output device and may include an image display device such as a display.

The user may input missing person information through the input device 340. The missing person information may include a color of a top and a bottom worn by the missing person. The missing person information may further include a shoe color. In addition, the missing person information may include the name of the missing person, the location of the missing person, the time of the disappearance, and a photo of the missing person.

The missing person information input by the user may be transmitted to the server 200 through, for example, an Internet browser. The server 200 may store missing person information in the memory 220. The missing person information may be transmitted to the server 200 through a dedicated program.

4 is a diagram illustrating an internal configuration of a processor of an unmanned aerial vehicle according to an embodiment.

Referring to FIG. 4, the processor 110 of the drone 100 includes a color image receiver 111, a thermal image receiver 112, and an input image generator 113.

The color image receiving unit 111 receives a color image generated by photographing a photographing area by the first camera 150. The thermal image receiving unit 112 receives a thermal image generated by photographing a photographing area by the second camera 160. The first camera 150 and the second camera 160 simultaneously photograph the same area, and the color image and the thermal image may substantially match each other.

The input image generator 113 generates an input image based on a color image and a thermal image area. According to an example, the input image generator 113 may generate an input image by masking a region of a color image below a preset reference temperature of the thermal image.

According to an example, the input image generator 113 includes a black region in which a temperature corresponding to a corresponding pixel is less than a reference temperature and a region in which a temperature corresponding to the pixel is higher than a reference temperature based on a preset reference temperature in the thermal image A filter image in which the white area is divided can be generated. The input image generator 113 may generate an input image by masking a part of the color image corresponding to the black region in the filter image as a black region, and leaving only the rest of the color image corresponding to the white region in the filter image. .

According to an example, the input image generator 113 may generate an input image by performing an AND operation on a filter image and a color image. When the AND operation is performed on the filter image and the color image, a pixel value of a region corresponding to the black region of the filter image in the color image may be changed to a value corresponding to black.

The input image generator 113 may reduce the size of the input image by masking an area in which the possibility of the presence of a person is low or not in the color image as a black area.

According to another example, the input image generator 113 may generate a modified filter image by expanding a white area of the filter image by a preset margin, and then combine the modified filter image with a color image to generate an input image.

The processor 110 may further include a location information generator 114. The location information generator 114 may generate location information indicating the current location of the unmanned aerial vehicle 100 using the location measurement module 170. The processor 110 may transmit to the server 200 in real time location information indicating the location of the unmanned aerial vehicle 100 that has captured the input image together with the input image.

5 is a diagram illustrating an internal configuration of a processor of a server according to an embodiment.

Referring to FIG. 5, the processor 210 of the server 200 includes a missing person information receiving unit 211, an input image receiving unit 212, an object searching unit 213, and an object image outputting unit 214.

The missing person information receiving unit 211 receives missing person information from the terminal 300. The missing person information may include color information of the clothes worn by the missing person. For example, the missing person information may include a color of a top worn by the missing person and a color of a bottom. In addition, the missing person information may include information such as the name, age, height, sex, photo, and location of the missing person and the date and time of the disappearance.

The input image receiving unit 212 receives an input image from the drone 100. The input image may be a color image in which a portion where no humans cannot exist is masked with black using a thermal image.

The object search unit 213 may search for an object that matches the color of the upper and lower colors in the input image. The object search unit 213 may search for an object using the learning model 213m trained in advance.

The learning model 213m may be designed to simulate a human brain structure on a computer. For example, the learning model 213m may include network nodes having weights that simulate neurons of a human neural network. Network nodes can each form a connection relationship to simulate synaptic activity in which neurons send and receive signals through synapses.

The learning model 213m may include, for example, an artificial intelligence neural network model or a deep learning network model developed from a neural network model.

The learning model 213m is a model that is trained to recognize an object matching the color of the top and bottom of the missing person information by supervised learning. Supervised learning aims to find a fixed answer through an algorithm. Accordingly, the learning model 213m may be a model in which a function is inferred from training data. In supervised learning, a labeled sample (data with a target output value) is used for training.

The supervised learning algorithm receives a series of training data and the corresponding target output value, finds errors through learning that compares the actual output value of the input data with the target output value, and corrects the model based on the result. It is done. Supervised learning is divided into regression and classification according to the type of outcome. The function derived through the supervised learning algorithm is again used to predict the new result value. As such, the learning model 213m optimizes the parameters of the learning model 213m through learning a number of training data.

The learning model 213m according to the present invention may have a multilayer perceptrons structure including a plurality of hidden layers. Perceptron is a term that refers to the mathematical model of each neuron (y=Wx+b), and multilayer perceptron is the accuracy of prediction through learning through error-backpropagation algorithm and gradient decent technique. Can increase. The way the learning model (213m) learns through the error-backpropagation algorithm is to start from the input layer and compare it with the reference label value when the y value is obtained through the output layer. This is a method of delivering values and updating each W and b values according to the calculated cost.

The training data set provided to the learning model 213m may be an image of a person wearing a top and bottom. The image of a person may include labels for the color of the top and bottom. Here, the label may indicate the class of the object. A class according to an embodiment may represent colors of tops and bottoms such as red, blue, yellow, white, and black.

On the other hand, after the learning model (213m) performs a learning process using training data, a model with optimized parameters is created, and when unlabeled data is input to the generated model, a result value corresponding to the input data (label ) Can be predicted.

The learning model 213m may include a feature extraction network and a multi-task network that performs a plurality of tasks.

When data labeled (y) for each input (x) is provided to the learning model (213m), the feature extraction network extracts the features of the input data, and the multi-task network is an object matching the color of the top and bottom. Is learned to recognize.

The feature extraction network may be a convolutional neural network ("CNN").

The feature extraction network can be used to extract "features" such as borders, line colors, and the like from complex input data. The feature extraction network may include a plurality of layers. Each layer may receive data, and may generate data output from a corresponding layer by processing data input to the corresponding layer. The data output from the layer may be a feature map generated by convolving an image into which a feature extraction network is input or an input feature map with one or more filters or one or more kernels. The initial layers of the feature extraction network can be operated to extract low level features such as edges or gradients from the input. The next layers of the feature extraction network can gradually extract more complex features.

One or more or all layers for receiving and outputting a feature map in the feature extraction network may be hidden layers (eg, hidden convolutional layers). Meanwhile, the feature extraction network may perform other processing operations in addition to the operation of applying the convolution kernel to the feature map. Examples of such other processing operations may include, but are not limited to, operations such as activation function, pooling, and resampling.

The input of the feature extraction network may be a pre-processed image obtained by pre-processing an input image received from the unmanned aerial vehicle 100. The preprocessed image may separate color regions not masked with black from the input image, and may be reduced or enlarged to a preset resolution. The preprocessed image may be a color image having preset lengths in the horizontal direction and the vertical direction.

The feature extraction network can extract features of the preprocessed image. The feature extraction network may include a plurality of convolutional layers, and each convolutional layer may include a Rectified Linear Unit ("ReLU") layer and a maximum pooling layer. . A drop out layer may be inserted in the feature extraction network.

The feature extraction network may receive a preprocessed image as an input and output output data. In this case, the output data output from the feature extraction network may be provided as an input of the multi-task network.

The multi-task network includes a plurality of task modules, and each module performs complementary tasks. The multi-task network detects/classifies objects matching the colors of the top and bottom on a single forward pass, and at the same time performs a task of determining whether the corresponding object is a person.

The model trained by the multi-task network can improve recognition accuracy by recognizing a plurality of elements (eg, whether or not a person is a person, a color of a top, a color of a bottom, etc.) through a plurality of tasks.

Each task of the multi-task network may share mid-level attributes output by the feature extraction network in the front end. Since the middle-level properties are robust to changes in weather or illumination, the multi-task network can improve recognition accuracy by jointly learning the middle-level properties.

The multi-task network consists of a multi-label classification module, an object mask module, and a grid regression module according to the type of task each network performs. Can be.

The grid regression module performs the grid detection task. Regression is an analysis technique that measures the fit after obtaining a model between two variables for continuous variables. It analyzes the model and given data. The grid detection task may be to detect a location of a detection target as a combination of grids.

The object mask module performs a binary classification task on objects. The object mask module classifies a given input value into two categories (for example, '0' or '1') according to a certain criterion. For example, the object mask module may perform a classification task such as whether the detected object is a person.

The multi-label classification module performs a class classification task. The multi-label classification module can classify the inputs into multiple categories. The multi-label classification module may perform a classification task of classifying an image color of the detected object. Also, the multi-label classification module may perform a classification task of classifying a color under the detected object.

Each of the multi-label classification module, the object mask module, and the grid regression module may include convolutional layers, fully connected layers, and a loss layer.

The convolutional layers may be filtering elements that filter input data. The convolutional layers may be composed of a convolutional filtering layer, a pooling layer, or a combination thereof.

The fully connected layers include a plurality of layers, and each layer may be composed of a plurality of nodes. In addition, dropout, which is a model regularization algorithm, may be applied to the fully connected layers. Dropout is an algorithm in which a predetermined percentage of nodes (eg, 50% of nodes) do not participate in learning randomly in the current learning epoch.

The lossy layer can predict a plurality of elements (eg, people, top/bottom color, etc.) from the outputs of the fully connected layers, and calculate the losses by comparing the predicted elements with the actual elements. Losses can be back propagated to fully connected layers and convolutional layers through a back propagation technique. Based on the back-propagated losses, the connection weights in the convolutional layers and the fully connected layers may be updated. On the other hand, the method of calculating the loss is not limited to a specific method. For example, Hinge Loss, Square Loss, Softmax Loss, Cross-entropy Loss, Absolute Loss, Insensitive Loss ), etc. can be used depending on the purpose.

According to another example, the learning model 213m is a first learning model 213m1 for classifying a person from an image, and a second learning model for classifying the color of a top worn by a corresponding person and the color of a bottom in a person's image. It may include a model (213m2). The object search unit 213 first determines whether a person is included in the input image received from the unmanned aerial vehicle 100 using the first learning model 213m1, and if it is determined that the image includes a person, the second learning model ( 213m2) can be used to determine the color of the top and bottom that the person is wearing.

The object image output unit 214 may output a portion including the object, that is, an object image, when the object search unit 213 searches for an object matching the colors of the top and bottom in the input image. The object image may be transmitted to the terminal 300.

6 is a diagram for explaining a method of searching for a missing person in a missing person search system including the unmanned aerial vehicle 100, the server 200, and the terminal 300 according to an exemplary embodiment.

Referring to FIG. 6, the terminal 300 may transmit missing person information to the server 200 (S10). Thereafter, the unmanned aerial vehicle 100 is started to search for the missing person, and generates an input image using the color image captured by the first camera 150 and the thermal image captured by the second camera 160, It can be transmitted to (200) (S20). The server 200 may determine whether the missing person information, in particular, a human object matching the color information of the top and bottom, exists in the input image using the learning model 213m of the object search unit 213 (S30). If an object matching the upper and lower color information exists, the server 200 may generate an object image including the object and transmit it to the terminal 300 (S40).

In addition, if there is an object matching the color information of the upper and lower portions, the server 200 may transmit a corresponding position to the unmanned aerial vehicle 100 together with a request for an enlarged image (S50). The unmanned aerial vehicle 100 may generate an enlarged image by magnifying and photographing an image of a corresponding location using the zoom function of the first camera 150, and may transmit the enlarged image to the server 200 (S60 ). The server 200 may transmit the received enlarged image to the terminal 300.

The user of the terminal 300 can directly check whether the missing person is correct based on the detected image and/or the enlarged image received from the server 200.

According to the present invention, the input image generated by the unmanned aerial vehicle 100 is masked with an image in which there is no possibility of a person present, so that the size of the image data can be reduced. Assuming that the bandwidth of the network 10 is constant, the resolution of the image data may be higher.

Since the amount of data to be searched for detecting the missing person has decreased, the server 200 can quickly detect the missing person. In addition, the face of the missing person with a large amount of computation was previously matched, but in the present invention, considering that the clothes worn by the missing person do not change well, the calculation amount is less than before and the missing person is searched based on the colors of the top and bottom with high accuracy You can search for the missing person more quickly.

The various embodiments described above are exemplary, and are not to be performed independently as they are distinguished from each other. The embodiments described in the present specification may be implemented in combination with each other.

The various embodiments described above may be implemented in the form of a computer program that can be executed through various components on a computer, and such a computer program may be recorded in a computer-readable medium. In this case, the medium may be one that continuously stores a program executable by a computer, or temporarily stores a program for execution or download. In addition, the medium may be a variety of recording means or storage means in a form in which a single piece of hardware or several pieces of hardware are combined. The medium is not limited to a medium directly connected to a computer system, but may be distributed on a network. Examples of media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical recording media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, And there may be ones configured to store program instructions, including ROM, RAM, flash memory, and the like. In addition, examples of other media include an app store that distributes applications, a site that supplies or distributes various software, and a recording medium or a storage medium managed by a server.

In the present specification, a "unit", a "module", etc. may be a hardware component such as a processor or a circuit, and/or a software component executed by a hardware configuration such as a processor. For example, "unit", "module", etc. are components such as software components, object-oriented software components, class components and task components, processes, functions, properties, It can be implemented by procedures, subroutines, segments of program code, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays and variables.

The above description of the present invention is for illustrative purposes only, and those of ordinary skill in the art to which the present invention pertains will be able to understand that other specific forms can be easily modified without changing the technical spirit or essential features of the present invention. will be. Therefore, it should be understood that the embodiments described above are illustrative and non-limiting in all respects. For example, each component described as a single type may be implemented in a distributed manner, and similarly, components described as being distributed may also be implemented in a combined form.

The scope of the present invention is indicated by the claims to be described later rather than the detailed description, and all changes or modified forms derived from the meaning and scope of the claims and their equivalent concepts should be construed as being included in the scope of the present invention. do.

100: drone
110: processor
111: color image receiver
112: thermal image receiving unit
113: input image generator
114: location information generation unit
120: memory
130: communication module
150: first camera
160: second camera
170: position measurement module
200: server
210: processor
211: missing person information receiving unit
212: input image receiving unit
213: object search section
213m: learning model
214: object image output unit
220: memory
230: communication module
240: input/output interface
300: terminal
310: processor
320: memory
330: communication module
340: input/output device

Claims (17)

In the method performed by the server,
Receiving missing person information including colors of the missing person's top and bottom;
Receiving an input image;
Searching for an object matching the colors of the top and bottom in the input image using a pre-trained learning model; And
And outputting a portion including the searched object in the input image. The method of claim 1,
The learning model is a missing person search method, characterized in that the training model is trained based on images of a person whose color of a top and a bottom worn is labeled. The method of claim 1,
The learning model includes a first learning model for classifying a person from an image, and a second learning model for classifying colors of a top and a bottom worn by the person from the image of the person. The method of claim 1,
The input image is received from the unmanned aerial vehicle,
And the processor of the unmanned aerial vehicle is configured to generate the input image based on a color image captured by a general camera and a thermal image captured by a thermal imaging camera. The method of claim 4,
And the processor of the unmanned aerial vehicle is configured to generate the input image by masking an area of the color image below a predetermined reference temperature of the thermal image. The method of claim 7,
And the object is searched by inputting a color part that is not masked from the color image into the learning model. The method of claim 1,
The missing person search method, wherein the missing person information is received from a terminal, and a portion including the searched object is transmitted to the terminal. A computer program stored in a medium to execute the method of any one of claims 1 to 7 using a server. Including a server and a drone that communicate with each other,
The server receives missing person information including the color of the top and bottom of the missing person, receives an input image from the unmanned aerial vehicle, and uses a pre-trained learning model to locate an object matching the top and bottom colors in the input image And outputting a portion including the searched object in the input image. The method of claim 9,
The learning model is a missing person search system, characterized in that the training model is trained based on the images of the person whose top and bottom colors are labeled. The method of claim 9,
The learning model includes a first learning model for classifying a person from an image, and a second learning model for classifying colors of a top and a bottom worn by the person from the image of the person. The method of claim 9,
The unmanned aerial vehicle,
A general camera that generates color images;
A thermal imaging camera that generates a thermal image; And
And a processor configured to generate the input image based on the color image and the thermal image. The method of claim 12,
The general camera and the thermal imaging camera are configured to photograph the same area. The method of claim 12,
And the processor is configured to generate the input image by masking an area of the color image below a predetermined reference temperature of the thermal image. The method of claim 14,
The object is searched for by inputting a color part that is not masked in the color image into the learning model. The method of claim 9,
The unmanned aerial vehicle further includes a location measurement module for generating location information from a satellite signal,
And the processor transmits the location information together with the input image to the server. The method of claim 9,
The missing person search system further comprises at least one terminal for transmitting the missing person information to the server and receiving a portion including the searched object.

Images