WO2019212237A1 - Abnormal entity detection device and method - Google Patents

Abstract

An abnormal entity detection device, according to one embodiment of the present invention, comprises: a communication processor for communicating with a learning server; and a control unit for receiving a captured image and outputting an image in which, among entities in the captured image, a pixel region where movement is not detected is masked, wherein the communication processor transmits, to the learning server, the captured image and a bitmap of the entity density of the captured image.

Description

Anomaly Object Detection Apparatus and Method

Embodiments of the present invention relate to an apparatus and a method for detecting abnormal objects.

Livestock raised in groups within a small kennel are very vulnerable to the spread of communicable disease. For example, legal epidemics such as foot-and-mouth disease and bird flu are spread through the air, so once they occur, the social costs of protection and prevention of infection are very high, and the social anxiety about food is rapidly spreading. . If abnormal signs are found in the kennel, it is important to isolate diseased livestock as soon as possible to prevent the spread of disease.

Attempts have been made to detect abnormal signs in the kennel based on learning. To this end, the local machine collects the image data photographed in the kennel, and transmits the collected image data to the learning server. The learning server may learn the image data received from the local machines and extract a parameter to be applied to the anomaly detection algorithm.

In general, the local machine includes a camera, and all image data photographed by the camera is transmitted to the learning server. At this time, excessive communication traffic is generated when the local machine transmits data to the learning server, and the learning server may perform an excessive amount of operations to learn all of them.

The present invention has been made in an effort to provide an abnormal object detecting apparatus and method capable of detecting an object having a high probability of disease in image data photographed inside a kennel.

An apparatus for detecting an anomaly according to an embodiment of the present invention includes a communication processor communicating with a learning server; And a controller configured to receive a captured image and to output an image of masking a pixel region in which no motion is detected among the objects of the captured image, wherein the communication processor includes a bitmap of the captured image and the object density of the captured image. Send to the learning server.

The bitmap may include information in which an object density of the captured image is modified.

The communication processor communicates with an administrator terminal, and the modified information may be modified by the administrator terminal.

The apparatus for detecting an abnormal object according to an embodiment of the present invention transmits a controller for extracting abnormal object information from a captured image in which a plurality of objects are photographed together, and transmits the abnormal object information to a manager terminal. And a learning unit for receiving feedback information about the learning information, and a learning preprocessing unit generating learning data based on the feedback information. The learning data includes image data extracted based on the feedback information. The controller extracts the abnormal entity information by using update information of the learning data learned by the learning server.

The feedback information may inform that there is an error in the abnormal entity information.

The feedback information may include information about an error time zone and information about a spatial area of the abnormal entity information.

The image data based on the feedback information may be image data in the time domain and the spatial domain extracted according to the feedback information of the captured image.

The control unit may include a first feature extracting unit extracting a position distribution of the plurality of objects in the captured image, a second feature extracting unit extracting the movements of the plurality of objects in the captured image, and the first feature extracting unit. And an abnormal entity information generating unit that generates abnormal entity data based on the position distribution extracted by the extracted feature and the movement extracted by the second feature extraction unit.

The update information includes at least one of first update information applied to the first feature extraction unit, second update information applied to the second feature extraction unit, and third update information applied to the abnormal object information generating unit. can do.

An object detection method according to an embodiment of the present invention comprises the steps of receiving an n-th image; outputting a first density map of the object in the n-th image; outputting a motion map in an nth image; Outputting an n-th or more object density map by using the first density map, the motion map, and a previously stored n-th or more object density map; And masking an area where the abnormal object is located in the n-th image using the n-th or more object density map, wherein the n-th image and the second density map are transmitted to a learning server, and the second density map. Is information in which the density of the object in the n-th image is corrected.

The second density map may be modified by the administrator terminal.

Receiving update information obtained by re-learning using the n-th image and the second density map from the learning server, and using the update information, an abnormal object density map of a predetermined image after the n-th image; The method may further include outputting.

The modified information includes at least one of error time space information and space area information of the n-th or more object density map, and the second density map uses at least one of the time area information and the space area information. Can be extracted.

The outputting of the abnormal object density map for the predetermined image may include extracting position distributions of the plurality of objects in the predetermined image by using the predetermined image and first update information received from the learning server. Extracting the movements of the plurality of objects in the predetermined image using the image and the second update information received from the learning server, and the position distribution, the movement, and the third update information received from the learning server. The method may include generating an abnormal object density map of the predetermined image using the image.

An anomaly object detection system according to an embodiment of the present invention includes an anomaly object detection device communicating with a learning server and the learning server, wherein the anomaly object detection device comprises: a communication processor communicating with the learning server; And a controller configured to receive a captured image and to output an image of masking a pixel region in which no motion is detected among the objects of the captured image, wherein the communication processor includes a bitmap of the captured image and the object density of the captured image. Send to the learning server.

The apparatus and method for detecting an abnormal object according to an exemplary embodiment of the present invention may detect an abnormal object having a high probability of disease on image data photographed inside a kennel.

In addition, according to one embodiment of the present invention, since the amount of data transmitted to the learning server can be reduced, it is possible to prevent the occurrence of excessive communication traffic, and to reduce the amount of computation in the learning server.

1 is a block diagram of an anomaly detection system according to an embodiment of the present invention.

2 is a block diagram of a learning system for detecting abnormal objects according to an embodiment of the present invention.

3 is a conceptual diagram of an apparatus for detecting an anomaly according to an embodiment of the present invention.

4 is a block diagram of an anomaly detection apparatus according to an embodiment of the present invention.

5 is a flowchart of a method for detecting an abnormal object of the apparatus for detecting an abnormal object according to an embodiment of the present invention.

6 is a view for explaining the principle of the object density prediction network.

7 is a diagram illustrating an example of displaying abnormal entity information on a manager terminal.

8 is a diagram for explaining an example in which abnormal entity information is displayed for each block on a manager terminal.

9 is a block diagram of a controller included in an anomaly detecting apparatus according to an embodiment of the present invention.

10 is a view for explaining an abnormal object detection algorithm of the abnormal object detection apparatus according to the embodiment of the present invention.

11 is a view for explaining a method for detecting an abnormal object by the apparatus for detecting an abnormal object according to an embodiment of the present invention by using the result of re-learning by the learning server.

12 is a result of testing the performance of the abnormal object detection apparatus according to an embodiment of the present invention.

As the invention allows for various changes and numerous embodiments, particular embodiments will be illustrated and described in the drawings. However, this is not intended to limit the present invention to specific embodiments, it should be understood to include all modifications, equivalents, and substitutes included in the spirit and scope of the present invention.

Terms including ordinal numbers, such as second and first, may be used to describe various components, but the components are not limited by the terms. The terms are used only for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, the second component may be referred to as the first component, and similarly, the first component may also be referred to as the second component. The term and / or includes a combination of a plurality of related items or any item of a plurality of related items.

When a component is referred to as being "connected" or "connected" to another component, it may be directly connected to or connected to that other component, but it may be understood that other components may be present in between. Should be. On the other hand, when a component is said to be "directly connected" or "directly connected" to another component, it should be understood that there is no other component in between.

The terminology used herein is for the purpose of describing particular example embodiments only and is not intended to be limiting of the present invention. Singular expressions include plural expressions unless the context clearly indicates otherwise. In this application, the terms "comprise" or "have" are intended to indicate that there is a feature, number, step, operation, component, part, or combination thereof described in the specification, and one or more other features. It is to be understood that the present invention does not exclude the possibility of the presence or the addition of numbers, steps, operations, components, components, or a combination thereof.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art. Terms such as those defined in the commonly used dictionaries should be construed as having meanings consistent with the meanings in the context of the related art and shall not be construed in ideal or excessively formal meanings unless expressly defined in this application. Do not.

DETAILED DESCRIPTION Hereinafter, exemplary embodiments will be described in detail with reference to the accompanying drawings, and the same or corresponding components will be given the same reference numerals regardless of the reference numerals, and redundant description thereof will be omitted.

1 is a block diagram of an abnormal object detection system according to an embodiment of the present invention, FIG. 2 is a block diagram of a learning system for detecting an abnormal object according to an embodiment of the present invention, and FIG. 3 is according to an embodiment of the present invention. It is a conceptual diagram of the abnormal object detection apparatus.

1 to 3, an abnormal object detection system 1000 according to an exemplary embodiment of the present invention includes an abnormal object detecting apparatus 100, an administrator terminal 200, an air conditioning apparatus 300, and a learning server 400. do.

The learning system 2000 for detecting an abnormal object according to an exemplary embodiment of the present invention includes a plurality of abnormal object detecting devices 100 and a learning server 400. Here, the plurality of abnormal object detecting apparatuses 100 may be a plurality of abnormal object detecting apparatuses installed in one breeding ground, or may be a plurality of abnormal object detecting apparatuses installed in the plurality of breeding grounds.

The abnormality object detecting apparatus 100 may detect an environment in the kennel 10 and transmit it to at least one of the manager terminal 200 and the air conditioning apparatus 300. Here, the kennel 10 means a livestock breeding barn. The livestock may be not only poultry such as chickens and ducks, but also various kinds of animals bred in groups such as cattle and pigs.

The abnormal object detection apparatus 100 extracts abnormal object information in the kennel 10. Here, the abnormal entity information may include at least one of information on the presence or absence of the abnormal entity, information on a spatial area in which the abnormal entity exists, and information on a time domain in which the abnormal entity exists. In this case, the abnormal subject may refer to an individual who is not in a normal state due to a disease, pregnancy, or the like.

To this end, the abnormal object detecting apparatus 100 may include a photographing unit, and may acquire image data of an image in which a plurality of objects are photographed together using the photographing unit. Here, the photographing unit may be mixed with the camera. The abnormal object detecting apparatus 100 may extract an abnormal object information on the image data by driving a pre-stored algorithm. Here, the prestored algorithm may include a trained model. The learned model may be a computer readable program and may be stored in a recording medium or a storage device executable by the computer. A processor in a computer may read a program stored in a recording medium or a storage device, execute a program, that is, a trained model, calculate input information, and output a calculation result. In this case, the input information may be image data, and the operation result may be abnormal object information.

As illustrated in FIG. 3A, the abnormality object detecting apparatus 100 may be arranged for each of the kennel 10. The abnormal object detecting apparatus 100 may include a plurality of photographing units 111, and the plurality of photographing units 111 may be disposed at various places in the kennel 10. For example, the plurality of photographing units 111 may be disposed at the upper and side portions of the kennel 10. The abnormal object detection apparatus 100 may extract the abnormal object information by collecting a plurality of image data acquired by the plurality of photographing units 111.

Alternatively, as illustrated in FIG. 3B, a plurality of abnormal object detection apparatuses 100 may be disposed in one kennel 10. The plurality of abnormal object detecting apparatuses 100 may be disposed at various places in the kennel 10, and each of the abnormal object detecting apparatuses 100 uses the individual image data obtained by each photographing unit 111. You can also extract information.

According to the exemplary embodiment of the present invention, the abnormality object detecting apparatus 100 may communicate with the manager terminal 200 and the air conditioning apparatus 300 by wire or wirelessly. Here, although the abnormality object detecting apparatus 100 is illustrated as communicating with the manager terminal 200 and the air conditioning apparatus 300, respectively, but is not limited thereto, and the abnormality object detecting apparatus 100 communicates with the manager terminal 200. In addition, the manager terminal 200 may communicate with the air conditioning apparatus 300.

The manager terminal 200 may be a personal computer (PC), a tablet PC, a mobile terminal, or the like, and may be mixed with a management server. When the abnormal object detecting apparatus 100 transmits at least one of the environment and the abnormal object information in the breeding ground 10 to the manager terminal 200, the manager may check the breeding ground 10 through a screen output to the manager terminal 200. At least one of environment information and abnormal object information within the server can be recognized. For example, when the abnormal object detecting apparatus 100 captures an abnormal situation in the kennel 10 and transmits the abnormal situation to the manager terminal 200, the manager is connected to the kennel 10 through a screen output to the manager terminal 200. It can be recognized that an abnormal situation has occurred in the first place and can respond to the initial situation. Here, the abnormal situation may be, for example, the generation of diseased livestock, pregnancy of the livestock, growth of the livestock, the humidity in the kennel 10, temperature, more than the concentration of a specific molecule and the like.

The air conditioning apparatus 300 is a device for controlling the temperature of the kennel 10. When the abnormality object detecting apparatus 100 captures a temperature abnormality in the kennel 10 and transmits it to the manager terminal 200, the manager generates a temperature abnormality in the kennel 10 through a screen output to the manager terminal 200. It can be recognized that, by controlling the air conditioning apparatus 300 can normalize the temperature in the kennel 10. Alternatively, when the abnormality object detecting apparatus 100 detects a temperature abnormality in the kennel 10 and directly delivers it to the air conditioning apparatus 300, the air conditioner 300 may directly normalize the temperature in the kennel 10. For example, if the temperature in the kennel 10 is lower or higher than the temperature suitable for the livestock, the movement of the livestock tends to be slowed down. Accordingly, the abnormality object detecting apparatus 100, the manager terminal 200, or the air conditioning apparatus 300 may detect a temperature abnormality in the kennel 10 and normalize the temperature in the kennel 10.

The air conditioning apparatus 300 may adjust the humidity of the kennel 10. When the humidity in the kennel 10 is abnormal, the air conditioning apparatus 300 may be controlled to normalize the humidity in the kennel 10.

According to an embodiment of the present invention, the abnormal object detection apparatus 100 extracts abnormal object information by driving a previously stored algorithm. To this end, the abnormal object detecting apparatus 100 may transmit the training data to the remote learning server 400 and extract the abnormal object information by applying the parameter received from the learning server 400 to the algorithm for detecting the abnormal object. have.

According to the exemplary embodiment of the present invention, the learning server 400 receives the training data from the plurality of abnormal object detection apparatus 100, and re-trains the training data to extract parameters. For example, the learning server 400 may learn the learning data using a deep learning technique, but is not limited thereto. The learning server 400 may learn the learning data using various techniques and extract parameters.

Here, the abnormal object detecting apparatus 100 may be mixed with the local machine, and the learning server 400 may collect the training data from the plurality of abnormal object detecting apparatuses installed in the plurality of kennels.

The abnormality object detecting apparatus 100 according to an exemplary embodiment of the present invention pre-processes the image data, selects the training data, and transmits only the selected training data to the training server 400. Accordingly, communication traffic between the abnormality object detecting apparatus 100 and the learning server 400 can be reduced, and the amount of computation of the learning server 400 can be reduced.

4 is a block diagram of an apparatus for detecting an abnormal object according to an embodiment of the present invention, and FIG. 5 is a flowchart of a method for detecting an abnormal object in an apparatus for detecting an abnormal object according to an embodiment of the present invention.

Referring to FIG. 4, the abnormal object detecting apparatus 100 may include a photographing unit 111, a control unit 112, a communication unit 113, a display unit 114, a user interface unit 115, an encoding unit 116, and a database. 117, a light source 118, a pan tilt unit 119, and a learning preprocessor 120. However, according to the exemplary embodiment of the present invention, at least one of the display unit 114, the user interface unit 115, the light source unit 118, and the pan tilt unit 119 may be omitted. Here, the photographing unit 111 may be implemented to include a lens and an image sensor, and at least one of the controller 112, the encoding unit 116, and the learning preprocessor 120 may be implemented by a computer processor or a chip. The database 117 may be mixed with a memory, and the communication unit 113 may be mixed with an antenna or a communication processor.

The photographing unit 111 may include one or a plurality of photographing units. For example, the photographing unit 111 may include at least one upper photographing unit disposed above the kennel 10 and at least one side photographing unit disposed at the side of the kennel 10. Each of the upper photographing unit and the side photographing unit may be an IP camera that can communicate in a wired or wireless manner and transmit real-time images.

The photographing unit 111 may generate image data by photographing an image including a plurality of objects. In the embodiment of the present invention, a plurality of individuals may mean poultry farmed in a kennel. In the present specification, image data photographed by the photographing unit 111 may be mixed with original data, original image, photographed image, and the like.

The photographing unit 111 may generate a plurality of image data using a plurality of images sequentially photographed. For example, the photographing unit 111 may generate first image data by capturing a first image including a plurality of objects, and generate second image data by capturing a second image including a plurality of objects. can do. Each of the first image and the second image may be an image continuously photographed in time, and one image data may mean a single frame. The photographing unit 111 may generate the first image data and the second image data by using the first image and the second image which are sequentially photographed.

The photographing unit 111 may be an image sensor photographing a subject using a complementary metal-oxide semiconductor (CMOS) module or a charge coupled device (CCD) module. At this time, the input image frame is provided to the CMOS module or CCD module in the photographing unit 111 through the lens, the CMOS module or CCD module converts the optical signal of the subject passing through the lens into an electrical signal (image data) and outputs it. do.

The photographing unit 111 may include a fisheye lens or a wide angle lens having a wide viewing angle. Accordingly, it is also possible for one photographing unit 111 to photograph the entire space inside the kennel 10.

In addition, the photographing unit 111 may be a depth camera. The photographing unit 111 may be driven by any one of various depth recognition methods, and the depth information may be included in the image photographed by the photographing unit 111. The photographing unit 111 may be, for example, a Kinect sensor. Kinect sensor is a depth camera of the structured light projection method, it is possible to obtain a three-dimensional information of the scene by projecting a pattern image defined using a projector or a laser, and obtains the image projected pattern through the camera. These Kinect sensors include infrared emitters that irradiate patterns using infrared lasers, and infrared cameras that capture infrared images. An RGB camera that functions like a typical webcam is disposed between the infrared emitter and the infrared camera. In addition to this, the Kinect sensor may further comprise a pan tilt unit 119 for adjusting the angle of the microphone array and the camera.

The basic principle of the Kinect sensor is that when the laser pattern irradiated from the infrared emitter is projected and reflected on the object, the distance to the surface of the object is obtained using the position and size of the pattern at the reflection point. According to this principle, the photographing unit 111 may generate the image data including the depth information for each object by irradiating the laser pattern to the space in the kennel and sensing the laser pattern reflected from the object.

The controller 112 controls the abnormal object detection apparatus 100 as a whole. In particular, according to an embodiment of the present invention, the control unit 112 extracts the abnormal object information from the image data for the image photographed with a plurality of objects by the photographing unit 111. The abnormal object information may include at least one of information on the presence or absence of an abnormal object in the image data, information on a spatial area in which the abnormal object exists in the image data, and information on a time domain in which the abnormal object exists in the image data. have. Here, the information about the spatial region in which the abnormal object exists in the image data may be coordinate information of the region in which the abnormal object exists or coordinate information of the abnormal object. In this case, the abnormal entity information may be represented by region, block, or pixel, and the abnormal entity information may further include a probability that there is an abnormal entity by region, block, or pixel. In the present specification, the abnormal entity information may be mixed with the abnormal entity data. Alternatively, the abnormal entity information may be information obtained from abnormal entity data. Alternatively, the abnormal entity information may include abnormal entity data. At this time, the control unit 112 may extract the abnormal object information by a method of driving the algorithm stored in advance in the database 117, the pre-stored algorithm and the parameters applied to the algorithm is the result learned by the learning server 400 Extracted algorithms and parameters. Details of how the control unit 112 extracts abnormal object information will be described later.

The controller 112 may perform an operation of the abnormality object detecting apparatus 100 by executing a command stored in the user interface 115 or the database 117, for example.

Alternatively, the controller 112 may control various operations of the abnormality object detecting apparatus 100 using a command received from the manager terminal 200.

The controller 112 may control the photographing unit 111 to track and capture a specific area in which an abnormal object is located. The tracking shooting target may be set through the user interface unit 115 or may be set through a control command of the manager terminal 200. The control unit 112 controls the photographing unit 111 to track and photograph a specific area in which an abnormal object exists to generate image data so that continuous monitoring can be performed.

In this case, the controller 112 may control the pan tilt unit 119 of the abnormal object detecting apparatus 100 to perform tracking shooting. The pan tilt unit 119 may control a photographing area of the photographing unit 111 by driving two motors, a pan and a tilt. The pan tilt unit 119 may adjust the directing direction of the photographing unit 111 to photograph a specific area under the control of the controller 112. In addition, the pan tilt unit 119 may adjust the directing direction of the photographing unit 111 to track a specific object under the control of the controller 112.

In an embodiment of the present invention, the controller 112 receives an image photographed by the photographing unit 111, and an area in which no motion is detected, for example, a motion, is not detected for a predetermined time among the objects in the input image. An image in which no pixels are masked may be generated and output. In this specification, an image in which a pixel in which no motion is detected is masked may mean an image in which a pixel in which no motion is detected is displayed to be distinguished from a neighboring pixel or an image which is not displayed in order to be distinguished from a peripheral pixel. That is, in the present specification, masking may mean an overall method of distinguishing from surrounding pixels, and may be interpreted to include not only masking processing generally applied in the field of image processing, but also mosaic processing, transparent pixel processing, and the like. have.

The communication unit 113 may perform data communication with at least one of the other abnormality object detecting apparatus, the manager terminal 200, or the learning server 400. For example, the communication unit 113 may include a wireless LAN (WLAN), Wi-Fi, WiBro, WiMAX, World Interoperability for Microwave Access (WMAX), and HSDPA (High Speed Downlink). Data communication may be performed using telecommunication technologies such as Packet Access, IEEE 802.16, Long Term Evolution (LTE), and Wireless Mobile Broadband Service (WMBS).

Alternatively, the communication unit 113 may include Bluetooth, Radio Frequency Identification (RFID), Infrared Data Association (IrDA), Ultra Wideband (UWB), Zigbee, and Near Field Communication (NFC). In addition, as a wired communication technology, data communication may be performed using a short-range communication technology such as USB communication, Ethernet, serial communication, and optical / coaxial cable.

For example, the communication unit 113 may perform data communication with another abnormality object detecting apparatus using a short range communication technology, and perform data communication with the manager terminal 200 or the learning server 400 using a long distance communication technology. However, the present invention is not limited thereto, and various communication technologies may be used in consideration of various aspects of the kennel 10.

The communication unit 113 transmits the image data photographed by the photographing unit 111 to the manager terminal 200, or transmits the abnormal object information extracted by the controller 112 to the manager terminal 200, or the image data and the abnormality. The result of matching the individual information may be transmitted to the manager terminal 200. In addition, the communication unit 113 may transmit a bitmap of an image photographed by the photographing unit 111 and a density of an object in the image to the learning server 400. Here, the bitmap transmitted to the learning server 400 may be a bitmap modified by the manager terminal 200.

The data transmitted through the communication unit 113 may be compressed data encoded through the encoding unit 116.

The display unit 114 includes a liquid crystal display (LCD), a thin film transistor liquid crystal display (TFT LCD), an organic light-emitting diode (OLED), and a flexible display. ), At least one of a 3D display and an e-ink display.

The display unit 114 may display at least one of image data and distribution data in which pixel values are adjusted through the control unit 112.

In addition, the display unit 114 may output the image data photographed by the photographing unit 111 to the screen, or may output the result of detecting the image data and the abnormal object information on the screen.

In addition, the display unit 114 may output various user interfaces or graphic user interfaces on the screen.

The user interface 115 generates input data for controlling the operation of the abnormality object detecting apparatus 100. The user interface unit 115 may include a keypad, a dome switch, a touch pad, a jog wheel, a jog switch, and the like. When the display unit 114 and the touch pad have a mutual layer structure and constitute a touch screen, the display unit 114 may be used as an input device in addition to the output device.

The user interface 115 may receive various commands for the operation of the abnormality object detecting apparatus.

The encoding unit 116 encodes the image data photographed by the photographing unit 111 or the processed image data processed through the control unit 112 into a digital signal. For example, the encoding unit 116 may encode image data according to H.264, H.265, Moving Picture Experts Group (MPEG), and Motion Joint Photographic Experts Group (M-JPEG) standards.

The database 117 may include a flash memory type, a hard disk type, a multimedia card micro type, a memory of a card type (for example, SD or XD memory). , Magnetic memory, magnetic disk, optical disk, random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (ROMEEP), programmable memory (PROM) Read-Only Memory) may include at least one storage medium. In addition, the anomaly detecting apparatus 100 may operate a web storage that performs a storage function of the database 117 on the Internet, or may operate in connection with the web storage.

The database 117 may store image data photographed by the photographing unit 111 and may store image data for a predetermined period of time.

In addition, the database 117 may store data and programs necessary for the abnormal object detection apparatus 100 to operate, and may store an algorithm and parameters applied to the controller 112 to extract the abnormal object information. have.

In addition, the database 117 may store various user interfaces (UIs) or graphical user interfaces (GUIs).

The light source unit 118 may irradiate light in a direction that is directed under the control of the controller 112. For example, the light source unit 118 may include at least one laser diode LD and a light emitting diode LED. The light source unit may emit light of various wavelength bands under the control of the controller.

For example, the light source unit 118 may irradiate light in the infrared wavelength band for night photographing. Alternatively, the light source unit 118 may irradiate light in the ultraviolet wavelength band for photochemotherapy of livestock in the kennel.

The learning preprocessor 120 extracts the learning data to be used for learning by the learning server 400 from the image data acquired by the photographing unit 111. A detailed method of extracting the training data by the learning preprocessor 120 will be described below.

Referring to FIG. 5, the apparatus 100 for detecting an abnormality acquires image data of an image in which a plurality of objects are photographed together using the photographing unit 111 (S500), and the controller 112 obtains the image data. The abnormal object information is extracted from the apparatus (S502). Here, one image data may mean a single frame, and the photographing unit 111 may generate a plurality of image data by using images sequentially photographed. In this case, the controller 112 may extract the abnormal entity information using a pre-stored algorithm and a parameter applied thereto. Here, the pre-stored algorithm is a motion among the objects using a first algorithm trained to display the area where the object is determined to be located in the image data and a second algorithm trained to detect motion by using an optical flow. This may mean an algorithm for displaying an object without. Here, the first algorithm may use a real-time object detection method using an algorithm indicating an area in which an object exists, or the first algorithm may indicate distribution information of an object, that is, density information of an area in which an object is located. Can also be used. Here, the first algorithm may be all or part of the learned model in this specification. Here, the learned model may be a computer readable program and may be stored in a recording medium or a storage device that can be executed by the computer. A processor in a computer may read a program stored in a recording medium or a storage device, execute a program, that is, a trained model, calculate input information, and output a calculation result. In this case, the input of the trained model may be image data photographing one or a plurality of objects, and the output of the trained model may be a density map of an object in the image data. Here, the density map may be mixed with position distribution data, density information, density image, and the like. 6 is a diagram for describing an object density prediction network, which is an example of a first algorithm according to an embodiment of the present invention. Here, the object density prediction network is an example of a deep learning algorithm designed to display density information of an area where an object is located. The object density prediction network may be an algorithm for inputting an original image into a convolution network-based learning machine and then outputting a density image represented by a gray scale probability map. Here, the convolutional network may be a neural network. In this case, the image data photographing one or more objects becomes an input layer of the neural network, and the neural network may perform an operation on the image data photographing one or more objects based on the learned weight coefficient. And, the output layer of the neural network may be a density map for the entity. According to this, abnormal object information can be easily extracted even in a kennel environment in which objects having a similar shape, such as a chicken house, are accommodated at high density.

The object density prediction network may be learned using the original image and the density image, and may be learned by the training server of FIG. 11 to be described later.

As shown in FIG. 6, the object density prediction network may include at least one convolution network (layer).

The convolutional network may classify the feature points of the image using at least one feature map (W).

The convolutional network may improve the performance of the object density prediction network through at least one pooler and / or activator.

The object density prediction network further includes a concatenator, and the concatenator can concatenate the output results of at least one convolutional network, rearrange them, and output density (distribution) information of the object using feature points of the original image. have.

At least one feature map (W) may be trained (tuned) to output density information of an object in a training server to be described later with reference to FIG. 11.

In an embodiment of the present invention, the input teacher data of the object density prediction network included in the learning server may be an original image and a learning label, and the output may be a density image. Here, the original image may be an image in which one or a plurality of objects are photographed by the photographing unit, and the learning label may be an object density estimation image represented by a pseudo density 8-bit gray image or a probability map. The density density network is configured to have the same or similar size (Width x Highet) as the original image, and the position of each pixel (or block) corresponds to each other, and the pixel value of the density image is It can represent the probability that an object (eg poultry) can be present in the corresponding pixel. For example, the higher the probability that an object exists in a pixel of the original image, the higher the value of the pixel of the density image may be. Here, the density image may be an 8-bit gray image or a probability map. For example, a density image may be generated using a method of taking a white dot in the middle of a labeled rectangular box and blurring the spot to blur each object.

The communication unit 113 of the abnormal object detecting apparatus 100 transmits the abnormal object information acquired by the control unit 112 to the manager terminal 200 (S504), and the manager terminal 200 outputs the abnormal object information (S506). ). Accordingly, the manager can recognize the abnormal object information output to the manager terminal 200. Here, the abnormal entity information may be mixed with the abnormal entity data.

7 is a diagram illustrating an example of displaying abnormal entity information on a manager terminal. The manager terminal 200 may display an image as shown in FIG. 7C. That is, the manager terminal 200 includes an image obtained by matching the original image photographed by the photographing unit 111 as shown in FIG. 7A and abnormal object data extracted by the controller 112 as shown in FIG. 7B. Can be displayed. Here, the abnormal object data may be a chicken density estimation image, and may be displayed as a pseudo density 8 bit gray image, a probability map, or a bit map. In this case, the abnormal object data is not data indicating an abnormal probability for each object, but data representing an abnormal probability of an object, a body, or a head in a region, block, or pixel corresponding to each divided area, block, or pixel in the image data. to be.

If there is an error in the abnormal object information exposed to the manager terminal 200, the manager may transmit feedback information to the abnormal object detection apparatus 100 through the manager terminal 200 (S508). For example, when it is determined that the abnormal object detecting apparatus 100 is an abnormal object even though it is not an abnormal object, or when the abnormal object detecting apparatus 100 is determined to be a normal object despite being an abnormal object, the manager terminal 200 is determined. ) May transmit feedback information indicating that there is an error in the abnormal object information to the abnormal object detecting apparatus 100. Here, the feedback information may include at least one of a temporal domain and a spatial domain in which the error information is abnormal. The error time zone and the space zone may be selected or designated by an administrator, and may be input through a user interface unit of the manager terminal 200. For example, as illustrated in FIG. 8C, the manager terminal 200 displays abnormal object information 802, 804, 806, and 808 for four areas of the screen 800, and the manager displays 4. If one or more pieces of abnormal object information (for example, 806) is determined to be an error, the administrator may select and feed back the abnormal object information 806 which is an error through the user interface. Alternatively, when the abnormal object information is not displayed despite the abnormal object in the screen output by the manager terminal 200, the manager may designate and feed back a time area and a spatial area which appear to be the abnormal object.

The learning preprocessor 120 of the object detecting apparatus 100 extracts training data using the feedback information received from the manager terminal 200 (S510) and transmits the extracted training data to the training server 400 ( S512).

Here, the training data may be part of the image data acquired in step S500. The learning preprocessor 120 of the abnormality object detecting apparatus 100 may extract the image data according to the time domain and the spatial domain included in the feedback information received from the manager terminal 200 among the image data acquired in step S500. . For example, when the manager terminal 200 determines that there is an error in a part of the nth frame (eg, 806 of FIG. 8C), the training data is part of the nth frame (eg, FIG. 8). 806 of (c). Alternatively, when the manager terminal 200 determines that there is an error in the abnormal object data of the n + 3 th frame in the n th frame among the entire frames, the training data may be the n + 3 th frame in the n th frame of the entire frames. . Accordingly, the abnormal object detecting apparatus 100 may extract only the image data having an error in detecting the abnormal object from the entire image data acquired by the photographing unit 111 and transmit the extracted image data to the learning server 400. Accordingly, communication traffic between the abnormality object detecting apparatus 100 and the learning server 400 may be significantly reduced. Here, the training data may further include at least one of error information and correction information as well as extracted image data. The error information may be information indicating that a determination by the controller is wrong, and the correction information may be information indicating a direction to be corrected. Alternatively, the training data may include the extracted original image data and the abnormal object data after the error is corrected. For example, if it is determined that there is an error in some region (for example, 806 of FIG. 8C) in the n th frame to the n + 3 th frame, the training data is determined for the n th frame to the n + 3 th frame. Image data of the original and abnormal object data for the nth to n + 3th frames, and the abnormal object data for the nth to n + 3th frames may be included in some areas determined to be in error (eg, This may be a gray image after the error of 806 of FIG. 8C is corrected. The training server 400 retrains using the training data received from the abnormal object detection apparatus 100 (S514). That is, the learning server 400 may detect an error in some areas determined to be in error in the original image data for the n th frame to the n + 3 th frame and the abnormal object data for the n th th frame to the n + 3 th frame. The relationship between the gray images after the correction can be learned. Accordingly, the learning server 400 transmits the update information for detecting the abnormal object to the abnormal object detecting apparatus 100 (S516). Here, the update information may be a parameter applied to an algorithm for detecting an abnormal object, which may be in the form of an adjustable matrix.

Thereafter, the apparatus 100 for detecting an abnormality acquires image data of an image in which a plurality of objects are taken together as shown in step S500 (S518), and the controller 112 obtains the image using an algorithm and update information stored in advance. The abnormal object information is extracted from the image data (S520).

Accordingly, the abnormal object detecting apparatus 100 may detect the abnormal object by reflecting the feedback information of the manager terminal 200, and may reduce communication traffic between the abnormal object detecting apparatus 100 and the learning server 400. .

As illustrated in FIG. 2, when the learning server 400 communicates with the plurality of abnormal object detecting apparatuses 100, the learning server 400 may transmit the same update information to the plurality of abnormal object detecting apparatuses 100. have. According to this, the result of re-learning by the error information obtained from one abnormal object detecting apparatus 100 can be reflected in the other abnormal object detecting apparatus 100. As a result, the plurality of abnormal object detecting apparatuses 100 Overall, the detection accuracy can be improved. Alternatively, since the abnormal object detection apparatus 100 may be in a different environment, the update information may be specific for the abnormal object detection apparatus 100.

According to an embodiment of the present invention, the learning server 400 also stores an algorithm such as an object density prediction network of the abnormal object detecting apparatus 100, and the learning server 400 uses the same as an original image corresponding to the error data. It is desirable to relearn.

Hereinafter, a method of detecting an abnormal object by the abnormal object detecting apparatus 100 according to an exemplary embodiment of the present invention will be described.

FIG. 9 is a block diagram of a controller included in an abnormal object detecting apparatus according to an embodiment of the present invention. FIG. 10 is a diagram illustrating an abnormal object detecting algorithm of the abnormal object detecting apparatus according to an embodiment of the present invention. Is a diagram for explaining a method of detecting an abnormal object by the apparatus for detecting an abnormal object according to an embodiment of the present invention by using the result of re-learning by the learning server.

Referring to FIG. 9, the control unit 112 includes a first feature extraction unit 600, a second feature extraction unit 602, and an abnormal entity information generation unit 604.

The first feature extracting unit 600 extracts a position distribution of a plurality of objects in the image data with respect to the image photographed by the photographing unit 111, and the second feature extracting unit 602 extracts a plurality of objects in the image data. Extract the movement. The abnormal object information generating unit 604 then generates the abnormal object information, for example, the pixel based on the position distribution extracted by the first feature extraction unit 600 and the movement extracted by the second feature extraction unit 602. Estimate the odds of a star anomaly.

In detail, the first feature extracting unit 600 may generate position distribution data indicating the position distribution of the plurality of objects using the image data. Here, the location distribution of the plurality of objects may mean a density distribution of objects for each location, and the location distribution data may be mixed with the density map. As described above, the first feature extraction unit 600 may use a real-time object detection method using a first algorithm trained to suggest an area in which the object is located in the image data, that is, an area suggestion algorithm. The first feature extraction unit 600 generates first position distribution data indicating a position distribution of the plurality of objects by using the first image data generated by including the plurality of objects, for example, and includes the plurality of objects. Second position distribution data representing a position distribution of a plurality of objects may be generated using the second image data generated by the method. Here, the first position distribution data and the second position distribution data may be position distribution data of image data generated in time series.

In the exemplary embodiment of the present invention, the position distribution data does not indicate individual individual positions but is data indicating a probability that an individual, a trunk, or a head may exist in a region or block corresponding to each divided region or block of the image data. The position distribution data may be a heat map expressing a probability that an object exists in each pixel in a different color.

In addition, the first feature extraction unit 600 may detect an animal object from the image data using the object detection classifier. At this time, the object detection classifier is trained by constructing a training DB from images of animal objects photographed by different postures or external environments of the animal object. The object detection classifier is a SVM (Support Vector Machine), a neural network, and an AdaBoost algorithm. Create a database of animal subjects through various learning algorithms, including In detail, the first feature extracting unit 600 detects an edge of the object corresponding to the foreground in the image data of the background in the previously photographed kennel, applies an edge of the foreground object detected in the image data, and applies the edge of the foreground object. Animal objects can be detected by applying the object detection classifier to the area of the image data to which is applied.

The second feature extraction unit 602 may generate motion data indicating the movement of the motion object among the plurality of objects using the image data. Here, the motion data is not data indicating the movement of an individual object, but data indicating whether a motion exists in a corresponding area or block for each divided area or block of the image data, and the motion data may be mixed with the motion map. . The motion data may be data indicating whether a motion exists in a pixel corresponding to each pixel. As described above, the second feature extraction unit 602 may use a second algorithm trained to detect motion using optical flow. The second feature extraction unit 602 may detect a movement at a specific point, a specific object, or a specific pixel on the distribution map using the single image data or the plurality of consecutive image data.

For example, the second feature extraction unit 602 generates first motion data indicating the movement of the motion object among the plurality of objects by using the first image data, and operates the motion among the plurality of objects by using the second image data. Second motion data representing the movement of the object may be generated. Here, the first motion data and the second motion data may be motion data for a plurality of image data generated in time series.

The second feature extraction unit 602 may detect the movement of the moving object using the Dense Optical Flow method. The second feature extraction unit 602 may detect a motion for each pixel by calculating a motion vector for all pixels on the image data. In the case of the Dense Optical Flow method, since the motion vector is calculated for all pixels, the detection accuracy is improved, but the amount of computation is relatively increased. Therefore, the Dense Optical Flow method can be applied to a specific area where detection accuracy is very high, such as a kennel where an abnormal situation is suspected or a kennel with a large number of individuals.

Alternatively, the second feature extraction unit 602 may detect the movement of the moving object using a sparse optical flow method. The second feature extraction unit 602 may detect a motion by calculating a motion vector only for some of the characteristic pixels that are easy to track, such as edges in the image. The sparse optical flow method reduces the amount of computation, but only results for a limited number of pixels. Therefore, the sparse optical flow method may be applied to a kennel having a small number of individuals or to a specific area where the objects do not overlap.

Alternatively, the second feature extraction unit 602 may detect movement of the moving object using block matching. The second feature extraction unit 602 may divide the image evenly or unequally, calculate a motion vector with respect to the divided region, and detect a motion. Block Matching reduces the amount of computation because it calculates the motion vector for each partition, but it can have a relatively low detection accuracy because it calculates the results for the motion vector for each region. Accordingly, the block matching method may be applied to a kennel with a small number of individuals or to a specific area where the objects do not overlap.

Alternatively, the second feature extraction unit 602 may detect the movement of the moving object by using a continuous frame difference method. The second feature extraction unit 602 may compare the consecutive image frames for each pixel and calculate a value corresponding to the difference to detect motion. Since the Continuous Frame Difference method detects motion by using the difference between frames, the overall computational amount is reduced, but the detection accuracy of a large object or a duplicate object may be relatively low. In addition, the Continuous Frame Difference method can not distinguish between the background image and the moving object may have a relatively low accuracy. Therefore, the Continuous Frame Difference method may be applied to a kennel with a small number of objects or to a specific area where the objects do not overlap.

Alternatively, the second feature extraction unit 602 may detect the movement of the moving object by using the background subtraction method. The second feature extraction unit 602 may compare successive image frames for each pixel in a state where the background image is initially learned, and calculate a value corresponding to the difference to detect motion. The background subtraction method is to learn the background image in advance so that the background image can be distinguished from the moving object. Therefore, a separate process of filtering the background image is required, thereby increasing the amount of computation but improving the accuracy. Therefore, the Background Subtraction method can be applied to a specific area where detection accuracy is very high, such as a kennel where an abnormal situation is suspected or a kennel with a large number of individuals. In the Background Subtraction method, the background image can be updated continuously.

The second feature extraction unit 602 detects the movement on the distribution chart using an appropriate method according to the environment in the kennel and the setting of the outside. The above motion detection method is just an example, and methods capable of displaying a region (for example, a pixel / block) in which a motion occurs in a frame may be used.

The process of generating position distribution data by the first feature extracting unit 600 and the process of generating motion data by the second feature extracting unit 602 may be performed simultaneously, in parallel, or sequentially. That is, the process of generating position distribution data by the first feature extracting unit 600 and the process of generating motion data by the second feature extracting unit 602 may be independently processed.

The abnormal object information generating unit 604 may generate the abnormal object data indicating the abnormal object by region, block, or pixel by comparing position distribution data and motion data of the image data by region, block, or pixel. The abnormal object information generating unit 604 may generate, for example, first abnormal object data indicating the abnormal object by comparing the first position distribution data and the first motion data. The abnormal object information generating unit 604 may generate first abnormal object data indicating information on an object for which movement is not detected on the first position distribution data by comparing the first position distribution data and the first motion data. . That is, the abnormal object information generating unit 604 may estimate that the object has no disease detected on the first position distribution data indicating the position of the object, and generate the first abnormal object data. That is, the first abnormal object data may refer to data obtained by determining whether an object is a disease by using an object's position distribution and motion detection information for single image data. Alternatively, the abnormal object information generating unit 604 compares the position distribution data and the motion data of the plurality of image data to calculate the cumulative number of motion detection and cumulative motion detection of the plurality of objects, and calculates the cumulative number of motion detection and the number of motion detection. The abnormal object data may be generated according to the cumulative number of motion non-detections. The abnormal object information generating unit 604 may generate the second abnormal object data by comparing the first abnormal object data, the second position distribution data, and the second motion data, for example. The abnormal object information generating unit 604 compares the first abnormal object data with the second position distribution data and the second motion data to calculate the cumulative number of motion detection of the plurality of objects and the cumulative number of motion non-detection of the plurality of objects, and detect the motion. The second or more entity data may be generated according to the cumulative number of times and the motion non-detection cumulative number of times. That is, the second abnormal object data may mean data obtained by determining whether an object is a disease by using position information and motion detection information of the object accumulated with respect to the plurality of image data.

The abnormal object information generating unit 604 may control the pixel display of the plurality of objects on the image data according to the motion detection accumulation count and the motion non-detection accumulation count of the abnormal object data. The abnormal object information generating unit 604 may control pixel display of the plurality of objects on the image data according to, for example, the second abnormal object data. Here, the display of the pixel may include all concepts for distinguishing and displaying a pixel corresponding to an arbitrary point from other pixels, such as the saturation of the pixel, the intensity of the pixel, the color of the pixel, the outline of the pixel, and the mark display. In the embodiment of the present invention, the display of the pixel may be controlled by adjusting the pixel value. The pixel value may be adjusted in stages, and in the case of a pixel having a high pixel value, the pixel value may be visually emphasized than a pixel having a low pixel value. However, the present invention is not limited thereto, and a pixel having a low pixel value may be set to be displayed to be highlighted than a pixel having a high pixel value. Herein, the pixel value may mean an abnormality probability of each pixel.

In the following description, one pixel represents one object in order to identify and identify the pixel and the object in which the motion is detected. This is for convenience of description and in practice, a plurality of pixels represent one object. That is, in order to determine an abnormal situation by detecting only movement of some body regions of poultry, a method of controlling the display of pixels by detecting movement for each pixel will be used.

The abnormal object information generating unit 604 may classify the object as an abnormal object as the movement of the specific object is not detected, and classify the pixel as a normal object as the motion is detected.

Here, at least one of the first feature extracting unit 600, the second feature extracting unit 602, and the abnormal object information generating unit 604 may apply the update information received from the learning server 400. Here, the update information may include first update information applied to the first feature extraction unit 600, second update information applied to the second feature extraction unit 602, and third application applied to the abnormal object information generation unit 604. It may include at least one of the update information. At least one of the first update information, the second update information, and the third update information may be update information extracted by the learning server 400 as a result of learning the training data transmitted by the abnormal object detection apparatus 100. Here, the training data may include a part of the image data acquired by the abnormal object detecting apparatus 100 and a part of the abnormal object data in which the error is corrected, which means that the abnormal object detecting apparatus 100 uses the algorithm for detecting the abnormal object. It may be obtained using the feedback information of the manager terminal 200 with respect to the abnormal object information obtained by driving. Here, at least one of the first update information, the second update information, and the third update information may include an adjustable matrix.

Referring to FIG. 10, first, n-th image data is obtained. Here, the image data may be, for example, RGB data having a size of W X H (S1101). Here, the n th image data may be mixed with the n th original data, the n th original image, the n th original image data, and the like.

The controller 112 of the abnormal object detection apparatus 100, for example, the first feature extraction unit 600 detects an object from the nth image data and generates position distribution data of the object with respect to the nth image data (S1102). ). Here, the position distribution data may be generated for each region, for each block, or for each pixel, and the controller 112 may use the first algorithm trained to suggest a region in which the object is located in the image data. Here, the first algorithm may use a real-time object detection method using an algorithm indicating an area where an object exists as described above, or the first algorithm may include distribution information of an object, that is, an area of an area where an object is located. It is also possible to use a method of representing density information. Here, the position distribution data may be a first density map.

In the following description, a method of calculating or displaying anomalous object data using a pixel-by-pixel operation of image data will be described.

The update parameter μ value is applied to the position distribution data. Alternatively, the update parameter μ value and the offset parameter ε value may be applied at the same time. μ is a very small value, for example 0.001. That is, the position distribution data is controlled to be gradually displayed on the pixels only after accumulating for a long time. The offset parameter ε is a parameter for adjusting the accumulation of the position distribution data and may have a value of 1/10 to 1/100 of μ (S1103).

The controller 112 of the abnormal object detecting apparatus 100, for example, the second feature extraction unit 602 detects the movement of the object with respect to the n-th image data by comparing the n-th image data and the n-th image data. do. In this case, the n−1th image data may be stored in the latch circuit or the buffer circuit. The control unit 112 of the abnormal object detection apparatus 100, for example, the second feature extraction unit 602, generates motion data according to the detected movement (S1104). Here, the motion data may be a motion map. Here, the motion data may be generated for each region, block, or pixel, and the controller 112 may use a second algorithm trained to detect motion by using an optical flow. The update parameter κ may be applied to the operation data. κ is a parameter for adjusting the accumulation of motion data (S1105).

The control unit 112 of the abnormal object detecting apparatus 100, for example, the abnormal object information generating unit 604 adds the n-1th or more object data to the position distribution data of the n-th image data (S1106), and the n-th image. The abnormal object data for the n-th image data is generated by subtracting the motion data of the data (S1107). Here, the abnormal object data for the nth image data may be an nth abnormal object density map.

The apparatus 100 for detecting an abnormal object according to an exemplary embodiment of the present invention repeats steps S1101 to S1107 to display the color of the detected object lightly or close to the original color, and accumulates the color of the detected object. Can be controlled to display dark or close to red.

Thereafter, the n-th or more entity data may be matched on the n-th image data, that is, the n-th original data, and the image in which the n-th or more entity data is matched on the n-th original data may be displayed on the manager terminal 200. have. That is, the region where the abnormal object is located in the n-th image may be masked using the n-th or more object density map, and the masked image may be displayed on the manager terminal 200.

The operation of the abnormality object detecting apparatus 100 according to the exemplary embodiment of the present invention may apply the following Equation 1.

[Equation 1]

Pixel _t = Pixel _t-1 (1-μ) + μW _t -F _t

Μ in Equation 1 may be changed according to a setting as an update parameter.

In Equation 1, Pixel _t and Pixelt-1 are abnormal object data, and may indicate concentration of pixels as a value for displaying the presence or absence of the abnormal object in the pixel. The higher the probability that an abnormal object exists, the darker the color can be displayed. For example, if there is no probability of anomalies, it is displayed in primary color (white). The higher the probability of anomaly is expressed, the closer it is to red. To make it happen. Accordingly, Pixel _t and Pixelt-1 may be set to have a value between 0 and 1, and the closer to 0, the closer to the primary color (white), and the closer to 1, the red.

In Equation 1, Pixel _t-1 is abnormal object data of a previous frame in which position distribution data and motion data are accumulated. In Equation 1, Pixel _t is abnormal object data updated by applying position distribution data and motion data of the current frame.

In Equation 1, Wt may be position distribution data of a current frame. The position distribution data may have a value between 0 and 1 as a probability that an object exists in a corresponding pixel. The update parameter μ may be applied to the position distribution data. μ is a very small value, for example 0.001. That is, the position distribution data is controlled to be gradually displayed on the pixels only after accumulating for a long time.

In Equation 1, F _t may be motion data of the current frame. The motion data is an absolute value of the motion vector and may have a value of 0 or more. Since the magnitude of the motion vector corresponds to the velocity of the object, it may have a value of 1 or more. Since a separate parameter is not reflected in the motion data, when the motion is detected in the pixel, the display of the pixel is initialized.

Alternatively, the operation of the abnormality object detecting apparatus according to the embodiment may apply the following Equation 2.

[Equation 2]

Pixel _t = Pixel _t-1 (1-μ + ε) + μW _t -F _t

※: update parameter, ε: offset parameter

In Equation 2, an offset parameter ε is added in Equation 1, and the same description as in Equation 1 is omitted. The offset parameter ε is a parameter for adjusting the accumulation of the position distribution data and may have a value of 1/10 to 1/100 of μ.

Alternatively, the operation of the apparatus for detecting an abnormality object according to the embodiment may apply Equation 3 or Equation 4 below.

[Equation 3]

Pixelt = Pixelt-1 (1-μ) + μWt-κFt

[Equation 4]

Pixel _t = Pixel _t-1 (1-μ + ε) + μW _t -κF _t

※: update parameter, ε: offset parameter, κ: update parameter

Equation 3 or 4 is a product of the operation data F _t multiplied by the update parameter κ and the same content as in Equation 1 and Equation 2 is omitted. constant κ is a parameter that adjusts the accumulation of motion data.

Alternatively, the operation of the abnormality object detecting apparatus according to the embodiment may apply Equation 5 below.

[Equation 5]

max (1, 2, 3, or 4, 0)

Equation 5 is an equation for preventing the value of Equations 1, 2, 3 or 4 from falling below zero. For example, the size of the motion data Ft is greater than the sum of other parameter values, and when the value of Equation 1, 2, 3 or 4 becomes a negative number less than 0, it is corrected to be displayed as 0. Control method.

Here, the first update information, the second update information, and the third update information received from the learning server 400 may be applied during the processes S1101 to S1107. For example, the first update information may be represented by at least one of the update parameter μ value and the offset parameter ε value as described above. In order to obtain at least one of the first update information, the second update information, and the third update information, the abnormality object detecting apparatus 100 according to an embodiment of the present invention may extract the training data and transmit the training data to the training server 400. . In this case, the training data may include a part of the image data acquired by the abnormal object detecting apparatus 100, which is a feedback on the abnormal object information extracted by the abnormal object detecting apparatus 100 by driving an algorithm for detecting the abnormal object. Can be obtained using the information. Here, the feedback information may include information modified by the manager terminal 200.

More specifically, referring to FIG. 11 (a), the apparatus for detecting an anomaly object 100 according to an exemplary embodiment of the present invention may extract training data and transmit the training data to the training server 400, and the training server 400 may perform abnormalities. The object density prediction network may be retrained, that is, updated using the training data received from the object detection apparatus 100. Here, the training data may include image data indicated as having an error and abnormal object data in which the error is corrected. For example, when it is detected that the abnormal object exists even though the abnormal object does not exist in the partial region 806 in the n th frame, the training data is the image data of the n th frame and the abnormal object data of the n th frame. The abnormality of the partial region 806 may include the object data. Here, the abnormality object data in which the error is corrected may be information in which the density of the object is corrected by the manager terminal 200, and the density distribution of the object for each region, block, or pixel, extracted by the abnormal object detection apparatus 100, may be determined. In the case of a first density map, a density distribution of an object for each region, block, or pixel modified by the manager terminal 200 may be referred to as a second density map. The learning server 400 compares the output image of the object density prediction network in the learning server 400 with the error-corrected image included in the training data, that is, the correct answer image, to obtain a loss, and to minimize the loss. In the trainer, variables (eg, feature maps) of the object density prediction network can be learned (corrected).

Subsequently, referring to FIG. 11B, the control unit 112 of the abnormality object detecting apparatus 100 generates the position distribution data, that is, in step S1102, the object density network re-learned by the learning server 400. It is available. That is, the control unit 112 of the abnormality object detecting apparatus 100 uses the update information obtained by re-learning the object density network using the training data, for example, the n-th image and the second density map, from the learning server 400. The abnormal object density map of the predetermined image after the nth image may be output.

12 is a result of testing the performance of the abnormal object detection apparatus according to an embodiment of the present invention.

12 (a) shows an input image. As shown in FIG. 12 (a), there are many errors in the input image in a method of detecting an individual object when a plurality of objects are distributed in a cluster. Therefore, detection of abnormal objects also causes a large error.

FIG. 12B illustrates a result of processing the input image of FIG. 12A using a first algorithm according to an exemplary embodiment of the present invention. For example, according to the first algorithm, a learning model that is trained by creating a probability map by taking a white dot in the middle of a labeled rectangular box and blurring the spot to detect each object is used. Accordingly, the trained model may output a probability map (for example, an 8-bit gray image representing the position distribution of the object) according to the possibility of the existence of the object in the input image. As a result, it is possible to detect a plurality of individuals by displaying the locations where the individual objects exist more accurately even in the group of individuals.

The learning model may be used to detect an individual in consideration of the density of the individual, and may determine whether there is an abnormality for each individual.

In another embodiment, since the clustered individuals have a large number of individuals, a large amount of overhead is involved in determining whether there is an abnormality for each individual (tracking method for each individual). Therefore, using the following second algorithm, it is possible to more simply and accurately determine the presence or absence of abnormalities on the clustered individuals.

FIG. 12C illustrates a result of processing the input image of FIG. 12A by the second algorithm according to the exemplary embodiment. For example, according to the second algorithm, the moving pixel may be found by extracting the optical flow from the input image using the dense optical flow method. Then, moving pixels were removed from the 8-bit gray image of FIG. 12 (b) to obtain an image representing a pixel without motion as shown in FIG. 12 (c).

FIG. 12 (d) is an image obtained by matching the image of FIG. 12 (b) and the image of FIG. 12 (c) with the input image of FIG. 12 (a). Referring to FIG. 12 (d), an image showing an area in which there is no motion in an area in which a plurality of objects in the input image are clustered and distributed may be obtained. Accordingly, it can be seen that a pixel in which motion is detected can be extracted with respect to the input image, and from this, it is possible to detect a plurality of objects moving in a cluster or a plurality of objects moving in a cluster.

The term '~ part' used in the present embodiment refers to software or a hardware component such as a field-programmable gate array (FPGA) or an ASIC, and '~ part' performs certain roles. However, '~' is not meant to be limited to software or hardware. '~ Portion' may be configured to be in an addressable storage medium or may be configured to play one or more processors. Thus, as an example, '~' means components such as software components, object-oriented software components, class components, and task components, and processes, functions, properties, procedures, and the like. Subroutines, segments of program code, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, and variables. The functionality provided within the components and the 'parts' may be combined into a smaller number of components and the 'parts' or further separated into additional components and the 'parts'. In addition, the components and '~' may be implemented to play one or more CPUs in the device or secure multimedia card.

Although described above with reference to a preferred embodiment of the present invention, those skilled in the art will be variously modified and changed within the scope of the invention without departing from the spirit and scope of the invention described in the claims below I can understand that you can.

Claims (14)

1. A communication processor in communication with the learning server; And

   A control unit configured to receive a captured image and output an image of masking a pixel area in which no motion is detected among the objects of the captured image;

   And the communication processor transmits the captured image and a bitmap of the object density of the captured image to the learning server.
2. The method of claim 1,

   And the bitmap includes information in which an object density of the captured image is modified.
3. A control unit for extracting abnormal object information from a captured image in which a plurality of objects are photographed together;

   A communication unit which transmits the abnormal object information to a manager terminal and receives feedback information on the abnormal object information from the manager terminal, and

   A learning preprocessor configured to generate learning data based on the feedback information,

   The learning data includes image data extracted based on the feedback information, and is transmitted to the learning server by the communication unit.

   And the controller extracts the abnormal object information by using update information of the training data learned by the learning server.
4. The method of claim 3,

   The feedback information includes an abnormality object detecting apparatus including information about a time domain in which there is an error among the abnormality object information and information about a spatial region.
5. The method of claim 3,

   The abnormality object detecting apparatus based on the feedback information, the image data is image data in a time domain and a spatial domain extracted according to the feedback information of the captured image.
6. The method of claim 3,

   The control unit

   A first feature extracting unit extracting a position distribution of the plurality of objects in the captured image;

   A second feature extraction unit for extracting movements of the plurality of objects in the captured image; and

   And an abnormal object information generating unit generating abnormal object data based on the position distribution extracted by the first feature extracting unit and the movement extracted by the second feature extracting unit.
7. receiving an nth image;

   outputting a first density map of the object in the n-th image;

   outputting a motion map in an nth image;

   Outputting an n-th or more object density map by using the first density map, the motion map, and a previously stored n-th or more object density map; And

   Masking an area in which the abnormal object is located in the n-th image using the n-th or more object density map,

   The nth image and the second density map is transmitted to the learning server,

   The second density map is abnormal object detection method of the density information of the object in the n-th image.
8. The method of claim 7, wherein

   Receiving update information obtained by re-learning using the nth image and the second density map from the learning server, and

   And outputting an abnormal object density map of the predetermined image after the nth image by using the update information.
9. The method of claim 7, wherein

   The modified information includes at least one of error time space information and space area information of the n-th or more object density map.

   And the second density map is extracted using at least one of the time domain information and the space domain information.
10. The method of claim 8,

    And the update information includes update information for extracting a location distribution of the plurality of objects in the predetermined image.
11. A learning server, and

    An abnormal object detection device in communication with the learning server;

    The abnormal object detection device,

    A communication processor in communication with the learning server; And

    A control unit configured to receive a captured image and output an image of masking a pixel area in which no motion is detected among the objects of the captured image;

    And the communication processor transmits the captured image and a bitmap of the object density of the captured image to the learning server.
12. Learned model for executing a computer to output a density map for the object based on image data of one or a plurality of objects.
13. The method of claim 12,

    The image data photographing the one or a plurality of objects becomes an input layer of a neural network, and the neural network performs an operation on the image data photographing the one or a plurality of objects based on the learned weight coefficient. A trained model that runs a computer such that a density map for an individual is the output layer of the neural network.
14. Receiving unit for receiving image data and abnormal object data, And

    A learning unit learning a relationship between the image data and the abnormal object data;

    The receiving unit receives the image data indicated as an error and the abnormal object data is corrected,

    And the learning unit relearns the relationship between the image data indicated as having the error and the abnormal entity data in which the error is corrected.

Images