KR20230069892A - Method and apparatus for identifying object representing abnormal temperatures - Google Patents

Abstract

The present disclosure relates to a method for identifying an object indicating an abnormal temperature and an electronic device for performing the method. According to one embodiment, a method for an electronic device to identify an object indicating an abnormal temperature may comprise the steps of: obtaining a first image and a second image of an object in a space captured by the electronic device from lights of different wavelength bands; identifying an object in the first image; identifying an object corresponding to the object in the first image in the second image, based on location information of the object identified in the first image; and identifying whether an object photographed by the electronic device is abnormal based on the temperature indicated by the object identified in the second image.

Description

Method and apparatus for identifying objects exhibiting abnormal temperature

The present disclosure relates to a method and apparatus for identifying an object exhibiting an abnormal temperature. More particularly, it relates to a method and apparatus for identifying an object exhibiting an abnormal temperature in an image using lights of different wavelength bands.

An artificial neural network may refer to a computing device or a method performed by a computing device to implement interconnected sets of artificial neurons. As an example of an artificial neural network, a deep neural network or deep learning may have a multi-layer structure, and each of the layers may be learned according to a plurality of data.

As the development of artificial neural network technology has recently been activated, technology for automatically recognizing objects in images is being actively researched in the field of artificial intelligence. In particular, technologies for identifying a person in an image obtained using a surveillance camera based on artificial intelligence technology are also being commercialized.

In addition, due to the recent spread of coronavirus, demand for monitoring equipment for monitoring fever patients is increasing, and technology development for automatically identifying people with abnormal temperatures (eg infected fever patients) is also being actively researched. there is.

5 However, most video or image analysis technologies still have limitations in accurately identifying a person's body temperature along with uniquely identifying a person. Therefore, there is a demand for developing a technology for effectively detecting a patient with fever while identifying individual people by analyzing an image or video.

Korea Patent No. 1242755

According to an embodiment, a method and electronic device for identifying an object exhibiting an abnormal temperature may be provided.

Also, according to an embodiment, a method and an electronic device for identifying an object exhibiting an abnormal temperature in an image using lights of different wavelength ranges may be provided.

According to an embodiment of the present disclosure for achieving the above technical problem, in a method for identifying an object exhibiting an abnormal temperature by an electronic device, from lights of different wavelength bands, an object in a space photographed by the electronic device is detected. Obtaining a first image and a second image related to; identifying an object in the first image; identifying an object corresponding to an object in the first image from the second image, based on location information of the object identified in the first image; and identifying whether the object photographed by the electronic device has an abnormality based on a temperature indicated by the object identified in the second image. A method including may be provided.

In addition, according to an embodiment, the method includes, as a result of identifying whether an object photographed by the electronic device is abnormal, outputting notification information for notifying that the object is an abnormal object when the object is identified as an abnormal object. ; may further include.

In addition, according to another embodiment of the present disclosure for solving the above technical problem, an electronic device for identifying an object exhibiting an abnormal temperature includes: a display for displaying at least one image; a storage unit for storing one or more instructions; and a processor executing the one or more instructions; The processor obtains a first image and a second image of an object in a space photographed by the electronic device from lights of different wavelength bands by executing the one or more instructions, and in the first image Identify an object, identify an object corresponding to the object in the first image in the second image based on location information of the object identified in the first image, and identify a temperature indicated by the object identified in the second image. Based on this, an electronic device may be provided that identifies whether or not the object photographed by the electronic device is abnormal.

Further, according to an embodiment, as a result of identifying whether an object photographed by the electronic device is abnormal, the processor may output notification information for notifying that the object is an abnormal object when the object is identified as an abnormal object. .

In addition, according to another embodiment of the present disclosure for solving the above technical problem, obtaining a first image and a second image of an object in a space photographed by an electronic device from lights of different wavelength bands; identifying an object in the first image; identifying an object corresponding to an object in the first image from the second image, based on location information of the object identified in the first image; and identifying whether the object photographed by the electronic device has an abnormality based on a temperature indicated by the object identified in the second image. A computer-readable recording medium recording a program for executing the method on a computer, including, may be provided.

1 is a diagram schematically illustrating a method of identifying an object exhibiting an abnormal temperature by an electronic device according to an exemplary embodiment.
2 is a flowchart of a method of identifying, by an electronic device, an object indicating an abnormal temperature, according to an exemplary embodiment.
3 is a diagram for describing in detail a method of identifying an object in a first image by an electronic device according to an exemplary embodiment.
4 is a diagram for explaining a process of identifying an object in a first image by an electronic device according to an exemplary embodiment.
5 is a diagram for explaining a process of identifying an object in a second image based on pixel information of pixels in a first image by an electronic device according to an embodiment.
6 is a diagram for explaining a process of converting a second image based on pixel information of pixels in a first image by an electronic device according to an exemplary embodiment.
7 is a diagram for explaining a process of converting a second image based on pixel information in a first image by an electronic device according to an exemplary embodiment.
8 is a diagram for explaining a process of identifying an object exhibiting an abnormal temperature in an image by an electronic device according to another embodiment.
9 is a block diagram of an electronic device according to an embodiment.
10 is a block diagram of an electronic device according to another embodiment.
11 is a block diagram of a server interworking with an electronic device according to an embodiment.

Terms used in this specification will be briefly described, and the present disclosure will be described in detail.

The terms used in the present disclosure have been selected from general terms that are currently widely used as much as possible while considering the functions in the present disclosure, but they may vary according to the intention or precedent of a person skilled in the art, the emergence of new technologies, and the like. In addition, in a specific case, there is also a term arbitrarily selected by the applicant, and in this case, the meaning will be described in detail in the description of the invention. Therefore, terms used in the present disclosure should be defined based on the meaning of the term and the general content of the present disclosure, not simply the name of the term.

When it is said that a certain part "includes" a certain component throughout the specification, it means that it may further include other components without excluding other components unless otherwise stated. In addition, terms such as "...unit" and "module" described in the specification mean a unit that processes at least one function or operation, which may be implemented as hardware or software or a combination of hardware and software. .

Hereinafter, with reference to the accompanying drawings, embodiments of the present disclosure will be described in detail so that those skilled in the art can easily carry out the present disclosure. However, the present disclosure may be embodied in many different forms and is not limited to the embodiments described herein. And in order to clearly describe the present disclosure in the drawings, parts irrelevant to the description are omitted, and similar reference numerals are attached to similar parts throughout the specification.

1 is a diagram schematically illustrating a method of identifying an object exhibiting an abnormal temperature by an electronic device according to an exemplary embodiment.

According to an embodiment, the electronic device 1000 may include an artificial intelligence model 110. According to one embodiment, the artificial intelligence model may include at least one artificial neural network model (Artificial Neural Network Model) or a machine learning model capable of being learned according to other artificial intelligence learning algorithms. The electronic device 1000 acquires images generated from light in different areas from at least one camera that captures the same space outside the electronic device, and uses the artificial intelligence model 110 to determine an abnormal temperature in the acquired images. The object it represents can be identified.

According to an embodiment, the electronic device 1000 may identify an object in the first image by extracting a feature vector from the first image 122 and inputting the extracted feature vector to the artificial intelligence model 110 . The electronic device 1000 may identify an object corresponding to the object identified in the first image 142 in the second image 142 by using information about the object identified in the first image 122 . The electronic device 1000 may determine whether the object has an abnormal temperature by identifying whether the temperature of the object identified in the second image is equal to or greater than a preset threshold temperature.

According to another embodiment, the electronic device 1000 may generate a third image by correcting image information in the second image based on information on the object identified in the first image 122 . The electronic device 1000 may determine whether the object has an abnormal temperature as the identification result 152 of the object currently located outside the electronic device by measuring the temperature indicated by the object identified from the third image. According to an embodiment, the first image may be a visible ray image generated from light in the visible ray region, and the second image may be a thermal infrared image generated from light in the thermal infrared region. However, the present invention is not limited thereto, and the first image and the second image may be images generated from light having different arbitrary wavelength regions.

Also, according to an embodiment, the electronic device 1000 may obtain a first image from a visible ray camera and a second image from a thermal infrared camera, but the electronic device 1000 may acquire a visible ray camera and a thermal infrared ray In the case of including a camera, a visible ray image and a thermal infrared ray image may be directly acquired. According to an embodiment of the present disclosure, a spectrum of light belonging to the visible ray region (eg, a region in which the wavelength of light is 380 nm to 780 nm) can be indicated, and a spectrum belonging to the thermal infrared region (eg, a wavelength of light is 0.78 um to 1000 um). It may represent a spectrum of light, but is not limited thereto, and may be arbitrarily set to represent different wavelength regions.

According to an embodiment, the electronic device 1000 is equipped with an AI program for image processing using an artificial intelligence model, and includes a smartphone, tablet PC, smart TV, mobile phone, media player, and server including an image capture function. , micro servers, or other mobile or non-mobile computing devices, but are not limited thereto. According to an embodiment, the artificial intelligence model used by the electronic device 1000 may include an artificial neural network model. An artificial neural network model may refer to a computing system based on a biological neural network. Unlike classical algorithms that perform tasks according to predefined conditions, artificial neural networks can learn to perform tasks by considering multiple samples.

An artificial neural network may have a structure in which artificial neurons are connected, and a connection between neurons may be referred to as a synapse. A neuron can process the received signal and transmit the processed signal to another neuron through a synapse. The output of a neuron may be referred to as an activation, and a neuron and/or synapse may have a weight that may be varied, depending on which weight the influence of a signal processed by a neuron may increase or decrease. . For example, an artificial neural network may be composed of a plurality of neural network layers. Each of the plurality of neural network layers has a plurality of weight values (weights), and a neural network operation is performed through an operation between an operation result of a previous layer and a plurality of weights. A plurality of weights possessed by a plurality of neural network layers may be optimized by a learning result of an artificial neural network.

For example, a plurality of weights may be modified and updated so that a loss value or a cost value obtained from a neural network model is reduced or minimized during a learning process. The neural network model according to the present disclosure may include a deep neural network (DNN), for example, a Convolutional Neural Network (CNN), a Deep Neural Network (DNN), a Recurrent Neural Network (RNN), and a Restricted Neural Network (RBM). Boltzmann Machine), Deep Belief Network (DBN), Bidirectional Recurrent Deep Neural Network (BRDNN), or Deep Q-Networks, but is not limited to the above examples.

According to an embodiment, the electronic device 1000 includes a memory and a processor in which at least one artificial intelligence model is loaded, object identification 104 for images input to the electronic device, and one of the input images. Based on the information of the object identified in the image of , image information in another image may be calibrated (calibration 102 ), and the temperature of the object in the image including the calibrated image information may be identified ( 106 ). The electronic device 1000 may output the type of the identified object and information on whether the temperature of the object is within an ideal range as an object identification result 152 for images generated from light in different regions. According to an embodiment, when the identified object has an abnormal temperature, the electronic device 1000 identifies the object identified in the image as an abnormal object, and as the object is identified as an abnormal object, the electronic device 1000 automatically outputs alarm information, or Alarm information may be transmitted to an external device connected to the electronic device 1000 .

According to an embodiment, the electronic device 1000 may be connected to the server 2000. According to an embodiment, the electronic device 1000 may interwork with the server 2000 to identify whether an object in an image has an abnormal temperature. The server 2000 may include other computing devices capable of transmitting and receiving data with the electronic device 1000 by being connected to the electronic device 1000 through a network. According to an embodiment, the server 2000 may include a Local Area Network (LAN), a Wide Area Network (WAN), a Value Added Network (VAN), a mobile radio communication network, It can be connected to the image matching device 1000 through a satellite communication network and their mutual combination. In addition, according to an embodiment, the server 2000 is a data communication network in a comprehensive sense that allows each network constituent entity (eg, an image matching device and a server) shown in FIG. 1 to communicate smoothly with each other, and is a wired Internet. , wireless Internet and mobile wireless communications networks themselves.

2 is a flowchart of a method of identifying, by an electronic device, an object indicating an abnormal temperature, according to an exemplary embodiment.

In operation S210, the electronic device 1000 may acquire a first image and a second image of an object in a space photographed by the electronic device from lights of different wavelength bands. According to an embodiment, the first image may be a visible ray image generated from light in the visible ray region, and the second image may be a thermal infrared image generated from light in the thermal infrared region. Hereinafter, for convenience, it is assumed that the first image is a visible ray image and the second image is a thermal infrared ray image.

According to an embodiment, the electronic device 1000 may acquire a visible ray image and a thermal infrared ray image from a visible ray camera and a thermal infrared camera that capture the same area of space outside the electronic device. However, according to another embodiment, when the electronic device 1000 includes a visible ray camera and a thermal infrared camera, the electronic device 1000 may directly acquire the first image and the second image.

In S220, the electronic device 1000 may identify an object in the first image. For example, the electronic device 1000 may include at least one artificial intelligence model. The electronic device 1000 extracts a feature vector representing a feature of an object in the first image based on pixel values in the first image, and inputs the extracted feature vector to a neural network model pre-stored in the electronic device, thereby generating the first image. I can identify my object. More specifically, the electronic device 1000 may store the object type or identification information in advance in the memory of the electronic device or on a server connected to the electronic device. The electronic device 1000 inputs the feature vector extracted from the first image to the artificial intelligence model, compares the output value of the artificial intelligence model output from the artificial intelligence model with the previously stored object type or identification information, An object on the first image may be recognized.

According to an embodiment, the electronic device 1000 inputs a feature vector extracted from the first image to an artificial intelligence model, and based on an output value of the artificial intelligence model output from the artificial intelligence model, the object in the first image It is also possible to identify a candidate object region with a high probability of existing. The electronic device 1000 may identify an object based on the probability that the object exists in the candidate object area. A detailed method of identifying an object in the first image by the electronic device 1000 using an artificial intelligence model will be described in detail with reference to FIG. 3 to be described later.

In operation S230, the electronic device 1000 may identify an object corresponding to the object in the first image in the second image, based on location information of the object identified in the first image. For example, the electronic device 1000 may identify an object in the first image by using an artificial intelligence model, and obtain information on the type of the object and the location of the object in the image as an object identification result. The electronic device 1000 may identify an object corresponding to the object identified in the first image in the second image by using location information of the object determined from the first image.

According to an embodiment, the electronic device 1000 uses the location information of the object obtained from the first image to determine coordinate values of pixels representing the object in the first image or to a boundary of a predetermined object area including pixels representing the object. Coordinate values of positioned pixels may be obtained. The electronic device 1000 identifies the location information of the object in the second image by applying the coordinate values of pixels representing the object obtained from the first image or the coordinate values of pixels positioned at the boundary of the object area to the second image. can do.

In operation S240, the electronic device 1000 may identify a temperature indicated by the object based on pixel information indicating the object identified in the second image. The electronic device 1000 may identify whether the object is abnormal by determining whether the temperature indicated by the object identified in the second image is equal to or greater than a predetermined threshold value. More specifically, the electronic device 1000 detects a candidate object region from the second image based on location information of the object identified from the first image, and based on pixel values of pixels in the candidate object region, the object is detected. You can also identify the temperature it represents. Although not shown in FIG. 2 , as will be described later, the electronic device 1000 generates a third image by correcting the second image based on pixel information of pixels in the first image, and creates an object in the third image. It is possible to determine whether a temperature indicated by an object outside the current electronic device is abnormal by using a pixel value for .

3 is a diagram for describing in detail a method of identifying an object in a first image by an electronic device according to an exemplary embodiment.

In S310, the electronic device 1000 may extract a feature vector based on pixel values in the first image. For example, the electronic device 1000 may extract feature points from the first image based on the amount of change in pixel values in the first image, and extract a feature vector using the extracted feature points. According to an embodiment, the electronic device 1000 uses a feature extraction algorithm of at least one of SIFT, Histogram of Oriented Gradient (HOG), Harr, Ferns, and Local Binary Pattern (LBP) from the first image to extract feature points. may be extracted, and a feature vector may be generated using the extracted feature points.

In S320, the electronic device 1000 may input the feature vector to the pretrained neural network model and identify the object using the output value of the neural network model output from the neural network model. More specifically, when a feature vector is input, the electronic device 1000 identifies an object in the first image by using a neural network model pre-learned to output information about the location and type of the identified object. can For example, the electronic device 1000 may access a database in which the type or identification information of an object matching the extracted feature vector is previously stored, and recognize the type or identification information of the object that matches the feature vector. According to an embodiment, the database may be previously stored in the electronic device 1000 or may be stored on a server connected to the electronic device 1000.

Also, the electronic device 1000 may obtain location information of an object in the first image from an output value of the neural network model. According to an embodiment, the location information of the object may include coordinate values of pixels representing the object in the first image or coordinate values of pixels located at a boundary of a predetermined object area including pixels representing the object. Also, according to an embodiment, as described later with reference to FIG. 4 , the electronic device 1000 may identify the width of the object area where the object in the first image is located or the probability that the object exists in the corresponding object area. The electronic device 1000 may select different neural network models based on at least one of the width of the object area included in the first image or the probability value of an object existing in the detected object area. The electronic device 1000 may identify an object from the first image by using the selected neural network model.

4 is a diagram for explaining a process of identifying an object in a first image by an electronic device according to an exemplary embodiment.

The electronic device 1000 according to an embodiment of the present disclosure acquires a first image (eg, a visible ray image) and a second image (eg, a thermal infrared image) generated from lights of different regions, and obtains the obtained first image. And using the second image, the type of object in the image and the temperature indicated by the object may be identified, and the type of identified object and the temperature indicated by the object may be displayed together. Also, the electronic device 1000 may acquire the first image and the second image, and output the obtained first image and the second image together. Hereinafter, with reference to FIG. 4 , it will be described assuming that the object is a person.

For example, the electronic device 1000 performs the method of identifying an object exhibiting an abnormal temperature described above with reference to FIGS. 2 and 3 , and marks an object region where an object exists in a first image (eg, a visible ray image) as a box ( 412), and the probability that an object exists in each object area can be displayed as a number. Also, referring to the boxes 402, 404, 406, 408, and 410 shown in FIG. 4, the electronic device 1000 uses the type of object identified in the object area and temperature information indicated by the object as an identification result, and the object area It can be displayed around the box.

In addition, according to an embodiment, the electronic device 1000 may identify objects using different neural network models according to an area of an object area in which an object exists in the first image or a probability value in which an object exists in the object area. may be The electronic device 1000 according to the present disclosure may pre-learn neural network models that identify objects matched to an area of an object area or a probability value of an object existing in the object area. The electronic device 1000 selects one neural network model from pre-learned neural network models according to the area of the object region in the first image or the probability value of the presence of an object in the object region, and uses the selected neural network model to generate the first neural network model. You can also identify objects from images.

The electronic device 1000 according to the present disclosure acquires a first image and a second image generated from lights of different wavelength regions, and obtains the first image (eg, visible ray image) and the second image (thermal infrared image). By using together, the type of object in the image and the temperature indicated by the object may be identified, and information on the type of identified object and temperature indicated by the object may be superimposed on the first image and displayed. The electronic device 1000 according to the present disclosure uses not only an image generated from light in a single area, but also a plurality of images generated from light in different areas to identify the type of object in the image and the temperature indicated by the object. can be accurately identified.

5 is a diagram for explaining a process of identifying an object in a second image based on pixel information of pixels in a first image by an electronic device according to an embodiment.

Hereinafter, for convenience, it is assumed that the first image is a visible ray image generated from light in the visible ray region and the second image is a thermal infrared image generated from light in the thermal infrared region.

In S520, the electronic device 1000 may convert pixel information of the second image based on pixel information of the first image. For example, resolution of the first image and the second image may vary according to a difference in characteristics between an image sensor that senses light in the visible ray region and an image sensor that senses light in the thermal infrared region. Accordingly, the number of pixels for representing objects in the first image and the second image may be different, and as a result, the same object may be displayed in different sizes and outlines in the first image and the second image.

The electronic device 1000 according to the present disclosure may convert pixel information of the second image based on pixel information of the first image in order to compensate for differences in image characteristics generated from lights in different regions. According to an embodiment, the electronic device 1000 may generate a third image by converting pixel information of the second image based on pixel information in the first image. A process of converting pixel information of the second image by the electronic device 1000 based on pixel information of the first image will be described in detail with reference to FIG. 6 .

In S540, the electronic device 1000 may identify an object corresponding to the object in the first image from the second image including the converted pixel information. According to another embodiment, the electronic device 1000 may identify an object corresponding to an object in the first image from a third image generated by converting pixel information of the second image. Since the second image or the third image including the converted pixel information generated by the electronic device 1000 is currently converted based on the pixel information in the first image (eg, visible ray image), the object in the first image An object can be represented with the same number of pixels as the number of pixels for expressing . Accordingly, the electronic device 1000 does not immediately identify an object in the second image (eg, a thermal infrared image), but the second image corrected based on pixel information of the first image or a second image generated by correcting the second image. 3 Since the temperature of the identified object is determined after identifying the object in the image, the temperature of the object can be recognized more accurately.

6 is a diagram for explaining a process of converting a second image based on pixel information of pixels in a first image by an electronic device according to an exemplary embodiment.

Hereinafter, for convenience, it is assumed that the first image is a visible ray image generated from the visible ray region and the second image is a thermal infrared image generated from the thermal infrared region. However, it is not limited thereto, and the first image and the second image may be arbitrary images generated from lights in different regions.

In S620, the electronic device 1000 may generate at least one pixel in the second image based on pixel information in the first image. For example, the first image is a visible ray image and may have a higher resolution than the second image, which is a thermal infrared ray image. Accordingly, the number of pixels for representing the object on the first image is the object on the second image. It may be more than the number of pixels to express. Accordingly, the shape of the object on the second image may be in a distorted state. Therefore, the electronic device 1000 further generates pixels as many as the resolution difference between the first image and the second image in order to correct the distortion of the object on the second image, and places the generated pixels on the second image. can reflect

According to an embodiment, the electronic device 1000 compares a pixel value difference between a pixel included in the first image and a pixel included in a second image corresponding to each pixel included in the first image, and then compares a pixel value If the difference between is greater than or equal to a predetermined threshold, a pixel may be additionally generated at at least one point around a pixel on the second image representing a difference in pixel values greater than or equal to the predetermined threshold.

In S640, the electronic device 1000 may determine pixel values of at least one pixel generated on the second image. For example, the electronic device 1000 may allocate a pixel value to at least one pixel generated on the second image in order to correct a resolution difference between the first image and the second image. The electronic device 1000 may identify pixel values around one or more pixels generated on the second image, and allocate a pixel value to the one or more generated pixels based on the identified pixel values. According to an embodiment, the electronic device 1000 may determine an average of pixel values of pixels around at least one pixel generated on the second image as a pixel value of the generated one pixel. According to another embodiment, the electronic device 1000 applies an interpolation algorithm to pixel values of pixels around at least one pixel generated on the second image, so that a pixel of a newly generated pixel in the second image is applied. values can also be estimated.

In S660, the electronic device 1000 may convert pixel values of pixels existing in the second image based on pixel information in the first image. For example, the electronic device 1000 further generates pixels in the second image and allocates pixel values to the generated pixels in order to compensate for the resolution difference between the first image and the second image, and also assigns pixel values to the second image. By converting the pixel values of pixels existing on the image, the object in the second image can be expressed more accurately.

According to the above-described steps S620 to S660, the electronic device 1000 may correct the second image by converting pixel information in the second image. The electronic device 1000 generates a third image by correcting at least one pixel information in the second image, identifies an object using the generated third image, and determines whether the temperature of the identified object is equal to or higher than a preset threshold. may be identified.

7 is a diagram for explaining a process of converting a second image based on pixel information in a first image by an electronic device according to an exemplary embodiment.

Referring to FIG. 7 , a process of generating a third image by correcting the second image based on pixel information in the first image by the electronic device 1000 will be described in detail. For example, the electronic device 1000 may obtain a visible ray image 720 from an RGB camera 722 and a thermal infrared image 740 from a thermal imaging camera 742 . Referring to the thermal infrared image 740, it can be seen that the area of the object is displayed thicker on the thermal infrared image, unlike the visible ray image 720, even if the same object is photographed. The electronic device 1000 corrects the thermal infrared image 740 according to the method according to FIG. 6 to compensate for distortion due to the difference in image characteristics between the visible ray image 720 and the thermal infrared image 740, and thereby, the second Distortion of the shape of an object on an image may be corrected.

For example, the electronic device 1000 generates at least one pixel in the second image and assigns a pixel value to the generated at least one pixel to compensate for a resolution difference between the first image and the second image. In addition, the second image may be corrected by converting pixel values already existing on the second image. The electronic device 1000 may correct the second image, identify the object using the corrected second image, and recognize the temperature of the object. However, according to another embodiment, the electronic device 1000 generates a third image by correcting the second image, identifies an object in the image using the generated third image 760, and identifies the image of the object. It can also detect temperature.

According to an embodiment, the electronic device 1000 may correct the second image based on the photographed chess board after photographing the chess board together with the subject. For example, since a chess board may not be well visible on a thermal infrared image, after heating the chess board to a preset temperature, photographing the chess board together with the subject, and generating a second image including the photographed chess board. A third image may be generated by correcting based on pixel information of one image.

8 is a diagram for explaining a process of identifying an object exhibiting an abnormal temperature in an image by an electronic device according to another embodiment.

Referring to FIG. 8 , the operation of the electronic device 1000 will be described in detail by dividing a case in which an abnormality is not detected (812) and a case in which an abnormality is detected (814). For example, the electronic device 1000 acquires visible ray images 802 and 806 and thermal infrared images 804 and 808, and uses the obtained visible ray images and thermal infrared images to determine whether an object in the image has an abnormal temperature. It can be identified whether it represents

For example, the electronic device 1000 may obtain a visible ray image 802 and a thermal infrared ray image 804 from a visible ray camera and a thermal imaging camera, respectively. The electronic device 1000 may obtain an output value of the artificial intelligence model by extracting a feature vector of an object from the visible ray image 802 and inputting the extracted feature vector to a pre-learned artificial intelligence model. The electronic device 1000 identifies the type of object or identification information in the visible ray image 802 by comparing the output value of the artificial intelligence model with a pre-stored object type or identification information database. Location information of an object can be identified.

The electronic device 1000 may identify an object corresponding to the object identified in the visible ray image in the thermal infrared image 804 by using the location information of the object acquired from the visible ray image. According to an embodiment, the electronic device 1000 may obtain information about a candidate object region in which an object is located, as object location information, from a visible ray image, and use the information about the candidate object region to obtain information about a candidate object region. Candidate object regions 842 and 844 where an object is located on the infrared image may be identified. The electronic device 1000 may identify a maximum temperature 832 and a minimum temperature 834 within the candidate object regions 842 and 844 identified on the thermal infrared image.

The electronic device 1000 may identify the maximum temperature 832 in the candidate object region identified on the thermal infrared image as the temperature of the object in the candidate object region. According to an embodiment, the electronic device 1000 may identify the temperature of the object in the visible ray image as a normal temperature when the temperature of the object identified in the thermal infrared image does not exceed a preset threshold (eg, 39). According to an embodiment, the electronic device 1000 may display identification information 822 and 824 of the recognized object and temperature values 826 and 828 of the recognized object together on the visible ray image.

Also, according to an embodiment, the electronic device 1000 may obtain a visible ray image 806 and a thermal infrared image 808 from a visible ray camera and a thermal imaging camera, respectively. The electronic device 1000 may obtain an output value of the artificial intelligence model by extracting a feature vector of the object from the visible ray image 806 and inputting the extracted feature vector to the artificial intelligence model. The electronic device 1000 may recognize object identification information 852 and 854 in the visible ray image 806 by retrieving the output value of the artificial intelligence model from a database in which previously stored object identification information is stored. The electronic device 1000 may display the identification information of the object identified in the visible ray image 806 by overlapping it on the visible ray image.

In addition, the electronic device 1000 obtains object position information from the output value of the artificial intelligence model, and based on the obtained object position information, the electronic device 1000 corresponds to the object identified in the visible ray image in the thermal infrared image 808. object can be identified. According to an embodiment, the electronic device 1000 may obtain information on a candidate object region where the object is located as object location information from the visible ray image 806, and use the information on the candidate object region , candidate object regions 866 and 868 in which objects are located on the thermal infrared image may be identified.

The electronic device 1000 may identify a maximum temperature 862 and a minimum temperature 864 within the identified candidate object regions 866 and 868 on the thermal infrared image. The electronic device 1000 may identify the maximum temperature 862 in the candidate object region identified on the thermal infrared image as the temperature of the object in the candidate object region. According to an embodiment, the electronic device 1000 may identify the temperature of the object in the visible ray image as an abnormal temperature when the temperature of the object identified in the thermal infrared image exceeds a preset threshold (eg, 39 degrees). For example, the electronic device 1000 may identify 58 degrees, which is the maximum temperature of the candidate object region 866 identified on the thermal infrared image, as the temperature of the object, and 30 degrees, which is the maximum temperature of the candidate object region 868. can be identified as the temperature of the object. Since the maximum temperature of the candidate object region 866 is 58 degrees, which is equal to or greater than a preset threshold (eg, 39 degrees), the electronic device 1000 may output notification information after identifying the temperature as abnormal.

In addition, the electronic device 1000 may output notification information and display identification information 852 and 854 of the recognized object and temperature values 856 and 858 of the recognized object on the visible ray image 806. there is. Also, according to an embodiment, when the temperature of the identified object belongs to the abnormal temperature range, the electronic device 1000 identifies an object exhibiting an abnormal temperature on the visible ray image as an abnormal object, and identifies the object identified as an abnormal object. You can also mask red.

9 is a block diagram of an electronic device according to an embodiment.

10 is a block diagram of an electronic device according to another embodiment.

As shown in FIG. 9 , the electronic device 1000 may include a processor 1400 and a memory 1402 . However, not all illustrated components are essential components. The electronic device 1000 may be implemented with more components than those illustrated, or the electronic device 1000 may be implemented with fewer components.

For example, as shown in FIG. 1 , the electronic device 1000 according to an embodiment includes a processor 1400 and a memory 1402, as well as a user input interface 1100, an output unit 1200, and a camera 1300. , may further include a network interface 1500.

The user input interface 1100 means a means through which a user inputs a sequence for controlling the electronic device 1000 . For example, the user input interface 1100 includes a key pad, a dome switch, a touch pad (contact capacitive method, pressure resistive film method, infrared sensing method, surface ultrasonic conduction method, A spray tension measuring method, a piezo effect method, etc.), a jog wheel, a jog switch, and the like may be included, but are not limited thereto. The user input interface 1100 may receive a user's input sequence for a screen output on the display by the electronic device 1000 . Also, the user input interface 1100 may receive a user's touch input touching the display or a key input through a graphic user interface on the display.

The output unit 1200 may output an audio signal, a video signal, or a vibration signal, and the output unit 1200 may include a display unit 1210, a sound output unit 1220, and a vibration motor 1230. there is. The display unit 1210 may display at least one of a first image, a second image, or a third image generated by correcting the second image acquired by the electronic device 1000 . According to an embodiment, the display unit 1210 may overlap the identification information of the object identified by the electronic device and information about the temperature of the object to display the first image.

The audio output unit 1220 outputs audio data received from the network interface 1500 or stored in the memory 1700 . Also, when the electronic device 1000 recognizes that an object indicating an abnormal temperature has been identified, the audio output unit 1220 may output notification information in the form of audio. Also, the sound output unit 1220 outputs sound signals related to functions performed by the electronic device 1000 .

The camera 1300 may acquire images generated from light of different wavelength regions. For example, the camera 1300 may include at least one image sensor for sensing light of different wavelength ranges, and generate images by sensing light of various wavelength ranges using the at least one image sensor. can According to an embodiment, the camera 1300 may include at least one of a visible ray camera, a thermal infrared ray camera, a near infrared ray camera, and an RGB camera. According to an embodiment, the camera 1300 may be included outside of the electronic device. In this case, the electronic device 1000 is communicatively connected to the camera 1300, so that a visible ray image, a near-infrared ray image, and a thermal An infrared image or the like may be received.

The processor 1400 may control overall operations of the electronic device 1000 by executing one or more instructions stored in the memory 1402 . For example, the processor 1400 may control operations of the user input interface 1100, the output unit 1200, the camera 1300, and the network interface 1500 by executing one or more instructions stored in the memory 1402. can Also, the processor 1400 may perform the functions of the electronic device 1000 described in FIGS. 1 to 8 by executing instructions stored in the memory 1402 .

According to an embodiment, the processor 1400 executes one or more instructions stored in the memory 1402 to obtain a first image and a second image of an object in a space captured by the electronic device from lights of different wavelength bands. Obtaining, identifying an object in the first image, identifying an object corresponding to the object in the first image in the second image based on location information of the object identified in the first image, and 2 Based on the temperature indicated by the object identified in the image, whether or not the object photographed by the electronic device may be identified as abnormal.

In addition, according to an embodiment, the processor 1400 executes one or more instructions stored in the memory 1402 to identify whether an object photographed by the electronic device is abnormal, and when the object is identified as an abnormal object, Notification information for notifying that the object is an abnormal object may be output.

Further, according to an embodiment, the processor 1400 extracts a feature vector based on pixel values in the first image by executing one or more instructions stored in the memory 1402, and when the feature vector is input, the first image The location information of the object in the first image may be identified using a neural network model that outputs location information of the object identified in 1 image and information about the type of the object.

Also, according to an embodiment, the processor 1400 converts pixel information of the second image based on pixel information of pixels in the first image by executing one or more instructions stored in the memory 1402, and An object corresponding to an object in the first image may be identified from the second image including the converted pixel information.

Also, according to an embodiment, the processor 1400 generates at least one pixel in the second image based on pixel information of pixels in the first image by executing one or more instructions stored in the memory 1402 and , Based on pixel values around the generated at least one pixel, the generated at least one pixel allocates pixel values, and based on the pixel information of the pixels in the first image, the pixel values in the second image You can transform the pixel values of existing pixels.

In addition, according to an embodiment, the processor 1400 executes one or more instructions stored in the memory 1402, based on location information of an object identified in the first image, to obtain a second image including the converted pixel information. In the image, a candidate object region may be determined, and an object corresponding to an object in the first image may be identified within the determined candidate object region.

Further, according to an embodiment, the processor 1400 identifies a temperature indicated by the object based on pixel values of pixels in the candidate object region by executing one or more instructions stored in the memory 1402, and identifies the identified temperature. When the temperature of the object is equal to or higher than a preset threshold, the object may be identified as an abnormal object.

Also, according to an embodiment, the processor 1400 executes one or more instructions stored in the memory 1402 to generate at least one of the first image, the second image, or the second image including the converted pixel information. output, output the type of the identified object and temperature information indicated by the identified object, and transmit the notification information to an external device connected to the electronic device.

The network interface 1500 may include one or more components that allow the electronic device 1000 to communicate with a camera or another electronic device (not shown) located outside the electronic device or the server 2000 . Another device (not shown) may be another computing device capable of communicating with the electronic device 1000 and processing an image.

For example, the network interface 1500 may include a wireless communication interface 1510, a wired communication interface 1520, and a mobile communication unit 1530. The wireless communication interface 1510 includes a short-range wireless communication unit, a Bluetooth communication unit, a near field communication unit, a WLAN (Wi-Fi) communication unit, a Zigbee communication unit, an infrared data association (IrDA) ) communication unit, WFD (Wi-Fi Direct) communication unit, etc., but is not limited thereto. The wired communication interface 1520 may connect the server 2000 or the electronic device 1000 by wire.

The mobile communication unit 1530 transmits and receives a radio signal with at least one of a base station, an external terminal, and a server on a mobile communication network. Here, the wireless signal may include a voice signal, a video call signal, or various types of data according to text/multimedia message transmission and reception.

According to an embodiment, the network interface 1500 may transmit the first image and the second image captured by the electronic device to the outside or, conversely, may receive the first image and the second image from the outside. Also, the network interface 1500 may transmit object identification information determined by the electronic device 1000 using the first image and the second image and information about the temperature of the object to the server 2000 .

The memory 1402 may store programs for processing and control of the processor 1400 and may store data input to or output from the electronic device 1000 . Also, the memory 1700 may store various instructions necessary for the electronic device 1000 to capture an image, identify an object in the captured image, and recognize the temperature of the identified object. In addition, the memory 1700 may further store information about an artificial intelligence model for identifying the type and position of an object in the input image when images generated from lights of different regions are input.

According to an embodiment, when an image is input, the memory 1402 is information about a neural network model that is learned to output information on the type and location of an object in the input image, and is related to layers and connection strength between the layers. You can also store the weight (weight) about. According to an embodiment, the memory 1402 may store weight values of a neural network model received by the electronic device 1000 from a server or an external electronic device, and when already stored weight values are modified or updated, the memory 1402 is modified and updated. You can also store the weight values again. According to an embodiment, the memory 1402 includes a learning algorithm for performing supervised learning on learning data, a learning model learned based on the learning algorithm, a feed forward neural network model, a support vector machine learning model, It can also store information about the linear regression learning model.

The memory 1402 is a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (eg SD or XD memory, etc.), RAM (RAM, Random Access Memory) SRAM (Static Random Access Memory), ROM (Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), PROM (Programmable Read-Only Memory), magnetic memory, magnetic disk , an optical disk, and at least one type of storage medium.

Programs stored in the memory 1402 may be classified into a plurality of modules according to their functions, for example, a UI module 1405, a touch screen module 1406, a notification module 1407, and the like. .

The UI module 1405 may provide a UI or GUI capable of controlling an operation of a camera or an electronic device by interlocking with the first image and the second image. The touch screen module 1406 may detect a user's touch gesture on the touch screen and transmit information about the touch gesture to the processor 1400 . The touch screen module 1406 according to some embodiments may recognize and analyze the touch code. The touch screen module 1406 may be configured as separate hardware including a controller.

The notification module 1407 may generate a signal for notifying occurrence of an event of the electronic device 1000 . The electronic device 1000 may include an instruction for outputting a notification signal when an object identified on the image is identified as an abnormal object having an abnormal temperature. Also, the notification module 1407 may store an instruction for outputting a notification when an event related to the other electronic device 1000 receiving a signal from an external server or receiving a message occurs. The notification module 1407 may output a notification signal in the form of a video signal through the display unit 1210 or may output a notification signal in the form of an audio signal through the audio output unit 1220 .

11 is a block diagram of a server interworking with an electronic device according to an embodiment.

The server 2000 may include a network interface 2100, a database 2200 and a processor 2300. The network interface 2100 may correspond to the network interface 1500 of the electronic device 1000 shown in FIG. 10 . For example, the network interface 2100 may receive identification information of an object identified by the electronic device 1000 and information about a temperature of the identified object from the electronic device 1000 . Also, according to an embodiment, the network interface 2100 may further receive information about the first image, the second image, or a third image generated by correcting the second image from the electronic device 1000 . In addition, the network interface 2100 receives at least one of the first image, the second image, and the third image from the electronic device, and the server performs object identification and temperature analysis of the identified object using the received images. may be transmitted to an electronic device.

The database 2200 may correspond to the memory 1402 of the electronic device 1000 shown in FIG. 10 . For example, the database 2200 includes learning data received from the electronic device 1000 and information on a learning model learned based on the learning data (eg, neural network model layers and weight values related to connection strength of the layers). ) can be stored.

The processor 2300 typically controls the overall operation of the server 2000. For example, the processor 2300 may generally control the DB 2200 and the network interface 2100 by executing programs stored in the DB 2200 of the server 2000 . Also, the processor 2300 may perform part of the operation of the electronic device 1000 in FIGS. 1 to 10 by executing programs stored in the DB 2100 . For example, the processor 2300 learns at least one artificial intelligence model based on the learning data received from the electronic device 1000, and uses the learned learning model to generate an object for an image received from the electronic device. An operation of identifying and identifying the temperature of the identified object may be performed.

The method according to an embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer readable medium. The computer readable medium may include program instructions, data files, data structures, etc. alone or in combination. Program commands recorded on the medium may be specially designed and configured for the present disclosure, or may be known and usable to those skilled in computer software.

In addition, a computer program device including a recording medium in which a program for performing a different method according to the above embodiment is stored may be provided. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. - includes hardware devices specially configured to store and execute program instructions, such as magneto-optical media, and ROM, RAM, flash memory, and the like. Examples of program instructions include high-level language codes that can be executed by a computer using an interpreter, as well as machine language codes such as those produced by a compiler.

Although the embodiments of the present disclosure have been described in detail above, the scope of the present disclosure is not limited thereto, and various modifications and improvements of those skilled in the art using the basic concepts of the present disclosure defined in the following claims are also included in the present disclosure. fall within the scope of the right

Claims (16)

A method for an electronic device to identify an object exhibiting an abnormal temperature,
obtaining a first image and a second image of an object in a space photographed by the electronic device from lights of different wavelength bands;
identifying an object in the first image;
A pixel value difference between a pixel included in the first image and a pixel included in a second image corresponding to each pixel included in the first image is compared, and when the difference between the pixel values is equal to or greater than a preset threshold, the generating additionally at least one pixel at at least one point around a pixel on a second image indicating a difference in pixel values equal to or greater than a set threshold;
allocating an average of pixel values of pixels around at least one pixel generated on the second image as pixel values of the generated at least one pixel;
converting pixel values of existing pixels in the second image based on pixel information of pixels in the first image;
identifying an object corresponding to an object in the first image from a second image including the generated pixels to which the pixel values are assigned and the existing pixels to which the pixel values are converted; and
identifying whether the object photographed by the electronic device has an abnormality based on a temperature indicated by the object identified in the second image; Including, method.
The method of claim 1, wherein the method
outputting notification information notifying that the object is an abnormal object when the object is identified as an abnormal object as a result of identifying whether the object photographed by the electronic device is abnormal; Further comprising a method.
3. The method of claim 2, wherein identifying an object in the first image comprises:
extracting a feature vector based on pixel values in the obtained first image; and
identifying location information of the object in the first image by using a neural network model that outputs location information of the object identified in the first image and information about the type of the object when the feature vector is input; Including, method.
The method of claim 1 , wherein identifying an object corresponding to an object in the first image comprises:
determining a candidate object region in a second image including the converted pixel information, based on location information of the object identified in the first image; and
identifying an object corresponding to an object in the first image within the determined candidate object region; Including, method.
The method of claim 4, wherein the step of identifying whether the object is abnormal
identifying a temperature indicated by the object based on pixel values of pixels in the candidate object area; and
identifying the object as an abnormal object when the temperature of the identified object is equal to or greater than a preset threshold; Including, method.
The method of claim 1, wherein the first image is an image generated from lights in a visible ray region, and the second image is an image generated from lights in a thermal infrared region.
The method of claim 3, wherein the neural network model
A deep neural network model, characterized in that it includes layers and weights for connection strengths between the layers, and is pre-learned by modifying and updating the weights.
The method of claim 1, wherein the method,
outputting at least one of the first image, the second image, or a second image including the converted pixel information; and
outputting the type of the identified object and temperature information indicated by the identified object; including,
In the step of outputting the notification information,
transmitting the notification information to an external device connected to the electronic device; Characterized in that it further comprises, the method.
An electronic device for identifying an object exhibiting an abnormal temperature,
a display that displays at least one image;
a storage unit for storing one or more instructions; and
a processor that executes the one or more instructions; including,
By executing the one or more instructions, the processor:
Obtaining a first image and a second image of an object in a space photographed by the electronic device from lights of different wavelength bands;
identify an object in the first image;
A pixel value difference between a pixel included in the first image and a pixel included in a second image corresponding to each pixel included in the first image is compared, and when the difference between the pixel values is equal to or greater than a preset threshold, the additionally generating at least one pixel at at least one point around a pixel on a second image representing a difference in pixel value equal to or greater than a set threshold;
assigning an average of pixel values of pixels around at least one pixel generated on the second image as pixel values of the generated at least one pixel;
Converting pixel values of existing pixels in the second image based on pixel information of pixels in the first image;
Identifying an object corresponding to an object in the first image from a second image including the generated at least one pixel to which the pixel values are assigned and the existing pixels to which the pixel values are converted;
Based on the temperature indicated by the object identified in the second image, the electronic device identifies whether or not the object captured by the electronic device is abnormal.
10. The method of claim 9, wherein the processor
The electronic device outputs notification information for notifying that the object is an abnormal object when the object is identified as an abnormal object as a result of identifying whether the object captured by the electronic device is abnormal.
11. The method of claim 10, wherein the processor
Extracting a feature vector based on pixel values in the obtained first image;
When the feature vector is input, identifying the location information of the object in the first image by using a neural network model that outputs location information of the object identified in the first image and information about the type of the object. Device.
10. The method of claim 9, wherein the processor
determining a candidate object region in a second image including the converted pixel information based on location information of an object identified in the first image;
Identifying an object corresponding to an object in the first image within the determined candidate object region.
13. The method of claim 12, wherein the processor
Based on pixel values of pixels in the candidate object region, identify a temperature indicated by the object;
When the temperature of the identified object is equal to or greater than a preset threshold, identifying the object as an abnormal object.
The method of claim 12, wherein the first image is an image generated from lights in a visible ray region, and the second image is an image generated from lights in a thermal infrared region,
The neural network model is a deep neural network model, and includes layers and weights related to connection strength between the layers, and is pre-learned by modifying and updating the weights.
10. The method of claim 9, wherein the processor
outputting at least one of the first image, the second image, or a second image including the converted pixel information;
Outputting the type of the identified object and temperature information indicated by the identified object;
The electronic device characterized in that for transmitting the notification information to an external device connected to the electronic device.
obtaining a first image and a second image of an object in a space photographed by the electronic device from lights of different wavelength bands;
identifying an object in the first image;
A pixel value difference between a pixel included in the first image and a pixel included in a second image corresponding to each pixel included in the first image is compared, and when the difference between the pixel values is equal to or greater than a preset threshold, the generating additionally at least one pixel at at least one point around a pixel on a second image indicating a difference in pixel values equal to or greater than a set threshold;
allocating an average of pixel values of pixels around at least one pixel generated on the second image as pixel values of the generated at least one pixel;
converting pixel values of existing pixels in the second image based on pixel information of pixels in the first image;
identifying an object corresponding to an object in the first image from a second image including the generated pixels to which the pixel values are assigned and the existing pixels to which the pixel values are converted; and
identifying whether the object photographed by the electronic device has an abnormality based on a temperature indicated by the object identified in the second image; A computer-readable recording medium recording a program for executing a method on a computer, including a.

Images