WO2022089064A1 - 一种图像识别的方法及电子设备 - Google Patents

一种图像识别的方法及电子设备 Download PDF

Info

Publication number
WO2022089064A1
WO2022089064A1 PCT/CN2021/118155 CN2021118155W WO2022089064A1 WO 2022089064 A1 WO2022089064 A1 WO 2022089064A1 CN 2021118155 W CN2021118155 W CN 2021118155W WO 2022089064 A1 WO2022089064 A1 WO 2022089064A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
spectral
information
spectrum
target substance
Prior art date
Application number
PCT/CN2021/118155
Other languages
English (en)
French (fr)
Inventor
郭宏伟
韩浩
吴泽仁
Original Assignee
华为技术有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 华为技术有限公司 filed Critical 华为技术有限公司
Publication of WO2022089064A1 publication Critical patent/WO2022089064A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
    • G01N21/00—Investigating or analysing materials by the use of optical means, i.e. using sub-millimetre waves, infrared, visible or ultraviolet light
    • G01N21/17—Systems in which incident light is modified in accordance with the properties of the material investigated
    • G01N21/25—Colour; Spectral properties, i.e. comparison of effect of material on the light at two or more different wavelengths or wavelength bands
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
    • G01N21/00—Investigating or analysing materials by the use of optical means, i.e. using sub-millimetre waves, infrared, visible or ultraviolet light
    • G01N21/17—Systems in which incident light is modified in accordance with the properties of the material investigated
    • G01N21/25—Colour; Spectral properties, i.e. comparison of effect of material on the light at two or more different wavelengths or wavelength bands
    • G01N21/31—Investigating relative effect of material at wavelengths characteristic of specific elements or molecules, e.g. atomic absorption spectrometry
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
    • G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/90—Determination of colour characteristics
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
    • G06V10/774—Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/10—Terrestrial scenes
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10024—Color image
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10048—Infrared image

Definitions

  • the present application relates to the technical field of image recognition, and in particular, to an image recognition method and electronic device.
  • the images collected by ordinary cameras cannot usually be used to identify the composition of the target substance in the images, so that people's requirements cannot be met. Therefore, in actual work or production, it is often necessary to obtain the composition information of the target substance in the image through a higher resolution image, and then identify the composition and content of the target substance. For example, by identifying the composition information of the fruit in the image, the composition and sugar content of the fruit in the image can be obtained.
  • hyperspectral images are multi-channel images subdivided in the wavelength domain. Therefore, when identifying hyperspectral images, more surface or intrinsic information of the target substance in the image can be obtained. For example, by acquiring the spectral information of the hyperspectral image, the vibrational information (such as O-H, C-H, N-H bonds, etc.) of a specific chemical group of the target substance in the image can be analyzed.
  • the vibrational information such as O-H, C-H, N-H bonds, etc.
  • the collected hyperspectral images it is often affected by the acquisition equipment, acquisition time, etc., resulting in the collected hyperspectral images being images of low-resolution spectral bands. Effective spectral information, and thus cannot more accurately identify the target material composition in the image. Therefore, it is necessary to reconstruct images of low-resolution spectral segments into images of high-resolution spectral segments through hyperspectral reconstruction technology to obtain more effective spectral information.
  • the RGB image is mainly reconstructed to obtain a hyperspectral image of the visible spectrum.
  • the wavelength range is mainly concentrated in the visible light region. Hyperspectral images in the visible spectrum cannot obtain sufficient spectral information. Therefore, when identifying the composition information of the target substance in the hyperspectral image of the visible spectrum, there is still a problem that the target substance composition in the image cannot be identified more accurately.
  • the present application provides an image recognition method and electronic device, which are used to accurately recognize the component information of a target substance in an image.
  • an embodiment of the present invention provides an image recognition method, which is suitable for an electronic device with image capturing and recognition functions.
  • Acquiring a first image containing at least one target substance where the first image is an image of a first spectral section; the first spectral section is a low-resolution spectral section; performing spectral reconstruction on the first image to obtain the first spectral section
  • Two images the second image is an image of the second spectral band; the second spectral band is a high-resolution spectral band with a wavelength range in the visible light region and the infrared light region; the second image is identified to obtain the composition information of at least one target substance; displaying the composition information of the at least one target substance.
  • a hyperspectral image of a low-resolution spectral segment containing at least one target substance is first acquired, and the spectral segment is reconstructed on the hyperspectral image of the low-resolution spectral segment to obtain a hyperspectral image of a high-resolution spectral segment, and the high-resolution spectral segment is obtained.
  • the wavelength range of the hyperspectral image of the resolution spectrum segment is mainly concentrated in the visible light region and the infrared light region. Therefore, through the reconstructed hyperspectral image of the high resolution spectrum segment, enough effective spectral information can be obtained, and then the reconstruction is performed.
  • the component information of at least one target substance in the hyperspectral image of the subsequent high-resolution spectral segment is identified, the at least one target substance component in the image can be identified more accurately.
  • performing spectral reconstruction on the first image to obtain a second image includes: acquiring a first spectral reconstruction model, and performing spectral reconstruction on the first image based on the first spectral reconstruction model. segment reconstruction to obtain a second image; wherein, the first spectrum segment reconstruction model is obtained by adding the penalty term of multi-channel supervision information of the color temperature sensor to the second spectrum segment reconstruction model obtained by pre-training; the second spectrum segment The reconstruction model is obtained by training a plurality of second images; the plurality of third images are images collected by the first collection device and the second collection device; the first collection device is an RGB camera, and the second collection device is a TOF camera and/or a color temperature sensor; the penalty item of the multi-channel supervision information of the color temperature sensor is used to adjust the channel of the second spectral segment reconstruction model.
  • the first image of the low-resolution spectral segment is reconstructed by spectral segment, and the second image of the high-resolution spectral segment is obtained.
  • the first spectral segment reconstruction model is obtained based on the pre-trained penalty term for adding multi-channel supervision information of the color temperature sensor in the second spectral segment reconstruction model.
  • the pre-trained second spectral segment reconstruction model is obtained by training based on the images collected by the first acquisition device and the second acquisition device
  • the first acquisition device is an RGB camera
  • the second acquisition device is a TOF camera and/or color temperature sensor
  • the second spectral segment reconstruction model obtained by training based on the images collected by the first acquisition device and the second acquisition device can be used to reconstruct a hyperspectral image of a high-resolution spectral segment
  • the penalty item of the multi-channel supervision information of the color temperature sensor is added to the second spectral segment reconstruction model to obtain the first spectral segment reconstruction model, so that the first spectral segment reconstruction model is more accurate and the reconstruction accuracy is higher high and has good adaptability.
  • recognizing the second image to obtain component information of the at least one target substance includes: acquiring spectral information of the second image, where the spectral information is a spectrum of the second image the spectral information corresponding to the segment; use the convolutional neural network model to identify the spectral information of the second image to determine the chemical bond of the at least one target substance; determine the at least one target according to the chemical bond of the at least one target substance Information on the composition of the substance.
  • the spectral information of the reconstructed second image is obtained, and the spectral information is identified by using the convolutional neural network model, and the chemical bond of at least one target substance in the second image can be determined.
  • the analysis obtains the components possessed by at least one target substance, and the related information of the components.
  • the method before using the convolutional neural network model to identify the spectral information of the second image, the method further includes: acquiring the second image according to the spectral information of the second image The first spectrum corresponding to the spectral range obtained from the fourth image, and the second spectrum corresponding to the spectral range obtained from the fourth image; wherein, the fourth image is an image obtained by the color temperature sensor collecting the at least one target substance; A spectrum and the second spectrum, when it is determined that the error value between the first spectrum and the second spectrum is greater than a set threshold, the spectrum information in the second image is adjusted.
  • the spectrum of the color temperature sensor is collected, and the spectrum of the second image is compared with the spectrum of the color temperature sensor.
  • the error between the two spectra is large, it means that the reconstruction of the second image is not accurate and requires
  • the spectrum of the reconstructed second image is adjusted so that the spectrum of the second image is consistent with the spectrum of the color temperature sensor.
  • adjusting the spectral information in the second image includes: acquiring N spectral values corresponding to N channels in the second image according to the first spectrum; Two spectra, obtain N spectral values corresponding to the N channels in the fourth image; the value of N is greater than or equal to 1; wherein, the N channels in the second image and the N channels in the fourth image are The channels are in a one-to-one correspondence; according to the N spectral values corresponding to the N channels in the second image and the N spectral values corresponding to the N channels in the fourth image, the N spectral values in the second image are channel to adjust.
  • the spectral value corresponding to each channel in the second image and the spectral value corresponding to each channel in the fourth image collected by the color temperature sensor are specifically obtained , and then adjust the N channels in the second image according to the spectral values corresponding to the N channels in the two types of images, so as to ensure the spectral accuracy of each channel in the adjusted second image.
  • adjusting the N channels in the second image includes: for the i-th channel in the N channels, determining a spectral value corresponding to the i-th channel in the second image is L i , the spectral value corresponding to the i-th channel in the fourth image collected by the color temperature sensor is l i , and i takes any integer value from 1 to N; when the difference between L i and l i is determined When the absolute value of is greater than the set threshold, the ith channel in the second image is adjusted.
  • the spectral value corresponding to the channel and the spectral value corresponding to the channel in the fourth image collected by the color temperature sensor are determined, and then when the two values If the error range is greater than the set threshold, it means that the spectrum of the channel in the second image is inaccurate, and then the channel in the second image is adjusted to ensure the accuracy of the spectrum of the channel.
  • the present application provides an apparatus for image recognition, the apparatus having the function of implementing the method described in the above first aspect or any possible design of the above first aspect.
  • This function can be implemented by hardware or by executing corresponding software by hardware.
  • the hardware or software includes one or more modules corresponding to the above functions. For example, it includes a display unit, a communication unit, and a processing unit.
  • an embodiment of the present application further provides a computer storage medium, where a software program is stored in the storage medium, and when the software program is read and executed by one or more processors, the above-mentioned first aspect or the above-mentioned first aspect can be implemented. Any method provided by the design.
  • the embodiments of the present application further provide a computer program product containing instructions, when the instructions are run on a computer, the computer can execute the method provided by the first aspect or any one of the designs.
  • an embodiment of the present application provides a chip system, where the chip system includes a processor for supporting a device to implement the functions involved in the first aspect above.
  • the chip system further includes a memory for storing necessary program instructions and data.
  • the chip system may be composed of chips, or may include chips and other discrete devices.
  • the present application provides a chip system, the chip system includes a processor and an interface, the interface is used to obtain a program or an instruction, and the processor is used to call the program or instruction to implement or support a device to implement the first
  • the functions involved in one aspect for example, determine or process at least one of the data and information involved in the methods described above.
  • the chip system further includes a memory for storing necessary program instructions and data of the electronic device.
  • the chip system may be composed of chips, or may include chips and other discrete devices.
  • an embodiment of the present application further provides an interface, wherein the device has a display screen, a memory, and a processor, and the processor is configured to execute a computer program stored in the memory, and the interface includes An interface displayed when the apparatus executes the method described in the first aspect or the third aspect.
  • FIG. 1 is a schematic diagram of a hardware structure of an electronic device according to an embodiment of the present invention.
  • FIG. 2 is an application scenario diagram to which an image recognition method provided by an embodiment of the present invention is applicable
  • FIG. 3 is a schematic diagram of a process flow of an image recognition method provided by an embodiment of the present invention.
  • FIG. 4 is a schematic diagram of an embodiment of an image recognition method provided by an embodiment of the present invention.
  • FIG. 5 is a schematic diagram showing a distribution map of a target substance according to an embodiment of the present invention.
  • the target substance involved in the examples of the present application refers to a kind of concrete objects containing different components and contents, such as apples, bananas, etc.
  • the images involved in the embodiments of the present application include ordinary images and hyperspectral images, where the ordinary images are images captured by ordinary cameras or cameras of electronic devices, and the hyperspectral images are in the wavelength domain.
  • Subdivided multi-channel images for example, through hyperspectral sensors mounted on different space platforms, namely imaging spectrometers, in the ultraviolet, visible, near-infrared and mid-infrared regions of the electromagnetic spectrum, multiple continuous and subdivided spectral bands to the target Regions are imaged simultaneously. Therefore, in hyperspectral images, not only more material surface or intrinsic information, but also spectral information can be obtained.
  • the reconstructed hyperspectral image of the high spectral segment involved in the embodiment of the present application is mainly aimed at the hyperspectral image of the low-resolution spectral segment, because the spectral information in the hyperspectral image of the low-resolution spectral segment is usually less and blurred, It cannot be used to more accurately identify the material components in the image, so that the hyperspectral image of the low-resolution spectrum is reconstructed into a hyperspectral image of the high-resolution spectrum.
  • the hyperspectral image of the high-resolution spectrum contains relatively more information of the common spectrum. And accurate, can more accurately identify the material components in the image.
  • the RGB (Red, Green, Blue) camera involved in the embodiment of the present application mainly provides three basic color (red, green, blue) components by three different cables.
  • this camera uses three independent charge coupled device image sensors (Charge Coupled Device, CCD) to obtain three color signals, and color images can be obtained through RGB cameras, but the resolution and clarity of images obtained through RGB cameras are limited.
  • CCD Charge Coupled Device
  • the time-of-flight (TOF) camera involved in the embodiment of the present application is a true depth-sensing lens, used for 3D recognition, and the TOF measurement principle (TOF image sensor) is used to determine the camera and the substance or the camera. The distance between the surrounding environment and generate a depth image or 3D image from the measured points.
  • the TOF measurement principle is to emit high-frequency light by modulating the light transmitter, and then reflect it back when it hits the object. The time difference can form a high-precision 3D three-dimensional image, and the identification can be completed by comparison.
  • TOF cameras are commonly used in applications including laser-based non-scanning lidar imaging systems, motion sensing and tracking, object detection for machine vision and autonomous driving, and terrain mapping.
  • the spectral information involved in the embodiments of the present application is simply the component information contained in the light.
  • Light can be classified according to the amount of energy (usually expressed by wavelength) corresponding to photons.
  • the wavelengths are from short to long, followed by three regions of ultraviolet light, visible light, and infrared light. Visible light of different colors corresponds to light of different wavelengths.
  • the spectrum is the proportional information of each wavelength component in the light.
  • the abscissa represents the wavelength
  • the ordinate represents the relative intensity.
  • the chemical composition and relative content of substances can be identified and determined by spectral information.
  • the embodiment of the present application provides an identification method, which is applicable to various electronic devices, such as mobile phones, cameras, tablet computers, and other devices.
  • Figure 1 shows a block diagram of a possible electronic device.
  • the electronic device 100 includes: a radio frequency (RF) circuit 101 , a power supply 102 , a processor 103 , a memory 104 , an input unit 105 , a display screen 106 , a camera 107 , a sensor 108 , and a communication interface 109 , and components such as a wireless fidelity (wireless fidelity, WiFi) module 110 .
  • RF radio frequency
  • the structure of the electronic device shown in FIG. 1 does not constitute a limitation on the electronic device, and the electronic device provided in this embodiment of the present application may include more or less components than those shown in the figure, or a combination of certain components may be included. some components, or a different arrangement of components.
  • the RF circuit 101 can be used for data reception and transmission during communication or conversation. Particularly, after receiving the downlink data of the base station, the RF circuit 101 sends the data to the processor 103 for processing; in addition, it sends the uplink data to be sent to the base station.
  • the RF circuit 101 includes, but is not limited to, an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier (LNA), a duplexer, and the like.
  • LNA low noise amplifier
  • the RF circuit 101 can also communicate with the network and other devices through wireless communication.
  • the wireless communication can use any communication standard or protocol, including but not limited to global system of mobile communication (GSM), general packet radio service (GPRS), code division multiple access (code division multiple access) division multiple access, CDMA), wideband code division multiple access (WCDMA), long term evolution (long term evolution, LTE), email, short message service (short messaging service, SMS), etc.
  • GSM global system of mobile communication
  • GPRS general packet radio service
  • code division multiple access code division multiple access
  • CDMA code division multiple access
  • WCDMA wideband code division multiple access
  • long term evolution long term evolution
  • email short message service
  • SMS short message service
  • the WiFi technology belongs to the short-distance wireless transmission technology, and the electronic device 100 can be connected to an access point (access point, AP) through the WiFi module 110, thereby realizing the access of the data network.
  • the WiFi module 110 can be used for data reception and transmission during communication.
  • the electronic device 100 can be physically connected to other devices through the communication interface 108 .
  • the communication interface 109 is connected with the communication interface of the other device through a cable to realize data transmission between the electronic device 100 and the other device.
  • the electronic device 100 can be used to implement communication services and send information to other electronic devices, so the electronic device 100 needs to have a data transmission function, that is, the electronic device 100 needs to include a communication module inside .
  • FIG. 1 shows communication modules such as the RF circuit 101 , the WiFi module 110 , and the communication interface 108 , it should be understood that the electronic device 100 has at least one of the above-mentioned components or other functions.
  • a communication module (such as a Bluetooth module) for realizing communication for data transmission.
  • the electronic device 100 when the electronic device 100 is a mobile phone, the electronic device 100 may include the RF circuit 101, and may also include the WiFi module 110; when the electronic device 100 is a computer, the electronic device 100 may The communication interface 109 is included, and the WiFi module 110 may also be included. When the electronic device 100 is a tablet computer, the electronic device 100 may include the WiFi module.
  • the memory 104 may be used to store software programs and modules.
  • the processor 103 executes various functional applications and data processing of the electronic device 100 by running the software programs and modules stored in the memory 104.
  • the memory 104 may mainly include a program storage area and a data storage area.
  • the storage program area may store operating systems, various application programs, etc.; the storage data area may store multimedia files such as pictures, videos, and the like.
  • the memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other volatile solid-state storage device.
  • the input unit 105 can be used to receive images, numbers or characters input by the user, or images captured by the camera 107 , and generate key signal input related to user settings and function control of the electronic device 100 .
  • the input unit 105 may include a touch panel 1051 and other input devices 1052 .
  • the touch panel 1051 can collect the user's touch operations on or near it (for example, the user uses any suitable objects or accessories such as fingers, stylus, etc. on the touch panel 1051 or on the touch panel 1051). operations near the touch panel 1051 ), and drive the corresponding connection device according to a preset program.
  • the touch panel 1051 may include two parts, a touch detection device and a touch controller. Among them, the touch detection device detects the user's touch orientation, detects the signal brought by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device, converts it into contact coordinates, and then sends it to the touch controller. to the processor 103, and can receive and execute commands sent by the processor 103.
  • the touch panel 1051 can be implemented in various types such as resistive, capacitive, infrared, and surface acoustic waves.
  • the other input devices 1052 may include, but are not limited to, one or more of physical keyboards, function keys (such as volume control keys, switch keys, etc.), trackballs, mice, joysticks, and the like.
  • function keys such as volume control keys, switch keys, etc.
  • trackballs mice, joysticks, and the like.
  • the display screen 106 may be used to display information input by or provided to the user and various menus of the electronic device 100 .
  • the display screen 106 is the display system of the electronic device 100, and is used for presenting an interface and realizing human-computer interaction.
  • the display screen 106 may include a display panel 1061 .
  • the display panel 1061 may be configured in the form of a liquid crystal display (liquid crystal display, LCD), an organic light-emitting diode (organic light-emitting diode, OLED) or the like.
  • the touch panel 1051 can cover the display panel 1061, and when the touch panel 1051 detects a touch operation on or near it, it transmits to the processor 103 to determine the type of the touch event, Then the processor 103 provides corresponding visual output on the display panel 1061 according to the type of the touch event.
  • the touch panel 1051 and the display panel 1061 are used as two independent components to realize the input and output functions of the electronic device 100, in some embodiments, the The touch panel 1051 is integrated with the display panel 1061 to realize the input and output functions of the electronic device 100 .
  • the processor 103 is the control center of the electronic device 100, uses various interfaces and lines to connect various components, runs or executes the software programs and/or modules stored in the memory 104, and invokes the software programs and/or modules stored in the
  • the data in the memory 104 executes various functions of the electronic device 100 and processes data, thereby realizing various services based on the electronic device.
  • the processor 103 may include one or more processing units.
  • the processor 103 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the like, and the modem processor mainly processes wireless communication. It can be understood that, the above-mentioned modulation and demodulation processor may not be integrated into the processor 103 .
  • the camera 107 is used for realizing the shooting function of the electronic device 100, and shooting pictures or videos.
  • the camera 107 can also be used to implement the scanning function of the electronic device 100 to scan the scanned object (two-dimensional code/barcode).
  • the camera 107 may be an RGB camera or a TOF camera.
  • the RGB camera is used to collect the target material, and a color image composed of three colors of RGB can be obtained.
  • the TOF camera can be used to collect the target material to generate a depth image or a 3D image.
  • the sensors 108 may include one or more sensors.
  • a touch sensor 1081 a color temperature sensor 1082, and the like.
  • the sensor 108 may also include a gyroscope, an acceleration sensor, a fingerprint sensor, an ambient light sensor, a distance sensor, a proximity light sensor, a bone conduction sensor, a pressure sensor, a positioning sensor (eg, a global positioning system). , GPS) sensor) etc., which are not limited.
  • the touch sensor 1081 may also be referred to as a "touch panel", which can be used to collect user touch operations on or near it (for example, the user uses a finger, a stylus, etc., any suitable object or accessory on the touch sensor 1081 or at any location). operation near the touch sensor 1081), and drive the corresponding connection device according to the preset program.
  • the touch sensor 1081 may include two parts, a touch detection device and a touch controller. Among them, the touch detection device detects the user's touch orientation, detects the signal brought by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device, converts it into contact coordinates, and then sends it to the touch controller. to the processor 103, and can receive and execute commands sent by the processor 103.
  • the touch sensor 1081 can be implemented in various types such as resistive, capacitive, infrared, and surface acoustic waves.
  • the touch sensor 1081 When the touch sensor 1081 is disposed on the display screen 106, the touch sensor 1081 and the display screen 106 form a touch screen, which may also be referred to as a "touch screen".
  • the touch sensor 1081 is used to detect a touch operation on or near it.
  • the touch sensor 1081 can pass the detected touch operation to the application processor to determine the type of touch event, so that the electronic device 100 can provide visual output related to the touch operation and the like through the display screen 106 .
  • the electronic device 100 may perform interface switching in response to the touch sensor 1081 detecting a touch operation on or near it, and display the switched interface on the display screen 106 .
  • the touch sensor 1081 may also be disposed on the surface of the electronic device 100 , which is different from the position where the display screen 106 is located.
  • the color temperature sensor 1082 is used to detect the color temperature of the environment when the camera 107 captures a photo, obtain a color temperature signal, and convert the color temperature signal into an electrical signal.
  • the color temperature sensor 1082 may be disposed on the display screen 106 .
  • the processor 103 receives an electrical signal of the color temperature, and then performs color temperature adjustment to ensure that the color temperature is consistent with the ambient light level, so that the color of the photos collected by the camera 107 is more accurate.
  • the electronic device 100 also includes a power source 102 (eg, a battery) for powering the various components.
  • a power source 102 eg, a battery
  • the power supply 102 may be logically connected to the processor 103 through a power management system, so as to implement functions such as managing charging, discharging, and power consumption through the power management system.
  • the electronic device 100 may also include an audio circuit and the like, which will not be repeated here.
  • FIG. 2 shows a schematic diagram of an application scenario of the embodiment of the present application.
  • the mobile phone is a smart phone that can collect hyperspectral images and spectra.
  • the user turns on the camera of the smart phone, and the current fruit, such as apples, is recorded by the mobile phone.
  • the collected hyperspectral image of the apple is automatically displayed on the left side of the display screen of the smart phone, and the hyperspectral image includes the image of the apple and the corresponding components and contents of the apple.
  • the ingredients include: water, sugar, protein (g), fiber (g), etc., and the corresponding contents are 85%, 12.3%, 0.2g, 1.7g, and the like.
  • the vibration information of specific chemical groups in a substance can be analyzed, such as O-H, C-H, N-H bonds, etc., so as to determine the composition of the substance.
  • RGB-Hypercube pairs of hyperspectral images of the same object need to be collected for training the reconstruction network. Since the reconstruction network is usually based on convolution, the residual connection or density is introduced by introducing the The higher Dense module is used to closely fuse the features of each layer, so as to complete the mapping from the limited 3 channels to the N channels, and complete the reconstruction.
  • the RGB single image is mainly reconstructed, since the wavelength range of the image is concentrated in the visible light region, while the effective information of more substances is concentrated in the near-infrared region.
  • Reconstruction from a single RGB image cannot obtain sufficient material information, and is often limited by the initial acquisition data set, resulting in the actual reconstruction process is likely to be biased, giving wrong material information (for example, when the object is different substances of the same color. Time).
  • the method includes: the first step: selecting the same kind of fruit to form a sample set, and randomly dividing it into a calibration set and a prediction set; the second step: collecting the original near-infrared spectra of all samples, dividing the spectrum into equal intervals, and dividing the absorbance of each interval into separate Summation; the third step: use chemical analysis method to determine the sugar content in the sample; the fourth step: use the BP neural network to construct a quantitative correction model between the sugar content of the calibration set and the near-infrared characteristic spectrum; the fifth step: combine the prediction set The near-infrared spectral information data of the sample is input into the model, and the sugar content of the sample in the prediction set is obtained.
  • the detection of material components is carried out. Since the current spectrum is concentrated in the near-infrared region, only the one-dimensional spectrum collected at a specific point can be analyzed to obtain a single-point result, but not a two-dimensional spectrum. dimensional distribution of matter.
  • the feature extraction ability is relatively weak, and it only recognizes the components of a single substance, and cannot classify and detect general chemical bonds.
  • an embodiment of the present application provides an image recognition method.
  • a first image containing at least one target substance is obtained first, and the first image is an image of a first spectral segment; the first spectral segment is low-resolution spectrum; then, performing spectrum reconstruction on the first image to obtain a second image, where the second image is an image of the second spectrum; the second spectrum is a wavelength range in the visible light region and high-resolution spectrum in the infrared light region; finally, identifying the second image to obtain the composition information of the at least one target substance and displaying the composition information of the at least one target substance.
  • spectrum reconstruction is performed on an image containing at least one low-resolution segment of the target substance, to obtain an image of a high-resolution spectrum with a wavelength range in the visible light region and the infrared light region. Therefore, more and more effective spectral information can be obtained through the image of the high-resolution spectral range, and then the composition information of at least one target substance in the image of the high-resolution spectral range can be accurately identified.
  • FIG. 3 is a flowchart of an image recognition method provided by an embodiment of the present application. Executing the method in this embodiment of the present application may be applicable to the electronic device 100 shown in FIG. 1 and to the scenario shown in FIG. 2 . As shown in Figure 3, the process of the method includes:
  • the electronic device acquires a first image including at least one target substance, where the first image is an image of a first spectral segment; the first spectral segment is a low-resolution spectral segment.
  • the first image is a hyperspectral image of a low-resolution spectral band, which can be acquired by a specific acquisition device, where the specific acquisition device includes: RGB camera+TOF camera+color temperature sensor ( at least two, which must include an RGB camera).
  • the target substance may be a substance containing different component contents, which is not specifically limited in this application.
  • the target substance may be a certain fruit, such as apples, bananas, and the like.
  • the electronic device can turn on its own camera when receiving a shooting instruction from the user, and capture the first image under the control of the user, for example, the user takes a mobile phone to shoot a flower or a fruit, etc.
  • a first image of a target substance for example, the electronic device can also receive the first image sent by other devices, for example, the first image sent by other electronic devices can be received through a WiFi module or an RF circuit, so that the first image containing at least one target substance can also be obtained. an image.
  • the electronic device reconstructs the spectral band of the first image to obtain a second image, where the second image is an image of the second spectral band; the second spectral band is the wavelength range in the visible light region and the infrared light region high-resolution spectrum.
  • a first spectral segment reconstruction model may be obtained first, and spectral segment reconstruction is performed on the first image based on the first spectral segment reconstruction model to obtain a second image; wherein , the first spectral segment reconstruction model is obtained by adding the penalty term of the multi-channel supervision information of the color temperature sensor to the second spectral segment reconstruction model obtained by pre-training; the second spectral segment reconstruction model is trained by a plurality of third images Obtained; the plurality of third images are images collected by the first collection device and the second collection device; the first collection device is an RGB camera, and the second collection device is a TOF camera and/or a color temperature sensor; The penalty item of the multi-channel supervision information of the color temperature sensor is used to adjust the channel of the second spectral segment reconstruction model.
  • a hyperspectral image to be reconstructed containing at least one target substance (the spectral resolution of the hyperspectral image is low) is input into the electronic device, and the first high spectral image obtained by the electronic device is
  • the spectral reconstruction model reconstructs the hyperspectral image to be reconstructed containing at least one target substance, and obtains a reconstructed hyperspectral image of the target substance (the reconstructed hyperspectral image has a higher spectral resolution, and the reconstructed hyperspectral image is The wavelength range of the resulting hyperspectral image is concentrated in the visible and infrared regions).
  • a hyperspectral image reconstruction of 400-1000 nm is performed on an unknown fruit object using the first hyperspectral reconstruction model (eg, a neural network for hyperspectral reconstruction).
  • the first hyperspectral reconstruction model eg, a neural network for hyperspectral reconstruction.
  • the first spectral segment reconstruction model is the first hyperspectral reconstruction model
  • the second spectral segment reconstruction model is the trained second hyperspectral reconstruction model. It is used to reconstruct the hyperspectral image of the low-resolution spectrum, so as to obtain the hyperspectral image of the high-resolution spectrum.
  • the second hyperspectral reconstruction model may be obtained by training a plurality of third images in advance, and may specifically include but not be limited to the following operations:
  • the first acquisition device may be an RGB camera
  • the second acquisition device may be a TOF camera and a color temperature sensor
  • M is an integer greater than 1
  • the M hyperspectral images (that is, the third image) of the target substance are trained to obtain a trained second hyperspectral reconstruction model
  • the A color temperature penalty item is added to the trained second hyperspectral reconstruction model to obtain the first hyperspectral reconstruction model, wherein the penalty item of the multi-channel supervision information of the color temperature sensor is used for the correction of the first hyperspectral reconstruction model. channel to adjust.
  • multiple hyperspectral image pairs of the target substance are collected by a first collection device and a second collection device
  • the first collection device is an RGB camera
  • the second collection device is a TOF camera and/or a color temperature sensor
  • the image collected by the RGB camera or the TOF camera or the color temperature sensor is the original 4-channel image, which can cover the image in the visible light to the near-infrared light region.
  • a second hyperspectral image reconstruction model (for example, a hyperspectral reconstruction neural network) is obtained by training using a plurality of acquired hyperspectral image pairs of the target substance.
  • the multi-modal information of RGB camera + TOF camera + color temperature sensor (at least two, which must include RGB camera) can be used to reconstruct hyperspectral images.
  • the hyperspectral image is a multi-channel image subdivided in the wavelength domain
  • the penalty term of the multi-channel supervision information of the color temperature sensor is added, and the first high spectral image can be constructed and obtained.
  • a first collection device RGB camera
  • a second collection device TOF camera and/or color temperature sensor
  • a high-precision device are used to collect a plurality of (here, the plurality can be many, such as 1000, etc., The value can be set larger) the hyperspectral image pair of the substance, and the hyperspectral image reconstruction model is obtained by training; the penalty term for the multi-channel supervision information of the color temperature sensor is added to the hyperspectral image reconstruction model obtained by training (such as the neural network of hyperspectral reconstruction), while It is ensured that the first hyperspectral reconstruction model obtained by final construction has higher accuracy and good adaptability.
  • the first hyperspectral reconstruction model to reconstruct the acquired hyperspectral image of the low-resolution spectral segment of the target substance to be identified, to obtain the hyperspectral image of the high-resolution spectral segment of the target substance, wherein the high-resolution spectral segment includes A high-resolution spectrum of wavelengths in the visible and infrared regions. Therefore, more and more effective spectral information can be obtained through the hyperspectral image of the high-resolution spectral segment, and then more accurate composition information of the target substance can be determined.
  • S303 The electronic device recognizes the second image to obtain component information of the at least one target substance, and displays the component information of the at least one target substance.
  • the spectral information in the second image can also be adjusted through a color temperature sensor, and specific implementations include but are not limited to the following:
  • the electronic device may also first obtain the first spectrum corresponding to the spectral range of the second image according to the spectral information of the second image, and obtain the second spectrum corresponding to the spectral range of the fourth image; the fourth image is:
  • the color temperature sensor in the electronic device collects the image obtained by the at least one target substance; the electronic device may further determine the first spectrum and the first spectrum according to the first spectrum and the second spectrum. When the error value between the two spectra is greater than the set threshold, the spectral information in the second image is adjusted.
  • the electronic device adjusts the spectral information in the second image, and specific embodiments include but are not limited to the following:
  • N spectral values corresponding to N channels in the second image according to the first spectrum and obtain N spectral values corresponding to N channels in the fourth image according to the second spectrum ; the value of N is greater than or equal to 1; wherein, there is a one-to-one correspondence between the N channels in the second image and the N channels in the fourth image; further, according to the corresponding N channels in the second image
  • the N spectral values of , and the N spectral values corresponding to the N channels in the fourth image are adjusted for the N channels in the second image.
  • the spectrum corresponding to each channel in the fourth image is a one-dimensional spectrum
  • the spectrum corresponding to each channel in the second image is a multi-dimensional spectrum. Therefore, it is also necessary to process the spectral dimension corresponding to each channel in the second image to obtain a one-dimensional spectrum. For example, for each channel in the second image, global average pooling is performed on each channel in the second image to obtain a one-dimensional spectrum.
  • obtaining N spectral values corresponding to N channels in the second image according to the first spectrum in the second image specifically including but not limited to the following:
  • the spectral dimension of the channel is processed to obtain a one-dimensional spectrum, and the spectral value corresponding to the one-dimensional spectrum obtained by processing in each channel is used as the spectral value of the corresponding channel.
  • the electronic device adjusts the N channels in the second image
  • specific implementation manners include but are not limited to the following:
  • the ith channel in the N channels to adjust the ith channel, first determine the spectral value L i corresponding to the ith channel in the second image, and determine the color temperature
  • the spectral value corresponding to the ith channel in the fourth image collected by the sensor is l i , and i takes any integer value from 1 to N; when it is determined that the absolute value of the difference between the Li and the li is greater than the set value.
  • the threshold is determined, the i-th channel in the second image is adjusted.
  • the absolute value of the difference between Li and Li is less than the set threshold, it means that the error between the i -th channel in the reconstructed second image and the i -th channel in the image captured by the color temperature sensor is relatively small , which means that the reconstruction of the i-th channel in the second image is accurate, and the electronic device does not need to adjust the channel.
  • adjusting the N channels in the second image through the above steps can realize the adjustment of the spectrum of the second image, thereby ensuring the accuracy of the spectral information in the second image.
  • Step S303 is performed after the second image is adjusted based on the color temperature sensor.
  • the electronic device may first acquire spectral information of the adjusted second image, where the spectral information is spectral information corresponding to the spectral segments of the adjusted second image; then, use a convolutional neural network
  • the model identifies the spectral information of the second image, and determines the chemical bond of the at least one target substance; finally, determines the composition information of the at least one target substance according to the chemical bond of the at least one target substance.
  • the electronic device may specifically use a convolutional neural network to classify and regress the spectral dimension in the adjusted second image according to the spectral information of the second image, determine the chemical bond of the at least one target substance, and determine the chemical bond of the at least one target substance.
  • component information of the at least one target substance analyzing the component information of the at least one target substance to obtain a distribution map of the at least one target substance.
  • step S303 when one of the target substances is fruit, its internal chemical bond information can be determined through step S303, so as to obtain the distribution of soluble solid substances in the fruit, such as fructose, blood oxygen, etc.
  • the present application also provides an embodiment of an image recognition method.
  • the method may be performed by an electronic device capable of supporting photographing and displaying.
  • the specific process is as follows.
  • the electronic device acquires a first hyperspectral image to be reconstructed containing at least one target substance.
  • the electronic device acquires the hyperspectral image to be reconstructed of at least one fruit through its camera, or the electronic device acquires the hyperspectral image to be reconstructed of at least one fruit from other electronic devices.
  • the electronic device acquires the adjusted hyperspectral image reconstruction network, and reconstructs the first hyperspectral image to obtain a second hyperspectral image.
  • the first hyperspectral image is reconstructed using the adjusted hyperspectral image reconstruction network to obtain a reconstructed second hyperspectral image of the unknown fruit.
  • the adjusted hyperspectral image reconstruction network is used to perform spectral segment reconstruction on the hyperspectral image to be reconstructed containing the target substance, A second hyperspectral image containing a high-resolution spectral band of the target substance is obtained, and the image wavelength range is in the visible light range and the infrared light range. Therefore, more and more effective spectral information can be obtained through the hyperspectral image of this high-resolution spectral band.
  • the adjusted hyperspectral image reconstruction network can be obtained through the following steps:
  • the RGB camera mainly shoots, and the TOF camera is the auxiliary.
  • the RGB camera (wavelength range is 400-780nm) is larger than 24 pixels (MegaPixel), it adopts 240P, and the OF camera (X: 960nm) adopts 240P.
  • the obtained photo is a composite RGBX four-channel image, and the number of photos here can be a large number, which is not specifically limited here.
  • a plurality of RGBX four-channel images obtained in step B1 are trained to obtain a trained hyperspectral image reconstruction network (equivalent to a hyperspectral image reconstruction model).
  • the hyperspectral image is a multi-channel image subdivided in the wavelength domain, on the basis of the hyperspectral image reconstruction model obtained after training, the penalty item of the multi-channel supervision information of the color temperature sensor is added to monitor and adjust the trained hyperspectral image reconstruction in real time. model, so that it has higher accuracy and better adaptability after adjustment.
  • Steps B1-B3 are the training and adjustment framework of the hyperspectral image reconstruction model, so as to subsequently reconstruct the first hyperspectral image to be reconstructed containing at least one target substance.
  • the electronic device acquires spectral information of the N channel of the reconstructed second hyperspectral image.
  • a404 Perform average pooling on each channel to obtain a one-dimensional spectrum and obtain the corresponding spectral value.
  • the spectrum corresponding to each channel in the second hyperspectral image is a multi-dimensional spectrum. Therefore, it is also necessary to process the spectral dimension corresponding to each channel in the second hyperspectral image to obtain a one-dimensional spectrum. For example, for each channel in the second hyperspectral image, global average pooling is performed on each channel in the second hyperspectral image to obtain a one-dimensional spectrum.
  • acquiring N spectral values corresponding to N channels in the second hyperspectral image according to the first spectrum in the second hyperspectral image specifically includes the following: After processing the spectral dimension of each channel in each channel to obtain a one-dimensional spectrum, the spectral value corresponding to the one-dimensional spectrum obtained by processing in each channel is used as the spectral value of the corresponding channel.
  • the electronic device collects a hyperspectral image of the at least one target substance through a color temperature sensor, and acquires the spectral value of the N channel in the hyperspectral image.
  • the spectrum corresponding to each channel is a one-dimensional spectrum in the hyperspectral image of the at least one target substance collected by the color temperature sensor, the spectral value corresponding to the spectrum of each channel can be directly obtained.
  • the electronic device calculates an error between the spectral value of each channel in the second hyperspectral image and the spectral value of each channel in the hyperspectral image acquired by the color temperature sensor.
  • the spectral value corresponding to the i-th channel in the second hyperspectral image is L i
  • the color temperature sensor collects the spectral value L i .
  • the spectral value corresponding to the i-th channel in the hyperspectral image is l i
  • i takes any integer value from 1 to N to calculate the absolute value of the difference between Li and l i .
  • a407(1) The error is greater than the set threshold, and the spectral channel is adjusted online in real time through the color temperature sensor to obtain the adjusted hyperspectral stereo image and high-precision N-channel hyperspectral stereo.
  • the ith channel in the second hyperspectral image is adjusted.
  • the ith channel in the second hyperspectral image is not adjusted.
  • the electronic device Step a408 is performed on the second hyperspectral image.
  • the electronic device analyzes the chemical bond information contained in the one-dimensional spectrum of each pixel by using the convolutional neural network, determines the composition of the substance, and obtains a distribution map of substance division.
  • the adjusted second hyperspectral image is obtained. Further, the electronic device performs step a408 on the adjusted second hyperspectral image.
  • the electronic device utilizes a convolutional neural network to analyze the chemical bond information contained in the one-dimensional spectrum of each pixel in the adjusted second hyperspectral image, determine the composition of the first substance, and obtain a distribution map into which the substance is divided, For example, when the first substance is an apple, the distribution of fructose in the apple is obtained, or when the first substance is blood, the distribution of blood oxygen in the blood is obtained.
  • High-precision N-channel hyperspectral stereo wavelength range: 400-1000nm, wavelength resolution is 10nm.
  • a hyperspectral image reconstruction model is used to reconstruct the image of the low-resolution spectral range of the target substance, and the high-resolution spectral range of the target substance is obtained.
  • the hyperspectral image reconstruction model is a model obtained by adding a color temperature penalty term constraint to a pre-trained hyperspectral image reconstruction model, wherein the pre-trained hyperspectral reconstruction model is based on an acquisition device (except for the RGB camera)
  • the pre-trained hyperspectral reconstruction model is based on an acquisition device (except for the RGB camera)
  • it also includes the training of multiple image data sets collected by RGB cameras and/or TOF cameras), so that the improved hyperspectral image reconstruction model can be guaranteed to have higher accuracy and better adaptability.
  • the reconstructed hyperspectral image of the high-resolution spectral band is further adjusted, so that the spectral information of the hyperspectral image of the high-resolution spectral band is more accurate. Therefore, not only more spectral information can be obtained, but also the accuracy of the spectral information can be ensured through the hyperspectral image of the high-resolution spectral segment finally obtained, so as to ensure that the target substance composition information determined by the spectral information with higher accuracy is more accurate. It is more and more accurate, so that the target material components in the image can be identified more accurately.
  • the specific structure of the electronic device can refer to the schematic diagram of the hardware structure of an electronic device 100 shown in FIG.
  • the at least one processor 103 is interconnected with the transceiver, the memory 104 , the display screen 106 and the camera 107 .
  • the at least one processor 103, the transceiver, the memory 104, the display screen 106, and the camera 107 can be connected to each other through a bus;
  • the bus can be a peripheral component interconnect standard (peripheral component interconnect, PCI) bus or Extended industry standard architecture (extended industry standard architecture, EISA) bus, etc.
  • PCI peripheral component interconnect
  • EISA Extended industry standard architecture
  • the bus can be divided into an address bus, a data bus, a control bus, and the like.
  • the transceiver is used to communicate and interact with other devices. For example, when the electronic device 100 collects an image of the current scene through an external camera, the electronic device 100 uses the transceiver to obtain a picture captured by other external cameras as the first image.
  • the transceiver may be a Bluetooth module, a WiFi module 110, an RF circuit 101 and the like.
  • the at least one processor 103 can use the camera 107 in the electronic device 100 to collect the current image of the at least one target substance as the obtained first image, or can also use a transceiver, such as the RF circuit 101 or the other
  • the transceiver 110 acquires an image containing at least one target substance sent by other electronic devices as the acquired first image.
  • the at least processor 103 is configured to implement the above-mentioned embodiment and example shown in FIG. 3 to provide an image recognition method. For details, refer to the description in the above-mentioned embodiment, which will not be repeated here.
  • the display screen 106 is used to display the image collected by the camera 107, or display the distribution map of at least one target substance component information obtained after image recognition.
  • the electronic device 100 may further include an audio circuit for receiving and sending out voice signals.
  • the memory 104 is used to store program instructions and data (for example, a hyperspectral image reconstruction model, a collected hyperspectral image) and the like.
  • the program instructions may include program code, which includes instructions for computer operation.
  • the memory 104 may include random access memory (RAM), and may also include non-volatile memory (non-volatile memory), such as at least one disk storage.
  • the at least one processor 103 executes the program stored in the memory 104, and implements the above functions through the above components, thereby finally implementing the methods provided in the above embodiments.
  • Computer-readable media includes both computer storage media and communication media including any medium that facilitates transfer of a computer program from one place to another.
  • a storage medium can be any available medium that a computer can access.
  • computer readable media may include RAM, ROM, electrically erasable programmable read only memory (EEPROM), compact disc read-Only memory (CD- ROM) or other optical disk storage, magnetic disk storage media, or other magnetic storage devices, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and that can be accessed by a computer. also. Any connection can be appropriately made into a computer-readable medium.
  • RAM random access memory
  • ROM read only memory
  • EEPROM electrically erasable programmable read only memory
  • CD- ROM compact disc read-Only memory
  • Any connection can be appropriately made into a computer-readable medium.
  • disks and discs include compact discs (CDs), laser discs, optical discs, digital video discs (DVDs), floppy disks, and Blu-ray discs, wherein Disks usually reproduce data magnetically, while discs use lasers to reproduce data optically. Combinations of the above should also be included within the scope of computer-readable media.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Health & Medical Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • Evolutionary Computation (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Artificial Intelligence (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Software Systems (AREA)
  • Computing Systems (AREA)
  • Spectroscopy & Molecular Physics (AREA)
  • Data Mining & Analysis (AREA)
  • Multimedia (AREA)
  • Analytical Chemistry (AREA)
  • Pathology (AREA)
  • Immunology (AREA)
  • Biochemistry (AREA)
  • General Engineering & Computer Science (AREA)
  • Chemical & Material Sciences (AREA)
  • Databases & Information Systems (AREA)
  • Medical Informatics (AREA)
  • Mathematical Physics (AREA)
  • Biophysics (AREA)
  • Molecular Biology (AREA)
  • Biomedical Technology (AREA)
  • Computational Linguistics (AREA)
  • Evolutionary Biology (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Investigating Or Analysing Materials By Optical Means (AREA)

Abstract

本申请提供了一种图像识别的方法及电子设备。该方法包括：获取包含至少一个目标物质的第一图像，第一图像为第一谱段的图像；第一谱段为低分辨的谱段；对第一图像进行谱段重建，得到第二图像，第二图像为第二谱段的图像；第二谱段为波长范围在可见光区域和红外光区域的高分辨的谱段；对第二图像进行识别得到至少一个目标物质的成分信息；显示至少一个目标物质的成分信息。该方法中，通过重建后的高分辨谱段的第二图像，可以获取足够且有效的光谱信息，进而对重建后的高分辨谱段的第二图像中的至少一个目标物质进行识别其成分信息时，可较精确的识别到图像中的至少一个目标物质成分。

Description

一种图像识别的方法及电子设备
相关申请的交叉引用
本申请要求在2020年10月31日提交中国专利局、申请号为202011197236.0、申请名称为“一种图像识别的方法及电子设备”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本申请涉及图像识别技术领域，尤其涉及一种图像识别的方法及电子设备。
背景技术
目前，普通相机采集的图像通常不能用于识别图像中目标物质的成分，从而无法满足人们的要求。因此，在实际工作或生产中，往往需要通过更高分辨率的图像，来获取图像中目标物质的成分信息，进而识别出该目标物质中的成分和含量。例如通过识别图像中水果的成分信息，从而可得到图像中水果的成分及糖度含量。
与普通相机采集的RGB图像相比，高光谱图像为在波长域上细分的多通道图像，因此对高光谱图像进行识别时，可以获取图像中目标物质的更多表面或内在信息。例如，通过获取高光谱图像的光谱信息，可以分析出图像中的目标物质的特定化学基团的振动信息(如O-H、C-H、N-H键等)。在采集高光谱图像时，往往会受到采集设备、采集时间等影响，导致采集的高光谱图像为低分辨谱段的图像，然而，通过低分辨谱段的图像，无法获取高光谱图像的足够且有效的光谱信息，进而无法较精确地识别图像中的目标物质成分。因此，需要通过高光谱重建技术将低分辨谱段的图像重建为高分辨谱段的图像，以获取更多有效的光谱信息。
现有高光谱重建技术中，主要是对RGB图像进行重建得到可见光谱段的高光谱图像，在重建的可见光谱段的高光谱图像的中，波长范围主要集中在可见光的区域，然而，通过重建可见光谱段的高光谱图像并不能获取足够有效的光谱信息。因此，对可见光谱段的高光谱图像中的目标物质进行识别其成分信息时，仍然存在无法较精确识别到图像中的目标物质成分的问题。
发明内容
本申请提供了一种图像识别的方法及电子设备，用于精准的识别图像中的目标物质成分信息。
第一方面，本发明实施例提供一种图像识别的方法，该方法适用于具有图像拍摄和识别功能的电子设备。获取包含至少一个目标物质的第一图像，所述第一图像为第一谱段的图像；所述第一谱段为低分辨的谱段；对所述第一图像进行谱段重建，得到第二图像，所述第二图像为第二谱段的图像；所述第二谱段为波长范围在可见光区域和红外光区域的高分辨的谱段；对所述第二图像进行识别得到所述至少一个目标物质的成分信息；显示所述至少一个目标物质的成分信息。
通过该设计，首先获取包含至少一个目标物质的低分辨谱段的高光谱图像，将该低分辨谱段的高光谱图像进行谱段的重建，得到高分辨谱段的高光谱图像，并且该高分辨谱段的高光谱图像谱段的波长范围主要集中在可见光的区域和红外光区域，因此，通过该重建后的高分辨谱段的高光谱图像，可以获取足够有效的光谱信息，进而对重建后的高分辨谱段的高光谱图像中的至少一个目标物质进行识别其成分信息时，可较精确的识别到图像中的至少一个目标物质成分。
在一个可能的设计中，对所述第一图像进行谱段重建，得到第二图像，包括：获取第一谱段重建模型，基于所述第一谱段重建模型对所述第一图像进行谱段重建，得到第二图像；其中，所述第一谱段重建模型为在预先训练得到的第二谱段重建模型中增加色温传感器多通道监督信息的惩罚项得到的；所述第二谱段重建模型由多个第二图像训练得到；所述多个第三图像为通过第一采集设备和第二采集设备采集到的图像；所述第一采集设备为RGB摄像头，所述第二采集设备为TOF摄像头和/或色温传感器；所述色温传感器多通道监督信息的惩罚项用于对所述第二谱段重建模型的通道进行调整。
通过该设计，基于所述第一谱段重建模型对将低分辨谱段的第一图像进行谱段的重建，得到高分辨谱段的第二图像。其中，所述第一谱段重建模型是基于预先训练得到第二谱段重建模型中增加色温传感器多通道监督信息的惩罚项得到的。其中，预先训练得到第二谱段重建模型是基于第一采集设备和第二采集设备采集到的图像进行训练得到的，所述第一采集设备为RGB摄像头，所述第二采集设备为TOF摄像头和/或色温传感器，从而可知基于所述第一采集设备和所述第二采集设备采集到的图像训练得到的第二谱段重建模型可用于重建出高分辨谱段的高光谱图像，进一步通过在所述第二谱段重建模型中增加色温传感器多通道监督信息的惩罚项，得到所述第一谱段重建模型，使得所述第一谱段重建模型更具有精准性，重建的准确性更高，且具有良好的适应性。
在一个可能的设计中，对所述第二图像进行识别得到所述至少一个目标物质的成分信息，包括：获取所述第二图像的光谱信息，所述光谱信息为所述第二图像的谱段对应的光谱信息；使用卷积神经网络模型对所述第二图像的光谱信息进行识别，确定所述至少一个目标物质的化学键；根据所述至少一个目标物质的化学键，确定所述至少一个目标物质的成分信息。
通过该设计，获取到重建后的第二图像的光谱信息，使用卷积神经网络模型对该光谱信息进行识别，可以确定出所述第二图像中至少一个目标物质的化学键，根据该化学键，可以分析得到至少一个目标物质具备的成分，以及成分的相关信息。
在一个可能的设计中，所述使用卷积神经网络模型对所述第二图像的光谱信息进行识别之前，所述方法还包括：根据所述第二图像的光谱信息，获取所述第二图像的谱段对应的第一光谱，以及获取第四图像的谱段对应的第二光谱；其中，所述第四图像为色温传感器对所述至少一个目标物质进行采集得到的图像；根据所述第一光谱和所述第二光谱，确定所述第一光谱和所述第二光谱之间的误差值大于设定的阈值时，对所述第二图像中的光谱信息进行调整。
通过该设计，通过采集色温传感器的光谱，将所述第二图像的光谱和色温传感器的光谱进行比较，当两者光谱之间的误差较大，则表示所述第二图像重建不准确，需要对重建后的所述第二图像的光谱进行调整，以使得所述第二图像的光谱和色温传感器的光谱保持一致。
在一个可能的设计中，对所述第二图像中的光谱信息进行调整，包括：根据所述第一光谱，获取所述第二图像中N个通道对应的N个光谱值；根据所述第二光谱，获取所述第四图像中的N个通道对应的N个光谱值；所述N的值大于等于1；其中，所述第二图像中N个通道与所述第四图像中N个通道为一一对应的关系；根据所述第二图像中N个通道对应的N个光谱值与所述第四的图像中N个通道对应的N个光谱值，对所述第二图像中N个通道进行调整。
通过该设计，当确定需要对所述第二图像的光谱进行调整时，具体的获取到所述第二图像中各通道对应的光谱值和色温传感器采集的第四图像中各通道对应的光谱值，进而根据两类图像中N通道对应的光谱值，实现对所述第二图像中N个通道的调整，以保证调整后的所述第二图像中各通道的光谱的准确性。
在一个可能的设计中，对所述第二图像中N个通道进行调整，包括：针对所述N个通道中的第i个通道，确定所述第二图像中第i个通道对应的光谱值为L i，所述色温传感器采集的第四图像中第i个通道对应的光谱值为l i，i取遍从1到N的任意一个整数值；当确定所述L i与l i之差的绝对值大于设定的阈值时，对所述第二图像中第i个通道进行调整。
通过该设计，针对所述第二图像中N个通道中的任意一个通道，确定该通道对应的光谱值和所述色温传感器采集的第四图像中该通道对应的光谱值，进而当两个值的误差范围大于设定的阈值，则说明所述第二图像中该通道的光谱不准确，进而对所述第二图像中该通道进行调整，保证该通道的光谱的准确性。
第二方面，本申请提供一种图像识别的装置，该装置具有实现上述第一方面或上述第一方面的任意一种可能的设计中所述方法的功能。该功能可以通过硬件实现，也可以通过硬件执行相应的软件实现。该硬件或软件包括一个或多个与上述功能相对应的模块。比如包括显示单元和通信单元、处理单元。
第三方面，本申请实施例中还提供一种计算机存储介质，该存储介质中存储软件程序，该软件程序在被一个或多个处理器读取并执行时，可实现上述第一方面或其中任意一种设计提供的方法。
第四方面，本申请实施例还提供一种包含指令的计算机程序产品，当指令在计算机上运行时，使得计算机执行上述第一方面或其中任一种设计提供的方法。
第五方面，本申请实施例提供了一种芯片系统，该芯片系统包括处理器，用于支持设备实现上述第一方面中所涉及的功能。
在一种可能的设计中，所述芯片系统还包括存储器，所述存储器，用于保存必要的程序指令和数据。该芯片系统，可以由芯片构成，也可以包含芯片和其他分立器件。
第六方面，本申请提供了一种芯片系统，该芯片系统包括处理器和接口，所述接口用于获取程序或指令，所述处理器用于调用所述程序或指令以实现或者支持设备实现第一方面所涉及的功能，例如，确定或处理上述方法中所涉及的数据和信息中的至少一种。
在一种可能的设计中，所述芯片系统还包括存储器，所述存储器，用于保存电子设备必要的程序指令和数据。该芯片系统，可以由芯片构成，也可以包括芯片和其他分立器件。
第七方面，本申请实施例还提供了一种界面，其中，所述装置具有显示屏、存储器，以及处理器，所述处理器用于执行存储在所述存储器中的计算机程序，所述界面包括所述装置执行第一方面或第三方面所述的方法时显示的界面。
上述第二方面至第七方面中可以达到的技术效果，可以参照上述第一方面或其中任意 一种设计可以达到的技术效果说明，这里不再重复赘述。
附图说明
图1为本发明一实施例提供的一种电子设备的硬件结构示意图；
图2为本发明实施例提供的一种图像识别的方法适用的应用场景图；
图3为本发明实施例提供的一种图像识别的方法流程的示意图；
图4为本发明实施例提供的一种图像识别的方法的实施例的示意图；
图5为本发明实施例提供的一种目标物质分布图显示的示意图。
具体实施方式
下面将结合本申请实施例中的附图，对本申请实施例中的技术方案进行描述。
以下，对本申请实施例中的部分用语进行解释说明，以便于本领域技术人员理解。
1)、本申请实施例中涉及的目标物质，指的是一种包含不同成分和含量的具体实物，例如苹果、香蕉等。
2)、本申请实施例中涉及的图像，包括普通图像和高光谱图像(Hyperspectral Image)，所述普通图像为普通的相机或电子设备的摄像头拍摄的图像，所述高光谱图像为在波长域上细分的多通道图像，例如通过搭载在不同空间平台的高光谱传感器，即成像光谱仪，在电磁波谱的紫外、可见光、近红外和中红外区域，多个连续且细分的光谱波段对目标区域同时成像。因此，高光谱图像中既可以获取更多物质表面或内在的信息，也可以获取光谱信息。
3)、本申请实施例中涉及的重建高谱段的高光谱图像，主要针对低分辨谱段的高光谱图像，由于低分辨谱段的高光谱图像中的谱段信息通常较少且模糊，无法用于较精确识别图像中的物质成分，从而将低分辨谱段的高光谱图像重建成高分辨谱段的高光谱图像，高分辨谱段的高光谱图像中的普段谱段信息相对较多且准确，可以较精确的识别出图像中的物质成分。
4)、本申请实施例中涉及的RGB(Red、Green、Blue)摄像头，主要由三根不同的线缆给出了三个基本彩色(红、绿、蓝)成分。通常这种摄像头用三个独立的电荷耦合器件图像传感器(Charge Coupled Device，CCD)获取三种彩色信号，通过RGB摄像头可以获得彩色图像，但通过RGB摄像头获得的图像的分辨率和清晰度有限。
5)、本申请实施例中涉及的飞行时间(Time of flight，TOF)摄像头，为一种原深感镜头，用于进行3D识别，利用TOF测量原理(TOF图像传感器)来确定摄像头与物质或周围环境之间的距离，并通过测量的点生成深度图像或3D图像。具体的，TOF测量原理为通过调制光发射器发高频光线，碰到物体后反射回，接收器会获取来回的时间，进而计算出与物体的距离，景深不同的地上光线传播时间不同，借时间差可以形成高精度的3D立体图，通过对比完成识别。TOF摄像头通常可以应用于包括基于激光的非扫描激光雷达成像系统、运动传感和追踪，机器视觉和自动驾驶的物体检测，以及地形测绘等。
6)、本申请实施例中涉及的光谱信息，光谱信息简单的说就是光包含的成分信息。根据光子对应的能量多少(一般用波长表示)，可以将光进行分类。波长从短到长，依次为三段区域紫外光，可见光，红外光。不同颜色的可见光为对应不同波长的光。光谱就是光 中各个波长成分的比例信息，在一般表现形式中，横坐标表示波长，纵坐标表示相对强度。通过光谱信息可以鉴别及确定物质的化学组成和相对含量。
7)、本申请实施例中涉及的多个，是指大于或等于两个。
另外，需要理解的是，在本申请的描述中，“第一”、“第二”等词汇，仅用于区分描述的目的，而不能理解为指示或暗示相对重要性，也不能理解为指示或暗示顺序。
本申请实施例提供了一种识别方法，该方法适用于各种电子设备中，比如应用于手机、相机、平板电脑等设备中。图1示出了一种可能的电子设备的结构图。参阅图1所示，所述电子设备100包括：射频(radio frequency，RF)电路101、电源102、处理器103、存储器104、输入单元105、显示屏106、摄像头107、传感器108、通信接口109、以及无线保真(wireless fidelity，WiFi)模块110等部件。本领域技术人员可以理解，图1中示出的电子设备的结构并不构成对电子设备的限定，本申请实施例提供的电子设备可以包括比图示更多或更少的部件，或者组合某些部件，或者不同的部件布置。
下面结合图1对所述电子设备100的各个构成部件进行具体的介绍：
所述RF电路101可用于通信或通话过程中，数据的接收和发送。特别地，所述RF电路101在接收到基站的下行数据后，发送给所述处理器103处理；另外，将待发送的上行数据发送给基站。通常，所述RF电路101包括但不限于天线、至少一个放大器、收发信机、耦合器、低噪声放大器(low noise amplifier，LNA)、双工器等。
此外，RF电路101还可以通过无线通信与网络和其他设备通信。所述无线通信可以使用任一通信标准或协议，包括但不限于全球移动通讯系统(global system of mobile communication，GSM)、通用分组无线服务(general packet radio service，GPRS)、码分多址(code division multiple access，CDMA)、宽带码分多址(wideband code division multiple access，WCDMA)、长期演进(long term evolution，LTE)、电子邮件、短消息服务(short messaging service，SMS)等。
WiFi技术属于短距离无线传输技术，所述电子设备100通过WiFi模块110可以连接接入点(access point，AP)，从而实现数据网络的访问。所述WiFi模块110可用于通信过程中，数据的接收和发送。
所述电子设备100可以通过所述通信接口108与其他设备实现物理连接。可选的，所述通信接口109与所述其他设备的通信接口通过电缆连接，实现所述电子设备100和其他设备之间的数据传输。
由于在本申请实施例中，所述电子设备100能够用于实现通信业务，向其他电子设备发送信息，因此所述电子设备100需要具有数据传输功能，即所述电子设备100内部需要包含通信模块。虽然图1示出了所述RF电路101、所述WiFi模块110、和所述通信接口108等通信模块，但是可以理解的是，所述电子设备100中存在上述部件中的至少一个或者其他用于实现通信的通信模块(如蓝牙模块)，以进行数据传输。
例如，当所述电子设备100为手机时，所述电子设备100可以包含所述RF电路101，还可以包含所述WiFi模块110；当所述电子设备100为计算机时，所述电子设备100可以包含所述通信接口109，还可以包含所述WiFi模块110；当所述电子设备100为平板电脑时，所述电子设备100可以包含所述WiFi模块。
所述存储器104可用于存储软件程序以及模块。所述处理器103通过运行存储在所述存储器104的软件程序以及模块，从而执行所述电子设备100的各种功能应用以及数据处 理。
可选的，所述存储器104可以主要包括存储程序区和存储数据区。其中，存储程序区可存储操作系统、各种应用程序等；存储数据区可存储多媒体文件比如图片、视频等。
此外，所述存储器104可以包括高速随机存取存储器，还可以包括非易失性存储器，例如至少一个磁盘存储器件、闪存器件、或其他易失性固态存储器件。
所述输入单元105可用于接收用户输入的图像、数字或字符信息等，或者由摄像头107采集到的图像，以及产生与所述电子设备100的用户设置以及功能控制有关的键信号输入。
可选的，输入单元105可包括触控面板1051以及其他输入设备1052。
其中，所述触控面板1051，也称为触摸屏，可收集用户在其上或附近的触摸操作(比如用户使用手指、触笔等任何适合的物体或附件在所述触控面板1051上或在所述触控面板1051附近的操作)，并根据预先设定的程式驱动相应的连接装置。可选的，所述触控面板1051可以包括触摸检测装置和触摸控制器两个部分。其中，触摸检测装置检测用户的触摸方位，并检测触摸操作带来的信号，将信号传送给触摸控制器；触摸控制器从触摸检测装置上接收触摸信息，并将它转换成触点坐标，再送给所述处理器103，并能接收所述处理器103发来的命令并加以执行。此外，可以采用电阻式、电容式、红外线以及表面声波等多种类型实现所述触控面板1051。
可选的，所述其他输入设备1052可以包括但不限于物理键盘、功能键(比如音量控制按键、开关按键等)、轨迹球、鼠标、操作杆等中的一种或多种。
所述显示屏106可用于显示由用户输入的信息或提供给用户的信息以及所述电子设备100的各种菜单。所述显示屏106即为所述电子设备100的显示系统，用于呈现界面，实现人机交互。
所述显示屏106可以包括显示面板1061。可选的，所述显示面板1061可以采用液晶显示屏(liquid crystal display，LCD)、有机发光二极管(organic light-emitting diode，OLED)等形式来配置。
进一步的，所述触控面板1051可覆盖所述显示面板1061，当所述触控面板1051检测到在其上或附近的触摸操作后，传送给所述处理器103以确定触摸事件的类型，随后所述处理器103根据触摸事件的类型在所述显示面板1061上提供相应的视觉输出。
虽然在图1中，所述触控面板1051与所述显示面板1061是作为两个独立的部件来实现所述电子设备100的输入和输出功能，但是在某些实施例中，可以将所述触控面板1051与所述显示面板1061集成而实现所述电子设备100的输入和输出功能。
所述处理器103是所述电子设备100的控制中心，利用各种接口和线路连接各个部件，通过运行或执行存储在所述存储器104内的软件程序和/或模块，以及调用存储在所述存储器104内的数据，执行所述电子设备100的各种功能和处理数据，从而实现基于所述电子设备的多种业务。
可选的，所述处理器103可包括一个或多个处理单元。可选的，所述处理器103可集成应用处理器和调制解调处理器，其中，应用处理器主要处理操作系统、用户界面和应用程序等，调制解调处理器主要处理无线通信。可以理解的是，上述调制解调处理器也可以不集成到所述处理器103中。
所述摄像头107，用于实现所述电子设备100的拍摄功能，拍摄图片或视频。所述摄像头107还可以用于实现电子设备100的扫描功能，对扫描对象(二维码/条形码)进行扫 描。
需要注意的是，本申请实施例中，所述摄像头107可以为RGB摄像头或TOF摄像头。
RGB摄像头用于采集目标物质，可以获得RGB三种颜色组成的彩色图像，TOF摄像头可以用于采集目标物质，生成深度图像或3D图像。
所述传感器108可以包括一个或多个传感器。例如，触摸传感器1081、色温传感器1082等。在另一些实施例中，传感器108还可以包括陀螺仪、加速度传感器、指纹传感器、环境光传感器、距离传感器、接近光传感器、骨传导传感器、压力传感器、定位传感器(如全球定位系统(global positioning system，GPS)传感器)等中的一个或多个，对此不作限定。
触摸传感器1081也可称为“触控面板”，可用于收集用户在其上或附近的触摸操作(比如用户使用手指、触笔等任何适合的物体或附件在所述触摸传感器1081上或在所述触摸传感器1081附近的操作)，并根据预先设定的程式驱动相应的连接装置。可选的，所述触摸传感器1081可以包括触摸检测装置和触摸控制器两个部分。其中，触摸检测装置检测用户的触摸方位，并检测触摸操作带来的信号，将信号传送给触摸控制器；触摸控制器从触摸检测装置上接收触摸信息，并将它转换成触点坐标，再送给所述处理器103，并能接收所述处理器103发来的命令并加以执行。此外，可以采用电阻式、电容式、红外线以及表面声波等多种类型实现所述触摸传感器1081。
当触摸传感器1081设置于显示屏106时，由触摸传感器1081与显示屏106组成触摸屏，也可以称为“触控屏”。触摸传感器1081用于检测作用于其上或附近的触摸操作。触摸传感器1081可以将检测到的触摸操作传递给应用处理器，以确定触摸事件类型，从而电子设备100可以通过显示屏106提供与触摸操作相关的视觉输出等。例如，电子设备100可以响应于触摸传感器1081检测到作用于其上或附近的触摸操作，进行界面切换，并在显示屏106上显示切换后的界面。在另一些实施例中，触摸传感器1081也可以设置于电子设备100的表面，与显示屏106所处的位置不同。
色温传感器1082，用于在所述摄像头107采集照片时，检测环境的色温，得到色温信号，可将色温信号转换成电信号。示例的，所述色温传感器1082可以设置于所述显示屏106。根据检测到的环境光的色温，所述处理器103接收到色温的电信号，进而对执行色温调节，保证色温与环境光水平一致，从而使得所述摄像头107采集到的照片色彩更加准确。
所述电子设备100还包括用于给各个部件供电的电源102(比如电池)。可选的，所述电源102可以通过电源管理系统与所述处理器103逻辑相连，从而通过电源管理系统实现管理充电、放电、以及功耗等功能。
需要注意的是，尽管未示出，所述电子设备100还可以包括音频电路等，在此不再赘述。
在介绍本申请实施例的图像识别的方法之前，先介绍本申请实施例适用的一种应用场景，示例性的，图2示出了本申请实施例的一种应用场景的示意图。
如图2所示，以手机为例，作为拍摄设备，该手机为一种可采集高光谱图像及光谱的智能手机，用户开启该智能手机的摄像头，通过该手机对当前的水果，例如苹果，进行拍摄并获得该苹果的高光谱图像。此时，在该智能手机的显示屏的左侧自动显示采集到的该苹果的高光谱图像，高光谱图像中包括该苹果的图像和该苹果对应的成分以及相应的含量。 例如成分包括：水、糖、蛋白质(g)、纤维(g)等等，对应的含量为85％、12.3％、0.2g、1.7g等。
通常下，通过采集水果的高光谱图像，获取光谱信息，可以分析物质(例如某种水果)中特定的化学基团的振动信息，例如O-H、C-H、N-H键等，从而以确定物质的成分。
然而，在实际采集水果的高光谱图像时，往往会受到采集设备、采集时间等因素的影响，导致采集的高光谱图像为低分辨谱段的图像，然而，通过低分辨谱段的图像，无法获取到足够且有效的谱段信息，进而识别出物质成分和含量缺失或不准确。因此，需要通过高光谱重建技术将低分辨谱段的图像重建出高分辨谱段的图像，以获取更多的谱段信息。
现有高光谱图像重建技术，通常的针对单一的RGB图像，采用基于一般的卷积神经网络以及加入残差连接或者Dense模块的卷积神经网络进行图像重建，或者采用基于GAN的生成技术进行高光谱图像重建。
在重建过程中，需要针对相同的对象，采集该对象的RGB-高光谱图像立体(Hypercube)对，用于训练重建网络，由于重建网络通常基于卷积的，因此，通过引入残差连接或者密度更高的Dense模块，来进行各层特征的紧密融合，从而完成从有限的3通道到N通道的映射，完成重建。
然而，在该高光谱重建过程中，主要是对RGB单张图像进行重建的，由于该图像的波长范围集中在可见光区域，而更多物质的有效信息集中在近红外区域，因此，通过重建对RGB单张图像进行重建并不能获取足够物质信息，并且常常受限于初始采集数据集，导致实际重建过程很可能出现偏差，从而给出错误的物质信息(例如，当对象为相同颜色的不同物质时)。
再例如，利用神经网络和近红外光谱检测物质成分(例如水果的糖度)。该方法包括为：第一步骤：选取同种类水果组成样本集，随机分为校正集和预测集；第二步骤：采集所有样本的原始近红外光谱，对光谱等区间划分，对各区间吸光度分别求和；第三步骤：利用化学分析法测定样本中糖度含量；第四步骤：使用BP神经网络构建校正集样本糖度含量与近红外特征光谱之间的定量校正模型；第五步骤：将预测集样品的近红外光谱信息数据输入模型，得到预测集样本的糖度含量。
然而，在该高光谱重建过程中，针对物质成分进行检测，由于当前光谱集中在近红外区域，因此，只能针对特定点采集的一维光谱进行分析，得到单点的结果，而不能得到二维的物质分布。此外，采用传统的多层反馈(Back Propagation，BP)网络，特征提取能力相对较弱，而且只针对单一物质进行成分进行识别，无法对通用的化学键进行分类和检测。
因此，本申请实施例提供一种图像识别的方法，该方法中，首先获取包含至少一个目标物质的第一图像，所述第一图像为第一谱段的图像；所述第一谱段为低分辨的谱段；然后，对所述第一图像进行谱段重建，得到第二图像，所述第二图像为第二谱段的图像；所述第二谱段为波长范围在可见光区域和红外光区域的高分辨的谱段；最后，对所述第二图像进行识别得到所述至少一个目标物质的成分信息并显示所述至少一个目标物质的成分信息。该方案中，对包含至少一个目标物质的低分辨分段的图像进行谱段重建，得到波长范围在可见光区域和红外光区域的高分辨谱段的图像。因此，通过该高分辨谱段的图像，可以获取更多且有效的光谱信息，进而可以准确的识别得到该高分辨谱段的图像中至少一个目标物质的成分信息。
请参见图3，为本申请实施例提供的一种图像识别的方法的流程图。执行本申请实施例方法可以适用于图1所示的电子设备100中且适用于图2所示的场景中。如图3所示，该方法的流程包括：
S301：电子设备获取包含至少一个目标物质的第一图像，所述第一图像为第一谱段的图像；所述第一谱段为低分辨的谱段。
在一种实施方式中，所述第一图像为低分辨的谱段的高光谱图像，可以通过特定的采集设备采集得到，所述特定的采集设备中包括：RGB摄像头+TOF摄像头+色温传感器(至少两种，其中必须包括RGB摄像头)。
其中，所述目标物质可以为包含不同成分含量的物质，本申请不做具体限定。示例性的，所述目标物质可以为某种水果，例如苹果、香蕉等。
示例性的，电子设备可以在接收到用户的拍摄指令时，打开自身的摄像头，在用户的控制下摄取第一图像，比如用户拿着手机拍摄一朵花或一个水果等，手机可以得到包含至少一个目标物质的第一图像。再示例性的，电子设备也可以接收其他设备发来的第一图像，比如通过WiFi模块或RF电路可以接收其他电子设备发来的第一图像，从而也可以获取到包含至少一个目标物质的第一图像。
S302：电子设备对所述第一图像进行谱段重建，得到第二图像，所述第二图像为第二谱段的图像；所述第二谱段为波长范围在可见光区域和红外光区域的高分辨的谱段。
在一种实施方式中，在执行步骤S302时，具体可先获取第一谱段重建模型，基于所述第一谱段重建模型对所述第一图像进行谱段重建，得到第二图像；其中，所述第一谱段重建模型为在预先训练得到的第二谱段重建模型中增加色温传感器多通道监督信息的惩罚项得到的；所述第二谱段重建模型由多个第三图像训练得到；所述多个第三图像为通过第一采集设备和第二采集设备采集到的图像；所述第一采集设备为RGB摄像头，所述第二采集设备为TOF摄像头和/或色温传感器；所述色温传感器多通道监督信息的惩罚项用于对所述第二谱段重建模型的通道进行调整。
示例性的，向所述电子设备中输入一张包含至少一个目标物质的待重建高光谱图像(该高光谱图像的谱段分辨率较低)，通过所述电子设备获取的所述第一高光谱重建模型对所述包含至少一个目标物质的待重建高光谱图像进行重建，得到所述目标物质的重建后的高光谱图像(重建后的高光谱图像的谱段分辨率较高，且该重建后的高光谱图像的波长范围集中在可见光区域和红外光区域)。
例如，采用所述第一高光谱重建模型(例如高光谱重建的神经网络)对未知水果对象进行400-1000nm(波长范围)的高光谱图像重建。
需要注意的是，通常情况下，所述第一谱段重建模型即为第一高光谱重建模型，所述第二谱段重建模型即为训练后的第二高光谱重建模型。用于对低分辨谱段的高光谱图像进行谱段重建，从而得到高分辨谱段的高光谱图像。
在一种实施方式中，所述第二高光谱重建模型可以预先由多个第三图像训练得到，具体可以包括但不限于以下操作方式：
使用第一采集设备和第二采集设备采集M张目标物质的高光谱图像(即第三图像)，所述第一采集设备可为RGB摄像头，所述第二采集设备可为TOF摄像头和色温传感器中的任意一个或组合，所述M为大于1的整数；对所述M张目标物质的高光谱图像(即第三图像)进行训练，得到训练后的第二高光谱重建模型；在所述训练后的第二高光谱重建 模型中增加色温惩罚项，得到所述第一高光谱重建模型，其中，所述色温传感器多通道监督信息的惩罚项用于对所述第一高光谱重建模型的通道进行调整。
示例性的，通过第一采集设备和第二采集设备(所述第一采集设备为RGB摄像头、所述第二采集设备为TOF摄像头和/或色温传感器)采集多张目标物质的高光谱图像对(即包含通过RGB摄像或TOF摄像头或色温传感器采集的图像与通过第一采集设备和第二采集设备采集的高光谱图像之间的对应关系)，例如，通过RGB摄像头和TOF摄像头采集到的图像为原始4通道的图像，可以覆盖可见光到近红外光区域的图像。进一步，使用采集到的多个所述目标物质的高光谱图像对，进行训练得到第二高光谱图像重建模型(例如高光谱重建的神经网络)。
其中，利用RGB摄像头+TOF摄像头+色温传感器(至少两种，其中必须包括RGB摄像头)的多模态信息，可以用于重建高光谱图像。
另外，由于高光谱图像为在波长域上细分的多通道图，在训练得到第二高光谱图像重建模型基础上，增加色温传感器多通道监督信息的惩罚项，可以构建得到所述第一高光谱重建模型。
该步骤中，通过第一采集设备(RGB摄像头)和第二采集设备(TOF摄像头和/或色温传感器)和高精度设备采集多个(此处多个可以为很多个，比如1000个等等，数值可以设置较大)物质的高光谱图像对，训练得到高光谱图像重建模型；对训练得到高光谱图像重建模型(例如高光谱重建的神经网络)增加色温传感器多通道监督信息的惩罚项，而保证最终构建得到所述第一高光谱重建模型具有更高的精度和良好的适应能力。使用所述第一高光谱重建模型对获取的待识别目标物质的低分辨谱段的高光谱图像进行重建，得到所述目标物质的高分辨谱段的高光谱图像，其中，高分辨谱段包括波长范围在可见光区域和红外光区域的高分辨的谱段。从而通过所述高分辨谱段的高光谱图像，可以获取更多且有效的光谱信息，进而确定更精准的目标物质的成分信息。
S303：电子设备对所述第二图像进行识别得到所述至少一个目标物质的成分信息，显示所述至少一个目标物质的成分信息。
在执行步骤S303之前，还可以通过色温传感器，对所述第二图像中的光谱信息进行调整，具体实施方式包括但不限于以下：
电子设备还可以先根据所述第二图像的光谱信息，获取所述第二图像的谱段对应的第一光谱，并获取第四图像的谱段对应的第二光谱；所述第四图像为所述电子设备中的色温传感器对所述至少一个目标物质进行采集得到的图像；所述电子设备可以进一步根据所述第一光谱和所述第二光谱，确定所述第一光谱和所述第二光谱之间的误差值大于设定的阈值时，对所述第二图像中的光谱信息进行调整。
在一种实施方式中，所述电子设备对所述第二图像中的光谱信息进行调整，具体实施方式包括但不限于以下：
先根据所述第一光谱，获取所述第二图像中N个通道对应的N个光谱值，并根据所述第二光谱，获取所述第四图像中的N个通道对应的N个光谱值；所述N的值大于等于1；其中，所述第二图像中N个通道与所述第四图像中N个通道为一一对应的关系；进一步根据所述第二图像中N个通道对应的N个光谱值与所述第四的图像中N个通道对应的N个光谱值，对所述第二图像中N个通道进行调整。
另外，由于所述第四图像中每个通道对应光谱为一维的光谱，所述第二图像中每个通 道对应的光谱为多维的光谱。因此，还需要将所述第二图像中每个通道对应的光谱维度进行处理，得到一维的光谱。例如，针对所述第二图像中每个通道，将所述第二图像中每个通道进行全局的平均池化，得到一维的光谱。
示例性的，根据所述第二图像中的第一光谱，获取所述第二图像中N个通道对应的N个光谱值，具体包括但不限于以下：在对所述第二图像中每个通道的光谱维度进行处理，得到一维的光谱之后，将每个通道中处理得到的一维的光谱对应的光谱值作为对应通道的光谱值。
在一种实施方式中，所述电子设备对所述第二图像中N个通道进行调整，具体实施方式包括但不限于以下：
例如，针对所述N个通道中的第i个通道，对所述第i个通道进行调整，先确定所述第二图像中第i个通道对应的光谱值为L i，并确定所述色温传感器采集的第四图像中第i个通道对应的光谱值为l i，i取遍从1到N的任意一个整数值；当确定所述L i与所述l i之差的绝对值大于设定的阈值时，对所述第二图像中第i个通道进行调整。
需要注意的是，当确定所述L i与l i之差的绝对值大于设定的阈值时，则表示重建后的所述第二图像中第i通道与色温传感器采集图像中第i通道之间的误差比较大，即表示所述第二图像中第i通道重建不准确，则该电子设备需对该通道进行调整。当确定所述L i与l i之差的绝对值小于设定的阈值时，则表示重建后的所述第二图像中第i通道与色温传感器采集图像中第i通道之间的误差比较小，即表示所述第二图像中第i通道重建准确，则该电子设备无需对该通道进行调整。
因此，通过上述步骤实现对所述第二图像中N个通道的调整，即可实现所述第二图像的光谱的调整，进而保证所述第二图像中的光谱信息准确。
基于色温传感器对所述第二图像进行调整之后再执行步骤S303。
在一种实施方式中，电子设备可以先获取调整后的第二图像的光谱信息，所述光谱信息为所述调整后的第二图像的谱段对应的光谱信息；然后，使用卷积神经网络模型对所述第二图像的光谱信息进行识别，确定所述至少一个目标物质的化学键；最后，根据所述至少一个目标物质的化学键，确定所述至少一个目标物质的成分信息。
示例性的，所述电子设备具体可以根据所述第二图像的光谱信息，使用卷积神经网络对调整后的第二图像中光谱维进行分类回归，确定所述至少一个目标物质的化学键，确定所述至少一个目标物质的成分信息；对所述至少一个目标物质的成分信息进行分析，得到所述至少一个目标物质的分成的分布图。
例如，其中一个目标物质为水果时，可以通过S303步骤，确定其内部化学键信息，从而得到水果内可溶固形物质分布，如果糖、血氧等。
基于以上实施例提供的一种图像识别的方法，本申请还提供了一种图像识别的方式的实施例。该方法可由能够支持拍摄和显示的电子设备执行。如图4所示，具体流程如下所示。
a401：电子设备获取包含至少一个目标物质的待重建的第一高光谱图像。
例如，所述电子设备通过其摄像头采集到至少一个水果的待重建的高光谱图像，或者所述电子设备从其它电子设备中获取到至少一个水果的待重建的高光谱图像。
a402：所述电子设备获取调整后的高光谱图像重建网络，对所述第一高光谱图像进行重建，得到第二高光谱图像。
例如，利用调整后的高光谱图像重建网络重建所述第一高光谱图像，得到该未知水果的重建后的第二高光谱图像。
由于包含目标物质的待重建的第一高光谱图像为低分辨谱段的高光谱图像，使用调整后的高光谱图像重建网络对所述包含目标物质的待重建的高光谱图像进行谱段重建，得到包含目标物质的高分辨谱段的第二高光谱图像，该图像波长范围在可见光范围和红外光范围。因此，通过该高分辨谱段的高光谱图像，可以获取更多且有效的光谱信息。
所述调整后的高光谱图像重建网络，可以通过以下步骤得到：
B1：通过RGB摄像头和TOF摄像头进行拍摄，得到多张合成RGBX四通道的图像对。
示例性的，该RGB摄像头拍摄为主，TOF摄像头为辅，该RGB摄像头(波长范围为400-780nm)大于24像素(MegaPixel)时采用至240P，OF摄像头(X:960nm)采用至240P。
通过RGB摄像头和TOF摄像头进行拍摄，得到的照片为合成RGBX四通道的图像，此处多张可以为较大的数量，此处不做具体限定。
B2：训练得到高光谱图像重建网络。
具体的，对在步骤B1获取到的多张RGBX四通道的图像进行训练，得到训练的高光谱图像重建网络(相当于高光谱图像重建模型)。
B3：训练得到高光谱图像重建网络增加色温传感器多通道监督信息的惩罚项，得到调整后的高光谱图像重建模型。
由于高光谱图像为在波长域上细分的多通道图，在训练后得到高光谱图像重建模型基础上，增加色温传感器多通道监督信息的惩罚项，实时监测和调整训练后的高光谱图像重建模型，使得其调整后具有更高的精度和较好的适应性。
步骤B1-B3为高光谱图像重建模型的训练和调整框架，以便后续对包含至少一个目标物质的待重建的第一高光图像进行重建。
a403：所述电子设备获取重建后的第二高光谱图像的N通道的光谱信息。
a404：对各通道进行平均池化，得到一维的光谱，并获得对应光谱值。
由于所述第二高光谱图像中每个通道对应的光谱为多维的光谱。因此，还需要将所述第二高光谱图像中每个通道对应的光谱维度进行处理，得到一维的光谱。例如，针对所述第二高光谱图像中每个通道，将所述第二高光谱图像中每个通道进行全局的平均池化，得到一维的光谱。
示例性的，根据所述第二高光谱图像中的第一光谱，获取所述第二高光谱图像中N个通道对应的N个光谱值，具体包括以下：在对所述第二高光谱图像中每个通道的光谱维度进行处理，得到一维的光谱之后，将每个通道中处理得到的一维的光谱对应的光谱值作为对应通道的光谱值。
a405：所述电子设备通过色温传感器采集所述至少一个目标物质的高光谱图像，并获取该高光谱图像中的N通道的光谱值。
由于通过色温传感器采集所述至少一个目标物质的高光谱图像中每个通道对应光谱为一维的光谱，因此可以直接获取每个通道光谱对应的光谱值。
a406：所述电子设备计算所述第二高光谱图像中每个通道的光谱值与由色温传感器获取的高光谱图像中每个通道的光谱值之间的误差。
示例性的，针对所述第二高光谱图像中N个通道中的第i个通道，确定所述第二高光谱图像中第i个通道对应的光谱值为L i，所述色温传感器采集的高光谱图像中第i个通道 对应的光谱值为l i，i取遍从1到N的任意一个整数值，计算所述L i与l i之差的绝对值。
a407(1)：误差大于设定阈值，通过色温传感器实时在线对光谱通道进行调整，得到调整后高光谱立体图像和高精度N通道的高光谱立体。
例如，当确定所述L i与l i之差的绝对值大于设定的阈值时，对所述第二高光谱图像中第i个通道进行调整。
a407(2)：误差小于设定阈值，则直接输出。
例如，当确定所述L i与l i之差的绝对值小于设定的阈值时，则对所述第二高光谱图像中第i通道不进行调整。
当确定所述第二高光谱图像中每个通道的光谱值与所述色温传感器采集的高光谱图像中每个通道的光谱值之差的绝对值均小于设定阈值时，则所述电子设备对所述第二高光谱图像执行步骤a408。
a408：所述电子设备利用卷积神经网络分析各像素一维光谱中所包含的化学键信息，确定物质的成分，并得到物质分成的分布图。
经过步骤a403-a407(2)，得到调整后的第二高光谱图像。进一步，所述电子设备对调整后的第二高光谱图像执行步骤a408。
示例性的，所述电子设备利用卷积神经网络分析调整后的第二高光谱图像中各像素一维光谱中所包含的化学键信息，确定第一物质成分，并得到该物质分成的分布图，例如第一物质为苹果时，得到该苹果中果糖分布，或者第一物质为血液时，得到该血液中血氧分布。高精度N通道的高光谱立体，波长范围：400-1000nm，波长分辨率为10nm。
例如，通过本申请实施例的方法对包含苹果的图像进行识别，得到识别结果图可参考图5所示。
综上所述，本申请实施例提供一种图像识别的方法，该方法中，使用高光谱图像重建模型，对所述目标物质的低分辨谱段的图像进行重建，得到高分辨谱段的高光谱图像。其中，所述高光谱图像重建模型为在预先训练的高光谱图像重建模型中增加色温惩罚项约束后得到的模型，其中，预先训练的高光谱重建模型是基于采集设备(该采集设备除了RGB摄像头以外，还包括RGB摄像头和/或TOF摄像头)所采集的多个图像数据集训练得到的，从而可以保证改进后的高光谱图像重建模型具有更高的精度和较好的适应能力。另外通过色温传感器，对重建得到的高分辨谱段的高光谱图像进一步调整，从而使得该高分辨谱段的高光谱图像的光谱信息较准确。因此，通过最终得到的高分辨谱段的高光谱图像，不仅可以获取更多的光谱信息，还可以保证光谱信息的准确性，从而保证通过准确性较高的光谱信息确定的目标物质成分信息较多且准确，进而可以较精确识别到图像中的目标物质成分。
基于相同的构思，本申请实施例提供的一种电子设备，所述电子设备具体结构可以参考图2所示的一种电子设备100的硬件结构示意图，包括至少一个处理器103、存储器104、显示屏106、摄像头107、收发器(例如RF电路101无线通信模块110)。
所述至少一个处理器103与收发器、存储器104、显示屏106、摄像头107之间相互连接。可选的，所述至少一个处理器103与收发器、存储器104、显示屏106、摄像头107可以通过总线相互连接；所述总线可以是外设部件互连标准(peripheral component interconnect，PCI)总线或扩展工业标准结构(extended industry standard architecture，EISA)总线等。所述总线可以分为地址总线、数据总线、控制总线等。
所述收发器，用于与其他设备进行通信交互。例如，当所述电子设备100通过外部的摄像装置采集当前所在场景的图像时，所述电子设备100通过所述收发器，获取外部其他摄像装置采集的画面作为第一图像。可选的，所述收发器可以为蓝牙模块、WiFi模块110，RF电路101等。
这样，所述至少一个处理器103可以通过所述电子设备100内的摄像头107采集当前至少一个目标物质的图像作为获取的第一图像，或者也可以通过收发器，例如所述RF电路101或所述收发器110获取其他电子设备发送的包含至少一个目标物质的图像作为获取的第一图像。
所述至少处理器103，用于实现上述如图3所示的实施例和实例提供一种图像识别方法，具体可以参见上述实施例中的描述，此处不再赘述。
所述显示屏106，用于显示所述摄像头107采集的图像，或者显示图像识别后得到的至少一个目标物质成分信息的分布图。
可选的，所述电子设备100还可以包括音频电路，用于接收和发出语音信号。
所述存储器104，用于存放程序指令和数据(例如高光谱图像重建模型、采集的高光谱图像)等。具体地，程序指令可以包括程序代码，该程序代码包括计算机操作的指令。存储器104可能包含随机存取存储器(random access memory，RAM)，也可能还包括非易失性存储器(non-volatile memory)，例如至少一个磁盘存储器。所述至少一个处理器103执行所述存储器104所存放的程序，并通过上述各个部件，实现上述功能，从而最终实现以上实施例提供的方法。
通过以上的实施方式的描述，所属领域的技术人员可以清楚地了解到本申请实施例可以用硬件实现，或固件实现，或它们的组合方式来实现。当使用软件实现时，可以将上述功能存储在计算机可读介质中或作为计算机可读介质上的一个或多个指令或代码进行传输。计算机可读介质包括计算机存储介质和通信介质，其中通信介质包括便于从一个地方向另一个地方传送计算机程序的任何介质。存储介质可以是计算机能够存取的任何可用介质。以此为例但不限于：计算机可读介质可以包括RAM、ROM、电可擦可编程只读存储器(electrically erasable programmable read only memory，EEPROM)、只读光盘(compact disc read-Only memory，CD-ROM)或其他光盘存储、磁盘存储介质或者其他磁存储设备、或者能够用于携带或存储具有指令或数据结构形式的期望的程序代码并能够由计算机存取的任何其他介质。此外。任何连接可以适当的成为计算机可读介质。例如，如果软件是使用同轴电缆、光纤光缆、双绞线、数字用户线(digital subscriber line，DSL)或者诸如红外线、无线电和微波之类的无线技术从网站、服务器或者其他远程源传输的，那么同轴电缆、光纤光缆、双绞线、DSL或者诸如红外线、无线和微波之类的无线技术包括在所属介质的定影中。如本申请实施例所使用的，盘(disk)和碟(disc)包括压缩光碟(compact disc，CD)、激光碟、光碟、数字通用光碟(digital video disc，DVD)、软盘和蓝光光碟，其中盘通常磁性的复制数据，而碟则用激光来光学的复制数据。上面的组合也应当包括在计算机可读介质的保护范围之内。
总之，以上所述仅为本申请的实施例而已，并非用于限定本申请的保护范围。凡根据本申请的揭露，所作的任何修改、等同替换、改进等，均应包含在本申请的保护范围之内。

Claims (14)

  1. 一种图像识别的方法，其特征在于，所述方法包括：
    获取包含至少一个目标物质的第一图像，所述第一图像为第一谱段的图像；所述第一谱段为低分辨的谱段；
    对所述第一图像进行谱段重建，得到第二图像，所述第二图像为第二谱段的图像；所述第二谱段为波长范围在可见光区域和红外光区域的高分辨的谱段；
    对所述第二图像进行识别得到所述至少一个目标物质的成分信息；
    显示所述至少一个目标物质的成分信息。
  2. 如权利要求1所述的方法，其特征在于，对所述第一图像进行谱段重建，得到第二图像，包括：
    获取第一谱段重建模型，基于所述第一谱段重建模型对所述第一图像进行谱段重建，得到第二图像；
    其中，所述第一谱段重建模型为在预先训练得到的第二谱段重建模型中增加色温传感器多通道监督信息的惩罚项得到的；所述第二谱段重建模型由多个第三图像训练得到；
    所述多个第三图像为通过第一采集设备和第二采集设备采集到的图像；所述第一采集设备为RGB摄像头，所述第二采集设备为TOF摄像头和/或色温传感器；
    所述色温传感器多通道监督信息的惩罚项用于对所述第二谱段重建模型的通道进行调整。
  3. 如权利要求1所述的方法，其特征在于，对所述第二图像进行识别得到所述至少一个目标物质的成分信息，包括：
    获取所述第二图像的光谱信息，所述光谱信息为所述第二图像的谱段对应的光谱信息；
    使用卷积神经网络模型对所述第二图像的光谱信息进行识别，确定所述至少一个目标物质的化学键；
    根据所述至少一个目标物质的化学键，确定所述至少一个目标物质的成分信息。
  4. 如权利要求3所述的方法，其特征在于，所述使用卷积神经网络模型对所述第二图像的光谱信息进行识别之前，所述方法还包括：
    根据所述第二图像的光谱信息，获取所述第二图像的谱段对应的第一光谱，以及获取第四图像的谱段对应的第二光谱；其中，所述第四图像为色温传感器对所述至少一个目标物质进行采集得到的图像；
    根据所述第一光谱和所述第二光谱，确定所述第一光谱和所述第二光谱之间的误差值大于设定的阈值时，对所述第二图像中的光谱信息进行调整。
  5. 如权利要求4所述的方法，其特征在于，对所述第二图像中的光谱信息进行调整，包括：
    根据所述第一光谱，获取所述第二图像中N个通道对应的N个光谱值；
    根据所述第二光谱，获取所述第四图像中的N个通道对应的N个光谱值；所述N的值大于等于1；其中，所述第二图像中N个通道与所述第四图像中N个通道为一一对应的关系；
    根据所述第二图像中N个通道对应的N个光谱值与所述第四的图像中N个通道对应的N个光谱值，对所述第二图像中N个通道进行调整。
  6. 如权利要求5所述的方法，其特征在于，对所述第二图像中N个通道进行调整，包括：
    针对所述N个通道中的第i个通道，确定所述第二图像中第i个通道对应的光谱值为L i，所述色温传感器采集的第四图像中第i个通道对应的光谱值为l i，i取遍从1到N的任意一个整数值；
    当确定所述L i与l i之差的绝对值大于设定的阈值时，对所述第二图像中第i个通道进行调整。
  7. 一种电子设备，其特征在于，包括显示屏、一个或多个处理器，一个或多个存储器；
    所述显示屏，用于显示信息；
    所述一个或多个存储器存储有一个或多个计算机程序，所述一个或多个计算机程序包括指令；当所述指令被所述处理器执行时，使所述电子设备执行：
    获取包含至少一个目标物质的第一图像，所述第一图像为第一谱段的图像；所述第一谱段为低分辨的谱段；
    对所述第一图像进行谱段重建，得到第二图像，所述第二图像为第二谱段的图像；所述第二谱段为波长范围在可见光区域和红外光区域的高分辨的谱段；
    对所述第二图像进行识别得到所述至少一个目标物质的成分信息；
    在所述显示屏上显示所述至少一个目标物质的成分信息。
  8. 如权利要求7所述的电子设备，其特征在于，当所述指令被所述处理器执行时，使所述电子设备具体执行：
    获取第一谱段重建模型，基于所述第一谱段重建模型对所述第一图像进行谱段重建，得到第二图像；
    其中，所述第一谱段重建模型为在预先训练得到的第二谱段重建模型中增加色温传感器多通道监督信息的惩罚项得到的；所述第二谱段重建模型由多个第三图像训练得到；
    所述多个第三图像为通过第一采集设备和第二采集设备采集到的图像；所述第一采集设备为RGB摄像头，所述第二采集设备为TOF摄像头和/或色温传感器；
    所述色温传感器多通道监督信息的惩罚项用于对所述第二谱段重建模型的通道进行调整。
  9. 如权利要求7所述的电子设备，其特征在于，当所述指令被所述处理器执行时，使所述电子设备具体执行：
    获取所述第二图像的光谱信息，所述光谱信息为所述第二图像的谱段对应的光谱信息；
    使用卷积神经网络模型对所述第二图像的光谱信息进行识别，确定所述至少一个目标物质的化学键；
    根据所述至少一个目标物质的化学键，确定所述至少一个目标物质的成分信息。
  10. 如权利要求9所述的电子设备，其特征在于，当所述指令被所述处理器执行时，使所述电子设备还执行：
    在使用卷积神经网络模型对所述第二图像的光谱信息进行识别之前，根据所述第二图像的光谱信息，获取所述第二图像的谱段对应的第一光谱，以及获取第四图像的谱段对应的第二光谱；其中，所述第四图像为色温传感器对所述至少一个目标物质进行采集得到的图像；
    根据所述第一光谱和所述第二光谱，确定所述第一光谱和所述第二光谱之间的误差值 大于设定的阈值时，对所述第二图像中的光谱信息进行调整。
  11. 如权利要求10所述的电子设备，其特征在于，当所述指令被所述处理器执行时，使所述电子设备具体执行：
    根据所述第一光谱，获取所述第二图像中N个通道对应的N个光谱值；
    根据所述第二光谱，获取所述第四图像中的N个通道对应的N个光谱值；所述N的值大于等于1；其中，所述第二图像中N个通道与所述第四图像中N个通道为一一对应的关系；
    根据所述第二图像中N个通道对应的N个光谱值与所述第四的图像中N个通道对应的N个光谱值，对所述第二图像中N个通道进行调整。
  12. 如权利要求11所述的电子设备，其特征在于，当所述指令被所述处理器执行时，使所述电子设备具体执行：
    针对所述N个通道中的第i个通道，确定所述第二图像中第i个通道对应的光谱值为L i，所述色温传感器采集的第四图像中第i个通道对应的光谱值为l i，i取遍从1到N的任意一个整数值；
    当确定所述L i与l i之差的绝对值大于设定的阈值时，对所述第二图像中第i个通道进行调整。
  13. 一种计算机可读存储介质，其特征在于，所述计算机可读存储介质中存储有计算机程序，当计算机程序在设备上运行时，使得所述设备执行如权利要求1-6任一项所述的方法。
  14. 一种芯片，其特征在于，所述芯片用于读取存储器中存储的计算机程序，执行如权利要求1-6任一项所述的方法。
PCT/CN2021/118155 2020-10-31 2021-09-14 一种图像识别的方法及电子设备 WO2022089064A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202011197236.0A CN114519814A (zh) 2020-10-31 2020-10-31 一种图像识别的方法及电子设备
CN202011197236.0 2020-10-31

Publications (1)

Publication Number Publication Date
WO2022089064A1 true WO2022089064A1 (zh) 2022-05-05

Family

ID=81381886

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2021/118155 WO2022089064A1 (zh) 2020-10-31 2021-09-14 一种图像识别的方法及电子设备

Country Status (2)

Country Link
CN (1) CN114519814A (zh)
WO (1) WO2022089064A1 (zh)

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105160332A (zh) * 2015-09-30 2015-12-16 刘毅 利用高光谱成像技术实现糖尿病食物摄入热量管理的方法
CN107643267A (zh) * 2017-09-01 2018-01-30 武汉大学 一种基于可见光谱成像的古代壁画颜料无损全面识别方法
CN109146787A (zh) * 2018-08-15 2019-01-04 北京理工大学 一种基于插值的双相机光谱成像系统的实时重建方法
CN110490799A (zh) * 2019-07-25 2019-11-22 西安理工大学 基于自融合卷积神经网络的高光谱遥感影像超分辨率方法
CN110530843A (zh) * 2019-08-22 2019-12-03 北京农业智能装备技术研究中心 一种玉米籽粒中淀粉含量的检测方法及系统
CN110627182A (zh) * 2019-09-24 2019-12-31 康玄谷(北京)国际生物科技有限公司 一种获取矿物质水的方法和装置

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105160332A (zh) * 2015-09-30 2015-12-16 刘毅 利用高光谱成像技术实现糖尿病食物摄入热量管理的方法
CN107643267A (zh) * 2017-09-01 2018-01-30 武汉大学 一种基于可见光谱成像的古代壁画颜料无损全面识别方法
CN109146787A (zh) * 2018-08-15 2019-01-04 北京理工大学 一种基于插值的双相机光谱成像系统的实时重建方法
CN110490799A (zh) * 2019-07-25 2019-11-22 西安理工大学 基于自融合卷积神经网络的高光谱遥感影像超分辨率方法
CN110530843A (zh) * 2019-08-22 2019-12-03 北京农业智能装备技术研究中心 一种玉米籽粒中淀粉含量的检测方法及系统
CN110627182A (zh) * 2019-09-24 2019-12-31 康玄谷(北京)国际生物科技有限公司 一种获取矿物质水的方法和装置

Also Published As

Publication number Publication date
CN114519814A (zh) 2022-05-20

Similar Documents

Publication Publication Date Title
CN108307125B (zh) 一种图像采集方法、装置和存储介质
CN107635101B (zh) 拍摄方法、装置、存储介质和电子设备
US10531069B2 (en) Three-dimensional image sensors
US10510136B2 (en) Image blurring method, electronic device and computer device
CN111340864A (zh) 基于单目估计的三维场景融合方法及装置
CN108366207B (zh) 控制拍摄的方法、装置、电子设备及计算机可读存储介质
CN108600712B (zh) 一种图像传感器、移动终端及图像拍摄方法
CN109086761B (zh) 图像处理方法和装置、存储介质、电子设备
US20240040205A1 (en) Method for Displaying Label in Image Picture, Terminal Device, and Storage Medium
KR20180126359A (ko) 생물 특징 데이터의 검출 방법, 생물 특징 인식 장치 및 전자 단말기
CN107968917A (zh) 图像处理方法和装置、计算机设备、计算机可读存储介质
CN107888829B (zh) 移动终端的对焦方法、移动终端及存储介质
WO2023005870A1 (zh) 一种图像处理方法及相关设备
US11348254B2 (en) Visual search method, computer device, and storage medium
WO2022089064A1 (zh) 一种图像识别的方法及电子设备
CN113691716A (zh) 图像传感器、图像处理方法、装置、电子设备和存储介质
CN108600623B (zh) 重聚焦显示方法以及终端设备
CN110163036B (zh) 一种图像识别方法及装置
CN112672021B (zh) 语言识别方法、装置及电子设备
CN112532904B (zh) 视频的处理方法、装置及电子设备
CN110971889A (zh) 一种获取深度图像的方法、摄像装置以及终端
CN107194363B (zh) 图像饱和度处理方法、装置、存储介质及计算机设备
CN108174101B (zh) 一种拍摄方法和装置
WO2022213288A1 (zh) 一种深度图像处理方法、装置及存储介质
CN114125300B (zh) 拍摄方法、拍摄装置、电子设备和可读存储介质

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21884794

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 21884794

Country of ref document: EP

Kind code of ref document: A1