WO2023103295A1 - Physiological state detection method and apparatus, electronic device, storage medium, and computer program product - Google Patents

Abstract

The present disclosure provides a physiological state detection method and apparatus, an electronic device, a storage medium, and a computer program product. The method comprises: acquiring a video stream in a vehicle cabin; performing face detection on a plurality of frames of images in the video stream, and extracting a plurality of frames of face images of a target object in the vehicle cabin; determining at least one preset smooth region in each frame of face image as a region of interest; and extracting physiological state information on the basis of image information of the regions of interest in the plurality of frames of face images to obtain a physiological state detection result of the target object.

Description

Physiological state detection method, device, electronic equipment, storage medium and computer program product

Cross References to Related Applications

The embodiment of the present disclosure is based on the Chinese patent application with the application number 202111506368.1, the application date is December 10, 2021, and the application name is "a physiological state detection method, device, electronic equipment and storage medium", and requires the Chinese patent Priority of the application, the entire content of the Chinese patent application is hereby incorporated by reference into this disclosure.

technical field

The present disclosure relates to but not limited to the field of computer technology, and in particular relates to a physiological state detection method, device, electronic equipment, storage medium and computer program product.

Background technique

Accurate physiological state data is the basis for analyzing the variability of the human body, so the detection of physiological state is of great significance, and thus is widely used in various application scenarios.

Taking the safe driving scene as an example, effective physiological state detection can help understand the physiological state of the occupants in the car, thereby providing auxiliary decision-making for safe driving. In related technologies, it mainly relies on special detection equipment, such as blood pressure meter, heart rate meter, oximeter and other equipment for physiological state detection. In addition, it is also possible to measure the physiological state with the help of wearable devices such as smart watches and smart bracelets integrated with relevant sensing components.

It can be seen that the above-mentioned detection scheme requires the use of special instruments for contact measurement, which brings inconvenience to the detection, and thus cannot well meet the needs of scenarios such as safe driving.

Contents of the invention

Embodiments of the present disclosure at least provide a physiological state detection method, device, electronic equipment, storage medium, and computer program product.

In a first aspect, an embodiment of the present disclosure provides a method for detecting a physiological state, including:

Obtain the video stream in the cabin;

Perform face detection on multiple frames of images in the video stream, and extract multiple frames of facial images of the target object in the cabin;

Determining at least one preset smooth area in each frame of the facial image as an area of interest;

Physiological state information is extracted based on the image information of each region of interest in the multiple frames of facial images to obtain a physiological state detection result of the target object.

Using the above physiological state detection method, when the video stream in the cabin is obtained, face detection can be performed on multiple frames of images in the video stream first, so as to extract multiple frames of facial images of the target object in the cabin , and then at least one preset smooth region in each frame of face image can be determined as the region of interest, and the physiological state information can be extracted based on the image information of each region of interest in the multi-frame face image, so as to obtain the target object Physiological state test results. In the present disclosure, before extracting the physiological state information, by determining the preset smooth area in each frame of the facial image, image information more suitable for physiological state analysis can be found, so that the determined physiological state detection results are also more accurate . At the same time, the present disclosure adopts a non-contact detection method to realize the detection of the physiological state of the object in the cabin, which is convenient for operation.

In the second aspect, the embodiment of the present disclosure also provides a physiological state detection device, including:

The acquisition part is configured to acquire the video stream in the cabin;

The extraction part is configured to perform face detection on multiple frames of images in the video stream, and extract multiple frames of facial images of the target object in the vehicle cabin;

A determining part configured to determine at least one preset smooth area in each frame of the facial image as an area of interest;

The detection part is configured to extract physiological state information based on the image information of each region of interest in the multiple frames of facial images, and obtain a detection result of the physiological state of the target object.

In a third aspect, an embodiment of the present disclosure further provides an electronic device, including: a processor, a memory, and a bus, the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the The processor communicates with the memory through a bus, and when the machine-readable instructions are executed by the processor, the steps of the physiological state detection method according to any one of the first aspect and its various implementations are executed.

In the fourth aspect, the embodiments of the present disclosure also provide a computer-readable storage medium, on which a computer program is stored, and the computer program is executed by a processor as in the first aspect and its various implementation modes The steps of any one of the physiological state detection methods.

In a fifth aspect, the present disclosure also provides a computer program product, the computer program product includes a computer program or an instruction, and when the computer program or instruction is run on an electronic device, the electronic device executes The steps of the physiological state detection method according to any one of the first aspect and its various implementations.

For the effect description of the above-mentioned physiological state detection device, electronic equipment, and computer-readable storage medium, please refer to the description of the above-mentioned physiological state detection method. It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.

In order to make the above-mentioned objects, features and advantages of the present disclosure more comprehensible, preferred embodiments will be described in detail below together with the accompanying drawings.

Description of drawings

In order to illustrate the technical solutions of the embodiments of the present disclosure more clearly, the following will briefly introduce the accompanying drawings used in the embodiments. The accompanying drawings here are incorporated into the specification and constitute a part of the specification. The drawings show the embodiments consistent with the present disclosure, and are used together with the description to explain the technical solutions of the present disclosure. It should be understood that the following drawings only show some embodiments of the present disclosure, and therefore should not be regarded as limiting the scope. For those skilled in the art, they can also make From these drawings other related drawings are obtained.

FIG. 1 shows a flowchart of a physiological state detection method provided by an embodiment of the present disclosure;

FIG. 2 shows a flow chart of a physiological state detection method provided by an embodiment of the present disclosure;

FIG. 3 shows a flow chart of a method for selecting a region of interest in the method for detecting a physiological state provided by an embodiment of the present disclosure;

FIG. 4 shows a flow chart of a method for detecting a physiological state provided by an embodiment of the present disclosure;

Fig. 5 shows a flow chart of a physiological state detection method provided by an embodiment of the present disclosure;

FIG. 6 shows a flowchart of a method for detecting a physiological state provided by an embodiment of the present disclosure;

FIG. 7 shows a flow chart of a method for detecting a physiological state provided by an embodiment of the present disclosure;

Fig. 8 shows a schematic diagram of a physiological state detection device provided by an embodiment of the present disclosure;

Fig. 9 shows a schematic diagram of an electronic device provided by an embodiment of the present disclosure.

Detailed ways

In order to make the purpose, technical solutions and advantages of the embodiments of the present disclosure clearer, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only It is a part of the embodiments of the present disclosure, but not all of them. The components of the disclosed embodiments generally described and illustrated in the figures herein may be arranged and designed in a variety of different configurations. Accordingly, the following detailed description of the embodiments of the present disclosure provided in the accompanying drawings is not intended to limit the scope of the claimed disclosure, but merely represents selected embodiments of the present disclosure. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without creative effort shall fall within the protection scope of the present disclosure.

It should be noted that like numerals and letters denote similar items in the following figures, therefore, once an item is defined in one figure, it does not require further definition and explanation in subsequent figures.

The term "and/or" in this article only describes an association relationship, which means that there can be three kinds of relationships, for example, A and/or B can mean: there is A alone, A and B exist at the same time, and B exists alone. situation. In addition, the term "at least one" herein means any one of a variety or any combination of at least two of the more, for example, including at least one of A, B, and C, which may mean including from A, Any one or more elements selected from the set formed by B and C.

After research, it is found that in related technologies, it mainly relies on special detection equipment, such as blood pressure meter, heart rate meter, oximeter and other equipment for physiological state detection. In addition, it is also possible to measure the physiological state with the help of wearable devices such as smart watches and smart bracelets integrated with relevant sensing components. It can be seen that the above-mentioned detection scheme requires the use of special instruments for contact measurement, which brings inconvenience to the detection, and thus cannot well meet the needs of scenarios such as safe driving.

As a new heart rate detection technology, remote photoplethysmography (Remote Photo Plethysmo Graphic, rPPG) has a very broad application prospect, and it belongs to non-contact detection. The detection can be completed without additional hardware cost, and it is very convenient to use. However, the current bottleneck of the remote photoplethysmography method is that the detection accuracy is inferior to some special detection equipment, and it is also easily affected by external light.

Based on the above research, the present disclosure provides a solution for physiological state detection based on facial region of interest, which improves the accuracy of physiological state detection while ensuring simple operation.

In order to facilitate the understanding of this embodiment, a physiological state detection method disclosed in the embodiment of the present disclosure is first introduced in detail. The execution subject of the physiological state detection method provided in the embodiment of the present disclosure is generally an electronic device with a certain computing capability. The electronic equipment includes, for example: a terminal device or a server or other processing equipment, and the terminal device may be a user equipment (User Equipment, UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (Personal Digital Assistant, PDA), handheld devices, computing devices, vehicle-mounted devices, etc. In some possible implementation manners, the method for detecting a physiological state may be implemented by a processor invoking computer-readable instructions stored in a memory.

Referring to FIG. 1 , which is a flowchart of a physiological state detection method provided by an embodiment of the present disclosure, the method includes steps S101 to S104, wherein:

S101: Obtain a video stream in the cabin;

S102: Perform face detection on multiple frames of images in the video stream, and extract multiple frames of facial images of the target object in the cabin;

S103: Determine at least one preset smooth area in each frame of the face image as the area of interest;

S104: Extract physiological state information based on the image information of each region of interest in the multiple frames of facial images, and obtain a physiological state detection result of the target object.

In order to facilitate understanding of the physiological state detection method provided by the embodiment of the present disclosure, an application scenario of the method is briefly introduced next. The physiological state detection method in the embodiments of the present disclosure can be applied to the field of automobile driving where physiological state detection is required, that is, the embodiments of the present disclosure can realize the physiological state detection for the cabin environment. In addition, the embodiments of the present disclosure may also be applied to any other relevant fields that require physiological state detection, such as medical treatment and home life, and there is no limitation here. Considering the wide range of applications in the field of automobile driving, the following examples will be given in the field of automobile driving.

Wherein, the video stream in the embodiments of the present disclosure may be collected by a camera mounted on a vehicle, may also be collected by a camera built in a user terminal, or may be collected by other methods, which are not limited here.

In order to realize physiological state detection for a specific target object, the installation position of the camera may be preset based on the specific target object. For example, in order to realize the physiological state detection of the driver in the vehicle, the camera here can be installed in a position where the shooting range covers the driving area, such as the inside of the A-pillar of the vehicle, on the console, or the position of the steering wheel; The physiological state detection of occupants with various attributes including the driver and passengers in the vehicle. The camera here can be installed on the interior rearview mirror, top decoration, reading lights, etc. The shooting range can cover many areas in the vehicle cabin. The location of the seating area.

In practical applications, the in-vehicle image acquisition device included in the driver monitoring system (Driver Monitoring System, DMS) can also be used to realize the acquisition of video streams related to the driving area, or the occupant monitoring system (Occupant Monitoring System, OMS) can be used The included in-vehicle image acquisition device enables the acquisition of video streams about the riding area.

Considering that the skin color and brightness changes caused by the flow of facial blood vessels can reflect physiological states such as heartbeat and breathing, here we can first perform face detection on multiple frames of images in the video stream, and extract multiple frames of the target object in the cabin The facial image, and then realize the extraction of physiological state information for the facial image.

The target object can be an object of specific car attributes, such as the driver, the occupant of the passenger seat; or, the target object can be an object whose identity is pre-registered with facial information, such as the owner of the car registered through the application; or, the target The object can also be any occupant in the car, at least one occupant can be located by performing face detection on the video stream in the car cabin, and one or more occupants detected can be used as the target object.

In the process of face detection, faces of multiple objects may appear on one frame of image. In some scenarios, it is possible to choose to detect the physiological state of the occupant of a certain riding position, that is, to take the occupant of the riding position as the target object. In order to realize the physiological state detection of the target object in the cabin, here, according to the face detection results of the multi-frame images and the specified riding position, the multi-frame face of the target object can be determined from the detected facial images An internal image in which the specified ride location is used to indicate the location of the target object being measured.

The relative position of the camera used to collect video streams in the vehicle cabin is fixed in the interior space of the vehicle. The images collected by the camera can be divided according to the seat area. For example, for a 5-seater private car, it can be divided into: The image area corresponding to the passenger seat, the image area corresponding to the rear left seat, the image area corresponding to the rear right seat, and the image area corresponding to the rear middle seat. According to the position of the face of each occupant object in the image and the coordinate range of each image area, the image area where the face of each occupant object falls can be determined, and then the occupant object at the specified riding position can be determined as the target object .

In practical applications, the occupant monitoring system generally captures images of the entire vehicle, and may capture multiple people. You can manually select the "parking space for the front car" and "parking space for the rear seat" to specify the target object to be measured. At this time, the embodiment of the present disclosure may measure the faces in the corresponding area in the image. The driver monitoring system is aimed at the main driving area, and if the captured object only includes the driver, it is not necessary to specify the object.

It should be noted that physiological states, such as heart rate, respiratory rate, blood oxygen, blood pressure, etc., often require a certain period of monitoring before they can be evaluated. Therefore, in the embodiment of the present disclosure, the image change information corresponding to the multi-frame facial images in the video stream that lasts for a period of time is used to realize the extraction of physiological state information, so that the extracted physiological state detection results are more in line with the actual scene. need.

Considering that in the process of image change information analysis based on facial images, the detection accuracy is disturbed due to the influence of various factors such as illumination and occlusion, the embodiment of the present disclosure provides a facial region-of-interest based (Region of Interest, ROI) image change information analysis scheme, because the region of interest contains more effective pixels, which can effectively improve the detection accuracy.

Among them, in machine vision and image processing, the area to be processed is outlined from the processed image in the form of boxes, graphs, ellipses, irregular polygons, etc., which is called the region of interest. For each frame of the facial image, a relevant region of interest may be determined, and the region of interest may be constituted by one or more preset smooth regions corresponding to the facial image. The preset smoothing area can be a smooth connected area, which has a more uniform reflectivity to a certain extent, so that it can capture more effective changes in skin color and brightness caused by the flow of facial blood vessels, and then achieve more accurate Physiological state detection.

In the case of determining the region of interest of each frame of facial images, the physiological state detection method provided by the embodiment of the present disclosure can perform physiological state detection based on the image change information corresponding to the image information of each region of interest in multiple frames of facial images. Information extraction, the physiological state detection result extracted here may be a detection result including at least one of heart rate, respiratory rate, blood oxygen, blood pressure, and the like.

Considering that the determination of the region of interest plays a key role in realizing the detection of the physiological state, the process of determining the region of interest can be emphatically described in detail next. In some implementations, as shown in FIG. 2, the above step S103 of "determining at least one preset smooth area in each frame of facial image as the area of interest" may include the following steps S201 and S202:

Step S201, extracting facial feature points for each frame of facial image;

Step S202, based on the extracted facial feature points, determine at least one preset smooth area in each frame of the facial image as the ROI of the facial image.

Here, facial feature points may be extracted from the facial image first, and then a preset smooth area in the facial image may be determined based on the extracted facial feature points as the region of interest.

Wherein, the above-mentioned process of extracting facial feature points can be realized by using a face key point detection algorithm. For example, the facial feature points of a standard face image can be preset. Face the face image captured by the camera, so that in the process of extracting facial feature points from the face image of the target object extracted from each frame image, it can be based on the extracted face image of the target object and the standard face image The comparison between them is used to determine each facial feature point.

In the embodiment of the present disclosure, one or more preset smoothing areas in the facial image may be determined based on the determined coordinate information of the facial feature points. The preset smoothing area here may be a rectangular area, or other area with a connected shape, which is not limited in this embodiment of the present disclosure, and the following description will mostly be made by taking a rectangular area as an example.

In practical applications, the preset smoothing area may be one or more of the forehead area, the left cheek area, the right cheek area, and the chin area. In order to facilitate further physiological state analysis, the left cheek area can be a complete area, or it can be divided into two or more areas. Similarly, the forehead area, right cheek area, and chin area can also be divided according to the above The way to set it will not be described in detail here.

In the case of no area occlusion, the above four areas can be extracted simultaneously on a frame of face image. In the case of area occlusion, the actual extracted area of a frame of face image can be based on the actual situation to make sure.

As shown in FIG. 3 , it is a schematic diagram of facial feature points that can be extracted from a facial image captured by a camera, and there are 106 feature points in total. Here, based on the coordinate information of facial feature points, 5 preset smooth areas can be screened, as can be seen in Figure 3, where area 1 can be a rectangular region of interest in area 1 constructed by two feature points of eyebrows on both sides; area 2 is a cheek area on the left side, and the rectangular region of interest of area 2 can be constructed by the position of the feature points on the left edge of the face, the bridge of the nose, and the feature points of the left eye; area 3 is a cheek area on the right side, which can be constructed by The position of the feature point on the right edge of the face, the feature point on the bridge of the nose, and the position of the feature point on the right eye construct a rectangular region of interest in area 3, and area 4 is another cheek area on the left side. The position of the feature points of the alar of the nose and the feature points of the left mouth corner constructs a rectangular region of interest in area 4, and area 5 is another cheek area on the right side, which can be obtained through the feature points of the right edge of the face, the feature points of the right alar of the nose, and the feature points of the right mouth corner The location of Construct Region 5 is a rectangular region of interest.

Considering that different regions of interest may be affected by different external factors, resulting in different physiological state detection performances. Here, in order to capture more effective facial features as much as possible to achieve more accurate physiological state detection, the image quality detection of the region of interest can be performed first, and then the selection of a high-quality region of interest can be achieved, thereby improving the physiological state. Accuracy of test results. In some implementations, as shown in FIG. 4, the above step S104 "extracts the physiological state information based on the image information of each region of interest in the multi-frame facial image, and obtains the physiological state detection result of the target object" This can be achieved through the following steps:

Step 401, determine the image quality of each ROI in the face image;

Step 402, selecting a region of interest whose image quality meets preset requirements as a target region of interest;

Step 403, extracting physiological state information based on the image information of one or more target ROIs in the multi-frame facial images.

Here, considering that in the actual cabin environment, the face area is easily affected by external factors such as face rotation, sunglasses occlusion, or light, which makes some areas of interest very small or even disappear or are too dark, too noisy, etc. etc. Therefore, the image quality of each region of interest can be determined by at least one of the brightness, occlusion, area, and signal-to-noise ratio of each region of interest, and then the region of interest with higher image quality can be selected As the target region of interest to achieve subsequent extraction of physiological state information.

Among them, regarding image brightness, the brightness of the region of interest should not be too high or too low, too high brightness will cause the region of interest to be overexposed, and too low brightness will cause the region of interest to be too dark, which makes the image quality significantly Decrease; Regarding the occlusion situation, such as the occlusion of sunglasses, masks, etc., this may cause some regions of interest to be invisible; Regarding the area of the region, the region of interest with a larger area has better image quality to a certain extent, conversely, the larger the area The image quality of a small region of interest is poor; regarding the signal-to-noise ratio, the signal-to-noise ratio is configured as an indicator for evaluating the noise size of the region of interest. The larger the noise, the smaller the corresponding signal-to-noise ratio to a certain extent, and vice versa, the smaller the noise , the larger the corresponding signal-to-noise ratio, the region of interest with a larger signal-to-noise ratio needs to be selected to reduce the influence of noise on the detection results.

The embodiments of the present disclosure can combine one or more of the above-mentioned various image quality evaluation indicators to evaluate the image quality of the region of interest, so that the image quality evaluation result is more accurate. In addition, it can also Image quality evaluation is performed in combination with other evaluation indicators, which is not limited here.

In the embodiment of the present disclosure, in addition to evaluating each region of interest of each frame of facial image by comprehensively considering the above evaluation indicators and then realizing the screening of the relevant region of interest, the screening operation can also be implemented in combination with facial gestures.

Here, firstly, the face pose of the target object can be detected according to each frame of face image, so that after determining at least one preset smooth region in each frame of face image as the region of interest, the region of interest can be obtained from the region of interest according to the face pose Remove the region of interest in the image whose visible range does not meet the preset visibility requirements. Wherein, the preset visibility requirement may include that the area of the area is not smaller than a predetermined threshold, or that the area of the area is not smaller than the predetermined threshold and the brightness of the area is within a range. Optionally, the predetermined threshold of the area area can be associated with the deflection angle representing the facial posture, and whether the preset visibility requirement is met can be judged by whether the deflection angle of the facial posture exceeds the threshold. In this way, a region of interest with a large area or a large area that is not overexposed or underexposed can be screened out for extraction of physiological state parameters, which helps to improve the reliability of the detection result.

Wherein, the above-mentioned facial posture may be determined based on a pre-trained facial posture detection network. The face pose detection network training can be the correspondence between the face image samples and the corresponding marked face poses. The face poses that can be marked here include information such as the rotation direction and rotation angle of the head relative to the camera. In this way, when the facial image is input into the trained facial posture detection network, the facial posture of the relevant human face can be determined, for example, it can be 45° to the left. In this case, according to the facial posture angle Calculate the area of the right cheek area in the image, or directly determine that the area of the right cheek area is smaller than the preset threshold according to the facial posture, which is almost invisible in the image, and its visible range does not meet the preset visibility requirements. At this time, the corresponding ROI can be directly removed, and the operation process is simple.

The physiological state detection method provided by the embodiments of the present disclosure is based on the extraction of physiological state information based on the image information of each region of interest in multiple frames of facial images. In order to realize the conversion from image information to physiological state information, it is necessary to perform related signal processing on the image information of the region of interest. In some implementations, as shown in FIG. 5, the above step S104 "extracts physiological state information based on the image information of each region of interest in the multi-frame facial images, and obtains the physiological state detection result of the target object" May include the following steps:

Step 501, based on the luminance values of at least one region of interest corresponding to three color channels in the multi-frame face image, determine the time-domain brightness signal corresponding to at least one region of interest corresponding to each color channel;

Step 502, performing principal component analysis on time-domain luminance signals of at least one region of interest corresponding to multiple different color channels, to obtain a time-domain signal representing the physiological state of the target object;

Step 503: Determine the physiological state information of the target object based on the time-domain signal representing the physiological state of the target object.

Considering that the physiological state directly affects the blood flow change of the target object, and the blood flow change can be characterized based on the brightness change of the image. Therefore, here, firstly, it is possible to determine the time-domain luminance signal corresponding to each of the three color channels of red, green and blue in the region of interest to form a three-dimensional blue-green-red (BGR) signal, and then analyze the three-dimensional signals of the three different color channels The time-domain brightness signal is subjected to principal component analysis, and the one-dimensional signal obtained after extracting the principal component (dimension reduction) is used as the time-domain signal representing the physiological state of the target object. The time-domain signal may be determined from a time-domain luminance signal of one of the color channels (for example, a green channel), and the selected channel may be a channel that best characterizes changes in blood flow. In addition, it may also be determined by other principal component analysis methods, which is not limited here.

In order to facilitate more accurate principal component analysis, before performing principal component analysis on the three-dimensional time-domain brightness signal, processing such as regularization and detrending (Detrend) filtering and denoising can be performed. In addition, after principal component analysis, the obtained time-domain signal can also be denoised by moving average filtering, thereby further improving the accuracy of the time-domain signal and improving the accuracy of subsequent physiological state detection

In order to further improve the accuracy of physiological state detection, here, the frequency domain conversion can be performed on the time domain signal, and more useful information can be analyzed based on the converted frequency domain signal, for example, the amplitude distribution of each frequency component can be determined and energy distribution to obtain the frequency values of the main amplitude and energy distributions. Here, the physiological state information of the target object may be determined based on the peak value of the frequency domain signal.

Taking heart rate detection as an example, the peak value (pmax) of the frequency domain signal can be determined here, and the original heart rate measurement value can be obtained by summing the peak value and the heart rate reference value, where the peak value represents the heart rate variation, and the heart rate reference value can be obtained by It is determined based on the lower limit of the empirical heart rate estimation range, and the heart rate reference value may also be adjusted by considering the influence of factors such as video frame rate and frequency domain signal length.

After the heart rate is determined, relevant physiological indicators such as blood oxygen saturation and heart rate variability can be measured. For blood oxygen saturation, red light (600 to 800nm) and near-red light region (800 to 1000nm) can be used to detect the time domain signals of oxygen, hemoglobin (HbO2) and hemoglobin (Hemoglobin, Hb) respectively, and then calculate the corresponding Ratio, you can get the blood oxygen saturation; for heart rate variability, after extracting the time-domain signal, calculate the distance between every two adjacent peaks and combine the frame rate to get several intervals, and then take the standard deviation of these intervals (Standard Deviation of NN Intervals, SDNN), that is, heart rate variability.

The respiratory rate detection method is similar to the heart rate detection method, the difference is that the respiratory frequency range is different from the heart rate range, and the corresponding reference value settings are different. Based on the same method above, the respiratory frequency detection can be realized.

The embodiment of the present disclosure realizes the physiological state detection of multiple frames of images, that is, the image change information corresponding to the multiple frames of images can represent the change of the physiological state. In practical applications, the physiological state detection results determined in relation to the video stream may be updated along with continuous collection of image frames.

Here, when one or more frames of images included in the new video stream are obtained, face detection can be performed on the images in the new video stream, and the face image of the target object in the cabin can be extracted, and determined At least one preset smooth area in the face image is used as the area of interest. In this way, the physiological state detection result can be updated based on the image information of the region of interest in the facial image. If the preset detection duration is not reached, it will be updated again based on the acquired new video stream until the preset detection duration is reached. , to obtain the updated physiological state detection result.

Here, the heart rate detection is still used as an example for illustration. In the case that the preset detection time is determined to be 30 seconds (s), the video stream can be acquired continuously within 30 seconds. Still within 30 seconds, where heart rate measurements are calculated based on multiple frames of the starting video stream (e.g., within the first 5 seconds of the video stream). At this time, with the collection of image frames, the number of image frames increases, and a new heart rate measurement value can be calculated for each additional frame or each additional n frames, and then smoothed by sliding average, and the measurement ends after 30 seconds, and we get final measurement.

In the cabin environment, in order to help the target object to measure the physiological state more quickly, here, during a physiological state detection process, it can be used to remind the target object according to the duration of the acquired video stream and the preset detection duration. The detection progress reminder signal of the detection duration. For example, if the duration of the acquired video stream (that is, the detection time of the current target object's physiological state detection) reaches 25 seconds, and the preset detection duration is 30 seconds, you can issue a message about "Please keep still, there are 5 seconds left." The test will be completed" voice or screen prompt; or, when the current physiological state detection time of the target object reaches 30 seconds, a voice or screen prompt "measurement has been completed" will be issued.

In order to better adapt to the cabin environment, the physiological state detection method provided by the embodiment of the present disclosure can also provide relevant reminders in combination with body posture detection and physiological state detection. As shown in FIG. 6, the method includes the following steps:

Step 601, obtain the video stream in the cabin;

Step 602, performing body posture detection on the target object in the video stream to obtain posture change information of the target object;

Step 603, performing face detection on multiple frames of images in the video stream, and extracting multiple frames of facial images of the target object in the cabin;

Step S604, determining at least one preset smooth area in each frame of the face image as the area of interest;

Step S605, extracting physiological state information based on the image information of each region of interest in the multi-frame facial images, and obtaining the physiological state detection result of the target object;

Step S606, when the posture change information indicates that the posture of the target object is abnormal and the physiological state detection result does not meet the preset physiological state value, generate a reminder signal for reminding the associated objects of the target object to perform emergency assistance.

Here, the body posture detection of the target object can refer to the above-mentioned related methods of facial posture detection, or can be realized by using a trained body posture detection neural network, which will not be described in detail here.

The embodiments of the present disclosure may provide reminders based on body posture detection results such as posture change information and physiological state detection results. For example, when the driver's body posture is abnormal (such as covering his chest or lying down on the seat) and the driver's heart rate exceeds 100 beats per minute, a reminder message including the vehicle location can be sent to The relatives of the driver or the management personnel of the nearest hospital for emergency assistance.

After the physiological state detection is realized, the embodiment of the present disclosure may also display the physiological state detection result to provide better cabin service for the target object through the displayed physiological state detection.

In the embodiment of the present disclosure, on the one hand, the detection result of the physiological state of the target object can be transmitted to the display screen in the cabin for display on the display screen, so that the cabin personnel can monitor their own physiological state in real time, and can also If there is an abnormality in one's own physiological state, seek medical attention in time or take other necessary measures; When the application requests to obtain the detection result, the physiological state detection result is sent to the terminal device used by the target object through the server.

That is, here, the physiological state detection results of the target object can be recorded on the server, and the physical state detection results can also be statistically analyzed on the server side, for example, the statistical results of the physiological state of the target object can be determined for a month or a week, so that in When the target object initiates a physiological state detection application request, the physiological state detection results, statistical results, etc. can be sent to the terminal device of the target object, so as to achieve a more comprehensive physiological state evaluation.

Wherein, the above-mentioned physiological state detection application may be an application program (Application, APP) for physiological state detection, and the application program can respond to the acquisition request of the detection result related to the target object, and then realize the result presentation on the application program, and further practical. Through the application program, the physiological characteristics detection can be carried out in the process of daily life, work and even sports. The non-contact detection method also avoids the cross-use of detection equipment, bringing a safer detection experience.

The above-mentioned physiological state detection method will be described below in conjunction with a specific embodiment. However, it should be noted that this specific embodiment is only for better illustrating the present disclosure, and does not constitute an improper limitation to the present disclosure.

Referring to FIG. 7 , it is a logic flow diagram of a physiological state detection method provided by an embodiment of the present disclosure. The method includes the following steps:

Step S701, obtaining the video stream in the cabin;

Step S702, extracting multiple frames of facial images of the target object from the video stream;

Step S703, dividing multiple candidate regions of interest in each frame of facial images by facial feature points;

Step S704, performing secondary processing on multiple regions of interest in combination with the detection scene;

Here, multiple candidate regions of interest are screened in combination with different detection scenarios such as face rotation, sunglasses occlusion, or complex lighting.

Step S705, extracting time-domain luminance signals based on the image information of multiple regions of interest;

Here, the region of interest corresponds to the luminance values of the three color channels, and the time-domain luminance signal is a three-dimensional signal representing the luminance values of the region of interest corresponding to the three color channels.

Step S706, performing regularization and denoising processing on the time-domain luminance signal;

Step S707, by performing principal component analysis on the time-domain luminance signal to obtain a time-domain signal representing the physiological state of the target object;

Here, the three-dimensional signal is actually reduced to a one-dimensional original signal.

Step S708, performing moving average filtering and denoising processing on the time domain signal;

Step S709, converting the time domain signal into a frequency domain signal through frequency domain transformation;

Step S710, extracting physiological state information of the target object from the frequency domain signal.

Here, the physiological state information includes the target subject's heart rate or breathing rate, for example, the target subject's heart rate is determined from the peak value of the frequency domain signal.

The embodiment of the present disclosure proposes the extraction of multiple ROIs based on facial feature points, and the one-dimensional original signal extracted on this basis contains more effective pixel samples, which can effectively improve the detection accuracy. At the same time, the detection method of multiple regions of interest can avoid the problem that the region of interest is lost due to face rotation, sunglasses occlusion or external light during the detection process, and then the detection cannot continue, making the physiological state detection method more widely used. It can ensure that the rPPG method does not fail, and greatly increases the anti-interference ability in the detection process.

Compared with the physiological state detection using the remote photoelectric heart rate detection method in the related art, the detected object needs to be kept still for a period of time, which can only be used for active detection. In the embodiments of the present disclosure, through multiple regions of interest, the detected object is no longer limited to a static state for a period of time, making passive detection possible.

The vision-based physiological feature detection method of the embodiment of the present disclosure is easy to operate, and can be applied in at least the following scenarios: 1) The physiological feature detection is performed in the form of software application programs in daily life, work, and even sports, and the non-contact detection method It also avoids the cross-use of detection equipment, bringing a safer detection experience. 2) In the scenario of the smart car cabin, use the on-board camera to actively monitor the driver's health status, and play the role of a health steward in the on-board system. 3) In some stimulating and dangerous entertainment projects, the health status of tourists is detected by the camera to prevent some high-risk groups from being harmed in these projects. 4) Let the elderly in daily life get rid of the complex operation of special physiological detection equipment, and grasp the physical condition of the elderly more conveniently and quickly.

Those skilled in the art can understand that in the above-mentioned method of specific implementation, the writing order of each step does not imply a strict execution order and constitutes any limitation on the implementation process, and the execution order of each step should be based on its function and possible internal Logically OK.

Based on the same inventive concept, the embodiment of the present disclosure also provides a physiological state detection device corresponding to the physiological state detection method. Since the problem-solving principle of the device in the embodiment of the present disclosure is similar to the above-mentioned physiological state detection method of the embodiment of the present disclosure, therefore For the implementation of the device, reference may be made to the implementation of the method, and the repeated parts will not be described in detail.

Referring to FIG. 8 , it is a schematic diagram of a physiological state detection device provided by an embodiment of the present disclosure. The device includes: an acquisition part 801, an extraction part 802, a determination part 803, and a detection part 804; wherein,

The obtaining part 801 is configured to obtain the video stream in the cabin;

The extraction part 802 is configured to perform face detection on multiple frames of images in the video stream, and extract multiple frames of facial images of the target object in the cabin;

The determination part 803 is configured to determine at least one preset smooth area in each frame of the face image as the area of interest;

The detection part 804 is configured to extract physiological state information based on the image information of each region of interest in the multiple frames of facial images, and obtain a detection result of the physiological state of the target object.

Using the above-mentioned physiological state detection device, in the case of obtaining the video stream in the cabin, face detection can first be performed on multiple frames of images in the video stream to extract multiple frames of facial images of the target object in the cabin, Then at least one preset smooth region in each frame of face image can be determined as the region of interest, and the physiological state information can be extracted based on the image information of each region of interest in the multi-frame face image, so as to obtain the physiological state of the target object Status detection result. In the present disclosure, before extracting the physiological state information, by determining the preset smooth area in each frame of the facial image, image information more suitable for physiological state analysis can be found, so that the determined physiological state detection results are also more accurate In addition, the present disclosure adopts a non-contact detection method to realize the physiological state detection, which is convenient for operation.

In a possible implementation manner, the determination part 803 is configured to determine at least one preset smooth area in each frame of facial image as the region of interest according to the following steps: for each frame of facial image, perform facial feature point extraction ; Based on the extracted facial feature points, determine at least one preset smooth area in each frame of the facial image as the region of interest of the facial image.

In a possible implementation manner, the preset smoothing area includes at least one of a forehead area, a left cheek area, a right cheek area, and a chin area.

In a possible implementation manner, when there are multiple regions of interest determined in each frame of facial images, the detection part 804 is configured to follow the steps below based on the Physiological state information extraction from image information: determine the image quality of each ROI in the face image; select the ROI whose image quality meets the preset requirements as the target ROI; based on one or Physiological state information is extracted from the image information of multiple target regions of interest.

In a possible implementation manner, the detection part 804 is configured to determine the image quality of each ROI in the facial image according to the following steps: based on the brightness, occlusion, area, and signal-to-noise ratio of each ROI At least one of , determines the image quality for each region of interest.

In a possible implementation manner, the above-mentioned device further includes: a removal part 805 configured to detect the facial pose of the target object according to each frame of facial image; and determine at least one preset smoothing area in each frame of facial image After the region of interest is used, the region of interest whose visible range in the image does not meet the preset visibility requirements is removed from the region of interest according to the facial pose.

In a possible implementation manner, the image information includes brightness values corresponding to three color channels, and the detection part 804 is configured to perform physiological state information based on the image information of each region of interest in multiple frames of facial images according to the following steps: Extraction to obtain the physiological state detection result of the target object: based on the brightness values of at least one region of interest corresponding to three color channels in the multi-frame face image, determine the temporal brightness of at least one region of interest corresponding to each color channel signal; at least one region of interest corresponding to a plurality of time-domain brightness signals of different color channels is subjected to principal component analysis to obtain a time-domain signal representing the physiological state of the target object; determined based on the time-domain signal representing the physiological state of the target object Physiological state information of the target subject.

In a possible implementation manner, the detecting part 804 is configured to determine the physiological state information of the target object based on the time-domain signal representing the physiological state of the target object according to the following steps: frequency-conducting the time-domain signal representing the physiological state of the target object domain conversion to obtain a frequency domain signal representing the physiological state of the target object; based on the peak value of the frequency domain signal, determine the physiological state information of the target object.

In a possible implementation manner, the above detection part 804 is further configured to: in the case of acquiring one or more frames of images included in the new video stream, repeatedly execute the following steps until the preset detection duration is reached, and obtain The updated physiological state detection result: perform face detection on the image in the new video stream, extract the face image of the target object in the cabin; determine at least one preset smooth area in the face image as the region of interest ; Based on the image information of the region of interest in the face image, the physiological state detection result is updated.

In a possible implementation manner, the above device further includes: a first reminding part 806 configured to generate a detection process reminder signal for reminding the target object of the required detection duration according to the duration of the acquired video stream and the preset detection duration .

In a possible implementation manner, the extracting part 802 is configured to extract multiple frames of facial images of the target object in the cabin according to the following steps: according to the face detection results of the multiple frames of images, and the specified riding position, Multiple frames of facial images of the target object are determined from the detected facial images, wherein the specified riding position is used to indicate the position of the measured target object.

In a possible implementation manner, the above-mentioned device further includes: a display part 807 configured to: transmit the detection result of the physiological state of the target object to the display screen in the vehicle cabin for display on the display screen; The server of the physiological state detection application transmits the physiological state detection result of the target object, so that when the target object requests to obtain the detection result through the physiological state detection application, the physiological state detection result is sent to the terminal device used by the target object through the server.

In a possible implementation manner, the above-mentioned device further includes: a second reminding part 808 configured to: after acquiring the video stream in the cabin, perform body posture detection on the target object in the video stream to obtain the posture of the target object Change information: when the posture change information indicates that the posture of the target object is abnormal and the physiological state detection result does not conform to the preset physiological state information, generate a reminder signal for reminding the associated object of the target object to perform emergency assistance.

For the description of the processing flow of each part in the device and the interaction flow between each part, reference may be made to the relevant description in the above method embodiment, and details are not described here again.

In the embodiments of the present disclosure and other embodiments, a "part" may be a part of a circuit, a part of a processor, a part of a program or software, etc., of course it may also be a unit, and may also be part or non-part.

An embodiment of the present disclosure also provides an electronic device, as shown in FIG. 9 , which is a schematic structural diagram of the electronic device provided by the embodiment of the present disclosure, including: a processor 901 , a memory 902 , and a bus 903 . The memory 902 stores machine-readable instructions executable by the processor 901 (for example, execution instructions corresponding to the acquisition part 801, the extraction part 802, the determination part 803, and the detection part 804 in the apparatus in FIG. , the processor 901 communicates with the storage 902 through the total 903, and when the machine-readable instructions are executed by the processor 901, the following processes are performed:

Obtain the video stream in the cabin;

Perform face detection on multiple frames of images in the video stream, and extract multiple frames of facial images of the target object in the cabin;

Determining at least one preset smooth area in each frame of face image as an area of interest;

Physiological state information is extracted based on the image information of each region of interest in the multiple frames of facial images to obtain a physiological state detection result of the target object.

Embodiments of the present disclosure also provide a computer-readable storage medium, on which a computer program is stored, and when the computer program is run by a processor, the steps of the physiological state detection method described in the foregoing method embodiments are executed. Wherein, the storage medium may be a volatile or non-volatile computer-readable storage medium.

An embodiment of the present disclosure also provides a computer program product, the computer program product carries a program code, and the instructions included in the program code can be used to execute the steps of the physiological state detection method described in the above-mentioned method embodiment, please refer to the above-mentioned method The embodiment will not be repeated here.

Wherein, the above-mentioned computer program product may be implemented by means of hardware, software or a combination thereof. In an optional embodiment, the computer program product is embodied as a computer storage medium, and in another optional embodiment, the computer program product is embodied as a software product, such as a software development kit (Software Development Kit, SDK) and the like.

Those skilled in the art can clearly understand that for the convenience and brevity of description, the working process of the above-described system and device can refer to the corresponding process in the foregoing method embodiments, which will not be repeated here. In the several embodiments provided in the present disclosure, it should be understood that the disclosed systems, devices and methods may be implemented in other ways. The device embodiments described above are only illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or May be integrated into another system, or some features may be ignored, or not implemented. In another point, the mutual coupling or direct coupling or communication connection shown or discussed may be through some communication interfaces, and the indirect coupling or communication connection of devices or units may be in electrical, mechanical or other forms.

The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place, or may be distributed to multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, each unit may exist separately physically, or two or more units may be integrated into one unit.

If the functions are implemented in the form of software function units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium executable by a processor. Based on this understanding, the technical solution of the present disclosure is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including Several instructions are used to make an electronic device (which may be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the methods described in various embodiments of the present disclosure. The aforementioned storage media include: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), magnetic disk or optical disc and other media that can store program codes. .

Finally, it should be noted that: the above-mentioned embodiments are only specific implementations of the present disclosure, and are used to illustrate the technical solutions of the present disclosure, rather than limit them, and the protection scope of the present disclosure is not limited thereto, although referring to the aforementioned The embodiments have described the present disclosure in detail, and those skilled in the art should understand that any person familiar with the technical field can still modify the technical solutions described in the foregoing embodiments within the technical scope disclosed in the present disclosure Changes can be easily imagined, or equivalent replacements can be made to some of the technical features; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present disclosure, and should be included in this disclosure. within the scope of protection. Therefore, the protection scope of the present disclosure should be defined by the protection scope of the claims.

Industrial Applicability

The present disclosure provides a physiological state detection method, device, electronic equipment, and storage medium, wherein the method includes: acquiring a video stream in a vehicle cabin; performing face detection on multiple frames of images in the video stream, and extracting the vehicle cabin Multi-frame facial images of the target object in the frame; determine at least one preset smooth area in each frame of facial images as the region of interest; perform physiological state based on the image information of each region of interest in the multi-frame facial images The information is extracted to obtain the detection result of the physiological state of the target object. The present disclosure can find image information that is more suitable for physiological state analysis by determining the preset smooth area in each frame of facial images, so that the determined physiological state detection results are also more accurate. In addition, the present disclosure It adopts the non-contact detection method to realize the physiological state detection, which is convenient for deployment and implementation.

Claims (29)

1. A method for detecting a physiological state, comprising:

   Obtain the video stream in the cabin;

   Perform face detection on multiple frames of images in the video stream, and extract multiple frames of facial images of the target object in the cabin;

   Determining at least one preset smooth area in each frame of the facial image as an area of interest;

   Physiological state information is extracted based on the image information of each region of interest in the multiple frames of facial images to obtain a physiological state detection result of the target object.
2. The method according to claim 1, wherein said determining at least one preset smooth area in the facial image of each frame as an area of interest comprises:

   For each frame of the facial image, facial feature point extraction is performed;

   Based on the extracted facial feature points, at least one preset smooth area in each frame of the facial image is determined as the region of interest of the facial image.
3. The method according to claim 1 or 2, wherein the preset smoothing area includes at least one of a forehead area, a left cheek area, a right cheek area, and a chin area.
4. The method according to any one of claims 1 to 3, wherein, in the case where there are multiple regions of interest determined in each frame of the face image, the method based on each of the multiple frames of face images The image information of the region of interest is used to extract physiological state information, including:

   determining the image quality of each of said regions of interest in said facial image;

   Selecting the region of interest whose image quality meets the preset requirements as the target region of interest;

   Physiological state information is extracted based on the image information of one or more target regions of interest in the multiple frames of facial images.
5. The method according to claim 4, wherein said determining the image quality of each said region of interest in said face image comprises:

   Based on at least one of the brightness, occlusion, area, and signal-to-noise ratio of each of the regions of interest, the image quality of each region of interest is determined.
6. The method according to any one of claims 1 to 5, wherein the method further comprises:

   detecting the facial pose of the target object according to each frame of the facial image; and

   After determining at least one preset smooth area in each frame of the face image as the area of interest, remove the visible area in the image that does not meet the preset visibility requirements from the area of interest according to the facial pose. area of interest.
7. The method according to any one of claims 1 to 6, wherein the image information includes brightness values corresponding to three color channels, and the image information based on each region of interest in the multi-frame face image Extracting the physiological state information to obtain the physiological state detection result of the target object, including:

   Based on the brightness values of at least one region of interest in the multi-frame facial images corresponding to the three color channels, determine that the at least one region of interest corresponds to a time-domain brightness signal of each of the color channels;

   performing principal component analysis on the time-domain luminance signal corresponding to each of the color channels in the at least one region of interest, to obtain a time-domain signal representing the physiological state of the target object;

   Physiological state information of the target subject is determined based on a time domain signal characterizing the physiological state of the target subject.
8. The method according to claim 7, wherein said determining the physiological state information of the target object based on the time domain signal characterizing the physiological state of the target object comprises:

   performing frequency-domain conversion on the time-domain signal representing the physiological state of the target subject to obtain a frequency-domain signal representing the physiological state of the target subject;

   Physiological state information of the target subject is determined based on the peak value of the frequency domain signal.
9. The method according to any one of claims 1 to 8, wherein the method further comprises:

   In the case of acquiring one or more frames of images included in the new video stream, repeat the following steps until the preset detection duration is reached, and an updated physiological state detection result is obtained:

   Perform face detection on the image in the new video stream, and extract the face image of the target object in the cabin;

   determining at least one preset smooth area in the face image as an area of interest;

   The physiological state detection result is updated based on the image information of the region of interest in the facial image.
10. The method according to claim 9, wherein the method further comprises:

    A detection process reminder signal for reminding the target object of the required detection duration is generated according to the acquired video stream duration and the preset detection duration.
11. The method according to any one of claims 1 to 10, wherein said extracting multiple frames of facial images of the target object in the cabin comprises:

    According to the face detection results of the multi-frame images and the designated riding position, determine the multi-frame facial images of the target object from the detected facial images, wherein the designated riding position is used to indicate the position of the target object being measured.
12. The method according to any one of claims 1 to 11, wherein the method further comprises:

    transmitting the detection result of the physiological state of the target object to a display screen in the vehicle cabin for display on the display screen; and/or

    transmitting the physiological state detection result of the target object to the server of the physiological state detection application, so that when the target object requests to obtain the detection result through the physiological state detection application, the server sends the detection result to the The terminal device used by the target object sends the physiological state detection result.
13. The method according to any one of claims 1 to 12, wherein, after acquiring the video stream in the cabin, the method further comprises:

    Performing body posture detection on the target object in the video stream to obtain posture change information of the target object;

    After obtaining the physiological state detection result of the target object, the method further includes:

    When the posture change information indicates that the posture of the target object is abnormal and the physiological state detection result does not conform to a preset physiological state value, a reminder signal for reminding an associated object of the target object to perform emergency assistance is generated.
14. A physiological state detection device, comprising:

    The acquisition part is configured to acquire the video stream in the cabin;

    The extraction part is configured to perform face detection on multiple frames of images in the video stream, and extract multiple frames of facial images of the target object in the vehicle cabin;

    A determining part configured to determine at least one preset smooth area in each frame of the facial image as an area of interest;

    The detection part is configured to extract physiological state information based on the image information of each region of interest in the multiple frames of facial images, and obtain a detection result of the physiological state of the target object.
15. The device according to claim 14, wherein the determining part is configured to determine at least one preset smooth area in each frame of the face image as an area of interest according to the following steps, comprising: image, extracting facial feature points; based on the extracted facial feature points, determining at least one preset smooth area in each frame of the facial image as the region of interest of the facial image.
16. The apparatus according to claim 14 or 15, wherein the preset smoothing area comprises at least one of a forehead area, a left cheek area, a right cheek area, and a chin area.
17. The device according to any one of claims 14 to 16, wherein, when there are multiple regions of interest determined in each frame of the face image, the detection part is configured to follow the steps below based on the multiple The image information of each region of interest in the frame facial image is extracted from the physiological state information: the image quality of each region of interest in the facial image is determined; the image quality of the image meets the preset requirements. The region is used as a target region of interest; physiological state information is extracted based on the image information of one or more target regions of interest in the multiple frames of facial images.
18. The apparatus according to claim 17, wherein the detection part is configured to determine the image quality of each of the regions of interest in the face image by the following steps: based on the brightness of each of the regions of interest, At least one of occlusion, area, and signal-to-noise ratio to determine the image quality of each region of interest.
19. The device according to any one of claims 14 to 18, wherein the device further comprises a removal part configured to detect the facial pose of the target object according to the facial image of each frame; After at least one preset smooth area in the face image is used as the area of interest, the area of interest whose visible range in the image does not meet the preset visibility requirements is removed from the area of interest according to the facial pose.
20. The device according to any one of claims 14 to 19, wherein the image information includes brightness values corresponding to three color channels, and the detection part is configured to follow the steps below based on Physiological state information is extracted from the image information of each region of interest, and the physiological state detection result of the target object is obtained: based on the brightness values corresponding to the three color channels in at least one region of interest in the multi-frame facial images , determining that the at least one region of interest corresponds to the time-domain luminance signal of each of the color channels; performing principal component analysis on the time-domain luminance signal of the at least one region of interest corresponding to each of the color channels, to obtain A time domain signal characterizing the physiological state of the target object; determining physiological state information of the target object based on the time domain signal characterizing the physiological state of the target object.
21. The apparatus according to claim 20, wherein the detection part is configured to determine the physiological state information of the target subject based on the time-domain signal characterizing the physiological state of the target subject according to the following steps: The time-domain signal of the physiological state is converted into the frequency domain to obtain a frequency-domain signal representing the physiological state of the target object; based on the peak value of the frequency-domain signal, the physiological state information of the target object is determined.
22. The device according to any one of claims 14 to 21, wherein the detection part is further configured to repeatedly execute the following steps when one or more frames of images included in the new video stream are acquired until the predetermined Assume the detection duration to obtain an updated physiological state detection result: perform face detection on the image in the new video stream, extract the face image of the target object in the cabin; determine the face At least one preset smooth area in the image is used as an area of interest; and the physiological state detection result is updated based on the image information of the area of interest in the face image.
23. The device according to claim 22, wherein the device further comprises a first reminding part configured to generate a detection duration required for reminding the target object according to the duration of the acquired video stream and the preset detection duration The detection process reminder signal.
24. The device according to any one of claims 14 to 23, wherein the extracting part is configured to propose multiple frames of facial images of the target object in the cabin according to the following steps, comprising: according to the multiple frames of images The face detection result, and the specified riding position, determine the multi-frame facial images of the target object from the detected facial images, wherein the specified riding position is used to indicate the measured target object s position.
25. The device according to any one of claims 14 to 24, wherein the device further includes a display part configured to transmit the detection result of the physiological state of the target object to a display screen in the vehicle cabin, so as to display on the display screen; and/or, transmit the physiological state detection result of the target object to the server of the physiological state detection application, so that when the target object requests to obtain the detection result through the physiological state detection application Next, send the physiological state detection result to the terminal device used by the target object through the server.
26. The device according to any one of claims 14 to 25, wherein the device further comprises a second reminding part configured to, after acquiring the video stream in the cabin, make a body gesture to the target object in the video stream detection to obtain the posture change information of the target object; when the posture change information indicates that the posture of the target object is abnormal and the physiological state detection result does not meet the preset physiological state value, generate a message for reminding the A reminder signal for emergency rescue by the associated object of the target object.
27. An electronic device, comprising: a processor, a memory, and a bus, the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor communicates with the memory through the bus , when the machine-readable instructions are executed by the processor, the steps of the physiological state detection method according to any one of claims 1 to 13 are executed.
28. A computer-readable storage medium, where a computer program is stored on the computer-readable storage medium, and when the computer program is run by a processor, the steps of the physiological state detection method according to any one of claims 1 to 13 are executed.
29. A computer program product, the computer program product comprising a computer program or an instruction, when the computer program or instruction is run on an electronic device, the electronic device is made to execute any one of claims 1 to 13 The steps of the physiological state detection method.

Images