WO2023123214A1

WO2023123214A1 - Electronic device, hand compression depth measurement method, system, and wearable device

Info

Publication number: WO2023123214A1
Application number: PCT/CN2021/143106
Authority: WO
Inventors: 焦旭
Original assignee: 焦旭
Priority date: 2021-12-30
Filing date: 2021-12-30
Publication date: 2023-07-06
Also published as: CN114556446A

Abstract

An electronic device, comprising a processor and executing the following steps: acquiring a video provided by an image acquisition device, the video comprising multiple images of a video frame sequence generated according to a time sequence, each image comprising a wearable device worn near a compressed part when a user performs a compression on a patient, and a positioning mark image of the wearable device is arranged on the wearable device to serve as a tracking box; and identifying, according to at least one image in the video frame sequence, the tracking box on the wearable device in the image; tracking the change in rise and fall of the position of the tracking box in a compression direction in the multiple images in the video frame sequence; and determining a depth of a compression action of the user according to the change in the rise and fall of the position, and outputting the depth of the compression action to the electronic device and/or the wearable device. The electronic device improves the accuracy of compression depth.

Description

Electronic device, hand pressing depth detection method, system and wearable device

technical field

The present application belongs to the technical field of image processing, and in particular, relates to an electronic device, a hand pressing depth detection method, a system, and a wearable device

Background technique

An estimated 330,000 deaths from out-of-hospital cardiac arrest (OHCA) occur each year in Canada and the United States. In treated OHCA, overall survival is poor, with discharge rates ranging from 3.0% to 16.3%. The difference in patient survival can be attributed in part to the following 5 important links: rapid emergency medical system (ENS) access, early cardiopulmonary resuscitation (CPR), early defibrillation, early advanced life support (ACLS), and effective post-resuscitative care. In recent years, after the communities and hospitals have vigorously strengthened the above-mentioned links, the survival rate has only slightly improved. It has been recognized that the quality, frequency, and timeliness of CPR during cardiac resuscitation are critical to survival in cardiac arrest. Related studies have found that the increase in compression depth during CPR is closely related to the degree of improvement in survival rate, and when the compression depth is 4.03-5.53cm (peak value 4.56cm), the patient's survival rate is the highest. However, in the actual process of cardiac resuscitation, it is difficult to evaluate the compression depth in cardiac resuscitation due to the lack of tools for measuring depth. The traditional contact measurement has high precision and good stability, but it is not widely used due to factors such as volume, quality, installation conditions, structure, and inconvenient operation; Distance measurement is carried out with the patient through contact instruments.

The accuracy of the current hand pressing depth detection method needs to be further improved.

Contents of the invention

In order to improve the deficiencies of the above-mentioned prior art, the present application proposes an electronic device, a hand pressing depth detection method, a system, and a wearable device.

In some implementations, an electronic device is proposed, which is used to detect the depth of the user's pressing action in a scene where the user provides cardiopulmonary resuscitation for the patient, and the electronic device includes at least a processor;

The processor is configured to perform the following steps:

Acquiring the video provided by the image acquisition device, the video includes multiple images of video frame sequences generated in time sequence; wherein each of the images includes a wearable device worn near the pressing part when the user presses the patient, The wearable device is provided with a positioning mark image of the wearable device as a tracking frame;

Identifying a tracking frame in the wearable device in the image according to at least one frame image in the sequence of video frames;

Tracking fluctuations in the position of the tracking frame in the pressing direction in multiple images in the sequence of video frames;

Determining the depth of the pressing action of the user according to the fluctuation of the position and outputting the depth of the pressing action to the electronic device and/or the wearable device.

In some embodiments, identifying a tracking frame in a wearable device in the image according to at least one frame image in the sequence of video frames specifically includes:

At the same time, the tracking frame is recognized according to the preset shape and color of the background image of the wearable device, and according to the preset color and shape of the partial image in the background image.

converting the at least one image to the HSV color space, and extracting color features in the image, the extracted color features are consistent with preset color features, and the preset color features include the preset The color characteristics of the background image and the preset color characteristics of the partial image;

In the image region corresponding to the extracted color feature, according to the shape and color of the preset background image and the shape and color of the partial image, extract the directional gradient histogram feature and the gray level co-occurrence matrix feature;

Using the directional gradient histogram feature and the gray level co-occurrence matrix feature as the input of the classifier to identify the tracking frame in the wearable device;

The classifier is pre-trained, and the pre-set shape and color of the background image and the shape and color of the partial image are used as input when the classifier is trained.

In some embodiments, the preset background image is a red rectangular area or a green rectangular area; the preset partial image includes multiple white rectangular areas in the red rectangular area or the green rectangular area and a black rectangular area;

One of the plurality of white rectangular areas is located at the center of the red rectangular area or the green rectangular area, and the plurality of white rectangular areas are at least partially located on at least two opposite sides of the red rectangular area or the green rectangular area, so The position where the white square area is located intersects with the median line of the rectangle.

In some implementations, the image acquisition device is a camera on the electronic device, and the user wearing the wearable device is photographed by the camera;

The wearable device is a wristband, the wristband includes a wristband, and the wristband is provided with a hard square area, the background color of the hard square area is red or green, and the hard square area A white square area is provided in the center, and/or white square areas are respectively provided on at least two opposite sides of the hard square area, and the position of the white square area intersects the median line of the rectangle.

In some implementation manners, tracking the ups and downs of the position of the tracking frame in the pressing direction in the plurality of images in the sequence of video frames specifically includes:

A KCF algorithm is used to track the tracking frame, and the KCF algorithm at least fuses the histogram feature of the directional gradient, the color domain and the score of the classifier.

In some implementations, the depth of the user's pressing action is determined according to the fluctuation of the position, and the depth of the pressing action is output to the display of the electronic device and/or the display of the wearable device, or on the display Or output voice prompt information on the wearable device.

In some embodiments of the present application, a wearable device is provided. The wearable device is used in the above-mentioned electronic device scene, and is used for the user to provide cardiopulmonary resuscitation for the patient. In the scene where the user provides cardiopulmonary resuscitation, the wearable device is worn near the user's compression site, so that the camera captures the user providing cardiopulmonary resuscitation for the patient. The recovered video image is provided to the electronic device, and the electronic device identifies the wearable device in the video image and tracks the tracking frame on the wearable device;

The wearable device is provided with a hard square area, the background color of the hard square area is red or green, and the hard square area is provided with a white square area.

In some embodiments, the hard square area is provided with a white square area, which specifically includes:

The center of the hard square area is provided with a white square area; and/or

A white square area is provided on at least two of the four sides of the hard square area; and/or

Black strip-shaped areas are arranged on two median lines of the rigid square area, and the black strip-shaped areas extend from the center of the rigid square area to one or more sides of the four sides.

In some embodiments, white lines are arranged around the red or green background image of the hard square area, and the white line is the outline of the red or green background image of the hard square area.

In some implementations, the tracking frame includes one of the outline of the white square area, the outline of the black bar area, the outline of the red or green background image, or any two or more combination.

In some embodiments, the wearable device is a wristband worn on the wrist, the wristband includes a wristband, and the hard square area is provided on the wristband.

In some implementations, the background colors of the plurality of hard square regions are different and do not constitute central symmetry or axis symmetry.

The embodiment of the present application provides a hand pressing depth detection method, including:

An embodiment of the present application provides a hand pressing depth detection system, including an electronic device and a wearable device;

The electronic equipment includes:

one or more processors;

A storage device, on which one or more programs are stored, and when the one or more programs are executed by the one or more processors, the one or more processors implement the described method.

An embodiment of the present application provides a non-transitory computer-readable storage medium on which a computer program is stored, wherein, when the computer program is executed by a processor, the method described in any one of the above-mentioned embodiments is implemented.

The electronic device provided by the present application is used to detect the depth of the user's pressing action in the scene where the user provides cardiopulmonary resuscitation for the patient. The electronic device includes at least a processor; the processor is configured to acquire the video provided by the image acquisition device, and the video Contains a plurality of images of video frame sequences generated in time order; wherein, each of the images contains a wearable device worn near the pressing part when the user presses the patient, and the wearable device is set on the wearable device The positioning mark image of the video frame sequence is used as a tracking frame; and the tracking frame in the wearable device in the image is identified according to at least one frame image in the video frame sequence; and the tracking frame in a plurality of images in the video frame sequence is tracked The ups and downs of the position of the frame in the pressing direction; and determining the depth of the user's pressing action according to the ups and downs of the position and outputting the depth of the pressing action to the electronic device and/or the wearable device, so as to improve the user experience. The accuracy of the depth detection of the pressing action.

Description of drawings

The features and advantages of the present application will be more clearly understood by referring to the accompanying drawings, which are schematic and should not be construed as limiting the application in any way. In the accompanying drawings:

Fig. 1 is a schematic diagram of the system architecture of the hand pressing depth detection method and detection device operation in some examples of the present application;

FIG. 2 is a schematic diagram of a video shot in a hand pressing depth detection method in some embodiments of the present application;

Fig. 3 is a schematic diagram of the structure of the bracelet in some embodiments of the present application;

Fig. 4 is a schematic structural diagram of a wristband in some embodiments of the present application;

Fig. 5 is a schematic structural diagram of a wristband in some embodiments of the present application;

Fig. 6 is a schematic flow chart of a hand pressing depth detection method in some embodiments of the present application;

Fig. 7 is a schematic diagram of the detection result of the tracking coordinate position in the hand pressing depth detection method in some embodiments of the present application;

Fig. 8 is a schematic diagram of the detection result of the tracking coordinate position in the hand pressing depth detection method in some embodiments of the present application;

FIG. 9 is a schematic diagram of a tracking frame detected by a hand pressing depth detection method in some embodiments of the present application;

Fig. 10 is a schematic flow diagram of the implementation of identifying the tracking frame in the wearable device in the image according to at least one frame image in the video frame sequence in some embodiments of the present application;

Fig. 11 is a schematic structural diagram of a computer system suitable for realizing the control device of the embodiment of the present application. .

Detailed ways

In order to better understand the above-mentioned purpose, features and advantages of the present application, the present application will be further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that, in the case of no conflict, the embodiments of the present application and the features in the embodiments can be combined with each other.

In the following description, many specific details are set forth in order to fully understand the application, but the application can also be implemented in other ways different from those described here, therefore, the protection scope of the application is not limited by the specific details disclosed below. EXAMPLE LIMITATIONS.

Fig. 1 shows an exemplary system architecture 100 that can be applied to embodiments of the present application, such as a hand pressing depth detection system, a hand pressing depth detection method, a hand pressing depth detection device, an electronic device, and a wearable device.

As shown in FIG. 1 , a system architecture 100 may include a terminal device 101 , a terminal device 102 , a terminal device 103 , a network 104 and a server 105 . The network 104 is used as a medium for providing communication links between the terminal device 101 , the terminal device 102 , the terminal device 103 and the server 105 . Network 104 may include various connection types, such as wires, wireless communication links, or fiber optic cables, among others.

A user may use one or more of the terminal device 101 , the terminal device 102 , and the terminal device 103 to interact with the server 105 through the network 104 to receive or send data (such as video) and the like. Various communication client applications can be installed on the terminal device 101, the terminal device 102, and the terminal device 103, such as video playback software, video processing applications, web browser applications, shopping applications, search applications, instant messaging tools, mailboxes, etc. Client, social platform software, etc.

The terminal device 101, the terminal device 102, and the terminal device 103 can be hardware, such as various electronic devices that have a display screen and support data transmission, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, Smart wearable devices and more. Among them, the smart wearable device may be smart glasses, smart bracelets, smart helmets, and the like.

When the terminal device 101, the terminal device 102, and the terminal device 103 are software, they can be installed in the electronic devices listed above. It can be implemented as multiple software or software modules (such as software or software modules for providing distributed services), or as a single software or software module. No specific limitation is made here.

The server 105 may be a server that provides various services, for example, a background server that provides support for videos displayed on the terminal device 101 , the terminal device 102 , and the terminal device 103 . The background server can analyze and process the received data such as slicing requests, and feed back the processing results (such as indexed slices or slicing sequences) to electronic devices (such as terminal devices) connected in communication with it.

It should be noted that the hand pressing depth detection method provided by the embodiment of the present application can be executed by a processor, and correspondingly, the computer program of the hand pressing depth detection method can be stored in a non-volatile computer-readable storage medium, The instructions of the computer-readable storage medium can be obtained and executed by a processor.

The processor and memory can be placed in a terminal device, such as a mobile phone or a computer or a wearable device, or the processor and memory can be placed in a server. Alternatively, the processor and the memory may be one or more, and if there are more than one, part of the processor or part of the memory may be used for the server, and part of the memory may be used for the terminal device.

It should be noted that the server may be hardware or software. When the server is hardware, it can be implemented as a distributed server cluster composed of multiple servers, or as a single server. When the server is software, it can be implemented as multiple software or software modules (such as software or software modules for providing distributed services), or as a single software or software module. No specific limitation is made here.

It should be understood that the numbers of terminal devices, networks and servers in Fig. 1 are only illustrative. According to the implementation needs, there can be any number of terminal devices, networks and servers. When the electronic device on which the live stream playback method runs does not need to perform data transmission with other electronic devices, the system architecture may only include the electronic devices on which the hand pressing depth detection method runs (such as

terminal devices

101, 102, 103 or server 105).

The hand compression depth detection method of the present application is used to detect the depth of the user's compression action in the scene where the user provides cardiopulmonary resuscitation for the patient. The compression depth is realized by ranging, and the accuracy of ranging accuracy affects the accuracy of compression depth.

There are many factors that affect the ranging accuracy, but the influence of hardware factors can be limited by selecting high-quality hardware such as high-resolution CCD cameras and high-sampling frequency image acquisition cards, and various environmental factors. Because the styles of mobile phones are different, it is a relatively effective way to improve the ranging accuracy of the system through software algorithms.

The currently tried solution is to realize compression depth detection through image distance measurement. For example, add a calibration object (for example, with obvious color characteristics and known length and width, etc.) in the pressing scene, and use this calibration object to calibrate the camera and correct the image, so as to obtain the distance accuracy of the image and the measurement distance of the target. However, it is difficult to set a calibration object whose length and width are known during heart compressions.

The distance measurement scheme of the embodiment of the present application is: combining the traditional scale measurement idea, through digital image processing, extracting the wearable device (such as a wristband) when pressing, and using the algorithm to assist the calculation to complete the guarantee through the vertical distance that the wearable device moves. Ranging with speed and precision.

In order to measure the depth of the user's pressing action more accurately and quickly, this application proposes a system, as shown in Figure 2, the system includes: electronic devices (such as mobile phone 01), wearable devices (such as bracelet 02);

Exemplarily, as shown in FIG. 2 , the mobile phone 01 has a shooting function or can shoot a video in real time to obtain video data, and the video content includes the action of the user providing cardiopulmonary resuscitation compressions for the patient. The mobile phone is also used for processing video data, and identifying the user's pressing action and pressing depth in the video data.

Identifying the pressing depth can identify the ups and downs of the pressing hand used by the user to determine the pressing depth. In order to improve the accuracy of recognition and avoid confusion between the color of the hand and the similar color in the video, the detection may be reckless, and a calibration object can also be worn on the pressing hand, and by tracking the ups and downs of the calibration object with the hand pressing, determine Depth of hand compressions.

In Fig. 2, the bracelet 02 worn on the user's wrist is a calibration object worn near the pressing action during the detection of the pressing depth, which is convenient for the mobile phone to quickly identify the calibration object in the picture and track the calibration object. According to the calibration object The variation fluctuation determines the compression depth, so as to improve the accuracy of compression depth detection.

The calibration object is a reference object when detecting the fluctuation position of the pressing action, and the calibration object can be the whole wristband, or a combination of one or more features in the pattern/color of the local position in the wristband.

In order to reduce the possibility of reckless detection and improve detection accuracy, the embodiment of the present application provides a wearable device as a calibration object.

A wearable device, which is used in the scene where the user provides cardiopulmonary resuscitation for the patient and is worn near the user's compression site, so that the camera captures the video image of the user providing cardiopulmonary resuscitation for the patient while shooting the wearable device, and provides the video to the electronic device, The electronic device identifies the wearable device in the video image and tracks the tracking frame on the wearable device, so that the electronic device determines the compression depth according to the up and down position changes of the tracking frame.

In order to prevent the wearable device from deforming easily when it is soft, which will increase the difficulty of detection or reduce the accuracy of detecting the pressing depth, the wearable device is provided with a hard square area, and the background color of the hard square area can be white or red. , yellow or green etc.

Exemplarily, referring to FIG. 3 to FIG. 5 , a wearable device is provided as a wristband, and the wristband is configured to be worn on a user's wrist. The wristband includes a wristband 20, and the wristband 20 is provided with a hard square area 21, and the background color of the hard square area 21 is a red or green area as shown in FIG. 4 or FIG. 5 .

In some embodiments, the hard square area is provided with a white square area, which specifically includes: the center of the hard square area is provided with a white square area; and/or at least two of the four sides of the hard square area are White square areas are arranged on the opposite sides; and/or black strip-shaped areas are arranged on the two median lines of the hard square area, and the black bar-shaped areas extend from the center of the hard square area to on one or more of the four sides.

Exemplarily, referring to FIGS. 3 to 5 , the center of the hard square area 21 is provided with a white square area 22 , and/or at least two opposite sides of the hard square area 21 are respectively provided with white A square area 23, where the white square area 23 intersects with the median line of the rectangle. Two median lines of the hard square area 21 are provided with black strip-shaped areas 24, and the black strip-shaped areas 24 extend from the center of the hard square area to four sides.

In some embodiments, white lines are arranged around the red or green background image of the hard square area, and the white lines are the outline of the red or green background image of the hard square area.

Exemplarily, referring to FIGS. 3 to 5 , white lines 25 are arranged around the red or green background image of the hard square area 21 . The white line is the outline of the red or green background image of the hard square area 21 .

The above-mentioned electronic device identifies the wearable device in the video image and tracks the tracking frame on the wearable device, so that the electronic device determines the compression depth according to the up and down position changes of the tracking frame, wherein the tracking frame includes the white One of the outline of the square area, the outline of the black strip area, the outline of the red or green background image, or any combination of two or more.

In some embodiments, the background colors of the plurality of hard square regions 21 are different and do not constitute central symmetry or axis symmetry. For example the top two are green, the bottom left is red, and the bottom right is blue. It should be understood by those skilled in the art that other asymmetrical color combinations are also possible. This asymmetric color combination can avoid the problem of inaccurate recognition caused by symmetry in some positioning algorithms (such as the feature point positioning scene of the harr i s algorithm), thereby further improving the recognition effect.

Because the figures in Figures 3 to 5 are the same, only the colors are different, therefore, each reference numeral shown in Figure 4 is applicable to the corresponding figures in Figures 3 and 5, and Figures 3 and 5 do not carry reference numerals.

The hand pressing depth detection method provided in the embodiment of the present application includes the steps shown in Figure 6:

S1: the processor acquires the video provided by the image acquisition device, the video includes a plurality of images of a sequence of video frames generated in time order;

The processor may be a processor on an electronic device, for example, a processor on a mobile terminal, or a processor on a server, or a processor on a wearable device.

The image acquisition device may be a camera on the electronic device or a camera independent of the electronic device.

The video is a video taken by a camera received by the processor in real time, or an offline video.

Wherein, each of the images includes a wearable device worn near the pressing part when the user presses the patient, and an image of a positioning mark of the wearable device is set on the wearable device as a tracking frame.

Exemplarily, it is assumed that the processor is a processor on a mobile phone, and the image acquisition device is a front camera or a rear camera on the mobile phone. As shown in Figure 2, the user provides cardiopulmonary resuscitation for the patient in real time, and the camera on the mobile phone captures the picture of pressing the patient in real time, and the user wears the above-mentioned wearable device, such as a bracelet, when pressing the patient. The picture is provided to the processor of the mobile phone for processing, and the processor obtains multiple images according to the acquired video.

S2: The processor recognizes a tracking frame in the wearable device in the image according to at least one frame image in the video frame sequence;

The positioning marker image that can be used as the tracking frame in step S1 is pre-set in the detection method.

Step S2, some implementations are: the tracking frame can be recognized at least according to the preset shape and color of the background image of the wearable device, and according to the preset color and shape of the partial image in the background image.

Exemplarily, after a period of video is captured by the mobile phone, the processor determines the tracking frame according to the received video of the period. For example, image 1, image 2, image 3, . Described image 1, image 2, image 3, ..., image 30 are the actual image according to time order, also can be sampling value, for example, every interval 5 actual pictures sample an image as image 1, image 2, image 3, ..., image 30.

S3: The processor tracks the ups and downs of the position of the tracking frame in the pressing direction in the plurality of images in the sequence of video frames;

In some implementation manners, the tracking frame is tracked using a KCF algorithm, and the KCF algorithm at least fuses the histogram feature of oriented gradients, the color domain, and the score of the classifier.

Exemplarily, after the tracking frame is determined in step S2, the tracking frame is then tracked in subsequent video images, and the coordinates of the highest point and the lowest point of the tracking frame in each image are determined to obtain a series of coordinate positions, as shown in Figure 7 Show.

In FIG. 7, the abscissa is a plurality of images of the sequence of video frames generated in time order, and the ordinate is the coordinates of the highest point and the lowest point of the motion track of the tracking frame.

After removing the abnormal points, only a set of extreme points (the highest point and the lowest point) are selected for each pressing. In this paper, the peak pressing values of the wristband are selected in the 1-100 and 500-600 frame intervals for measurement, and the results are shown in the figure As shown in 8, it is the peak and valley information of the motion height. X and Y in accompanying drawing 8 are the abscissa of data, and Y is the ordinate of numerical value.

S4: The processor determines the depth of the user's pressing action according to the fluctuation of the position and outputs the depth of the pressing action to the electronic device and/or the wearable device;

In some implementations, the depth of the user's pressing action is determined according to the fluctuation of the position, and the depth of the pressing action is output to the display of the electronic device and/or the display of the wearable device, or on the display or Voice prompt information is output on the wearable device.

Exemplarily, the pressing depth is output to at least one or a combination of the following, such as output to the display screen of the mobile phone, output in the form of voice broadcast, output to the user's wearable device or other devices, such as smart glasses or smart watches worn by the user , and other terminal devices for monitoring or reference by others, such as telemedicine doctors, or family members of patients, etc.

In the above step 2, the tracking frame in the wearable device in the image is identified according to at least one frame image in the video frame sequence.

In an actual application scenario, for example, as shown in FIG. 9 , the tracking frame of the wristband is obtained, and the tracking frame includes a hard square area and a part of the wristband.

In the above steps 3 to 4, firstly, the trajectory map is given for the extreme points, and according to the number of frames corresponding to the trajectory, the corresponding calibration object (such as a wristband) is processed to obtain the corresponding calibration object (such as a wristband), and the pixel width of the wristband appearing in the image is calculated. The ratio of the actual length of the ring to the pixel width in the image, the width of the bracelet can be effectively extracted through the canny edge and the circumscribed border.

Specifically, in step 2, identify the tracking frame in the wearable device in the image according to at least one frame image in the video frame sequence, specifically including the steps shown in Figure 10:

S21: The processor converts the original image in the at least one acquired video to the HSV color space, and extracts the color features in the image, the extracted color features are consistent with the preset color features, and the preset The color features include the preset color features of the background image and the preset color features of the partial image;

Exemplarily, the image when the bracelet shown in FIG. 4 or 5 is worn by the user is acquired, and the green/red color in the image is extracted. In order to further quickly and accurately propose a target red or target green rectangular area, it is necessary to extract a green or red area with black and white lines nearby, so it is necessary to extract white, black, red, and green areas. The preset background color is rectangular red or rectangular green, and the preset color feature of the partial image is at least one of black and white.

S22: In the image region corresponding to the extracted color feature, the processor filters the color according to the shape and color of the preset background image and the shape and color of the partial image, and extracts a directional gradient histogram features and gray level co-occurrence matrix features;

Exemplarily, for the wristband as shown in FIG. 4 or 5, after the color features are obtained in step 21, the target features are further retained according to the following preset features, and the interference area is filtered.

Referring to Figures 4-5, the preset features that need to be preserved are as follows:

the red or green area is rectangular or square; and/or

The red or green area has a white square at its center; and/or

At least two of the four sides of the red or green rectangular area are provided with white squares on opposite sides; and/or

Black strip-shaped areas are arranged on the two median lines of the red or green rectangular area, and the black strip-shaped areas extend from the center of the hard square area to one or more sides of the four sides .

On the basis of the color-filtered features, the directional gradient histogram feature and the gray level co-occurrence matrix feature are extracted based on the extracted color and graphic features.

The Histogram of Oriented Gradient (HOG) feature is a feature descriptor used for object detection in computer vision and image processing. The HOG feature constitutes a feature by calculating and counting the gradient orientation histogram of the local area of the image. The HOG feature is a feature of an image.

Gray level co-occurrence matrix is a common method to describe texture by studying the spatial correlation characteristics of gray level. The gray level histogram is the statistical result of a single pixel on the image having a certain gray level, while the gray level co-occurrence matrix is obtained by making statistics on the condition that two pixels with a certain distance on the image have a certain gray level respectively.

S23: The processor uses the directional gradient histogram feature and the gray level co-occurrence matrix feature as an input of the classifier to identify the tracking frame in the wearable device;

The method described above in FIG. 6 can be executed by a processor on an electronic device, and the present application provides an electronic device for detecting the depth of a user's pressing action in a scene where a user provides cardiopulmonary resuscitation for a patient. The electronic device includes at least a processing device;

The processor is configured to perform the following steps:

The method described in FIG. 6 above can be applied to a hand pressing depth detection system, and the system includes electronic equipment and wearable equipment;

The electronic equipment includes:

one or more processors;

A storage device, on which one or more programs are stored, and when the one or more programs are executed by the one or more processors, the one or more processors implement the following steps:

The method described above in FIG. 6 may be an instruction stored in a non-transitory computer-readable storage medium, and the above steps are implemented when the instruction is executed, that is, to realize:

Referring now to FIG. 11 , it shows a schematic structural diagram of a computer system 800 suitable for implementing the control device of the embodiment of the present application. The control device shown in FIG. 11 is only an example, and should not limit the functions and scope of use of this embodiment of the present application.

As shown in FIG. 11 , a computer system 800 includes a central processing unit (CPU) 801, which can operate according to a program stored in a read-only memory (ROM) 802 or a program loaded from a storage section 808 into a random access memory (RAM) 803 Instead, various appropriate actions and processes are performed. In the RAM 803, various programs and data required for the operation of the system 800 are also stored. The CPU 801, ROM 802, and RAM 803 are connected to each other via a bus 804. An input/output (I/O) interface 805 is also connected to the bus 804 .

The following components are connected to the I/O interface 805: an input section 806 including a keyboard, a mouse, etc.; an output section 807 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker; a storage section 808 including a hard disk, etc. and a communication section 809 including a network interface card such as a LAN card, a modem, or the like. The communication section 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to the I/O interface 805 as needed. A removable medium 811, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is mounted on the drive 810 as necessary so that a computer program read therefrom is installed into the storage section 808 as necessary.

In particular, according to an embodiment of the present disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product, which includes a computer program carried on a computer-readable medium, where the computer program includes program codes for executing the methods shown in the flowcharts. In such an embodiment, the computer program may be downloaded and installed from a network via communication portion 809 and/or installed from removable media 811 . When the computer program is executed by the central processing unit (CPU) 801, the above-mentioned functions defined in the method of the present application are performed.

It should be noted that the computer-readable medium described in this application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the two. A computer readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, electrical connections with one or more wires, portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable Programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination of the above. In this application, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, however, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, in which computer-readable program codes are carried. Such propagated data signals may take many forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device. . Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

Computer program code for carrying out the operations of this application may be written in one or more programming languages, or combinations thereof, including object-oriented programming languages—such as Python, Java, Smalltalk, C++, and Includes conventional procedural programming languages - such as the "C" language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In cases involving a remote computer, the remote computer can be connected to the user computer through any kind of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (such as through an Internet service provider). Internet connection).

The flowchart and block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present application. In this regard, each block in a flowchart or block diagram may represent a module, program segment, or portion of code that contains one or more logical functions for implementing specified executable instructions. It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or they may sometimes be executed in the reverse order, depending upon the functionality involved. It should also be noted that each block of the block diagrams and/or flowchart illustrations, and combinations of blocks in the block diagrams and/or flowchart illustrations, can be implemented by a dedicated hardware-based system that performs the specified functions or operations , or may be implemented by a combination of dedicated hardware and computer instructions.

The units involved in the embodiments described in the present application may be implemented by means of software or by means of hardware. The described units may also be set in a processor, for example, it may be described as: a processor includes an acquiring unit, a dividing unit, a determining unit and a selecting unit. Wherein, the names of these units do not constitute a limitation on the unit itself under certain circumstances, for example, the acquisition unit may also be described as "a unit for acquiring picture book images to be processed".

As another aspect, the present application also provides a computer-readable medium. The computer-readable medium may be included in the electronic device described in the above embodiments; it may also exist independently without being assembled into the electronic device. middle. The above-mentioned computer-readable medium carries one or more programs, and when the above-mentioned one or more programs are executed by the electronic device, the electronic device: reads the target video, and the target video is pressed by the fixed shooting device against the hand Actions are collected to obtain the video, and the calibration object is worn on the hand; the calibration object is detected on the target video, and the tracking frame of the calibration object is obtained; the tracking frame is tracked, and the calibration object is obtained The highest point and the lowest point in the pressing direction; according to the coordinates of the highest point and the lowest point, the pressing depth of the hand is obtained.

The above description is only a preferred embodiment of the present application and an illustration of the applied technical principle. Those skilled in the art should understand that the scope of application involved in this application is not limited to the technical solutions formed by the specific combination of the above technical features, but also covers the technical solutions made by the above technical features or Other technical solutions formed by any combination of equivalent features. For example, a technical solution formed by replacing the above-mentioned features with technical features with similar functions disclosed in (but not limited to) this application.

Claims

An electronic device, characterized in that it is used to detect the depth of the user's pressing action in the scene where the user provides cardiopulmonary resuscitation for the patient, and the electronic device includes at least a processor;

The processor is configured to perform the following steps:

Acquiring the video provided by the image acquisition device, the video includes multiple images of video frame sequences generated in time sequence; wherein each of the images includes a wearable device worn near the pressing part when the user presses the patient, The wearable device is provided with a positioning mark image of the wearable device as a tracking frame;

Identifying a tracking frame in the wearable device in the image according to at least one frame image in the sequence of video frames;

Tracking fluctuations in the position of the tracking frame in the pressing direction in multiple images in the sequence of video frames;

Determining the depth of the pressing action of the user according to the fluctuation of the position and outputting the depth of the pressing action to the electronic device and/or the wearable device.
The electronic device according to claim 1, wherein identifying the tracking frame in the wearable device in the image according to at least one frame image in the sequence of video frames specifically includes:

At the same time, the tracking frame is recognized according to the preset shape and color of the background image of the wearable device, and according to the preset color and shape of the partial image in the background image.
The electronic device according to claim 2, wherein identifying the tracking frame in the wearable device in the image according to at least one frame image in the sequence of video frames specifically includes:

converting the at least one image to the HSV color space, and extracting color features in the image, the extracted color features are consistent with preset color features, and the preset color features include the preset The color characteristics of the background image and the preset color characteristics of the partial image;

In the image region corresponding to the extracted color feature, according to the shape and color of the preset background image and the shape and color of the partial image, extract the directional gradient histogram feature and the gray level co-occurrence matrix feature;

Using the directional gradient histogram feature and the gray level co-occurrence matrix feature as the input of the classifier to identify the tracking frame in the wearable device;

The classifier is pre-trained, and the pre-set shape and color of the background image and the shape and color of the partial image are used as input when the classifier is trained.
The electronic device according to claim 3, wherein the preset background image is a red rectangular area or a green rectangular area; the preset partial image includes the red rectangular area or the green rectangular area Multiple white and black rectangular areas in the area;

One of the plurality of white rectangular areas is located at the center of the red rectangular area or the green rectangular area, and the plurality of white rectangular areas are at least partially located on at least two opposite sides of the red rectangular area or the green rectangular area, so The position where the white square area is located intersects with the median line of the rectangle.
The electronic device according to claim 3, wherein the image acquisition device is a camera on the electronic device, and the user wearing the wearable device is photographed by the camera;

The wearable device is a wristband, the wristband includes a wristband, and the wristband is provided with a hard square area, the background color of the hard square area is red or green, and the hard square area A white square area is provided in the center, and/or white square areas are respectively provided on at least two opposite sides of the hard square area, and the position of the white square area intersects the median line of the rectangle.
The electronic device according to claim 1, characterized in that tracking the fluctuation of the position of the tracking frame in the pressing direction in the plurality of images in the sequence of video frames specifically comprises:

A KCF algorithm is used to track the tracking frame, and the KCF algorithm at least fuses the histogram feature of the directional gradient, the color domain and the score of the classifier.
The electronic device according to claim 1, wherein the depth of the user's pressing action is determined according to the fluctuation of the position and the depth of the pressing action is output to the display of the electronic device and/or the wearable device display, or output voice prompt information on the display or the wearable device.
A wearable device, characterized in that, the wearable device is the wearable device described in claim 1, which is used in the scene where the user provides cardiopulmonary resuscitation for the patient and is worn near the pressing part of the user, so that the camera captures the user providing cardiopulmonary resuscitation for the patient. The recovered video image is provided to the electronic device, and the electronic device identifies the wearable device in the video image and tracks the tracking frame on the wearable device;

The wearable device is provided with a hard square area, the background color of the hard square area is red or green, and the hard square area is provided with a white square area.
The wearable device according to claim 8, wherein the hard square area is provided with a white square area, specifically comprising:

The center of the hard square area is provided with a white square area; and/or

A white square area is provided on at least two of the four sides of the hard square area; and/or

Black strip-shaped areas are arranged on two median lines of the rigid square area, and the black strip-shaped areas extend from the center of the rigid square area to one or more sides of the four sides.
The wearable device according to claim 9, wherein white lines are arranged around the red or green background image of the hard square area, and the white lines are the red or green background image of the hard square area Outline.
The wearable device according to any one of claims 8-10, wherein the tracking frame includes the outline of the white square area, the outline of the black bar area, and the outline of the red or green background image one or any combination of two or more.
The wearable device according to claim 11, wherein the wearable device is a wristband worn on the wrist, the wristband includes a wristband, and the hard square area is provided on the wristband.
The wearable device according to claim 8, characterized in that the background colors of the plurality of rigid square regions are different and do not constitute central symmetry or axis symmetry.
A hand pressing depth detection method is characterized in that, comprising:

Obtain the video provided by the image acquisition device, the video contains a plurality of images of video frame sequences generated in time sequence; wherein, each of the images contains a wearable device worn near the pressing part when the user presses the patient, the The wearable device is provided with a positioning mark image of the wearable device as a tracking frame;

Identifying a tracking frame in the wearable device in the image according to at least one frame image in the sequence of video frames;

Tracking fluctuations in the position of the tracking frame in the pressing direction in multiple images in the sequence of video frames;

Determining the depth of the user's pressing action according to the fluctuation of the position and outputting the depth of the pressing action to other electronic devices and/or the wearable device.
A hand pressing depth detection system, characterized by comprising the electronic device according to any one of claims 1-7 and the wearable device according to any one of claims 8-13.
A non-transitory computer-readable storage medium, characterized in that a computer program is stored thereon, wherein the computer program implements the method according to claim 14 when executed by a processor.