WO2021233051A1 - Procédé et dispositif d'alerte d'interférence - Google Patents

Procédé et dispositif d'alerte d'interférence Download PDF

Info

Publication number
WO2021233051A1
WO2021233051A1 PCT/CN2021/088737 CN2021088737W WO2021233051A1 WO 2021233051 A1 WO2021233051 A1 WO 2021233051A1 CN 2021088737 W CN2021088737 W CN 2021088737W WO 2021233051 A1 WO2021233051 A1 WO 2021233051A1
Authority
WO
WIPO (PCT)
Prior art keywords
area
human body
joint point
interference
background
Prior art date
Application number
PCT/CN2021/088737
Other languages
English (en)
Chinese (zh)
Inventor
姜永航
Original Assignee
华为技术有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 华为技术有限公司 filed Critical 华为技术有限公司
Publication of WO2021233051A1 publication Critical patent/WO2021233051A1/fr

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features

Definitions

  • This application relates to the technical field of gesture recognition, and in particular to an interference prompt method and device.
  • Human posture recognition is mainly to input the image related to the human posture as input data into the posture recognition algorithm, and then recognize possible human actions from the input image through the posture recognition algorithm.
  • the accuracy of the recognition result is largely restricted by the input image. If the input image has high definition and few interference factors, the accuracy of the recognized human action is high; on the contrary, if there are many interference factors in the input image, the accuracy of the recognition result will be reduced. Therefore, in a gesture recognition scenario, if there are interference factors in the acquired image and the interference factors may affect the result of gesture recognition, how to determine and prompt the user of the interference factors becomes a technical problem to be solved urgently.
  • the present application provides an interference prompt method and device to determine the interference of background factors on the gesture recognition result in a gesture recognition scene and prompt the user with interference information.
  • this technical solution provides an interference notification method, including:
  • the method of the embodiment of the present application can be applied to a human body gesture recognition scene.
  • recognizing a user's gesture first determine the degree of interference of the background factor to the identified user's action, and send an interference prompt message to the user according to the determined degree of interference.
  • the interference prompt information can be used to prompt the user that the background of the current scene affects the accurate recognition of the action, and it is recommended to change clothes or change the background of the scene.
  • determining the human body region from the detected user image includes:
  • the target joint point or the area connected by the target joint point is determined as the human body area.
  • all or part of the limb joint points in the user image may be used as the target joint points.
  • the area formed by the target joint points can be determined as the human body area.
  • the left shoulder joint point, the right shoulder joint point, the left hip joint point, and the right hip joint point are determined as the target joint points; and the left shoulder joint point, the right shoulder joint point, the left hip joint point, and the right hip joint point are connected together.
  • the area is determined as the human body area.
  • the target joint point can be determined as the human body area. It can be understood that each limb joint point corresponds to a sub-region on the contour of the human body, and a sub-region contains only one limb joint point, and the size of the sub-region can be set according to actual needs.
  • determining the background area from the user image includes:
  • the expansion area is determined as the background area.
  • determining the extended area of the human body area in the user image includes:
  • a first area including the motion range area is determined in the user image, and an expansion area of the human body area is determined from the first area.
  • determining the extended area of the human body area in the user image includes:
  • the sub-expansion area of each limb joint point is determined as the expansion area of the human body area.
  • determining the degree of interference of the background area on recognizing user actions from the human body area includes:
  • determining the degree of interference of the background area on recognizing user actions from the human body area includes:
  • the degree of interference of the background area on recognizing user actions from the human body area is determined according to the complexity.
  • determining the degree of interference of the background area on recognizing user actions from the human body area includes:
  • the degree of interference of the background area to the recognition of the user's action from the human body area is determined according to the accuracy of each limb joint point.
  • determining the accuracy of each limb joint point contained in the human body area according to the information of the background area includes:
  • the accuracy of each limb joint point is determined according to the background complexity of each limb joint point and the color difference value between each limb joint point and the background area.
  • the present technical solution provides an interference prompting device, including: a processing unit, configured to determine a human body area and a background area from a detected user image; determining that the background area is appropriate for identifying a user from the human body area The interference degree of the action; the prompt unit is used to generate interference prompt information according to the interference degree.
  • the processing unit is specifically configured to identify target joint points from the user image; connect the target joint points or the target joint points to form The area is determined as the human body area.
  • the processing unit is specifically configured to determine an extended area of the human body area in the user image; and determine the extended area as the background area .
  • the processing unit is specifically configured to determine the action range area in the user image according to the user's current and subsequent actions; in the user image A first area including the motion range area is determined, and an expansion area of the human body area is determined from the first area.
  • the processing unit is specifically configured to determine, in the user image, the sub-expansion area of each limb joint point contained in the human body area;
  • the sub-expansion area of the limb joint point is determined as the expansion area of the human body area.
  • the processing unit is specifically configured to determine a color difference value between the background area and the human body area; and determine the background area according to the color difference value The degree of interference in recognizing user actions from the human body area.
  • the processing unit is specifically configured to determine the complexity of the background area; determine that the background area is paired from the human body area according to the complexity Identify the degree of interference in user actions.
  • the processing unit is specifically configured to determine the accuracy of each limb joint point contained in the human body area according to the information of the background area;
  • the accuracy of the limb joint points determines the degree of interference of the background area on recognizing user actions from the human body area.
  • the processing unit is specifically configured to determine the background complexity of each limb joint point contained in the human body area, and the difference between each limb joint point and the background area. Color difference value; the accuracy of each limb joint point is determined according to the background complexity of each limb joint point and the color difference value between each limb joint point and the background area.
  • the technical solution provides an electronic device, including: a display screen; a camera; one or more processors; a memory; and one or more computer programs, wherein the one or more computer programs are stored in In the memory, the one or more computer programs include instructions, which when executed by the device, cause the device to perform the following steps: determine the human body area and the background area from the detected user image; determine The degree of interference of the background area to identifying user actions from the human body area; and generating interference prompt information according to the degree of interference.
  • the present technical solution provides an electronic device, the device includes a storage medium and a central processing unit, the storage medium may be a non-volatile storage medium, and the storage medium stores a computer executable program.
  • the central processing unit is connected to the non-volatile storage medium and executes the computer executable program to implement the first aspect or the method in any possible implementation manner of the first aspect.
  • the present technical solution provides a chip that includes a processor and a data interface, and the processor reads instructions stored in a memory through the data interface, and executes the first aspect or any of the first aspects.
  • the processor reads instructions stored in a memory through the data interface, and executes the first aspect or any of the first aspects.
  • the chip may further include a memory in which instructions are stored, and the processor is configured to execute instructions stored on the memory.
  • the processor is configured to execute the first aspect or the method in any possible implementation manner of the first aspect.
  • the present technical solution provides a computer-readable storage medium that stores program code for device execution, and the program code includes the program code for executing the first aspect or any one of the first aspect.
  • FIG. 1 is a schematic structural diagram of an electronic device provided by an embodiment of the present application.
  • FIG. 2 is a schematic flowchart of the interference notification method provided by an embodiment of the present application.
  • FIG. 3 is another schematic flowchart of the interference notification method provided by an embodiment of the present application.
  • FIG. 4 is a schematic diagram of determining a human body area and a background area according to an image of a user according to an embodiment of the present application
  • FIG. 5 is another schematic flowchart of an interference prompt method provided by an embodiment of the present application.
  • FIG. 6 is another schematic flowchart of the interference notification method provided by an embodiment of the present application.
  • FIG. 7 is a schematic structural diagram of an electronic device provided by an embodiment of the present application.
  • interference factors in the input image will affect the accuracy of the gesture recognition result.
  • the image background is the background of the scene where the user is located when the user's actions are captured, and it is an important interference factor that affects the accuracy of the gesture recognition result.
  • the color, shape, and texture of the image background will affect the accurate recognition of user actions. For this reason, in gesture recognition, a scheme is needed to identify background interference and send interference prompts to users.
  • the embodiment of the present application provides an interference prompt method, which is applied to a human body gesture recognition scene.
  • an interference prompt method When recognizing a user's gesture, first determine the degree of interference of the background factor to the identified user's action, and send an interference prompt message to the user according to the determined degree of interference.
  • the interference prompt information can be used to prompt the user that the background of the current scene affects the accurate recognition of actions, and it is recommended to change the background of the scene or change clothes.
  • the interference prompt method of the embodiment of the present application can be applied to mobile phones, tablet computers, computers, smart screens, wearable devices, in-vehicle devices, smart home devices, augmented reality (AR)/virtual reality (virtual reality) with display screens. , VR) devices and other electronic devices, the embodiments of this application do not impose any restrictions on the specific types of electronic devices.
  • AR augmented reality
  • VR virtual reality
  • FIG. 1 shows a schematic structural diagram of an electronic device 100 provided in an embodiment of the present application.
  • the electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, and a battery 142 , Antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, audio module 170, speaker 170A, receiver 170B, microphone 170C, earphone jack 170D, sensor module 180, button 190, motor 191, indicator 192, camera 193 , Display screen 194, subscriber identification module (subscriber identification module, SIM) card interface 195, etc.
  • SIM subscriber identification module
  • the sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, and ambient light Sensor 180L, bone conduction sensor 180M, etc.
  • the structure illustrated in the embodiment of the present invention does not constitute a specific limitation on the electronic device 100.
  • the electronic device 100 may include more or fewer components than shown, or combine certain components, or disassemble certain components, or arrange different components.
  • the illustrated components can be implemented in hardware, software, or a combination of software and hardware.
  • the processor 110 may include one or more processing units.
  • the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), and an image signal processor. (image signal processor, ISP), controller, video codec, digital signal processor (digital signal processor, DSP), baseband processor, and/or neural-network processing unit (NPU), etc.
  • AP application processor
  • modem processor modem processor
  • GPU graphics processing unit
  • image signal processor image signal processor
  • ISP image signal processor
  • controller video codec
  • digital signal processor digital signal processor
  • DSP digital signal processor
  • NPU neural-network processing unit
  • the different processing units may be independent devices or integrated in one or more processors.
  • the controller may be the nerve center and command center of the electronic device 100.
  • the controller can generate operation control signals according to the instruction operation code and timing signals to complete the control of fetching instructions and executing instructions.
  • a memory may also be provided in the processor 110 to store instructions and data.
  • the memory in the processor 110 is a cache memory.
  • the memory can store instructions or data that have just been used or recycled by the processor 110. If the processor 110 needs to use the instruction or data again, it can be directly called from the memory. Repeated accesses are avoided, the waiting time of the processor 110 is reduced, and the efficiency of the system is improved.
  • the electronic device 100 implements a display function through a GPU, a display screen 194, an application processor, and the like.
  • the GPU is an image processing microprocessor, which is connected to the display screen 194 and the application processor.
  • the GPU is used to perform mathematical and geometric calculations and is used for graphics rendering.
  • the processor 110 may include one or more GPUs that execute program instructions to generate or change display information.
  • the display screen 194 is used to display images, videos, and the like.
  • the display screen 194 includes a display panel.
  • the display panel can use liquid crystal display (LCD), organic light-emitting diode (OLED), active matrix organic light-emitting diode or active-matrix organic light-emitting diode (active-matrix organic light-emitting diode).
  • LCD liquid crystal display
  • OLED organic light-emitting diode
  • active-matrix organic light-emitting diode active-matrix organic light-emitting diode
  • AMOLED flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, quantum dot light-emitting diode (QLED), etc.
  • the electronic device 100 may include one or N display screens 194, and N is a positive integer greater than one.
  • the electronic device 100 can implement a shooting function through an ISP, a camera 193, a video codec, a GPU, a display screen 194, and an application processor.
  • the ISP is used to process the data fed back from the camera 193. For example, when taking a picture, the shutter is opened, the light is transmitted to the photosensitive element of the camera through the lens, the light signal is converted into an electrical signal, and the photosensitive element of the camera transmits the electrical signal to the ISP for processing and is converted into an image visible to the naked eye.
  • ISP can also optimize the image noise, brightness, and skin color. ISP can also optimize the exposure, color temperature and other parameters of the shooting scene.
  • the ISP may be provided in the camera 193.
  • the camera 193 is used to capture still images or videos.
  • the object generates an optical image through the lens and is projected to the photosensitive element.
  • the photosensitive element may be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor.
  • CMOS complementary metal-oxide-semiconductor
  • the photosensitive element converts the optical signal into an electrical signal, and then transfers the electrical signal to the ISP to convert it into a digital image signal.
  • ISP outputs digital image signals to DSP for processing.
  • DSP converts digital image signals into standard RGB, YUV and other formats of image signals.
  • the electronic device 100 may include one or N cameras 193, and N is a positive integer greater than one.
  • NPU is a neural-network (NN) computing processor.
  • NN neural-network
  • applications such as intelligent cognition of the electronic device 100 can be realized, such as image recognition, face recognition, voice recognition, text understanding, and so on.
  • the camera 193 may be used to capture an image of the user in the current gesture recognition scene.
  • the ISP converts the electrical signal about the user image fed back by the camera 193 into a digital image signal.
  • ISP outputs digital image signals to DSP for processing.
  • DSP can convert digital image signals into standard RGB, YUV and other formats.
  • the above-mentioned image signal about the user image generated based on the DSP processing can be sent to the NPU for recognition of the human body posture, for example, the user's joint points and user actions are recognized based on the user image.
  • the posture of the human body recognized by the DSP can be displayed on the display screen 194.
  • reference actions can also be displayed on the display screen 194.
  • the display screen 194 may display fitness guidance actions.
  • the user action is displayed on the display screen 194.
  • the user action and the fitness guidance action may be displayed on the display screen 194 at the same time.
  • the NPU or the controller can execute the interference prompt method of the embodiment of the present application based on the user image, that is, the NPU or the controller can determine the interference degree of the background factor on the user's action recognition from the user image, and then provide the user with the interference Disturbance notice is issued.
  • the interference prompt may be displayed on the display screen 194, or the interference prompt information may be played through the audio module 170 and the speaker 170A.
  • the internal memory 121 may be used to store computer executable program code, where the executable program code includes instructions.
  • the internal memory 121 may include a storage program area and a storage data area.
  • the storage program area can store an operating system, an application program (such as a sound playback function, an image playback function, etc.) required by at least one function, and the like.
  • the data storage area can store data (such as image data, phone book, etc.) created during the use of the electronic device 100.
  • the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, a universal flash storage (UFS), and the like.
  • the processor 110 executes various functional applications and data processing of the electronic device 100 by running instructions stored in the internal memory 121 and/or instructions stored in a memory provided in the processor.
  • FIG. 2 is a schematic flowchart of the interference notification method provided by an embodiment of the present application.
  • This application will use a mobile phone or a display device as an electronic device to introduce in detail the interference notification method provided by this application.
  • the display device may be, for example, a television, a smart screen, or other equipment.
  • Smart screen is a new type of large-screen device.
  • the interference prompt method in the embodiment of the present application can be implemented.
  • the interference prompt method in the embodiment of this application can also be implemented on the mobile phone.
  • the user's image is captured through the camera on the electronic device.
  • the user's image is captured by the camera on the mobile phone.
  • the user's image is captured by the camera on the display device.
  • the captured user image also includes the background part of the scene where the user is located.
  • the human body area and the background area are determined from the user image.
  • the method of recognizing the human body area and the background area from the user image may be: first recognizing the human body area from the user image; after recognizing the human body area, determine the background area in the user image according to the human body area.
  • recognizing the human body area from the user image includes: using a human body gesture recognition algorithm to determine the position of the human body and the contour of the human body from the user image. Part or all of the outline of the human body may be used as the human body area in the embodiment of the present application.
  • the human body gesture recognition algorithm may be an algorithm such as open-pose and poseNet. When the human body gesture recognition algorithm is used to recognize the contour of the human body from the user image, the user's limb joint nodes can be recognized first, and the human body contour can be determined according to the recognized limb joint points.
  • the target joint node When determining the human body region from the contour of the human body, the target joint node can be determined first, and then the human body region can be determined according to the target joint point. Specifically, the area connected by the target joint point on the contour of the human body can be determined as the human body area.
  • the target joint points may also be determined as the human body area in this embodiment of the application.
  • all the identified limb joint points may be used as the target joint point. In addition, it is also possible to select some joint points from all the recognized joint points of the limbs as the target joint points.
  • the limb joint points corresponding to the human trunk may be used as the target joint points.
  • the left shoulder joint point, right shoulder joint point, left hip joint point, and right hip joint point are determined as the target joint points, and the left shoulder joint point, right shoulder joint point, left hip joint point, and right hip joint point can be surrounded by The rectangular area is determined as the human body area.
  • the left shoulder joint point, the right shoulder joint point, the left hip joint point, and the right hip joint point are directly determined as the human body area in the embodiment of the application.
  • the target joint point it may be preset or determined according to the user's current and/or subsequent actions. For example, if the user is doing a squat exercise, the limb joint points related to the user's lower limbs may be used as the target joint points.
  • the background region can be further determined in the user image according to the human body region.
  • the area related to the human body area in the user image can be determined as the background area.
  • the extended area of the human body area can be determined from the user image, and the extended area is determined as the background area in the embodiment of the application.
  • the human body area can be expanded in any one or more directions up, down, left, and right to expand out The part of the area can be determined as the background area.
  • the action range area may be determined in the user image according to the user's current and subsequent actions; the first area including the action range area may be determined in the user image, and the expansion area of the human body area may be determined from the first area.
  • the current user image detected is a squatting posture, and it can be judged that the user's subsequent action is a posture of gradually getting up from the squatting posture.
  • the user's action range from the squatting posture to the standing posture can be determined as the first area; the area in the first area excluding the user's current body range can be determined as the extended area, that is, the background area.
  • the manner of determining the expansion area may also be: determining the sub-expansion area of each limb joint point contained in the human body area in the user image; determining the sub-expansion area of each limb joint point as the expansion of the human body area area.
  • each target joint point can be expanded separately, and each sub-expansion area associated with the target joint point is determined as a background area.
  • the degree of interference of the background area on the recognition of user actions can be determined according to the color difference between the background area and the human body area, the complexity of the background area area, and the joint point accuracy derived based on the color difference and complexity.
  • determining the interference degree of the background area on the recognition of the user action according to the color difference between the background area and the human body area includes: determining the color difference value between the background area and the human body area; Identify the degree of interference in user actions.
  • the color difference value between the background area and the human body area is less than the first threshold, it can be considered that the color difference between the background area and the human body area is small, and the background area has a high degree of interference in recognizing user actions.
  • the degree of interference of the background area on the recognition of user actions can be determined according to the complexity of the background area; when the complexity of the background area is greater than the second threshold, it can be considered that the complexity of the background area is greater, and the degree of interference of the background area on recognizing user actions Higher.
  • the color difference value of the background area and the human body area can be used as the independent basis to determine the interference degree of the background area to the recognition of the user's action
  • the color difference value and the complexity can also be determined simultaneously.
  • the degree determines the degree of interference of the background area on recognizing user actions from the human body area.
  • the color difference between the background area and the human body area is greater than the first threshold
  • the accuracy of each limb joint point contained in the human body area can also be determined according to the background area information; the degree of interference of the background area on the recognition of user actions from the human body area can be determined according to the accuracy of each limb joint point .
  • the accuracy of each limb joint point contained in the human body area is determined, including: determining the background complexity of each limb joint point contained in the human body area and the color difference between each limb joint point and the background area; The background complexity of the limb joint points and the color difference between each limb joint point and the background area determine the accuracy of each limb joint point.
  • each limb joint point may be used as the target joint point.
  • the way to determine the background area can be: connect the target joint points into the second area; determine a third area that includes the second area in the user image, and determine the part of the third area except the second area as Background area.
  • the background area can be divided into multiple non-overlapping sub-areas, each sub-area contains at least one target joint point, calculate each eye joint point and its location The color difference value between the sub-regions.
  • Calculating the background complexity of each target joint point includes: calculating the complexity of the subregion where the target joint point is located.
  • another possible way of determining the background area of the target joint point may be: separately determining the sub-expansion area of each target joint point; and determining the sub-expansion area of each limb joint point as the background area of the human body area.
  • calculating the color difference value of each target joint point and the background area includes: calculating the color difference value between the target joint point and its corresponding sub-expansion area.
  • calculating the background complexity of each target joint point includes: calculating the complexity of the sub-expansion area where the target joint point is located.
  • the degree of interference of the background area on identifying the user's action can be determined according to the accuracy of each limb joint point. For example, when the accuracy of one or more limb joint points is less than the third threshold, it is considered that the background area has a high degree of interference with the one or more joint points, and a prompt may be issued to the user.
  • the total accuracy of the joint points can also be calculated according to the accuracy of the joint points of each limb, and the interference degree of the background area can be determined according to the total accuracy of the joint points.
  • the above-mentioned interference degree is used to characterize the degree of interference of the background area on the action of the user to be identified.
  • the above-mentioned interference degree may be a specific parameter.
  • the interference degree takes the value b1; when the complexity of the background area is in the range of a2-a3, the interference The value of the degree is b2, etc.;
  • the aforementioned interference degree can also be characterized by other parameters, for example, when the complexity of the background area is used to determine the interference of the background area on identifying user actions, when the complexity of the background area exceeds the second threshold At this time, it is considered that the interference degree of the background area has reached the level that needs to be prompted to the user.
  • the interference degree can be characterized by the complexity of the background area, and no specific assignment of the interference degree is required.
  • the interference prompt information when it is determined that the background area interferes with the action of identifying the user according to the calculated interference degree, the interference prompt information can be generated and the interference prompt information can be provided to the user.
  • the user when the color difference between the background area and the human body area is small, which affects the user's action recognition, the user can be prompted that the current dress is close to the background area in color, which affects the action recognition, and it is recommended to change the dress.
  • the pattern of the background area is too complicated, it can be prompted to change the background of the scene.
  • the user can be prompted about the interference of the current background scene in recognizing the user's action, and the accuracy of the user's action recognition can be improved.
  • the degree of interference of the background region to the user's action is determined according to the color difference value of the background region and the human body region.
  • the processing steps of the method in this embodiment include:
  • Identify the user's position and human body contour from the user's image can be used to determine the user position from the user image, and the recognized user position is shown in the rectangular box in Figure 4(a).
  • the joint points of each limb of the user are further recognized, and the joint points of the recognized limbs are connected to the contour of the human body.
  • the target joint points can be determined first, and the region connected by the target joint points can be determined as the human body region.
  • the left shoulder joint point, the right shoulder joint point, the left hip joint point, and the right hip joint point can be determined as the target joint points.
  • the rectangular area A enclosed by the left shoulder joint point, the right shoulder joint point, the left hip joint point, and the right hip joint point can be determined as the human body area.
  • a background area from the user image.
  • an expanded area B can be determined on the basis of the human body area A, and the part of the expanded area B where the human body area A is removed is the background area.
  • the size of the enlarged area can be set according to actual needs, for example, the size of the human body area is enlarged according to a certain size ratio.
  • the human body area and the background area determined from the user image may be a color image.
  • the pixel points of the human body area and the background area can be assigned values based on the three color channels of RGB, and the value range of each color channel is [0, 255].
  • the color difference between the human body area and the background area C
  • C 1 is the color value of the human body region, and the color value of the human body region can represent the color of the user's clothes or skin.
  • C 1 may be the average value of the color values of the pixels in the human body area or the median of the color values of the pixels in the human body area.
  • C 2 is the color value of the background area.
  • C 2 can be the mean value of the color values of the pixels in the background area or the median of the color values of the pixels in the background area.
  • the difference value C may take the average value, the modulus value or the maximum value of the three-channel color difference of C 1 and C 2.
  • the above-mentioned human body area and background area may be grayscale images.
  • the user image is converted into a grayscale image, and then the human body area and background area are identified from the grayscale image.
  • the human body area and the background area are respectively converted into grayscale images, and then the grayscale image is used to calculate the color difference between the human body area and the background area C.
  • the human body area and the background area are grayscale images, the gray values of the pixels in the human body area and the background area are determined based on their original RGB three-channel color values.
  • the formula for converting color pixel values into gray values is only an example, and can be adjusted according to actual needs in specific implementation.
  • the color difference between the human body area and the background area C
  • C 1 is the gray value of the human body area.
  • C 1 may be the mean value of the gray values of pixels in the human body area or the median of the gray values of pixels in the human body area.
  • C 2 is the gray value of the background area.
  • C 2 can be the mean value of the gray values of the pixels in the background area or the median of the gray values of the pixels in the background area.
  • the color difference value C can take the modulus of the difference between C 1 and C 2.
  • the color difference value C between the human body region and the background region is lower than the threshold c, it can be considered that the colors of the background region and the human body region are relatively close, and the background region will affect the accurate recognition of user actions by the human body gesture recognition algorithm.
  • the fitness guidance action can be played on the display device.
  • the camera on the display device captures user images, and the display device recognizes user actions based on the user images captured by the camera. Among them, it can be judged whether the user's actions are standard or not based on the actions recognized from the user's image.
  • prompt information can be generated, for example, prompting the user that the color of the clothes is close to the background color, suggesting to change the color of the clothes or change the venue, etc.
  • the prompt information may be displayed through a display device, or voice prompt may be performed through an audio module.
  • the degree of interference of the background background area to the user's action is determined according to the complexity of the background area.
  • the processing steps of the method in this embodiment include:
  • the complexity of the background area can be calculated according to the following formula.
  • n i is the number of pixels with a gray level of i in the background area
  • N is the total number of pixels in the background area
  • the complexity of the background area can also be measured by information entropy, where the calculation formula of the information entropy H is as follows:
  • n i is the number of pixels with a gray level of i in the background area
  • N is the total number of pixels in the background area
  • interference prompt information Generate interference prompt information, and send an interference prompt to the user.
  • the complexity of the background area is higher than the threshold h, it is considered that the background in the current scene is relatively complicated, which affects the recognition of user actions.
  • the information entropy is used to represent the complexity of the background area
  • the background area is considered to be relatively complex, which will affect the recognition of user actions by the human body gesture recognition algorithm.
  • interference prompt information can be generated, such as prompting the user to change the venue or change the background layout.
  • judging the interference of the background area on the recognition of user actions based on the complexity of the background can be used in scenes where the color difference of the background is not enough to affect the current action recognition, but the shape and texture of the background will still have an impact.
  • the degree of interference of the background area in recognizing the user's action is determined.
  • a joint point accuracy calculation model is first constructed, and the joint point accuracy calculation model is used to calculate the accuracy of the limb joint points in the currently captured user image.
  • the specific processing steps include:
  • the above-mentioned image training set contains a large number of images with the position of the limb joint points already marked.
  • the joint point accuracy calculation model can be trained to recognize the limb joint points.
  • the color difference value of each limb joint point is calculated.
  • the calculation can be performed according to the method of the first embodiment. Specifically, the area of the human body where each limb joint point is located can be determined first. For example, the human body is divided into a trunk area, a left upper limb area, a right upper limb area, a left lower limb area, and a right lower limb area.
  • the joint point to be calculated is the left elbow joint point, it is determined that the human body area where it is located is the left upper limb area.
  • the limb joint points are used as the human body area for calculation. It can be understood that the limb joint point corresponds to a sub-region on the contour of the human body, and a sub-region contains only one limb joint point, and the size of the sub-region can be set according to actual needs.
  • the expansion area of the limb joint point can be determined. After that, the color difference between the joint point and the corresponding expansion area is calculated.
  • the background complexity of each limb joint point is calculated.
  • the determination method of the background area of each limb joint point refer to the description of step 402.
  • the calculation method of the background complexity refer to the second embodiment, which will not be repeated here.
  • the accuracy P i of each limb joint point in each image is calculated.
  • P i a i *C i +b i *H i +c i ; where i is the number of the limb joint point, C i is the color difference value of the i-th limb joint point, and H i is the The background complexity of i limb joint points.
  • the accuracy P of each image may be determined according to the accuracy of the joint points of each limb in the image.
  • P is the mean value of the minimum value of each limb joint point P i accuracy or precision of each limb joint point P i in.
  • the target joint point can be determined according to the user's current or subsequent actions; the overall accuracy can be determined according to the accuracy of the target joint point.
  • the action to be recognized is a squat
  • the target joint points can be determined as the left knee joint point, right knee joint point, and hip joint point; according to the left knee joint point, right knee joint point, and hip joint point
  • the accuracy determines the overall accuracy of the image.
  • step 405 is executed.
  • calculating the image accuracy based on the joint point calculation model includes: identifying each limb joint point in the user image, and calculating the accuracy of each limb joint point. After the accuracy of each limb joint point is determined, the image accuracy is determined according to the accuracy of each limb joint point.
  • the minimum accuracy of each limb joint point can be used as the image accuracy, and an interference prompt can be generated when the image accuracy is less than the threshold; in addition, it can also be prompted according to the accuracy of the target joint point according to the user's current or subsequent actions . For example, when the user performs a squat action, if the overall accuracy determined based on the accuracy of the left knee joint point, right knee joint point, and hip joint point is less than the threshold, it is considered that the human body pose recognition algorithm may recognize the leg under the current background. Inaccurate. At this time, an interference prompt can be generated, and it is recommended to change the background or change the pants.
  • the action range area is determined in the user image according to the user’s current action and/or subsequent actions; in the user image, the background area is determined according to the action range area; after the background area is determined, it can be based on the above-mentioned embodiments 1 to 4 Either method determines whether the current background interferes with the recognition of the user's actions, and generates an interference prompt message when it is determined to cause interference.
  • determining the background area in the user image according to the action range area includes: determining the area in the action range area where the current user contour is removed as the background area; or, determining the first area larger than the action range area, and setting the first area The area where the current user contour is removed in the area is determined as the background area.
  • the foregoing manner of determining the current action and/or subsequent actions of the user may include: determining the current action and/or subsequent actions of the user through application information. For example, in a fitness scene, the fitness guidance action is played in the electronic device, and the current and subsequent fitness guidance actions to be played are preset in the application, so the current and/or subsequent actions of the user can be determined according to the information in the application. For another example, the current and/or subsequent actions of the user may be determined according to the acquired user image.
  • the user’s activity range is judged according to the user’s current and subsequent actions, and the influence of the background related to the user’s activity scope on user action recognition is judged.
  • the influence of the entire activity scope on identifying the user can be estimated, for example, when the user When squatting, it can predict the influence of standing background on lower limb motion recognition; on the other hand, it can also avoid the interference of the gesture recognition algorithm in the area where the user will not reach.
  • an electronic device includes hardware and/or software modules corresponding to each function.
  • the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a certain function is executed by hardware or computer software-driven hardware depends on the specific application and design constraint conditions of the technical solution. Those skilled in the art can use different methods for each specific application in combination with the embodiments to implement the described functions.
  • the electronic device may be divided into functional modules according to the foregoing method examples.
  • each functional module may be divided corresponding to each function, or two or more functions may be integrated into one processing module.
  • the above-mentioned integrated modules can be implemented in the form of hardware. It should be noted that the division of modules in this embodiment is illustrative, and is only a logical function division, and there may be other division methods in actual implementation.
  • FIG. 7 shows a schematic diagram of a possible composition of the electronic device involved in the foregoing embodiment.
  • the electronic device can be integrated with a functional module for interference prompting in human body gesture recognition.
  • the functional module specifically includes: a processing unit 701 and a prompting unit 702; among them:
  • the processing unit 701 is configured to determine a human body area and a background area from the detected user image; determine the degree of interference of the background area on recognizing user actions from the human body area; and the prompt unit 702 is configured to determine the degree of interference according to the degree of interference Generate interference prompt messages.
  • the processing unit 701 is specifically configured to identify a target joint point from the user image; and determine the target joint point or an area connected by the target joint points as the human body area.
  • the processing unit 701 is specifically configured to determine an extended area of the human body area in the user image; and determine the extended area as the background area.
  • the processing unit 701 is specifically configured to determine an action range area in the user image according to the current action and subsequent actions of the user;
  • the first area is to determine the expansion area of the human body area from the first area.
  • the processing unit 701 is specifically configured to determine, in the user image, the sub-expansion area of each limb joint point included in the human body area; determine the sub-expansion area of each limb joint point Is the extended area of the human body area.
  • the processing unit 701 is specifically configured to determine the color difference value between the background area and the human body area; according to the color difference value, the background area pair is determined from the human body area. The degree of interference of the user's actions.
  • the processing unit 701 is specifically configured to determine the complexity of the background area; and determine the degree of interference of the background area on identifying user actions from the human body area according to the complexity.
  • the processing unit 701 is specifically configured to determine the accuracy of each limb joint point contained in the human body area according to the information of the background area; and determine the accuracy of each limb joint point according to the accuracy of each limb joint point The degree of interference of the background area on recognizing user actions from the human body area.
  • the processing unit 701 is specifically configured to determine the background complexity of each limb joint point contained in the human body area and the color difference value between each limb joint point and the background area;
  • the background complexity of the limb joint points and the color difference value between the limb joint points and the background area determine the accuracy of the limb joint points.
  • the electronic device here is embodied in the form of a functional unit.
  • the term "unit” herein can be implemented in the form of software and/or hardware, which is not specifically limited.
  • a "unit” may be a software program, a hardware circuit, or a combination of the two that realizes the above-mentioned functions.
  • the hardware circuit may include an application specific integrated circuit (ASIC), an electronic circuit, and a processor for executing one or more software or firmware programs (such as a shared processor, a dedicated processor, or a group processor). Etc.) and memory, merged logic circuits and/or other suitable components that support the described functions.
  • An embodiment of the present application also provides an electronic device, including: a display screen; a camera; one or more processors; a memory; and one or more computer programs, wherein the one or more computer programs are stored in the In the memory, the one or more computer programs include instructions, and when the instructions are executed by the device, the device executes the following steps: determining the human body area and the background area from the detected user image; determining the The degree of interference of the background area on recognizing user actions from the human body area; and generating interference prompt information according to the degree of interference.
  • the device includes a storage medium and a central processing unit.
  • the storage medium may be a non-volatile storage medium.
  • a computer executable program is stored in the storage medium.
  • the central processing unit It is connected to the non-volatile storage medium and executes the computer executable program to realize the interference prompt method shown in FIGS. 2 to 6 above.
  • the present application also provides a chip, the chip includes a processor and a data interface, the processor reads instructions stored on the memory through the data interface, and executes the interference prompt method shown in FIGS. 2 to 6 above.
  • the chip may further include a memory in which instructions are stored, and the processor is configured to execute instructions stored on the memory. When the instructions are executed, the The processor is configured to execute the interference prompt method in the foregoing embodiments.
  • the present technical solution provides a computer-readable storage medium, the computer-readable medium storing program code for device execution, and the program code includes Method of instruction.
  • the memory can be read-only memory (ROM), other types of static storage devices that can store static information and instructions, random access memory (RAM), or other types that can store information and instructions
  • Dynamic storage devices can also be electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM), or other optical disk storage, optical disc storage ( Including compact discs, laser discs, optical discs, digital versatile discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or can be used to carry or store desired program codes in the form of instructions or data structures and can Any other medium accessed by the computer, etc.
  • EEPROM electrically erasable programmable read-only memory
  • CD-ROM compact disc read-only memory
  • optical disc storage Including compact discs, laser discs, optical discs, digital versatile discs, Blu-ray discs, etc.
  • magnetic disk storage media or other magnetic storage devices or can be used to carry or store desired program codes in the form of instructions or data structures and can Any other medium
  • At least one refers to one or more, and “multiple” refers to two or more.
  • And/or describes the association relationship of the associated objects, which means that there can be three relationships, for example, A and/or B, which can mean that A exists alone, A and B exist at the same time, and B exists alone. Among them, A and B can be singular or plural.
  • the character “/” generally indicates that the associated objects before and after are in an “or” relationship.
  • the following at least one item” and similar expressions refer to any combination of these items, including any combination of single items or plural items.
  • At least one of a, b, and c can represent: a, b, c, a-b, a-c, b-c, or a-b-c, where a, b, and c can be single or multiple.
  • any function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer readable storage medium.
  • the technical solution of the present application essentially or the part that contributes to the existing technology or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including Several instructions are used to make a computer device (which may be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the methods described in the various embodiments of the present application.
  • the aforementioned storage media include: U disk, mobile hard disk, read-only memory (read-only memory, ROM), random access memory (random access memory, RAM), magnetic disks or optical disks and other media that can store program codes. .
  • U disk mobile hard disk
  • read-only memory read-only memory
  • RAM random access memory

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Multimedia (AREA)
  • Theoretical Computer Science (AREA)
  • Image Analysis (AREA)
  • User Interface Of Digital Computer (AREA)

Abstract

La présente demande concerne le domaine technique de la reconnaissance d'attitude, et en particulier un procédé et un dispositif d'alerte d'interférence. Le procédé comprend les étapes consistant à : déterminer une zone de corps humain et une zone d'arrière-plan à partir d'une image d'utilisateur détectée ; déterminer le degré d'interférence de la zone d'arrière-plan pour la reconnaissance d'une action d'utilisateur à partir de la zone de corps humain ; et générer des informations d'alerte d'interférence en fonction du degré d'interférence. Selon le procédé et le dispositif d'alerte d'interférence dans les modes de réalisation de la présente demande, dans un scénario de reconnaissance d'attitude, l'interférence de facteurs d'arrière-plan sur un résultat de reconnaissance d'attitude peut être déterminée, et une alerte d'informations d'interférence peut être donnée à un utilisateur.
PCT/CN2021/088737 2020-05-21 2021-04-21 Procédé et dispositif d'alerte d'interférence WO2021233051A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202010437508.3A CN113705283A (zh) 2020-05-21 2020-05-21 干扰提示方法和设备
CN202010437508.3 2020-05-21

Publications (1)

Publication Number Publication Date
WO2021233051A1 true WO2021233051A1 (fr) 2021-11-25

Family

ID=78645877

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2021/088737 WO2021233051A1 (fr) 2020-05-21 2021-04-21 Procédé et dispositif d'alerte d'interférence

Country Status (2)

Country Link
CN (1) CN113705283A (fr)
WO (1) WO2021233051A1 (fr)

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115177755A (zh) * 2022-07-07 2022-10-14 中国人民解放军军事科学院军事医学研究院 在线智能紫外辐射消毒系统和方法

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104850219A (zh) * 2014-02-19 2015-08-19 北京三星通信技术研究有限公司 估计附着物体的人体姿势的设备和方法
US20160027149A1 (en) * 2010-11-13 2016-01-28 Super Booths Inc. System and Method for Storing Images Captured from a Booth
CN110033007A (zh) * 2019-04-19 2019-07-19 福州大学 基于深度姿态预估和多特征融合的行人衣着属性识别方法
CN110619285A (zh) * 2019-08-29 2019-12-27 福建天晴数码有限公司 人体骨骼关键点提取方法及计算机可读存储介质

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20160027149A1 (en) * 2010-11-13 2016-01-28 Super Booths Inc. System and Method for Storing Images Captured from a Booth
CN104850219A (zh) * 2014-02-19 2015-08-19 北京三星通信技术研究有限公司 估计附着物体的人体姿势的设备和方法
CN110033007A (zh) * 2019-04-19 2019-07-19 福州大学 基于深度姿态预估和多特征融合的行人衣着属性识别方法
CN110619285A (zh) * 2019-08-29 2019-12-27 福建天晴数码有限公司 人体骨骼关键点提取方法及计算机可读存储介质

Also Published As

Publication number Publication date
CN113705283A (zh) 2021-11-26

Similar Documents

Publication Publication Date Title
US9195304B2 (en) Image processing device, image processing method, and program
CN103310186B (zh) 校正图像中用户的注视方向的方法和便携式终端
WO2020063009A1 (fr) Procédé et appareil de traitement d'image, support d'informations, et dispositif électronique
WO2021114814A1 (fr) Procédé et appareil de reconnaissance d'attribut de corps humain, dispositif électronique et support d'informations
CN111242090B (zh) 基于人工智能的人脸识别方法、装置、设备及介质
CN107231529A (zh) 图像处理方法、移动终端及存储介质
WO2021042364A1 (fr) Procédé et dispositif permettant de prendre une photo
WO2021143216A1 (fr) Procédé de détection de vivacité de visage et appareil associé
CN109190509A (zh) 一种身份识别方法、装置和计算机可读存储介质
WO2020248950A1 (fr) Procédé pour déterminer la validité d'une caractéristique faciale, et dispositif électronique
WO2021233051A1 (fr) Procédé et dispositif d'alerte d'interférence
CN110956571B (zh) 基于slam进行虚实融合的方法及电子设备
CN113539439B (zh) 医学图像的处理方法、装置、计算机设备和存储介质
CN115049819A (zh) 注视区域识别方法及装置
CN115482359A (zh) 目标的尺寸测量方法、电子设备及其介质
CN113342157B (zh) 眼球追踪处理方法及相关装置
CN111277751B (zh) 拍照方法、装置、存储介质及电子设备
CN108804996B (zh) 人脸验证方法、装置、计算机设备及存储介质
CN117132515A (zh) 一种图像处理方法及电子设备
CN111277752B (zh) 提示方法、装置、存储介质及电子设备
CN115273196A (zh) 基于人脸关键点的肤色识别方法及电子设备
CN114758354A (zh) 坐姿检测方法、装置、电子设备、存储介质及程序产品
CN113313009A (zh) 连拍的输出图像方法、装置、终端及可读存储介质
WO2021238502A1 (fr) Procédé et dispositif de synchronisation d'estampille temporelle
WO2021244174A1 (fr) Procédé et dispositif de détection de peau

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21807636

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 21807636

Country of ref document: EP

Kind code of ref document: A1