CN116301362A - Image processing method, electronic device and storage medium - Google Patents

Abstract

The application provides an image processing method, electronic equipment and a storage medium, which relate to the field of electronic technology. This application adds a new intelligent perception algorithm platform based on the original AON algorithm platform. In the AON intelligent perception scene, the front camera is always on and collects images to intelligently perceive user services, which can be widely used in air gestures and intelligent code scanning. , smart gaze and other scenarios, which greatly meet the needs of users. When the application detects that the collected image meets the first preset trigger condition, it first judges whether the triggering scene is triggered in the bright screen scene or in the off-screen scene, performs business classification and identification, and then distributes the business to the Intelligent perception algorithm platform, and further judge the specific business content. The AON camera of this application operates with extremely low power consumption, can accurately perceive user services, and improves user experience.

Description

Image processing method, electronic device and storage medium

technical field

The present application relates to the field of electronic technology, and in particular to an image processing method, electronic equipment and a storage medium.

Background technique

At present, smart electronic devices such as mobile phones can use an always-on camera (AON) function, which can quickly unlock the screen with a face when the mobile phone is in an off-screen state. Specifically, when the AON function is applied to the mobile phone, the front camera of the mobile phone camera is always on, and images can be collected in real time, and face recognition can be performed through image analysis, which can achieve faster response to face unlocking the screen, without the need for users to Touch the phone to unlock the phone screen.

As mentioned above, the AON function can be applied to face unlock screen scenarios when the screen of the mobile phone is off. However, since users need to use the mobile phone with the screen on in most cases, the application scenarios of the AON function are too limited.

Contents of the invention

This application provides an image processing method, electronic equipment, and storage media. Based on the original AON algorithm platform, a new intelligent perception algorithm platform is added. In the AON intelligent perception scene, the front camera is always on and collects images to intelligently perceive users. It can be widely used in various scenarios such as air gestures, smart code scanning, and smart gazing, which can greatly meet the needs of users.

In order to achieve the above object, the application adopts the following technical solutions:

In the first aspect, the present application provides an image processing method, which is applied to the system architecture of electronic equipment. The system architecture includes a first algorithm platform and a second algorithm platform, and the business supported by the first algorithm platform is different from that of the second algorithm platform. The business that supports processing, the business that the second algorithm platform supports processing is face unlocking screen business, the method includes:

The electronic device turns on the first function, and after the first function is turned on, the camera module of the electronic device is in a normally-on state;

Outputting the multi-frame first image of the first size specification through the camera module;

Invoking the first algorithm platform to perform image analysis on multiple frames of the first image to obtain the first image analysis result;

When the first image analysis result satisfies the first preset trigger condition, the camera module outputs multi-frame second images of the second size specification; calls the first algorithm platform to perform image analysis on the multi-frame second images to obtain the second image analysis Result; making a response according to a preset strategy corresponding to the second image analysis result;

When the first image analysis result satisfies the second preset trigger condition, the camera module outputs a multi-frame third image of a third size specification; the second algorithm platform is called to perform image analysis on the multi-frame third image to obtain a third image analysis Result; making a response according to a preset strategy corresponding to the third image analysis result;

Wherein, the resolution corresponding to the first size specification is lower than the resolution corresponding to the second size specification, and the resolution corresponding to the second size specification is lower than or equal to the resolution corresponding to the third size specification.

Through the proposal of this application, this application adds a new intelligent perception algorithm platform on the basis of the AON algorithm platform provided by the original chip. In the AON intelligent perception scene, the camera module is always on and collects images to intelligently perceive user services, which is not limited to the realization of native The face unlocking screen service supported by the AON algorithm platform provided by the chip can also be widely used in various other scenarios (such as air gestures, smart scanning codes, smart gaze, etc.), which greatly meet user needs. Moreover, the solution of the present application uses different image resolutions for image analysis at different stages, which not only can ensure the accuracy of gesture recognition, but also can effectively reduce the power consumption and memory space required in the gesture recognition process. The AON camera of this application operates with extremely low power consumption, can accurately perceive user services, and improves user experience.

In some implementations, the camera module is a front camera. Among them, in the initial stage, the camera module (AON camera) continues to output lower-resolution images, so that the AON camera operates with extremely low power consumption, senses user services, and improves user experience.

In some implementations, the first preset trigger condition is any of the following: (1) the image contains the first preset feature, and the image is an image collected when the screen is turned on and unlocked; Assume that the features include any one of face features, human eye features, gesture features and two-dimensional code features; (2) the image contains face features, and the image is an image collected when the screen is off.

Through the solution of this application, when the collected images are monitored to meet the first preset trigger condition, it is possible to first judge whether the triggering scene is triggered in the bright screen scene or in the off-screen scene, perform business classification and identification, and further judge the specific business Content can accurately perceive user services and improve user experience.

In some implementations, the second preset trigger condition is that the image contains facial features, and the image is an image collected when the screen is on and locked.

In some implementations, the services supported by the first algorithm platform include air gesture services, intelligent code recognition services, services that keep the screen on while watching the screen, services that reduce the volume while watching the screen, services on intelligent horizontal and vertical screens, auxiliary photo taking services, and intelligent screen off. Show business.

Through the proposal of this application, this application adds a new intelligent perception algorithm platform on the basis of the original AON algorithm platform. In the AON intelligent perception scene, the front camera is always on and collects images to intelligently perceive user services, which can be widely used in the air Various scenarios such as gestures, smart code scanning, and smart gazing can greatly meet user needs.

In some implementations, the above method further includes: after outputting multiple frames of first images of the first size specification through the camera module of the electronic device, storing the multiple frames of first images in the first memory; After the multi-frame second images of the second size specification, the multi-frame second images are stored in the first memory and the second memory; after the multi-frame third images of the third size specification are output by the camera module, the multi-frame second images are output. Three images are stored in the second memory.

Wherein, in the case of storing the same image, the power consumption caused by the first memory is less than the power consumption caused by the second memory, and the power consumption caused by the first memory and the second memory together is less than that caused by the second memory power consumption.

In some implementations, the first memory is a TCM, and the second memory is a double-rate synchronous dynamic random access memory (DDR).

Through the solution of this application, low-power memory is used for data caching in the initial stage, and after the business is determined to be triggered, low-power memory combined with normal power memory is used for data caching. The AON camera of this application runs at extremely low power consumption. It can accurately perceive user services and improve user experience.

In some implementations, the third size specification is Video Graphics Array VGA, and the corresponding resolution is 640×480. The second size specification is QVGA, and the corresponding resolution is 320×240. The first size specification is QQVGA, and the corresponding resolution is 160×120.

In some implementations, the second algorithm platform is a framework provided by the original chip that supports the camera always-on AON algorithm, and the first algorithm platform is an AON algorithm integration framework that supports multiple services created based on the second algorithm platform.

In some implementation manners, the system framework further includes a smart sensing application, and the first function is a function provided by the smart sensing application. After the electronic device starts the first function, the above method further includes: the smart sensing application receives the first operation of the user to start the first service, and the first service is a service supported by the smart sensing application; in response to the first operation, the smart sensing application Sending a first message to the first algorithm platform, where the first message is used to indicate subscription to the first service; the first algorithm platform subscribes to the first service in response to the first message. Among them, the first service is at least one of gestures in the air, intelligent code recognition, keeping the screen on while watching the screen, lowering the volume while watching the screen, intelligent horizontal and vertical screens, auxiliary photo taking, and intelligent screen-off display.

In some implementation manners, after subscribing to the first service, the above method further includes: the first algorithm platform sends a second message to the camera module, where the second message is used to request the camera module to output images according to the first size specification.

In some implementations, after obtaining the first image analysis result, the above method further includes: the first algorithm platform judges that the first image analysis result satisfies the first preset trigger condition; the first algorithm platform sends a third message to the camera module, The third message is used to request the camera module to output images according to the second size specification.

In some implementations, after the first algorithm platform judges that the first image analysis result satisfies the first preset trigger condition, the above method further includes: the first algorithm platform reports the fourth message to the smart perception application; the smart perception application responds to the first The fourth message is to trigger the electronic device to display the prompt information of the first service in the preset area of the screen, and the fourth message is used to indicate that the first service supported by the smart perception application is triggered.

In some implementation manners, before making a response according to the preset policy corresponding to the second image analysis result, the above method further includes: the first algorithm platform reports the second image analysis result to the smart perception application.

Wherein, the responding according to the preset strategy corresponding to the second image analysis result includes: the smart perception application responds according to the preset strategy corresponding to the second image analysis result; At least one of the following: page or answer incoming calls, smart code recognition, watch the screen without turning off the screen, watch the screen to reduce the volume, smart horizontal and vertical screen, auxiliary photo taking, and smart screen-off display.

In some implementations, the system framework also includes an intelligent perception interface. The smart perception application and the first algorithm platform perform data interaction through the smart perception interface.

In some implementations, the system framework also includes a business management module, a distributor, a smart perception client and a native platform client.

The above method further includes: when the first image analysis result satisfies the first preset trigger condition, the business management module determines the first processing task, and sends the first processing task to the smart perception client through the distributor, and the smart perception client sends the first processing task to the smart perception client. The first processing task is forwarded to the first algorithm platform. Wherein, the first processing task is a task of processing multiple frames of the second image.

The above method further includes: when the first image analysis result satisfies the second preset trigger condition, the business management module determines the second processing task, and sends the second processing task to the native platform client through the distributor, and the native platform client will The second processing task is forwarded to the second algorithm platform. Wherein, the second processing task is a task of processing multiple frames of the third image.

In some implementation manners, the system framework further includes an algorithm library, which includes the first image processing algorithm and the second image processing algorithm.

The performing image analysis on multiple frames of the first image by the first algorithm platform to obtain the first image analysis result includes: calling the first image processing algorithm in the algorithm library by the first algorithm platform to perform image analysis on the multiple frames of the first image , to obtain the first image analysis result.

The performing image analysis on multiple frames of second images through the first algorithm platform to obtain the second image analysis results includes: calling the second image processing algorithm in the algorithm library by the first algorithm platform to perform image analysis on multiple frames of second images , to obtain the second image analysis result. Wherein, the requirement of the first image processing algorithm on the image resolution is lower than the requirement of the second image processing algorithm on the image resolution.

Through the proposal of this application, this application adds a new intelligent perception algorithm platform on the basis of the AON algorithm platform provided by the original chip. In the AON intelligent perception scene, the camera module is always on and collects images to intelligently perceive user services, which is not limited to the realization of native The face unlocking screen service supported by the AON algorithm platform provided by the chip can also be widely used in various other scenarios (such as air gestures, smart scanning codes, smart gaze, etc.), which greatly meet user needs.

In a second aspect, the present application provides an image processing device, which includes a unit for executing the method in the first aspect above. The device may correspond to executing the method described in the first aspect above. For the relevant description of the units in the device, please refer to the description in the first aspect above, and details are not repeated here for brevity.

Wherein, the method described in the first aspect above may be realized by hardware, or may be realized by executing corresponding software by hardware. Hardware or software includes one or more modules or units corresponding to the functions described above. For example, a processing module or unit, a display module or unit, etc.

In a third aspect, the present application provides an electronic device, the electronic device includes a processor, a processor, and a computer program or instruction stored in a memory, and the processor is used to execute the computer program or instruction, so that the method in the first aspect is executed .

In a fourth aspect, the present application provides a computer-readable storage medium on which is stored a computer program (also referred to as an instruction or code) for implementing the method in the first aspect. For example, when the computer program is executed by a computer, the computer can execute the method in the first aspect.

In a fifth aspect, the present application provides a chip, including a processor. The processor is used to read and execute the computer program stored in the memory, so as to execute the method in the first aspect and any possible implementation manners thereof. Optionally, the chip further includes a memory, and the memory is connected to the processor through a circuit or wires.

In a sixth aspect, the present application provides a chip system, including a processor. The processor is used to read and execute the computer program stored in the memory, so as to execute the method in the first aspect and any possible implementation manners thereof. Optionally, the chip system further includes a memory, and the memory is connected to the processor through a circuit or wires.

In a seventh aspect, the present application provides a computer program product, where the computer program product includes a computer program (also referred to as an instruction or code), and when the computer program is executed by an electronic device, the electronic device implements the method in the first aspect .

It can be understood that, for the beneficial effects of the above-mentioned second aspect to the seventh aspect, reference can be made to the relevant description in the above-mentioned first aspect, and details are not repeated here.

Description of drawings

FIG. 1 is a schematic diagram of an application scenario of an image processing method provided in an embodiment of the present application;

FIG. 2 is a schematic structural diagram of an electronic device provided in an embodiment of the present application;

FIG. 3 is a schematic diagram of the relationship between the camera module and the processor in the embodiment of the present application;

Fig. 4 is the schematic diagram of the software framework adopted in the embodiment of the present application;

FIG. 5 is a timing diagram of interaction between modules in the image processing method provided by the embodiment of the present application;

FIG. 6 is a schematic diagram of function settings of an image processing method provided in an embodiment of the present application;

FIG. 7 is a schematic interface diagram of the image processing method provided by the embodiment of the present application during application;

FIG. 8 is a first schematic flow diagram of the image processing method provided by the embodiment of the present application;

FIG. 9 is a second schematic flow diagram of the image processing method provided by the embodiment of the present application;

Fig. 10 shows a schematic interface diagram of an intelligent AOD scene applied in the embodiment of the present application

FIG. 11 is a schematic flow diagram III of the image processing method provided by the embodiment of the present application;

Fig. 12 shows a schematic interface diagram of an air screenshot scene applied in the embodiment of the present application.

Detailed ways

In order to make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments It is a part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the scope of protection of this application.

The term "and/or" in this article is an association relationship describing associated objects, which means that there can be three relationships, for example, A and/or B can mean: A exists alone, A and B exist simultaneously, and B exists alone These three situations. The symbol "/" in this document indicates that the associated object is an or relationship, for example, A/B indicates A or B.

The terms "first" and "second" and the like in the specification and claims herein are used to distinguish different objects, rather than to describe a specific order of objects. In the description of the embodiments of the present application, unless otherwise specified, "multiple" means two or more, for example, multiple processing units refer to two or more processing units, etc.; multiple A component refers to two or more components or the like.

In the embodiments of the present application, words such as "exemplary" or "for example" are used as examples, illustrations or illustrations. Any embodiment or design scheme described as "exemplary" or "for example" in the embodiments of the present application shall not be interpreted as being more preferred or more advantageous than other embodiments or design schemes. Rather, the use of words such as "exemplary" or "such as" is intended to present related concepts in a concrete manner.

At present, smart electronic devices such as mobile phones can use an always-on camera (AON) function, which can quickly unlock the screen with a face when the mobile phone is in an off-screen state. Specifically, when the AON function is applied to the mobile phone, the front camera of the mobile phone camera is always on, and images can be collected in real time, and face recognition can be performed through image analysis, which can achieve faster response to face unlocking the screen, without the need for users to Touch the phone to unlock the phone screen.

As mentioned above, the AON function can be applied to face unlock screen scenarios when the screen of the mobile phone is off. However, since users need to use the mobile phone with the screen on in most cases, the application scenarios of the AON function are too limited.

In view of this, the embodiment of the present application provides an image processing method and electronic equipment, which are improved at the bottom layer of the mobile phone system, and a new intelligent perception algorithm platform is built on the basis of the original AON platform. Based on the support of the intelligent perception algorithm platform, the AON function not only It can be applied to the face unlock screen scene when the mobile phone screen is off, and the always on display (AOD) scene after face recognition when the mobile phone screen is off, and can also be applied to gesture control when the mobile phone screen is on Screen, smart code recognition, keep watching the screen, lower the volume when watching, assist in taking pictures, intelligent horizontal and vertical screens and other scenarios, so it can improve the user experience.

Fig. 1 shows a schematic diagram of application scenarios involved in various exemplary embodiments of the present application. As shown in (a) in Figure 1, the mobile phone is in a bright screen state, and the front camera of the mobile phone is always on and collects images in real time. As shown in (b) in Figure 1, the mobile phone is in the off-screen state, and the front camera of the mobile phone is always on and collects images in real time. In the embodiment of this application, the AON function allows the camera device (front camera) of the electronic device to be in an active state all the time, which can realize the camera is always on under low power consumption, and can make the scanning, gesture recognition, face unlocking and other scenes complete faster .

This application adds a new intelligent perception algorithm platform based on the original AON algorithm platform. In the AON intelligent perception scene, the front camera is always on and collects images to intelligently perceive user services, which can be widely used in air gestures and intelligent code scanning. , smart gaze and other scenarios, which greatly meet the needs of users. When the application detects that the collected image meets the first preset trigger condition, it first judges whether the triggering scene is triggered in the bright screen scene or in the off-screen scene, performs business classification and identification, and then distributes the business to the Intelligent perception algorithm platform, and further judge the specific business content. The AON camera of this application operates with extremely low power consumption, can accurately perceive user services, and improves user experience.

Among them, the AON function provided by the embodiment of the present application can not only be widely used in various scenarios to meet user needs, but also through the improved algorithm of the present application, the mobile phone camera operates with extremely low power consumption, saving mobile phone power consumption. The specific algorithm will be described in detail below.

The image processing method provided in the embodiment of the present application may be applied to an electronic device with a front-facing camera shooting function. Electronic equipment includes various terminal equipment, and the terminal equipment may also be called a terminal (terminal), user equipment (user equipment, UE), mobile station (mobile station, MS), mobile terminal (mobile terminal, MT) and so on. The terminal device can be a mobile phone, a smart TV, a wearable device, a tablet computer (Pad), a computer with a wireless transceiver function, a virtual reality (virtual reality, VR) terminal device, an augmented reality (augmented reality, AR) terminal Equipment, wireless terminals in industrial control, wireless terminals in self-driving, wireless terminals in remote medical surgery, wireless terminals in smart grid, transportation Wireless terminals in transportation safety, wireless terminals in smart city, wireless terminals in smart home, etc. The embodiment of the present application does not limit the specific technology and specific device form adopted by the terminal device.

Referring to FIG. 2 , it is a schematic structural diagram of an electronic device provided by an embodiment of the present application. The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (universal serialbus, USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, Mobile communication module 150, wireless communication module 160, audio module 170, speaker 170A, receiver 170B, microphone 170C, earphone jack 170D, sensor module 180, button 190, motor 191, indicator 192, camera 193, display screen 194, and user An identification module (subscriber identification module, SIM) card interface 195 and the like. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a touch sensor 180K, and an ambient light sensor 180L.

It can be understood that, the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 100 . In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown in the figure, or combine certain components, or separate certain components, or arrange different components. The illustrated components can be realized in hardware, software or a combination of software and hardware.

The processor 110 may include one or more processing units, for example: the processor 110 may include an application processor (application processor, AP), a modem processor, a graphics processing unit (graphics processing unit, GPU), an image signal processor ( image signal processor, ISP), controller, memory, video codec, digital signal processor (digital signal processor, DSP), baseband processor, and/or neural network processor (neural-network processing unit, NPU), etc. . Wherein, different processing units may be independent devices, or may be integrated in one or more processors. For example, the processor 110 is configured to execute the image processing method in the embodiment of the present application.

In the embodiment of this application, the implementation of the AON function depends on the ISP. ISP can run various algorithm programs and process image signals in real time. FIG. 3 shows the connection relationship among the camera module, ISP, and AP.

(a) in FIG. 3 shows an architecture diagram of an external ISP, the camera module is connected to the ISP through a camera serial interface (cameraserial interface, CSI), and the ISP is connected to the AP. Wherein, the ISP is arranged separately, and the ISP is independent from the AP. After the ISP receives the image collected by the camera module, the ISP performs image signal processing on the image collected by the camera module. The AP can control the working mode of the ISP through a built-in integrated circuit (inter-integrated circuit, I2C), and obtain the working status of the ISP, and the like. (b) in FIG. 3 shows the architecture diagram of the built-in ISP, wherein the camera module is connected to the AP, and the ISP is built in the AP. After the AP receives the image collected by the camera module, the image signal is processed by the ISP.

Wherein, the controller may be the nerve center and command center of the electronic device 100 . The controller can generate an operation control signal according to the instruction opcode and timing signal, and complete the control of fetching and executing the instruction.

A memory may also be provided in the processor 110 for storing instructions and data. In some embodiments, the memory in processor 110 is a cache memory. The memory may hold instructions or data that the processor 110 has just used or recycled. If the processor 110 needs to use the instruction or data again, it can be directly called from the memory. Repeated access is avoided, and the waiting time of the processor 110 is reduced, thereby improving the efficiency of the system.

The external memory 120 generally refers to an external memory. In the embodiment of the present application, the external memory refers to a storage other than the internal memory of the electronic device and the cache memory of the processor, and the storage is generally a non-volatile memory.

The internal storage 121, which may also be referred to as "memory", may be used to store computer-executable program codes including instructions. The internal memory 121 may include an area for storing programs and an area for storing data. Wherein, the stored program area can store an operating system, at least one application program required by a function (such as a sound playing function, an image playing function, etc.) and the like.

In the embodiment of the present application, the memory of the electronic device adopts a tightly coupled memory (tightly coupled memory, TCM) and a DDR memory.

Among them, DDR memory is a double rate synchronous dynamic random access memory, and the full name of DDR memory is DDR SDRAM (double data rate SDRAM, double rate SDRAM). SDRAM is the abbreviation of synchronous dynamic random access memory (synchronous dynamic random access memory). The data transfer rate of DDR memory is very high.

Among them, the TCM is included in the address mapping space of the memory and can be accessed as a fast memory. TCM is used to provide low-latency memory to the processor without the unpredictability characteristic of caches. You can use the TCM to hold important routines, such as interrupt handling routines or real-time tasks that are critically avoiding cache non-determinism. In addition, the TCM can be used to hold scratchpad data, data types whose local attributes do not fit in the cache, and important data structures such as the interrupt stack.

It should be noted that from the perspective of storage capacity, TCM memory capacity is relatively small, while DDR memory capacity is relatively large. From the perspective of data transmission rate, the data transmission rate of DDR memory is higher than that of TCM memory. From the perspective of power consumption, the power consumption of DDR memory is greater than that of TCM memory. Electronic devices usually use DDR memory, so the power consumption is relatively large; in some scenarios of the present application, the TCM memory is used for data caching, which can reduce power consumption.

Exemplarily, in the embodiment of this application, in some pre-pre-sensing service scenarios, both the dynamic memory and the static memory in the electronic device use TCM memory to store continuous multi-frame images collected by the front camera. These images are An image of QQVGA image size specification (also called image specification or size specification), where the QQVGA image specification is a smaller image size, such as 160×120, which can be expressed as QQVGA (160×120), understandably, the QVGA image specification The image resolution is lower and the image takes up less memory.

Exemplarily, in some later scenarios for identifying specific services, the electronic device's dynamic memory and static memory combine DDR memory and TCM memory to jointly store continuous multi-frame images captured by the front camera, and these images are consecutive QVGA image specifications The multi-frame image, wherein, compared with the QQVGA image specification, the QVGA image specification is a larger image size, such as 320×240, which can be expressed as QQVGA(320×240). It can be understood that, compared with the QQVGA image specification, the image resolution of the QVGA image specification is relatively higher, and the image occupies a larger memory.

Wherein, VGA (video graphics array) refers to a video graphics array, and the corresponding resolution is 640×480 pixels. QVGA refers to the 1/4 size of VGA resolution, and the corresponding resolution is 320×240 pixels. QQVGA refers to the 1/4 screen of VGA resolution, and the corresponding resolution is 160×120 pixels.

It can be understood that the image corresponding to the QQVGA specification has the smallest resolution and occupies the smallest memory.

In the embodiment of the present application, when pre-sensing services in the early stage, on the one hand, the front camera collects images of smaller size in QQVGA specification and stores them in the TCM memory, thus occupying a very small memory space; on the other hand, the processor Analyze continuous multi-frame images to determine whether the user has triggered a certain service. For example, if it is detected through image analysis that there are hand features and the initial gesture changes, the processor can determine that the user has triggered the air gesture service; and then For example, if it is detected through image analysis that there are facial features and the screen orientation of the electronic device is deflected, it can be determined that the user has triggered the smart horizontal and vertical screen service.

After sensing that a service is triggered, on the one hand, the front camera collects larger-sized images of QVGA specifications and stores them in the DDR memory and TCM memory; The specific content of the service to be triggered, for example, if it is detected through image analysis that the palm changes from the extended state to the grasped state, then the processor can recognize the grasping gesture and use it to trigger a screen capture.

It can be seen that in the scene where the front camera is always on, although the front camera will continue to collect a large number of images, through the solution of this application, in the process of pre-sensing services in the early stage, by acquiring continuous multiple frames of smaller size Images are stored in low-power TCM memory, and image analysis of continuous multi-frame images of smaller size can greatly reduce power consumption.

The display screen 194 is used to display images, videos and the like. The display screen 194 includes a display panel. The display panel may use an organic light-emitting diode (OLED). In some embodiments, the electronic device 100 may include 1 or N display screens 194 , where N is a positive integer greater than 1.

The electronic device 100 also includes various sensors that can convert various physical signals into electrical signals. Proximity light sensor 180G may include, for example, light emitting diodes (LEDs) and light detectors, such as photodiodes. The proximity sensor can turn off the display screen 194 and/or the backlight when the electronic device 100 is moved to the ear. The ambient light sensor 180L is used for sensing ambient light brightness. The electronic device 100 can adaptively adjust the brightness of the display screen 194 according to the perceived ambient light brightness.

Touch sensor 180K, also known as "touch panel". The touch sensor 180K can be disposed on the display screen 194, and the touch sensor 180K and the display screen 194 form a touch screen, also called a “touch screen”. The touch sensor 180K is used to detect a touch operation on or near it. The touch sensor can pass the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 194 . In other embodiments, the touch sensor 180K may also be disposed on the surface of the electronic device 100 , which is different from the position of the display screen 194 .

Exemplarily, in the embodiment of the present application, the touch sensor 180K can detect the user's click operation on the icon of the application program, and transmit the detected click operation to the application processor, and determine that the click operation is used to start or run the application. program, and then execute the running operation of the application.

The electronic device 100 realizes the display function through the GPU, the display screen 194 , and the application processor. The GPU is a microprocessor for image processing, and is connected to the display screen 194 and the application processor. GPUs are used to perform mathematical and geometric calculations for graphics rendering. Processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

The electronic device 100 can realize the shooting function through the ISP, the camera 193 , the video codec, the GPU, the display screen 194 and the application processor.

The above is a specific description of the embodiment of the present application by taking the electronic device 100 as an example. It should be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 100 . Electronic device 100 may have more or fewer components than shown in the figures, may combine two or more components, or may have a different configuration of components. The various components shown in the figures may be implemented in hardware, software, or a combination of hardware and software including one or more signal processing and/or application specific integrated circuits.

In addition, an operating system runs on top of the above components. For example, the iOS operating system developed by Apple, the Android open source operating system developed by Google, and the Windows operating system developed by Microsoft. Applications can be installed and run on this operating system.

The operating system of the electronic device 100 may adopt a layered architecture, an event-driven architecture, a micro-kernel architecture, a micro-service architecture, or a cloud architecture. The embodiment of the present application takes the Android system with a layered architecture as an example to illustrate the software structure of the electronic device 100 .

FIG. 4 is a software architecture diagram of the electronic device 100 according to the embodiment of the present application. Various embodiments of the present application will be discussed based on the following technical architecture. It should be noted that, for the convenience of explaining the logic, only a schematic block diagram is used to illustrate the business logic relationship, without strictly expressing the specific location of the technical architecture of each business. Moreover, the naming of each module in the software architecture diagram is an exemplary example. The embodiment of the present application does not limit the naming of each module in the software architecture diagram. In actual implementation, the specific naming of modules can be determined according to actual needs.

The layered architecture divides the software into several layers, and each layer has a clear role and division of labor. Layers communicate through software interfaces. In some embodiments, the Android system includes four layers, which are applications, application framework, hardware abstraction layer (HAL), and kernel layer (kernel) from top to bottom. .

Wherein, the application program layer may include a series of application program packages. For example, the application program layer may include application programs such as a camera and smart perception (application programs may be referred to as applications for short), which is not limited in this embodiment of the present application.

The smart perception application provided by the embodiment of the present application supports various services. For example, the services supported by the smart perception application may include air gesture services, smart code recognition services, continuous screen services while watching, volume down services while watching, smart horizontal and vertical Screen business, auxiliary camera business, and smart AOD business. These services can be collectively referred to as smart perception services.

It should be noted that the realization of these services supported by smart perception applications depends on the front camera of the electronic device being always on, collecting images in real time, and obtaining data related to smart perception services. When the smart sensing application monitors business-related data, the smart sensing application sends the business-related data to the smart sensing algorithm platform. The smart sensing algorithm platform analyzes the collected images and judges whether to execute the related business of the smart sensing application according to the analysis results, or Which specific business to perform.

Among them, air gesture business: refers to the business in which electronic devices recognize user gestures and respond according to the preset strategy corresponding to the gestures to realize human-computer interaction. When the electronic device is in the bright screen state, the electronic device can support the recognition of various preset air gestures, and different gestures can correspond to different preset strategies, for example: air grasping gesture→screen capture, air up/down gesture → Turn pages, press gestures in the air → answer calls.

For example, taking the gesture of grasping in the air as an example, the user changes from the state of stretching the palm to the state of making a fist, and the manipulation mode corresponding to the gesture is preset to take a screenshot of the display interface of the electronic device. When the electronic device is in a bright screen state, the electronic device will automatically perform a screen capture operation when the electronic device detects an air-grip gesture through the front camera. Wherein, the air grasp gesture may also be called an air screenshot gesture, and this scene may be called an air screenshot scene.

For example, taking the up/down gesture in the air as an example, the fingers are changed from the state of closing together to the state of bending down (swipe up gesture). The close-up and bending state changes to the upward-expanding state (sliding gesture), and the manipulation method corresponding to this gesture is preset to turn the page down or slide the screen down. When the electronic device is in the bright screen state, the electronic device will automatically perform the operation of turning the page up when it detects the gesture of sliding up through the front camera, and automatically perform the operation of turning the page down when it detects the gesture of sliding down, without the need for users Human-computer interaction can be completed by touching electronic equipment. Wherein, the gesture of up/down in the air may also be called the gesture of sliding the screen in the air, and this scene may be called the scene of sliding the screen in the air.

For another example, take the air press gesture as an example. The user’s palm approaches the screen of the electronic device from far to near. The control method corresponding to this gesture is preset to answer an incoming call. When the air press gesture is recognized through the front camera, the operation of answering the incoming call will be automatically performed. This scene may be called an air press scene.

In addition, in the embodiment of this application, the air gesture service also supports user-defined operations, for example, the user cancels the "air press gesture → answer the phone" and resets it to "air press gesture → jump to the payment code interface" .

For example, when the screen of the electronic device is on and the main screen is displayed, the user can trigger a predefined shortcut service through an air press gesture, such as quickly jumping from the main screen to the payment code interface. In this scenario, the payment interface can be quickly called up by pressing gestures in the air, so this scenario can be called a smart payment scenario. It should be noted that the user can also reset it to "air press gesture → jump to the scan interface" or "air press gesture → jump to the ride code interface".

Among them, intelligent code recognition business: when the electronic device is in the bright screen state, the user aligns the front camera of the electronic device with the QR code, and the electronic device can automatically scan the code and analyze it, and quickly jump to the payment code interface or take a car Code interface, without the need for the user to click the scan entry to trigger the code scan.

Among them, the business of not turning off the screen when watching the electronic device: when the electronic device is in the bright state, if the electronic device detects that the user is looking at the electronic device through the front camera, the display screen of the electronic device will not be turned off until the user no longer looks at the electronic device .

Among them, watching and reducing the volume service: when the electronic device is in the bright screen state, if the electronic device detects that the user is looking at the electronic device through the front camera, the electronic device will automatically reduce the volume of the ringtone of the incoming call or the volume of the SMS prompt tone. For example, when a user receives an incoming call, the user can trigger the mobile phone to automatically lower the volume of the ringtone by looking at the screen of the mobile phone.

Optionally, the services of not turning off the screen while watching and reducing the volume of watching can be enabled by default, or can be enabled or disabled separately.

Among them, the smart horizontal and vertical screen business: when the electronic device is in the bright screen state, if the electronic device detects that the screen of the electronic device is deflected, and the electronic device detects the direction of the user's face through the front camera, then the electronic device automatically Rotate the screen so that the content displayed on the screen is always positive relative to the direction of the user's face. Compared with the automatic screen rotation function implemented by the gravity sensor in the prior art, the intelligent horizontal and vertical screen function realized by the present application can better meet the needs of users.

Among them, auxiliary camera service: when the electronic device is in the bright screen state, if the electronic device detects that the electronic device has turned on the camera function, and the electronic device detects the direction of the user's face through the front camera, then the electronic device automatically rotates according to the direction of the user's face screen, so that the content displayed on the screen is always positive relative to the direction of the user's face. Moreover, when the electronic device displays the photos taken, the electronic device automatically rotates the screen according to the orientation of the user's face, so that the photos displayed on the screen are always facing forward relative to the orientation of the user's face.

Among them, the smart AOD business: when the electronic device is in the off-screen state, if the electronic device detects facial features through the front camera, the electronic device will automatically display the screen-off pattern. When the electronic device displays the screen-off pattern, some pixels of the screen are lit to display information such as clock, date, notification, etc., which is convenient for users to view the above information. Since the mobile phone screen is partially lit, the power consumption of the electronic device can be reduced . It should be noted that the prerequisite for realizing the smart AOD service is that the electronic device supports the off-screen display function and the off-screen display function has been turned on.

It should be noted that the above-mentioned various services supported by the smart perception application are illustrative examples. It can be understood that in actual implementation, the smart perception application in the embodiment of the present application can also support other possible services, which can be specifically determined according to actual use requirements. Definitely, the embodiment of this application is not limited.

The application framework layer provides an application programming interface (application programming interface, API) and a programming framework for applications in the application layer. The application framework layer includes some predefined functions. Exemplarily, the application framework layer may include a camera management module, an AON management module, etc., which are not limited in this embodiment of the present application.

The hardware abstraction layer is the encapsulation of the Linux kernel driver, providing an interface upward, which hides the details of the hardware interface of a specific platform, and provides a virtual hardware platform for the operating system. In the embodiment of this application, the hardware abstraction layer includes modules such as smart perception interface and service, business management, business conflict management, business capability management, distributor, smart perception client, and native platform client.

Among them, the intelligent perception interface and service module provides an interface program for upper-level intelligent perception applications to access the intelligent perception algorithm platform, and provides services related to the intelligent perception algorithm platform (also known as the intelligent perception algorithm integration framework). This application adds a set of self-owned AON services in the camera HAL, and supports the development of features such as gesture features and code scanning features from the platform software infrastructure.

Among them, the service management module is used to manage multiple services such as air gesture service, intelligent code recognition service, watch the screen service, watch the volume down service, intelligent horizontal and vertical screen service, auxiliary camera service, and intelligent AOD service. Moreover, the business management module can call the business conflict management module to make a decision.

Wherein, the business conflict management module is used to make decisions on the business of the on-screen scene and the business of the off-screen scene. The business conflict management module can make a business judgment based on the business-related data monitored by the upper-layer smart perception application: whether the business-related data is an image collected when the screen is on or when the screen is off. In some embodiments, if the business-related data is an image collected when the screen of the electronic device is on, the business management module may send the business-related data to the smart perception client. If the service-related data is an image collected when the screen of the electronic device is off and there are facial features in the image, the service management module can determine that the smart AOD service is triggered, and send the service-related data to the smart perception client. Among them, the intelligent perception client has the function of acting as an agent and forwarding data.

Among them, the business capability management module is used to manage the capability information of each business supported by the intelligent perception algorithm platform. In some embodiments, the capability information of the service can be represented by numbers. For example, the capability value of the intelligent code recognition service is preset to 0; the capability value of the air gesture service is preset to 1, wherein the capability of taking screenshots The value is preset to 1-1, the ability value of air sliding screen is preset to 1-2, and the ability value of air pressing is preset to 1-3. It should be noted that the capability information here is an example, and may be specifically set according to actual usage requirements, and is not limited in this embodiment of the present application.

Specific to this application scheme, when a user triggers to start a certain service, the smart perception application can obtain the capability information of the service, and then the smart perception application registers with the smart perception algorithm platform according to the service capability information and subscribes to the corresponding service. After the service subscription is successful, the smart perception application will monitor whether the service is triggered in real time, and when the smart perception application detects that the service is triggered, the smart perception application will send the business data to the smart perception algorithm platform, through the smart perception algorithm The platform conducts algorithm analysis, and then executes the business according to the analysis results.

Wherein, the distributor is used for distributing business-related data according to the instruction of the business management module.

The kernel layer is the layer between hardware and software. Exemplarily, the kernel layer may include a display driver, a camera driver and a sensor driver.

Specific to the solution of this application, the kernel layer may include API interface and sensor interface, intelligent perception algorithm integration framework (that is, intelligent perception algorithm platform), and native AON algorithm platform.

Among them, the smart perception algorithm platform can include decision-making units, control units, and execution units. These units work together to complete the low-power algorithm analysis of business-related data, obtain image analysis results, and report the image analysis results to smart perception. application. The smart perception algorithm platform can also be called a low-power subsystem.

It should be noted that, in the embodiment of the present application, in the early business perception stage (first stage), the smart perception client performs image analysis on the business-related data to monitor whether the business-related data meets the first preset trigger condition, Therefore, the image quality requirements are lower. In the later stage of business identification (the second stage), the smart perception algorithm platform identifies specific business content through image analysis of business-related data that meets the first preset trigger condition, so it has high requirements for image quality.

In the first stage, business-related data includes continuous multi-frame images, all of which are small in size and low in resolution, and the low-power TCM memory described above is used for data caching. In the second stage, business-related data includes continuous multi-frame images, all of which are large in size and high in resolution, and the normal power consumption DDR memory and low power consumption TCM memory described above are used for data caching. The solution of the present application performs image analysis in stages, which not only can accurately monitor services, but also effectively reduces power consumption.

It should be noted that in the embodiment of this application, on the basis of the original AON algorithm platform, the smart perception service and the smart perception algorithm integration framework are mainly added. This application cooperates with the smart sensing service and smart sensing algorithm integration framework and the native AON algorithm platform to complete the low-power AON smart sensing function and business.

On the one hand, the smart perception service provides a unified service interface to support smart perception-related businesses using the AON capability of the native AON algorithm platform; at the same time, the interface of the native AON algorithm platform is shielded, and the native interface is no longer open to the outside world.

On the other hand, the smart perception service relies on the platform communication channel to establish a complete business interaction with the smart perception algorithm integration framework to use the low-power computing capabilities of AON cameras and native AON algorithm platforms.

On the other hand, the intelligent perception algorithm integration framework builds a unified low-power computing platform to support AON services by integrating AON camera resources, traditional sensor data, and integrated business algorithms.

The hardware layer provides various hardware devices. For example, the hardware devices involved in the embodiments of the present application include AON ISP, camera sensors, and cameras. Among them, the camera and the camera sensor are used to collect images in real time, and the AON ISP is used to process the image signal of the collected images. These hardware devices provide hardware support for the intelligent perception algorithm platform.

It should be noted that FIG. 4 only shows modules related to the embodiment of the present application, and each layer may also include any other possible modules, and each module may also include one or more sub-modules, which are not limited in this application.

It should also be noted that although the embodiment of the present application uses the Android system as an example for illustration, its basic principles are also applicable to electronic devices based on operating systems such as iOS or Windows.

The following will describe the sequence diagram of each module interaction in the above-mentioned technical framework provided by the embodiment of the present application through FIG. ), the intelligent perception client, the intelligent perception algorithm platform and the front camera for information interaction, which exemplarily illustrates the process of electronic devices realizing AON smart perception through the front camera.

S101. After the electronic device is turned on, the smart perception service module interacts with the smart perception client to start a camera provider process.

S102. The smart perception service module applies for registration to the smart perception client.

S103. The smart sensing application receives an operation of enabling the smart sensing function by the user.

It should be noted that the smart sensing function here can be any of the following: air gesture function, smart code recognition function, gazing function that does not turn off the screen, gazing function that reduces volume, smart horizontal and vertical screen function, auxiliary camera function, and smart AOD function.

Exemplarily, FIG. 6 shows a schematic diagram of an interactive interface for a user to activate the smart perception function. As shown in (a) and (b) of FIG. 6 , the electronic device displays a setting main interface in response to the user's operation of clicking the setting application icon on the desktop. Accessibility options are displayed on the main setting interface. As shown in (b) and (c) of FIG. 6 , the electronic device displays an auxiliary function interface in response to the user's operation of clicking the auxiliary function option. The smart perception option is displayed in the accessibility interface. As shown in (c) and (d) of FIG. 6 , the electronic device displays a smart perception setting interface in response to the user's operation of clicking the smart perception option.

As shown in (d) of FIG. 6 , the intelligent perception setting interface displays a smart gaze setting bar, an air gesture setting bar and other setting bars. The smart gaze setting bar includes a switch to keep the screen on while watching the screen, and a switch to reduce the volume while watching the screen. The air gesture setting bar includes the switch of sliding screen in the air, the switch of screenshot in the air, the switch of pressing in the air, and the switch of smart payment. The other settings bar includes the switch of smart AOD, the switch of smart horizontal and vertical screen, and the switch of auxiliary camera.

Users can turn on or off one or more of the above-mentioned functions according to actual usage needs. For the convenience of description, the following takes the user triggering to enable the function of taking a screenshot through the air as an example for an exemplary description.

S104, the smart perception application queries the capability information of the space screenshot service from the smart perception client in response to the user triggering the operation of enabling the space screenshot function.

Among them, the smart perception application sends a message of querying business capabilities to the framework layer module, and the framework layer module sends the message to the smart perception interface and service module, and then the smart perception interface and service module deliver the message to the smart perception client. The smart perception client can call the capability management module to query the capability information of the air screenshot service.

Wherein, different services have different capability values. For example, the ability value of the screen capture function in the air is 1-1, the ability value of sliding the screen in the air is 1-2, and the ability value of pressing the screen in the air is 1-3.

S105. The smart perception client feeds back the queried service capability information to the smart perception application.

S106. The smart perception application registers the AON service event with the smart perception algorithm platform according to the service capability value.

Among them, the registered data flow starts from the smart perception application, passes through the framework layer module, smart perception interface and service module, smart perception client, and reaches the smart perception algorithm platform.

S107, the smart perception algorithm platform subscribes to the AON service event according to the service capability value.

For example, assuming that the business capability value is 1-1, as mentioned above, the capability value of the air-screen capture function is 1-1, then the AON service event subscribed by the smart perception algorithm platform is an air-space screen capture.

Among them, in the case of subscribing to AON service events, the intelligent perception algorithm platform will adopt lower resolution image specifications in some cases, such as QQVGA (160×120), and higher resolution in other cases. Image format, such as QVGA (320×240). In addition, the intelligent perception algorithm platform will use TCM memory for data caching in some cases according to requirements, and use DDR memory and TCM memory for data storage in other cases.

The following S108-S117 is the cyclic execution process of AON smart perception.

S108, the front camera collects continuous multi-frame images when the screen of the device is turned on and off, and transmits the collected continuous multi-frame images to the intelligent perception algorithm platform.

Wherein, the front camera refers to an AON camera.

S109, the intelligent perception algorithm platform acquires continuous multi-frame images of QQVGA specification, stores the images in the TCM memory, and analyzes the continuous multi-frame images.

Exemplarily, the AON camera (front camera) outputs images with lower resolution according to the QQVGA (160×120) specification. Correspondingly, the smart perception algorithm platform acquires images with a lower resolution of the QQVGA (160×120) specification for image analysis.

S110, the intelligent perception algorithm platform determines whether the first preset trigger condition is met according to the analysis result.

Exemplarily, the above-mentioned first preset trigger condition may include: the image includes hand features, or the image includes two-dimensional code features, or the image includes human eye features, or the image includes human face features. It should be noted that the first preset trigger condition here is described as an example, and may be specifically set according to actual usage requirements, and is not limited in this embodiment of the present application.

For example, assuming that the analysis result shows that there are hand features in the image and the first preset trigger condition is met, it can be determined that the air gesture service is triggered.

For another example, assuming that the analysis result shows that there are human eye features in the image and the first preset trigger condition is met, it can be determined that the smart gaze service is triggered.

S111, when it is judged according to the analysis result that the first preset trigger condition is not satisfied, the smart perception client does not respond, and releases the invalid frame.

After it is judged in S110 that the first preset trigger condition is met, continue to execute S112-S113, and after it is judged in S110 that the first preset trigger condition is met, continue to execute S114.

S112. The smart perception client reports a message that the service is triggered to the smart perception application.

S113, the smart perception application displays prompt information corresponding to the triggered service on the screen.

Exemplarily, FIG. 7 shows a schematic diagram of an interactive interface when the air gesture service is triggered. As shown in (a) in Figure 7, the front-facing camera captures an image of the user's palm spreading out. After image analysis, it can be determined that the air gesture service is triggered. At this time, the image 11 of the palm spreading out can be displayed on the screen of the mobile phone. As shown in (b) in Figure 7, the front-facing camera captures the image that the user folds and expands. After image analysis, it can be determined that the air gesture service triggered by the user is specifically an air screenshot service. Correspondingly, the mobile phone screen at this time An image 12 of palms together can be displayed, and a prompt message of "screen capture" can be displayed. In this way, after seeing the prompt information displayed on the mobile phone, the user can know that the air gesture made by him/her is being recognized by the mobile phone, which improves the user's interactive experience.

S114, the intelligent perception algorithm platform acquires continuous multi-frame images of QVGA specification, stores the images in DDR memory and TCM memory, and analyzes the continuous multi-frame images to obtain image analysis results.

Exemplarily, the AON camera (front camera) outputs images with higher resolution according to the QVGA (320×240) specification. Correspondingly, the intelligent perception algorithm platform acquires higher-resolution images of the QVGA (320×240) specification for image analysis.

It should be noted that the DDR memory and the TCM memory run in different hardware buffer areas. Among them, the DDR memory can access the data in the TCM memory, but the TCM memory does not support accessing the data in the DDR memory.

S115. The intelligent perception algorithm platform reports the image analysis result to the intelligent perception application.

S116. The smart perception application responds according to the preset policy corresponding to the image analysis result.

For example, after analyzing multiple consecutive frames of images, it is found that the user's gesture is specifically a grasping gesture, that is, the analysis result of service 1 is a grasping gesture, and the preset strategy corresponding to the grasping gesture is an air screenshot. Therefore, the smart perception application responds according to the preset strategy corresponding to the grasping gesture, such as taking a screenshot in the air.

The above describes the process of the user triggering the smart perception function and executing AON smart perception. In some embodiments, the user can trigger the shutdown of the smart perception function according to usage requirements. For the specific process, refer to the following S117-S119.

S117. The smart sensing application receives an operation of disabling the smart sensing function from the user.

Here, the smart sensing function can be any of the following: air gesture function, smart code recognition function, gazing at the screen without turning off, gazing at the volume down function, smart horizontal and vertical screen function, auxiliary camera function, and smart AOD function. Users can turn off one or more of the above-mentioned functions according to actual usage needs. For the convenience of description, the following takes the user's trigger to close the air screenshot function as an example for an exemplary description.

S118. In response to the user operation, the smart perception application sends a message of canceling the AON service event to the smart perception algorithm platform.

S119, the intelligent perception algorithm platform stops subscribing to AON service events.

Exemplarily, the AON service event is an air screenshot. After the subscription to the air screenshot is stopped, the air screenshot service will not be triggered, that is, the mobile phone will no longer respond to the air screenshot gesture.

Based on the software architecture shown in FIG. 4 above, combined with the sequence diagram shown in FIG. 5 , the service perception and recognition in the image processing method provided by the embodiment of the present application are described from the partial perspective of the newly added smart algorithm platform. The following describes the image processing method provided by the embodiment of the present application in stages from the perspective of the original AON algorithm platform and the newly added smart algorithm platform in combination with the flow chart shown in FIG. 8 .

First of all, it needs to be explained that the original AON algorithm platform supports the processing of face unlocking screen business, and the newly added smart algorithm platform supports the processing of business including air gesture business, smart code recognition business, watching the screen without turning off the screen business, watching The business of lowering the volume of the screen, the business of intelligent horizontal and vertical screen, the business of auxiliary photo taking, and the business of intelligent screen-off display.

S201, collecting continuous multi-frame images in real time through an AON camera (front camera) of the electronic device.

S202. Acquire continuous multi-frame images with lower resolution (such as QQVGA specification), use low-power TCM memory for data buffering, and analyze the continuous multi-frame images.

Among them, the AON camera (front camera) outputs images with higher resolution according to the QVGA (320×240) specification. Correspondingly, the intelligent perception algorithm platform acquires higher-resolution images of the QVGA (320×240) specification for image analysis.

S203. Determine whether a first preset trigger condition is met according to the image analysis result.

Wherein, it may be judged whether a service is triggered by judging whether the analysis result satisfies the first preset trigger condition.

Exemplarily, the first preset trigger condition may be any of the following: the image contains the first preset feature, and the image is an image collected when the screen is turned on and unlocked; or, the image contains human face features , and the image is an image captured when the screen is off. Wherein, the first preset feature may include any one of human face feature, human eye feature, gesture feature and two-dimensional code feature.

On the one hand, when it is judged in S204 that the first preset trigger condition is satisfied, the following S204-S207 are continued to be executed. That is, business data processing is performed through the newly added intelligent perception algorithm platform. As mentioned above, the preset services supported by the newly added intelligent perception algorithm platform can include air gesture services, intelligent code recognition services, watch-on screen services, watch-down volume services, intelligent horizontal and vertical screen services, auxiliary camera services, smart AOD business can.

S204. Acquire continuous multi-frame images with higher resolution (for example, QVGA specification), and use DDR memory and TCM memory for data buffering.

S205, analyzing continuous multi-frame images through the newly added intelligent perception algorithm platform, and obtaining image analysis result 1.

S206, the newly added intelligent perception algorithm platform reports the image analysis result 1 to the application layer.

S207. Make a response according to the preset strategy corresponding to the image analysis result 1.

For example, the multi-frame continuous images collected by the front-facing camera are analyzed. At this time, the multi-frame continuous images use low-resolution images and use low-power memory for caching. If it is found that multiple frames of continuous images satisfy the first preset trigger condition, it can be determined that a service is triggered, for example, multiple frames of continuous images include a two-dimensional code feature, and thus it can be determined that the triggered service is an intelligent code recognition service. It can be seen from the judgment that the intelligent code recognition service is a preset service in the newly added intelligent perception algorithm platform, so the business analysis task is assigned to the newly added intelligent perception algorithm platform, and the newly added intelligent perception algorithm platform analyzes the multi-frame Continuous images are used for image analysis. At this time, multi-frame continuous images use high-resolution images, and normal power consumption memory and low power consumption memory are used for caching.

On the other hand, when it is judged in S204 that the first preset trigger condition is not met, the following S208-S212 are continued to be executed. That is, business data processing is performed through the native AON algorithm platform.

S208, first judge whether the second preset trigger condition is met according to the image analysis result.

Exemplarily, the second preset trigger condition is that the image contains facial features, and the image is an image collected when the screen is on and locked.

S209, when it is judged that the second preset trigger condition is met, acquire continuous multi-frame images with higher resolution (for example, VGA specification), and use DDR memory for data buffering.

S210, analyzing continuous multi-frame images through native AON algorithm platform to obtain image analysis result 2.

S211. The native AON algorithm platform reports the image analysis result 2 to the application layer.

S212. Make a response according to the preset strategy corresponding to the image analysis result 2.

For example, the multi-frame continuous images collected by the front-facing camera are analyzed. At this time, the multi-frame continuous images use low-resolution images and use low-power memory for caching. If it is found that multiple frames of continuous images satisfy the first preset trigger condition, it can be determined that a service is triggered, for example, the multiple frames of continuous images collected when the screen of the electronic device is turned on and locked include facial features, so it can be determined that it is triggered The business is face unlocking screen business. It can be seen from the judgment that the face unlocking screen service is a preset service in the native AON algorithm platform, so the business analysis task is assigned to the native AON algorithm platform, and the native AON algorithm platform performs image analysis on multiple frames of continuous images. Multi-frame continuous images use high-resolution images, and use normal power consumption memory and low power consumption memory for caching.

It should be noted that, in the embodiment of the present application, when it is judged that neither the first preset trigger condition nor the second preset trigger condition is satisfied, that is, when no service is triggered, the collected continuous multi-frame images are discarded, Then return to execute the above S201.

Among them, the embodiment of the present application implements AON smart perception together through the newly added smart perception algorithm platform and the original AON algorithm platform, wherein the image processing process completed through the smart perception algorithm platform can be divided into two stages:

The first stage: as in S201-S203 above, roughly analyze the multi-frame continuous images collected by the front camera, and determine whether the multi-frame continuous images meet the first preset trigger condition, so as to determine whether any service is triggered. In the first stage, low-resolution images are used for multi-frame continuous images, and low-power memory is used for data caching.

The second stage: as in S204-S207 above, finely analyze the multi-frame continuous images collected by the front camera to identify the specific content of the triggered service. In the second stage, multi-frame continuous images use high-resolution images, and use normal power consumption memory and low power consumption memory together for data caching.

The above describes the image processing method provided by the embodiment of the present application in stages with reference to the flow chart shown in FIG. 8 . The following describes the service identification process in the image processing method provided in the embodiment of the present application by making a judgment according to the on-screen/off-screen scene with reference to the flow chart shown in FIG. 9 .

It should be noted that the following assumes that the newly-added smart perception algorithm platform supports the gesture business, smart code recognition business, watch the screen service, watch the volume down service, intelligent horizontal and vertical screen business, auxiliary camera service, smart AOD and other services All have been enabled, among which the smart AOD service is applied to the off-screen scene, and the other services are applied to the on-screen scene.

As shown in Figure 9, the service identification process includes S301-S316.

S301, collecting continuous multi-frame images in real time through the front camera of the electronic device.

S302. Acquire continuous multi-frame images with lower resolution, and use low-power consumption TCM memory for data buffering, and analyze the continuous multi-frame images.

Exemplarily, as mentioned above, the continuous multi-frame images can be analyzed by the intelligent perception algorithm platform.

S303. Determine whether a first preset trigger condition is met according to the analysis result.

Similar to the above S203, by judging whether the analysis result satisfies the first preset trigger condition, it is judged whether any service is triggered.

When it is judged in S303 that the first preset trigger condition is not met in the image, the captured image frame is discarded, and the execution returns to S301.

S304. When it is judged that the first preset trigger condition is satisfied, judge whether the service is triggered in the bright screen scene.

When it is judged that the service is triggered in the bright screen scene, continue to execute the following S305. When it is judged that the trigger is not in the bright screen state, continue to execute the following S307.

S305, judging whether the screen is locked.

If the screen is locked, continue to execute S306-S307; if the screen is not locked, that is, the screen is in an unlocked state, continue to execute S308-S3311.

S306. When it is judged that the service is triggered when the screen is turned on and unlocked, obtain higher-resolution continuous multi-frame images, use DDR memory and TCM memory for data caching, and analyze the images through the intelligent perception algorithm platform.

S307. Respond according to the execution strategy corresponding to the image analysis result, such as taking screenshots/turning pages/answering calls in the air, intelligent code recognition, keeping the screen on while watching, lowering the volume while watching, intelligent horizontal and vertical screens, and/or assisting in taking pictures.

It should be noted that the specific image processing method when the screen is turned on and unlocked will be described in detail in FIG. 11 .

S308. When it is determined that the service is triggered when the screen is on and locked, determine whether there are facial features in the image.

S309, when it is judged that there is a face feature in the image, determine that the triggered service is a face unlocking screen service.

S310, analyzing the image through the intelligent perception algorithm platform.

S311, make a response according to the execution policy corresponding to the image analysis result, that is, unlock the screen.

Next, let's look at the image processing method in the off-screen scene.

S312. Determine whether a service is triggered in the off-screen scene.

S313. When it is judged that the service is triggered in the screen-off scene, judge whether there is a human face feature in the image.

When it is judged that there is no human face feature in the image, no response is made, and the execution returns to S301.

S314. When judging that there is a face feature in the image, determine that the triggered service is an intelligent AOD service.

S315 acquires continuous multi-frame images with higher resolution, uses DDR memory and TCM memory for data buffering, and analyzes images through the intelligent perception algorithm platform.

S316, make a response according to the execution policy corresponding to the image analysis result, that is, the smart AOD.

Exemplarily, FIG. 10 shows a schematic interface of an intelligent AOD scene applied in the embodiment of the present application.

Combined with the above-mentioned figure 9, through the branch judgment of the bright screen scene and the branch judgment of the off-screen scene, the services in different scenes can be distinguished and identified. The image processing method provided by the embodiment of the present application can support more AON intelligent perception functions and business, improving the user experience.

The above FIG. 9 exemplarily illustrates the image analysis process in the on-screen scene and off-screen scene provided by the embodiment of the present application. From the perspective of the newly added intelligent perception algorithm platform, the image analysis process provided by the embodiment of the present application in the scene of turning on the screen and unlocking is exemplarily described below in combination with the flow chart shown in FIG. 11 .

S401, collecting continuous multi-frame images in real time through the front camera of the electronic device.

S402. Acquire continuous multi-frame images with lower resolution, and use low-power TCM memory for data buffering, and analyze the continuous multi-frame images.

Exemplarily, as mentioned above, the continuous multi-frame images can be analyzed by the intelligent perception algorithm platform.

S403. Determine whether a first preset trigger condition is met according to the analysis result.

Wherein, by judging whether the analysis result satisfies the first preset trigger condition, it is judged whether any service is triggered.

When it is judged in S403 that the image does not meet the first preset trigger condition, the captured image frame is discarded, and the execution returns to S401.

S404, when it is determined that the first preset trigger condition is satisfied, determine whether the screen is turned on and unlocked.

When it is judged that the screen is on and unlocked, continue to execute the following S406-S415.

S405, when it is determined that the trigger is in the bright screen state, obtain continuous multi-frame images with higher resolution, use DDR memory and TCM memory for data buffering, and analyze the images through the intelligent perception algorithm platform.

S406, judging whether the image contains facial features.

S407, when it is determined in S406 that the image contains facial features, determine whether the camera is turned on.

S408, when it is determined in S407 that the camera is turned on, determine that the image analysis result is: the triggered services include the auxiliary photographing service and the intelligent horizontal and vertical screen service.

S409, when it is determined in S407 that the camera is not turned on, determine that the image analysis result is: the triggered services include the service of not turning off the screen while watching the screen, the service of reducing the volume while watching the screen, and the service of smart horizontal and vertical screens.

S410, when it is determined in S406 that the image does not contain human face features, determine whether the image contains a two-dimensional code.

S411. When it is determined in S410 that the image contains a two-dimensional code, determine that the image analysis result is: the triggered service is an intelligent code identification service.

S412, when it is determined in S410 that the image does not contain a two-dimensional code, determine whether the image contains a preset gesture.

S413, when it is determined in S412 that the image contains a preset gesture, determine that the image analysis result is: the triggered service is an air gesture service.

When it is judged in S412 that the image does not contain the preset gesture, no response is made, and the execution returns to S401.

After the above S408, S408, S411, and S413, continue to execute S414-S415.

S414. Acquire an image analysis result.

S415. Make a response according to the preset policy corresponding to the image analysis result.

Exemplarily, if the image analysis result is: the triggered service includes the auxiliary photographing service and the intelligent horizontal and vertical screen service, then the corresponding preset strategy includes implementing the auxiliary photographing and intelligent horizontal and vertical screen services based on the orientation of the face.

Exemplarily, if the image analysis result is: the triggered services include the service of keeping the screen off while watching the screen, the service of reducing the volume while watching the screen, and the service of intelligent horizontal and vertical screens, then the corresponding preset policies include keeping the screen on while watching the screen, reducing the volume while watching the screen, and Smart horizontal and vertical screen.

For example, if the image analysis result is: the triggered service is an intelligent code recognition service, then the corresponding preset strategy is to scan the code and read the code, and jump to the payment code interface or the ride code interface.

For example, if the image analysis result is: the triggered service is an air gesture service and is a grasping gesture, then the corresponding preset strategy is screen capture; if the image analysis result is: the triggered service is an air gesture service and /sliding gesture, the corresponding default strategy is page turning; if the image analysis result is: the triggered service is an air gesture business and an air press gesture, then the corresponding default strategy is to confirm answering when an incoming call signal is received Incoming calls, or the corresponding preset strategy is to quickly jump to the payment code interface when the electronic device displays the desktop. Exemplarily, FIG. 12 shows a schematic interface diagram of an air screenshot scene applied in the embodiment of the present application.

Through the above solution, in the bright screen scene, the triggered business will be allocated to the newly added intelligent perception algorithm platform, and the newly added intelligent perception algorithm platform will perform image analysis to obtain the image analysis results, and then can follow the preset corresponding to the image analysis results The strategy responds. Therefore, the image processing method provided in the embodiment of the present application can support more AON smart perception functions and services, and improve user experience.

Table 1 below shows the power consumption of each item and the total power consumption when the solution of the present application is applied to different services or characteristics. As shown in Table 1, assuming that the image output resolution is 320×240, the frame rate is 10, the total power consumption in the smart scanning scene is 19.03mA, and the total power consumption in the air gesture scene is 19.03mA. The total power consumption in the face-assisted camera scene is 11.03mA. However, when the related technology is adopted, the total power consumption of each application scenario is about 100 mA. It can be seen that the solution of this application has low power consumption when it is applied to different services or characteristics, and realizes the low-power consumption AON smart sensing function and services.

Table 1

This application adds a new intelligent perception algorithm platform based on the original AON algorithm platform. In the AON intelligent perception scene, the front camera is always on and collects images to intelligently perceive user services, which can be widely used in air gestures and intelligent code scanning. , smart gaze and other scenarios, which greatly meet the needs of users. When the application detects that the collected image meets the first preset trigger condition, it first judges whether the triggering scene is triggered in the bright screen scene or in the off-screen scene, performs business classification and identification, and then distributes the business to the Intelligent perception algorithm platform, and further judge the specific business content. The AON camera of this application operates with extremely low power consumption, can accurately perceive user services, and improves user experience.

It should be noted that, in this embodiment of the application, "greater than" can be replaced with "greater than or equal to", "less than or equal to" can be replaced with "less than", or "greater than or equal to" can be replaced with "greater than", "Less than" can be replaced with "less than or equal to".

The various embodiments described herein may be independent solutions, or may be combined according to internal logic, and these solutions all fall within the protection scope of the present application.

The solutions provided by the embodiments of the present application are mainly described above from the perspective of method steps. It can be understood that, in order to realize the above functions, the electronic device implementing the method includes hardware structures and/or software modules corresponding to each function. Those skilled in the art should be aware that, in combination with the units and algorithm steps of the examples described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a certain function is executed by hardware or computer software drives hardware depends on the specific application and design constraints of the technical solution. Professionals may use different methods to implement the described functions for each specific application, but such implementation should not be regarded as exceeding the protection scope of the present application.

The present application also provides a chip, the chip is coupled with a memory, and the chip is used to read and execute computer programs or instructions stored in the memory, so as to execute the methods in the above-mentioned embodiments.

The present application also provides an electronic device, which includes a chip, and the chip is used to read and execute computer programs or instructions stored in a memory, so that the methods in each embodiment are executed.

This embodiment also provides a computer-readable storage medium, where computer instructions are stored in the computer-readable storage medium, and when the computer instructions are run on the electronic device, the electronic device executes the above-mentioned relevant method steps to realize the steps in the above-mentioned embodiments. image processing method.

This embodiment also provides a computer program product, the computer-readable storage medium stores program codes, and when the computer program product runs on a computer, the computer executes the above-mentioned related steps to realize the image processing in the above-mentioned embodiment method.

In addition, an embodiment of the present application also provides a device, which may specifically be a chip, a component or a module, and the device may include a connected processor and a memory; wherein the memory is used to store computer-executable instructions, and when the device is running, The processor can execute the computer-executable instructions stored in the memory, so that the chip executes the image processing methods in the above method embodiments.

Wherein, the electronic device, computer-readable storage medium, computer program product or chip provided in this embodiment is all used to execute the corresponding method provided above, therefore, the beneficial effects it can achieve can refer to the above-mentioned The beneficial effects of the corresponding method will not be repeated here.

In the several embodiments provided in this application, it should be understood that the disclosed devices and methods may be implemented in other ways. For example, the device embodiments described above are only illustrative. For example, the division of modules or units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or It may be integrated into another device, or some features may be omitted, or not implemented. In another point, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection of devices or units may be in electrical, mechanical or other forms.

The above is only a specific implementation of the application, but the scope of protection of the application is not limited thereto. Anyone familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the application. Should be covered within the protection scope of this application. Therefore, the protection scope of the present application should be determined by the protection scope of the claims.

Claims (18)

1. An image processing method, characterized in that it is applied to the system architecture of electronic equipment, the system architecture includes a first algorithm platform and a second algorithm platform, and the business supported by the first algorithm platform is different from the first algorithm platform The business supported by the second algorithm platform is the face unlocking screen business, and the method includes: The electronic device starts the first function, and after the first function is started, the camera module of the electronic device is in a normally-on state; Outputting multiple frames of first images of a first size specification through the camera module; Invoking the first algorithm platform to perform image analysis on the multi-frame first images to obtain a first image analysis result; When the first image analysis result satisfies the first preset trigger condition, the camera module outputs multiple frames of second images of a second size specification; the first algorithm platform is invoked to process the multiple frames of second images Image analysis to obtain a second image analysis result; make a response according to a preset strategy corresponding to the second image analysis result; When the analysis result of the first image satisfies the second preset trigger condition, the camera module outputs multiple frames of third images of a third size specification; the second algorithm platform is invoked to process the multiple frames of third images Image analysis to obtain a third image analysis result; make a response according to a preset strategy corresponding to the third image analysis result; Wherein, the resolution corresponding to the first size specification is lower than the resolution corresponding to the second size specification, and the resolution corresponding to the second size specification is lower than or equal to the resolution corresponding to the third size specification. 2. The method according to claim 1, wherein the first preset trigger condition is any of the following: The image contains a first preset feature, and the image is an image collected when the screen is turned on and unlocked; wherein, the first preset feature includes a face feature, a human eye feature, a gesture feature, and a two-dimensional code feature any of The image contains facial features, and the image is an image collected when the screen is off; The second preset trigger condition is that the image contains facial features, and the image is an image collected when the screen is on and locked. 3. The method according to any one of claims 1 to 2, characterized in that, The services supported by the first algorithm platform include air gesture service, intelligent code recognition service, screen-on-off service while watching the screen, volume-down service while watching the screen, intelligent horizontal and vertical screen service, auxiliary photo-taking service, and intelligent screen-off display service. 4. The method according to any one of claims 1 to 3, wherein the method further comprises: After the camera module of the electronic device outputs multiple frames of first images of a first size specification, storing the multiple frames of first images in a first memory; After outputting multiple frames of second images of a second size specification through the camera module, storing the multiple frames of second images in the first memory and the second memory; After outputting multiple frames of third images of a third size specification through the camera module, storing the multiple frames of third images in the second memory; Wherein, in the case of storing the same image, the power consumption caused by the first memory is less than the power consumption caused by the second memory, and the power consumption caused by the first memory and the second memory together consumption is less than the power consumption caused by the second memory. 5. The method according to claim 4, wherein the first memory is a TCM, and the second memory is a double rate synchronous dynamic random access memory (DDR). 6. The method according to any one of claims 1 to 5, characterized in that, The third size specification is a video graphics array VGA, and the corresponding resolution is 640×480; The second size specification is QVGA, and the corresponding resolution is 320×240; The first size specification is QQVGA, and the corresponding resolution is 160×120. 7. The method according to any one of claims 1 to 6, wherein the camera module is a front camera. 8. The method according to any one of claims 1 to 7, wherein the system framework further includes a smart perception application, and the first function is a function provided by the smart perception application; After the electronic device starts the first function, the method further includes: The smart perception application receives a first operation of starting a first service by the user, and the first service is a service supported by the smart perception application; In response to the first operation, the smart perception application sends a first message to the first algorithm platform, where the first message is used to indicate subscription to the first service; The first algorithm platform subscribes to the first service in response to the first message; Wherein, the first service is at least one of gestures in the air, intelligent code recognition, keeping the screen on while watching the screen, reducing the volume while watching the screen, intelligent horizontal and vertical screens, auxiliary photo taking, and intelligent screen-off display. 9. The method according to claim 8, characterized in that, after subscribing to the first service, the method further comprises: The first algorithm platform sends a second message to the camera module, and the second message is used to request the camera module to output an image according to the first size specification. 10. The method according to claim 8 or 9, characterized in that, after said obtaining the first image analysis result, said method further comprises: The first algorithm platform judges that the first image analysis result satisfies the first preset trigger condition; The first algorithm platform sends a third message to the camera module, and the third message is used to request the camera module to output an image according to the second size specification. 11. The method according to claim 10, characterized in that, after the first algorithm platform judges that the first image analysis result satisfies the first preset trigger condition, the method further comprises: The first algorithm platform reports a fourth message to the smart sensing application, where the fourth message is used to indicate that the first service supported by the smart sensing application is triggered; In response to the fourth message, the smart sensing application triggers the electronic device to display the prompt information of the first service in a preset area of the screen. 12. The method according to any one of claims 8 to 11, wherein before responding according to the preset policy corresponding to the second image analysis result, the method further comprises: The first algorithm platform reports the second image analysis result to the smart perception application; Wherein, the responding according to the preset strategy corresponding to the second image analysis result includes: The smart perception application makes a response according to a preset strategy corresponding to the second image analysis result; Wherein, the preset strategy includes at least one of taking screenshots in the air or turning pages or answering calls, intelligent code recognition, watching the screen without turning off the screen, watching the screen to reduce the volume, intelligent horizontal and vertical screens, auxiliary photo taking, and intelligent screen-off display. 13. The method according to any one of claims 8 to 12, wherein the system framework further comprises an intelligent perception interface; Data interaction is performed between the intelligent perception application and the first algorithm platform through the intelligent perception interface. 14. The method according to any one of claims 1 to 13, wherein the system framework further includes a business management module, a distributor, a smart perception client and a native platform client; When the first image analysis result satisfies the first preset trigger condition, the service management module determines a first processing task, and sends the first processing task to the smart perception client through the distributor terminal, the smart perception client forwards the first processing task to the first algorithm platform; When the first image analysis result satisfies the second preset trigger condition, the business management module determines a second processing task, and sends the second processing task to the native platform client through the distributor end, the native platform client forwards the second processing task to the second algorithm platform; Wherein, the first processing task is a task of processing the multiple frames of second images, and the second processing task is a task of processing the multiple frames of third images. 15. The method according to any one of claims 1 to 14, wherein the system framework further comprises an algorithm library, and the algorithm library includes a first image processing algorithm and a second image processing algorithm; Using the first algorithm platform to perform image analysis on the multiple frames of first images to obtain a first image analysis result includes: calling the first image processing algorithm in the algorithm library by the first algorithm platform , performing image analysis on the multiple frames of first images to obtain the first image analysis results; The performing image analysis on the multi-frame second images by the first algorithm platform to obtain the second image analysis results includes: calling the second image processing algorithm in the algorithm library by the first algorithm platform , performing image analysis on the multiple frames of second images to obtain the second image analysis results; Wherein, the requirement of the first image processing algorithm for image resolution is lower than the requirement of the second image processing algorithm for image resolution. 16. A method according to any one of claims 1 to 15, wherein The second algorithm platform is a framework provided by the original chip that supports the camera's always-on AON algorithm, and the first algorithm platform is an AON algorithm integration framework that supports multiple services created based on the second algorithm platform. 17. An electronic device, characterized in that it comprises a processor, a memory, and a computer program stored on the memory, the processor is used to execute the computer program, so that the electronic device realizes the requirements of claims 1 to 10. The method of any one of 16. 18. A computer-readable storage medium, characterized in that the computer-readable storage medium stores a computer program, and when the computer program is run on an electronic device, the electronic device executes the electronic device according to claims 1 to 16. any one of the methods described.

Images