CN116301362A

CN116301362A - Image processing method, electronic device and storage medium

Info

Publication number: CN116301362A
Application number: CN202310222795.XA
Authority: CN
Inventors: 管一凡
Original assignee: Honor Device Co Ltd
Current assignee: Honor Device Co Ltd
Priority date: 2023-02-27
Filing date: 2023-02-27
Publication date: 2023-06-23
Anticipated expiration: 2043-02-27
Also published as: CN116301362B

Abstract

The application provides an image processing method, electronic equipment and a storage medium, and relates to the technical field of electronics. The intelligent perception algorithm platform is newly added on the basis of the original AON algorithm platform, in an AON intelligent perception scene, the front camera is used for normally opening and collecting images, intelligent perception user service can be widely applied to various scenes such as air-isolation gestures, intelligent code scanning and intelligent gazing, and the user requirements are greatly met. When the acquired image is monitored to meet the first preset triggering condition, whether the triggering scene is triggered in the bright screen scene or the off screen scene is judged, service classification and identification are carried out, then the service is distributed to the intelligent perception algorithm platform provided by the application, and specific service content is further judged. The AON camera runs with extremely low power consumption, can accurately sense user service, and improves user experience.

Description

Image processing method, electronic device and storage medium

Technical Field

The present disclosure relates to the field of electronic technologies, and in particular, to an image processing method, an electronic device, and a storage medium.

Background

At present, intelligent electronic devices such as mobile phones can be applied to a camera real-time on-line (AON) function, and a face unlocking screen can be rapidly realized when the mobile phones are in a screen-off state. Specifically, under the condition that the AON function is applied to the mobile phone, the front camera of the mobile phone camera is in a normally open state, images can be acquired in real time, face recognition is carried out through image analysis, a face unlocking screen can be responded more quickly, and a user does not need to touch the mobile phone, so that the mobile phone screen can be unlocked.

As described above, the AON function may be applied to a face unlock screen scene when the mobile phone is in a screen-off state, but since the user needs to use the mobile phone in a screen-on state in most cases, the application scene of the AON function is too limited.

Disclosure of Invention

The application provides an image processing method, electronic equipment and a storage medium, wherein an intelligent perception algorithm platform is newly added on the basis of a native AON algorithm platform, and in an AON intelligent perception scene, an image is normally opened and acquired through a front camera, so that user services are intelligently perceived, and the intelligent perception method can be widely applied to various scenes such as a blank gesture, intelligent code scanning, intelligent gazing and the like, and greatly meets user requirements.

In order to achieve the above purpose, the present application adopts the following technical scheme:

in a first aspect, the present application provides an image processing method, applied to a system architecture of an electronic device, where the system architecture includes a first algorithm platform and a second algorithm platform, where a service supported by the first algorithm platform is different from a service supported by the second algorithm platform, and the service supported by the second algorithm platform is a face unlock screen service, and the method includes:

the electronic equipment starts a first function, and after the first function is started, a camera module of the electronic equipment is in a normally-on state;

Outputting, by the camera module, a multi-frame first image of a first size specification;

calling a first algorithm platform to perform image analysis on multiple frames of first images to obtain a first image analysis result;

outputting a multi-frame second image with a second size specification through the camera module when the first image analysis result meets a first preset trigger condition; invoking a first algorithm platform to perform image analysis on a plurality of frames of second images to obtain a second image analysis result; responding according to a preset strategy corresponding to the second image analysis result;

outputting a multi-frame third image with a third dimension specification through the camera module when the first image analysis result meets a second preset trigger condition; invoking a second algorithm platform to perform image analysis on a plurality of frames of third images to obtain a third image analysis result; responding according to a preset strategy corresponding to the third image analysis result;

the resolution corresponding to the first size specification is lower than the resolution corresponding to the second size specification, and the resolution corresponding to the second size specification is lower than or equal to the resolution corresponding to the third size specification.

Through the scheme of the application, the intelligent perception algorithm platform is newly added on the basis of the AON algorithm platform provided by the original chip, in an AON intelligent perception scene, the image is normally opened and collected through the camera module, the intelligent perception user service is not limited to the face unlocking screen service supported by the AON algorithm platform provided by the original chip, and the intelligent perception algorithm platform can be widely applied to other various scenes (such as a blank gesture, an intelligent code scanning, intelligent watching and the like), so that the user requirements are greatly met. In addition, the scheme of the application applies different image resolutions to analyze the images at different stages, so that not only can the accuracy of gesture recognition be ensured, but also the power consumption and the memory space required in the gesture recognition process can be effectively reduced. The AON camera runs with extremely low power consumption, can accurately sense user service, and improves user experience.

In some implementations, the camera module is a front-facing camera. The camera module (AON camera) continuously outputs images with lower resolution in an initial stage, so that the AON camera operates with extremely low power consumption, perceives user services and improves user experience.

In some implementations, the first preset trigger condition is any one of: (1) The image comprises a first preset feature, and the image is acquired when the screen is on and unlocked; the first preset features comprise any one of face features, eye features, gesture features and two-dimensional code features; (2) The image contains facial features, and is an image acquired when the screen is off.

According to the method and the device for classifying and identifying the service, when the acquired image is monitored to meet the first preset triggering condition, whether the triggering scene is triggered in the bright screen scene or the off screen scene is judged, service classification and identification are carried out, specific service content is further judged, user service can be accurately perceived, and user experience is improved.

In some implementations, the second preset trigger condition is that a face feature is included in an image, and the image is an image acquired when the screen is on and locked.

In some implementations, the services supported by the first algorithm platform include a space gesture service, an intelligent code recognition service, a screen gazing not to put out service, a screen gazing volume reducing service, an intelligent horizontal and vertical screen service, an auxiliary photographing service and an intelligent screen put out display service.

Through this application scheme, this application has newly increased wisdom perception algorithm platform on native AON algorithm platform basis, in AON intelligence perception scene, normally open and gather the image through leading camera, wisdom perception user's business, can wide application in separate various scenes such as empty gesture, intelligent sweep the sign indicating number, intelligent gazing, greatly satisfies user's demand.

In some implementations, the above method further includes: after outputting a plurality of frames of first images of a first size specification through a camera module of the electronic device, storing the plurality of frames of first images in a first memory; after outputting the multi-frame second image of the second size specification by the camera module, storing the multi-frame second image in the first memory and the second memory; after outputting the multi-frame third image of the third size specification by the camera module, storing the multi-frame third image in the second memory.

In the case of storing the same image, the power consumption caused by the first memory is smaller than the power consumption caused by the second memory, and the power consumption caused by the first memory and the second memory together is smaller than the power consumption caused by the second memory.

In some implementations, the first memory is a tightly coupled memory TCM and the second memory is a double rate synchronous dynamic random access memory DDR.

According to the scheme, the low-power-consumption memory is adopted for data caching in the initial stage, and the low-power-consumption memory is adopted to combine with the normal-power-consumption memory for data caching after the service is triggered, so that the AON camera can accurately sense the user service and improve the user experience.

In some implementations, the third dimension is a video graphics array VGA, corresponding to a resolution of 640 x 480. The second size specification is QVGA, with a corresponding resolution of 320 x 240. The first size specification is QQVGA, with a corresponding resolution of 160 x 120.

In some implementations, the second algorithm platform is a framework supporting a camera always on AON algorithm provided by the native chip, and the first algorithm platform is an AON algorithm integration framework supporting multiple services created based on the second algorithm platform.

In some implementations, the system framework further includes a smart aware application, the first function being a function provided by the smart aware application. After the electronic device turns on the first function, the method further includes: the intelligent perception application receives a first operation of starting a first service by a user, wherein the first service is supported by the intelligent perception application; responsive to a first operation, the smart awareness application issues a first message to the first algorithm platform, the first message indicating a subscription to a first service; the first algorithm platform subscribes to a first service in response to the first message. The first service is at least one of a blank gesture, intelligent code recognition, screen fixation, volume reduction, intelligent horizontal and vertical screen fixation, auxiliary photographing and intelligent screen fixation display.

In some implementations, after subscribing to the first service, the method further includes: the first algorithm platform sends a second message to the camera module requesting the camera module to output an image according to the first size specification.

In some implementations, after obtaining the first image analysis result, the method further includes: the first algorithm platform judges that a first image analysis result meets a first preset triggering condition; the first algorithm platform sends a third message to the camera module requesting the camera module to output an image according to the second dimensional specification.

In some implementations, after the first algorithm platform determines that the first image analysis result meets the first preset trigger condition, the method further includes: the first algorithm platform reports a fourth message to the intelligent perception application; the intelligent perception application responds to a fourth message, the electronic equipment is triggered to display prompt information of the first service at a preset area of the screen, and the fourth message is used for indicating that the first service supported by the intelligent perception application is triggered.

In some implementations, before responding according to the preset policy corresponding to the second image analysis result, the method further includes: the first algorithm platform reports the second image analysis result to the intelligent perception application.

The responding according to the preset strategy corresponding to the second image analysis result comprises the following steps: the intelligent sensing application responds according to a preset strategy corresponding to the second image analysis result; the preset strategy comprises at least one of screen capturing or page turning or incoming call answering at intervals, intelligent identification, screen non-screen extinction fixation, screen volume reduction fixation, intelligent horizontal and vertical screen fixation, auxiliary photographing and intelligent screen extinction display.

In some implementations, the system framework further includes a smart awareness interface. And the intelligent perception application and the first algorithm platform conduct data interaction through an intelligent perception interface.

In some implementations, the system framework further includes a business management module, a distributor, a smart awareness client, and a native platform client.

The method further comprises the following steps: when the first image analysis result meets a first preset trigger condition, the business management module determines a first processing task and sends the first processing task to the intelligent perception client through the distributor, and the intelligent perception client forwards the first processing task to the first algorithm platform. The first processing task is a task for processing a plurality of frames of second images.

The method further comprises the following steps: when the first image analysis result meets a second preset trigger condition, the service management module determines a second processing task and sends the second processing task to the native platform client through the distributor, and the native platform client forwards the second processing task to the second algorithm platform. The second processing task is a task for processing a multi-frame third image.

In some implementations, the system framework further includes an algorithm library including a first image processing algorithm and a second image processing algorithm.

The image analysis is carried out on a plurality of frames of first images through a first algorithm platform to obtain a first image analysis result, and the method comprises the following steps: the first algorithm platform calls a first image processing algorithm in the algorithm library, and performs image analysis on multiple frames of first images to obtain a first image analysis result.

The image analysis is carried out on the multi-frame second image through the first algorithm platform to obtain a second image analysis result, and the method comprises the following steps: and the first algorithm platform calls a second image processing algorithm in the algorithm library, and performs image analysis on a plurality of frames of second images to obtain a second image analysis result. Wherein the first image processing algorithm requires less image resolution than the second image processing algorithm.

Through the scheme of the application, the intelligent perception algorithm platform is newly added on the basis of the AON algorithm platform provided by the original chip, in an AON intelligent perception scene, the image is normally opened and collected through the camera module, the intelligent perception user service is not limited to the face unlocking screen service supported by the AON algorithm platform provided by the original chip, and the intelligent perception algorithm platform can be widely applied to other various scenes (such as a blank gesture, an intelligent code scanning, intelligent watching and the like), so that the user requirements are greatly met.

In a second aspect, the present application provides an image processing apparatus comprising means for performing the method of the first aspect described above. The apparatus may correspond to performing the method described in the first aspect, and the relevant descriptions of the units in the apparatus are referred to the description of the first aspect, which is omitted herein for brevity.

The method described in the first aspect may be implemented by hardware, or may be implemented by executing corresponding software by hardware. The hardware or software includes one or more modules or units corresponding to the functions described above. Such as a processing module or unit, a display module or unit, etc.

In a third aspect, the present application provides an electronic device comprising a processor, a computer program or instructions stored in the processor and in a memory, the processor being for executing the computer program or instructions such that the method of the first aspect is performed.

In a fourth aspect, the present application provides a computer-readable storage medium having stored thereon a computer program (which may also be referred to as instructions or code) for implementing the method in the first aspect. For example, the computer program, when executed by a computer, causes the computer to perform the method of the first aspect.

In a fifth aspect, the present application provides a chip comprising a processor. The processor is configured to read and execute a computer program stored in the memory to perform the method of the first aspect and any possible implementation thereof. Optionally, the chip further comprises a memory, and the memory is connected with the processor through a circuit or a wire.

In a sixth aspect, the present application provides a system-on-chip comprising a processor. The processor is configured to read and execute a computer program stored in the memory to perform the method of the first aspect and any possible implementation thereof. Optionally, the chip system further comprises a memory, and the memory is connected with the processor through a circuit or a wire.

In a seventh aspect, the present application provides a computer program product comprising a computer program (which may also be referred to as instructions or code) which, when executed by an electronic device, causes the electronic device to carry out the method of the first aspect.

It will be appreciated that the advantages of the second to seventh aspects may be found in the relevant description of the first aspect, and are not described here again.

Drawings

Fig. 1 is an application scenario schematic diagram of an image processing method provided in an embodiment of the present application;

Fig. 2 is a schematic structural diagram of an electronic device according to an embodiment of the present application;

FIG. 3 is a schematic diagram illustrating a relationship between a camera module and a processor according to an embodiment of the present application;

FIG. 4 is a schematic diagram of a software architecture employed in an embodiment of the present application;

FIG. 5 is a timing chart of interaction of each module in the image processing method according to the embodiment of the present application;

fig. 6 is a schematic diagram of a functional setting of an image processing method according to an embodiment of the present application;

fig. 7 is an interface schematic diagram of an image processing method according to an embodiment of the present application when the image processing method is applied;

fig. 8 is a flowchart of an image processing method according to an embodiment of the present application;

fig. 9 is a second flowchart of an image processing method according to an embodiment of the present application;

FIG. 10 illustrates an interface schematic diagram of a smart AOD scene applied by an embodiment of the present application

Fig. 11 is a flowchart illustrating a third image processing method according to an embodiment of the present application;

FIG. 12 illustrates an interface schematic of a blank screen capture scenario employed by embodiments of the present application.

Detailed Description

For the purposes of making the objects, technical solutions and advantages of the embodiments of the present application more clear, the technical solutions of the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is apparent that the described embodiments are some embodiments of the present application, but not all embodiments. All other embodiments, which can be made by one of ordinary skill in the art without undue burden from the present disclosure, are within the scope of the present disclosure.

The term "and/or" herein is an association relationship describing an associated object, and means that there may be three relationships, for example, a and/or B may mean: a exists alone, A and B exist together, and B exists alone. The symbol "/" herein indicates that the associated object is or is a relationship, e.g., A/B indicates A or B.

The terms "first" and "second" and the like in the description and in the claims are used for distinguishing between different objects and not for describing a particular sequential order of objects. In the description of the embodiments of the present application, unless otherwise specified, the meaning of "a plurality of" means two or more, for example, a plurality of processing units means two or more processing units and the like; the plurality of elements means two or more elements and the like.

In the embodiments of the present application, words such as "exemplary" or "such as" are used to mean serving as examples, illustrations, or descriptions. Any embodiment or design described herein as "exemplary" or "for example" should not be construed as preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "such as" is intended to present related concepts in a concrete fashion.

In view of this, the embodiment of the application provides an image processing method and an electronic device, which are improved at the bottom layer of a mobile phone system, a smart perception algorithm platform is newly constructed on the basis of an original AON platform, and based on the support of the smart perception algorithm platform, the AON function not only can be applied to face unlocking screen scenes in a mobile phone screen-off state, but also can be applied to gesture control screen, intelligent code recognition, gazing without screen extinction, gazing volume reduction, auxiliary photographing, intelligent horizontal and vertical screen and other scenes in a mobile phone screen-off state, so that the user experience can be improved.

Fig. 1 shows a schematic view of an application scenario according to various exemplary embodiments of the present application. As shown in fig. 1 (a), the mobile phone is in a bright screen state, and the front camera of the mobile phone is normally open and acquires images in real time. As shown in fig. 1 (b), the mobile phone is in a screen-off state, and the front camera of the mobile phone is normally open and acquires images in real time. In the embodiment of the application, the AON function allows the camera device (front camera) of the electronic equipment to be in an active state all the time, so that the camera is normally open under low power consumption, and scenes such as code scanning, gesture recognition, face unlocking and the like can be completed more quickly.

The intelligent perception algorithm platform is newly added on the basis of the original AON algorithm platform, in an AON intelligent perception scene, the front camera is used for normally opening and collecting images, intelligent perception user service can be widely applied to various scenes such as air-isolation gestures, intelligent code scanning and intelligent gazing, and the user requirements are greatly met. When the acquired image is monitored to meet the first preset triggering condition, whether the triggering scene is triggered in the bright screen scene or the off screen scene is judged, service classification and identification are carried out, then the service is distributed to the intelligent perception algorithm platform provided by the application, and specific service content is further judged. The AON camera runs with extremely low power consumption, can accurately sense user service, and improves user experience.

The AON function provided by the embodiment of the application not only can be widely applied to various scenes and meets the requirements of users, but also can be operated with extremely low power consumption through the improved algorithm of the application, and the power consumption of the mobile phone is saved. The specific algorithm will be described in detail below.

The image processing method provided by the embodiment of the application can be applied to the electronic equipment with the front camera shooting function. The electronic devices include various terminal devices, which may also be called terminals (terminals), user Equipment (UEs), mobile Stations (MSs), mobile Terminals (MT), and the like. The terminal device may be a mobile phone, a smart television, a wearable device, a tablet (Pad), a computer with wireless transceiving function, a Virtual Reality (VR) terminal device, an augmented reality (augmented reality, AR) terminal device, a wireless terminal in industrial control (industrial control), a wireless terminal in unmanned driving (self-driving), a wireless terminal in teleoperation (remote medical surgery), a wireless terminal in smart grid (smart grid), a wireless terminal in transportation safety (transportation safety), a wireless terminal in smart city (smart city), a wireless terminal in smart home (smart home), or the like. The embodiment of the application does not limit the specific technology and the specific equipment form adopted by the terminal equipment.

Referring to fig. 2, a schematic structural diagram of an electronic device according to an embodiment of the present application is provided. The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (universal serial bus, USB) interface 130, a charge management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, keys 190, a motor 191, an indicator 192, a camera 193, a display 194, and a subscriber identity module (subscriber identification module, SIM) card interface 195, etc. The sensor modules 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a touch sensor 180K, an ambient light sensor 180L, and the like.

It is to be understood that the structure illustrated in the embodiments of the present application does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, electronic device 100 may include more or fewer components than shown, or certain components may be combined, or certain components may be split, or different arrangements of components. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

The processor 110 may include one or more processing units, such as: the processor 110 may include an application processor (application processor, AP), a modem processor, a graphics processor (graphics processing unit, GPU), an image signal processor (image signal processor, ISP), a controller, a memory, a video codec, a digital signal processor (digital signal processor, DSP), a baseband processor, and/or a neural network processor (neural-network processing unit, NPU), etc. Wherein the different processing units may be separate devices or may be integrated in one or more processors. For example, the processor 110 is configured to perform the image processing method in the embodiment of the present application.

In the present embodiment, the implementation of the AON function depends on ISPs. The ISP can run various algorithm programs to process the image signal in real time. Fig. 3 shows the connection relationship among the camera module, ISP, and AP.

Fig. 3 (a) shows a structure diagram of an external ISP, a camera module is connected to the ISP through a camera serial interface (camera serial interface, CSI), and the ISP is connected to an AP. Wherein the ISPs are separately disposed, the ISPs being independent of the AP. After the ISP receives the image collected by the camera module, the ISP performs image signal processing on the image collected by the camera module. The AP can control the working mode of the ISP through an internal integrated circuit (inter-integrated circuit, I2C) to acquire the working state of the ISP and the like. Fig. 3 (b) shows a configuration diagram of an ISP built in which a camera module is connected to an AP, the ISP is built in the AP, and after the AP receives an image collected by the camera module, the image signal processing is performed on the image by the ISP.

The controller may be a neural hub and a command center of the electronic device 100, among others. The controller can generate operation control signals according to the instruction operation codes and the time sequence signals to finish the control of instruction fetching and instruction execution.

A memory may also be provided in the processor 110 for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may hold instructions or data that the processor 110 has just used or recycled. If the processor 110 needs to reuse the instruction or data, it may be called directly from memory. Repeated accesses are avoided and the latency of the processor 110 is reduced, thereby improving the efficiency of the system.

External memory 120 is generally referred to as external memory, which in the present embodiment refers to storage other than memory of an electronic device and a cache of a processor, which is generally non-volatile memory.

Internal memory 121, which may also be referred to as "memory," may be used to store computer-executable program code that includes instructions. The internal memory 121 may include a storage program area and a storage data area. The storage program area may store an application program (such as a sound playing function, an image playing function, etc.) required for at least one function of the operating system, etc.

In the embodiment of the application, the memory of the electronic device adopts a tightly coupled memory (tightly coupled memory, TCM) and DDR memory.

The DDR memory is a double rate synchronous dynamic random access memory, and is called DDR SDRAM (double data rate SDRAM, double rate SDRAM). SDRAM is an abbreviation for synchronous dynamic random access memory (synchronous dynamic random access memory). The data transfer rate of DDR memory is very high.

Wherein the TCM is contained in an address mapping space of the memory and is accessible as a flash memory. TCM is used to provide low latency memory to the processor that has no cache-specific unpredictability. TCM may be used to house important routines such as interrupt handling routines or real-time tasks that are highly desirable to avoid cache uncertainty. In addition, TCM may be used to hold temporary register data, data types whose local attributes are not suitable for caching, and important data structures such as interrupt stacks.

From the viewpoint of storage capacity, TCM memory capacity is smaller and DDR memory capacity is larger. From the data transfer rate perspective, the data transfer rate of DDR memory is greater than the data transfer rate of TCM memory. From a power consumption perspective, DDR memory consumes more power than TCM memory. The electronic equipment generally uses DDR memory, so that the power consumption is larger; in some scenarios, the embodiment of the application uses the TCM memory for data caching, so that the power consumption can be reduced.

In this embodiment of the present application, in some scenarios where the service is perceived in advance, the dynamic memory and the static memory in the electronic device both use the TCM memory to store the continuous multi-frame images acquired by the front camera, where the images are images with QQVGA image size specifications (also referred to as image specifications or size specifications), where the QQVGA image specifications are smaller image sizes, for example 160×120, and may be represented as QQVGA (160×120), and it can be understood that the image resolution of the QVGA image specifications is lower, and the image occupies less memory.

Illustratively, in some scenarios of later identifying specific services, the dynamic memory and the static memory of the electronic device use DDR memory and TCM memory in combination to collectively store consecutive multi-frame images acquired by the front-end camera, where the QVGA image specification is a larger image size, such as 320×240, relative to the QVGA image specification, which may be represented as qqqvga (320×240). It will be appreciated that the image resolution of the QVGA image specification is relatively high and the image occupies a relatively large amount of memory relative to the QQVGA image specification.

Where VGA (video graphics array) refers to a video graphics array, corresponding to 640 x 480 pixels in resolution. QVGA refers to 1/4 of the size of VGA resolution, corresponding to 320X 240 pixels. QQVGA refers to 1/4 screen of VGA resolution, corresponding to 160X 120 pixels.

It can be understood that the resolution corresponding to the image of the QQVGA specification is the smallest, and the occupied memory is the smallest.

In the embodiment of the application, when the service is perceived in advance, on one hand, the front camera collects the smaller-sized image of the QQVGA specification and stores the smaller-sized image in the TCM memory, so that the front camera occupies a very small memory space; on the other hand, the processor analyzes the continuous multi-frame images to judge whether the user triggers a certain service, for example, if the hand features are detected through image analysis and the initial gesture changes, the processor can judge that the user triggers a space gesture service; for another example, if facial features are detected through image analysis and the screen direction of the electronic device is deflected, it may be determined that the user triggered the intelligent landscape/portrait service.

After sensing that the service is triggered, on one hand, the front camera collects images with larger size and QVGA specification and stores the images in the DDR memory and the TCM memory; on the other hand, the processor analyzes the continuous multi-frame images to identify the specific content of the triggered business, for example, if the palm is detected to change from the unfolded state to the gripping state through image analysis, the processor can identify a gripping gesture for triggering screen capturing.

Therefore, in a scene that the front camera is in a normally open state, although the front camera continuously collects a large number of images, through the scheme, in the process of sensing business in advance, the power consumption can be greatly reduced by acquiring continuous multi-frame images with smaller size and storing the continuous multi-frame images in a low-power consumption TCM memory and carrying out image analysis on the continuous multi-frame images with smaller size.

The display screen 194 is used to display images, videos, and the like. The display 194 includes a display panel. The display panel may employ an organic light-emitting diode (OLED). In some embodiments, the electronic device 100 may include 1 or N display screens 194, N being a positive integer greater than 1.

The electronic device 100 also includes various types of sensors that can convert various physical signals into electrical signals. The proximity light sensor 180G may include, for example, a Light Emitting Diode (LED) and a light detector, such as a photodiode. The proximity sensor may turn off the display screen 194 and/or the backlight when the electronic device 100 is moved to the ear. The ambient light sensor 180L is used to sense ambient light level. The electronic device 100 may adaptively adjust the brightness of the display screen 194 based on the perceived ambient light level.

The touch sensor 180K, also referred to as a "touch panel". The touch sensor 180K may be disposed on the display screen 194, and the touch sensor 180K and the display screen 194 form a touch screen, which is also called a "touch screen". The touch sensor 180K is for detecting a touch operation acting thereon or thereabout. The touch sensor may communicate the detected touch operation to the application processor to determine the touch event type. Visual output related to touch operations may be provided through the display 194. In other embodiments, the touch sensor 180K may also be disposed on the surface of the electronic device 100 at a different location than the display 194.

For example, in the embodiment of the present application, the touch sensor 180K may detect a click operation of an icon of an application program by a user, and transmit the detected click operation to the application processor, determine that the click operation is used to start or run the application program, and further perform a running operation of the application program.

The electronic device 100 implements display functions through a GPU, a display screen 194, an application processor, and the like. The GPU is a microprocessor for image processing, and is connected to the display 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

The electronic device 100 may implement photographing functions through an ISP, a camera 193, a video codec, a GPU, a display screen 194, an application processor, and the like.

The above is a specific description of the embodiment of the present application taking the electronic device 100 as an example. It should be understood that the structures illustrated in the embodiments of the present application do not constitute a specific limitation on the electronic device 100. The electronic device 100 may have more or fewer components than shown in the figures, may combine two or more components, or may have a different configuration of components. The various components shown in the figures may be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and/or application specific integrated circuits.

In addition, an operating system is run on the components. Such as the iOS operating system developed by apple corporation, the Android open source operating system developed by google corporation, the Windows operating system developed by microsoft corporation, etc. An operating application may be installed on the operating system.

The operating system of the electronic device 100 may employ a layered architecture, an event driven architecture, a microkernel architecture, a microservice architecture, or a cloud architecture. In this embodiment, taking an Android system with a layered architecture as an example, a software structure of the electronic device 100 is illustrated.

Fig. 4 is a software architecture diagram of the electronic device 100 of the embodiment of the present application. Embodiments of the present application will be discussed based on the following technical architecture. It should be noted that, for convenience of description of logic, only the service logic relationship is described by using a schematic block diagram, and the specific location of the technical architecture where each service is located is not strictly expressed. In addition, the naming of each module in the software architecture diagram is taken as an exemplary example, the naming of each module in the software architecture diagram is not limited in the embodiment of the present application, and in actual implementation, the specific naming of the module may be determined according to actual requirements.

The layered architecture divides the software into several layers, each with distinct roles and branches. The layers communicate with each other through a software interface. In some embodiments, the Android system includes four layers, from top to bottom, an application layer (applications), an application framework layer (application framework), a hardware abstraction layer (hardware abstraction layer, HAL), and a kernel layer (kernel), respectively.

The application layer may include a series of application packages, among other things. For example, the application layer may include applications (applications may be simply referred to as applications) such as cameras, smart sensor, etc., which are not limited in any way by the embodiments of the present application.

The intelligent perception application provided by the embodiment of the application supports various services, and the services supported by the intelligent perception application can comprise an air gesture service, an intelligent code recognition service, a watching non-screen-off service, a watching volume reduction service, an intelligent horizontal and vertical screen service, an auxiliary photographing service and an intelligent AOD service by way of example. These services may be collectively referred to as intelligent awareness services.

It should be noted that, the implementation of these services supported by the intelligent sensing application depends on that the front camera of the electronic device is in a normally open state, and images are collected in real time to obtain relevant data of the intelligent sensing service. When the intelligent perception application monitors the service related data, the intelligent perception application sends the service related data to the intelligent perception algorithm platform, and the intelligent perception algorithm platform judges whether to execute the related service of the intelligent perception application or specifically execute which service by analyzing the acquired image according to the analysis result.

Wherein, separate the space gesture business: the electronic equipment responds according to a preset strategy corresponding to the gesture by recognizing the gesture of the user, so that the service of man-machine interaction is realized. When the electronic device is in a bright screen state, the electronic device can support to recognize various preset blank gestures, and different gestures can correspond to different preset strategies, for example: a space grasp gesture, a screen capture, a space up/down gesture, a page turn, a space press gesture, and a call receiving.

For example, taking a blank grip gesture as an example, the user changes from a palm stretching state to a fist-making state, and the control mode corresponding to the gesture is preset to perform screen capturing processing on the display interface of the electronic device. When the electronic equipment is in a bright screen state, the electronic equipment automatically executes screen capturing operation when the front-facing camera detects a blank holding gesture. Wherein the pinch-off gesture may also be referred to as a pinch-off screen capture gesture, the scene may be referred to as a pinch-off screen capture scene.

For example, taking a space up/down gesture as an example, a finger is changed from a closed and spread state to a downward bent state (up gesture), a manipulation manner corresponding to the gesture is preset to turn up a page or slide up a screen, and a finger is changed from a closed and bent state to an upward spread state (down gesture), a manipulation manner corresponding to the gesture is preset to turn down a page or slide down a screen. When the electronic equipment is in a bright screen state, the electronic equipment can automatically execute the upward page turning operation when the upward sliding gesture is detected through the front-facing camera, and can automatically execute the downward page turning operation when the downward sliding gesture is detected, so that the man-machine interaction can be completed without the need of a user to contact the electronic equipment. Among other things, the pan up/down gesture may also be referred to as a pan swipe screen gesture, and the scene may be referred to as a pan swipe screen scene.

For another example, taking a blank pressing gesture as an example, a palm of a user is far from or near to a screen of the electronic device, and a control mode corresponding to the gesture is preset to answer an incoming call, when the electronic device receives an incoming signal and displays an incoming call interface, the electronic device automatically executes an operation of answering the incoming call when the blank pressing gesture is identified through the front camera. This scenario may be referred to as a space-compression scenario.

In addition, in the embodiment of the application, the space gesture service also supports user-defined operations, such as the user cancelling the space pressing gesture, answering a call, resetting to the space pressing gesture, and jumping to a payment code interface.

For example, when the electronic device is in a bright screen state and the electronic device displays the primary desktop, the user may trigger a predefined shortcut service, such as a quick jump from the primary desktop to the pay code interface, via a blank press gesture. In this scenario, the payment interface is quickly brought up by pressing the hand at a distance, so this scenario may be referred to as a smart payment scenario. The user may also reset to "skip to swipe interface" or "skip to ride interface" and the like.

Wherein, intelligent code identifying service: when the electronic equipment is in a bright screen state, a user aims the front camera of the electronic equipment at the two-dimensional code, the electronic equipment can automatically finish code scanning and analysis, and the electronic equipment quickly jumps to a payment code interface or a riding code interface without the need of clicking a scanning entrance by the user to trigger code scanning.

Wherein, look at the business of not putting out screen: when the electronic equipment is in a bright screen state, if the electronic equipment detects that the user looks at the electronic equipment through the front-facing camera, the display screen of the electronic equipment cannot be extinguished until the user does not look at the electronic equipment any more.

Wherein, gazing at the volume-reducing service: when the electronic equipment is in a bright screen state, if the electronic equipment detects that a user looks at the electronic equipment through the front-facing camera, the electronic equipment can automatically reduce the volume of the incoming call bell or the volume of the short message prompt tone. For example, when the user receives an incoming call, the user triggers the mobile phone to automatically turn down the volume of the incoming call bell by looking at the mobile phone screen.

Alternatively, the gaze non-quench service and the gaze volume-down service may both be on by default, or may be turned on or off separately.

Wherein, intelligent horizontal and vertical screen business: when the electronic equipment is in a bright screen state, if the electronic equipment detects that the screen of the electronic equipment deflects and the electronic equipment detects the face direction of the user through the front-facing camera, the electronic equipment automatically rotates the screen according to the face direction of the user, so that the display content of the screen always keeps forward relative to the face direction of the user. For the screen autogiration function that adopts the realization of gravity inductor among the prior art, the intelligent horizontal and vertical screen function that this application realized can more satisfy user's user demand.

Wherein, auxiliary photographing business: when the electronic equipment is in a bright screen state, if the electronic equipment detects that the electronic equipment starts a camera function and the electronic equipment detects the face direction of a user through the front-facing camera, the electronic equipment automatically rotates the screen according to the face direction of the user, so that the display content of the screen always keeps forward relative to the face direction of the user. And when the electronic device displays the shot photo, the electronic device automatically rotates the screen according to the face direction of the user, so that the photo displayed on the screen always keeps forward relative to the face direction of the user.

Wherein, intelligent AOD service: when the electronic equipment is in the screen-off state, if the electronic equipment detects the face characteristics through the front-facing camera, the electronic equipment automatically displays the screen-off pattern. When the electronic equipment displays the screen-extinguishing pattern, part of pixel points of the screen are lightened, information such as a clock, a date, a notice and the like can be displayed, so that a user can conveniently look up the information. It should be noted that, the preconditions for implementing the intelligent AOD service are: the electronic device supports the off-screen display function and has turned on the off-screen display function.

It should be noted that, the above various services supported by the smart sensor application are exemplary examples, and it can be understood that, in actual implementation, the smart sensor application in the embodiment of the present application may also support other possible services, which may be specifically determined according to actual use requirements, and the embodiment of the present application is not limited.

The application framework layer provides an application programming interface (application programming interface, API) and programming framework for application programs of the application layer. The application framework layer includes a number of predefined functions. By way of example, the application framework layer may include a camera management module, an AON management module, etc., which embodiments of the present application do not impose any limitation.

The hardware abstraction layer is the encapsulation of the Linux kernel driver, provides an interface upwards, hides the hardware interface details of a specific platform, and provides a virtual hardware platform for an operating system. In this embodiment of the present application, the hardware abstraction layer includes modules such as a smart-aware interface and services, service management, service conflict management, service capability management, a distributor, a smart-aware client, and a native platform client.

The intelligent perception interface and the service module provide an interface program for accessing the intelligent perception algorithm platform for the upper intelligent perception application and provide services related to the intelligent perception algorithm platform (also called as intelligent perception algorithm integration framework). The application adds an own AON service in the camera HAL, and supports characteristic development such as gesture characteristics, code scanning characteristics and the like on a platform software infrastructure.

The service management module is used for managing a plurality of services such as a space gesture service, an intelligent code identification service, a watching non-screen-quenching service, a watching volume reduction service, an intelligent horizontal and vertical screen service, an auxiliary photographing service, an intelligent AOD service and the like. And the service management module can call the service conflict management module to make decisions.

The business conflict management module is used for making a decision on the business of the bright screen scene and the business of the off-screen scene. The business conflict management module can judge the business according to the business related data obtained by the monitoring of the upper intelligent perception application: the service related data is an image acquired in a bright screen state or an image acquired in a dead screen state. In some embodiments, if the service related data is an image acquired when the electronic device is in a bright screen state, the service management module may issue the service related data to the smart sensor client. If the service related data is an image acquired when the electronic equipment is in the off-screen state and the image has face characteristics, the service management module can determine that the intelligent AOD service is triggered and send the service related data to the intelligent perception client. The intelligent perception client has the functions of proxy and data forwarding.

The business capability management module is used for managing capability information of each business supported by the intelligent perception algorithm platform. In some embodiments, the capability information of the service may be represented by a number, and illustratively, the capability value of the smart identification service is preset to 0; the capability value of the space gesture service is preset to be 1, wherein the capability value of the space screen capture is preset to be 1-1, the capability value of the space sliding screen is preset to be 1-2, and the capability value of the space pressing is preset to be 1-3. It should be noted that, the capability information is exemplary, and may be specifically set according to actual use requirements, which is not limited in this embodiment.

Specifically, according to the scheme, under the condition that a user triggers to start a certain service, the intelligent perception application can acquire the capability information of the service, and then the intelligent perception application registers and subscribes to the corresponding service to the intelligent perception algorithm platform according to the capability information of the service. After the service subscription is successful, the intelligent perception application monitors whether the service is triggered in real time, and sends service data to the intelligent perception algorithm platform under the condition that the intelligent perception application perceives that the service is triggered, and carries out algorithm analysis through the intelligent perception algorithm platform, and then executes the service according to an analysis result.

The distributor is used for distributing the service related data according to the indication of the service management module.

The kernel layer is a layer between hardware and software. Illustratively, the kernel layer may contain display drivers, camera drivers, and sensor drivers.

In particular to the present application, the kernel layer may include an API interface and a sensor interface, a smart aware algorithm integration framework (i.e., a smart aware algorithm platform), and a native AON algorithm platform.

The intelligent perception algorithm platform can comprise a decision unit, a control unit, an execution unit and the like, wherein the decision unit, the control unit, the execution unit and the like are combined to complete low-power consumption algorithm analysis of business related data, obtain an image analysis result and report the image analysis result to intelligent perception application. The intelligent awareness algorithm platform may also be referred to as a low power subsystem.

In the embodiment of the present application, in the early service awareness stage (the first stage), the intelligent awareness client performs image analysis on the service related data to monitor whether the service related data meets the first preset trigger condition, so that the requirement on image quality is lower. In the later service identification stage (second stage), the intelligent perception algorithm platform identifies specific service content by carrying out image analysis on service related data meeting the first preset trigger condition, so that the requirement on image quality is higher.

In the first stage, the service related data comprises continuous multi-frame images, are images with smaller size and lower resolution, and adopt the low-power consumption TCM memory for data caching. In the second stage, the service related data comprises continuous multi-frame images, are images with larger size and higher resolution, and adopt the DDR memory with normal power consumption and the TCM memory with low power consumption for data caching. According to the scheme, the image analysis is carried out in stages, so that the service can be accurately monitored, and the power consumption is effectively reduced.

It should be noted that, in the embodiment of the present application, on the basis of the native AON algorithm platform, a smart perception service and a smart perception algorithm integration framework are mainly added. The intelligent perception service and the intelligent perception algorithm integration framework are matched with the native AON algorithm platform, so that the low-power consumption AON intelligent perception function and service are completed.

On one hand, the intelligent perception service provides a unified service interface to the outside and supports the AON capability of the intelligent perception related service for using the native AON algorithm platform; and meanwhile, the interfaces of the native AON algorithm platform are shielded, and the native interfaces are not opened to the outside.

On the other hand, the intelligent perception service relies on a platform communication path to establish complete business interaction with an intelligent perception algorithm integration framework so as to use the low-power-consumption computing capacity of the AON camera and the native AON algorithm platform.

In yet another aspect, the intelligent perception algorithm integration framework constructs a unified low-power computing platform supporting AON services by integrating AON camera resources, traditional sensor data, and integrated business algorithms.

The hardware layer provides various hardware devices, such as those involved in embodiments of the present application including AON ISPs, camera sensors, and camera cameras, among others. The camera and the camera sensor are commonly used for acquiring images in real time, the AON ISP is used for processing image signals of the acquired images, and the hardware equipment provides hardware support for the intelligent perception algorithm platform.

It should be noted that fig. 4 only shows modules related to the embodiments of the present application, each layer may further include any other possible modules, each module may further include one or more sub-modules, and the present application is not limited thereto.

It should be noted that, although the embodiment of the present application is described taking an Android system as an example, the basic principle is equally applicable to electronic devices based on an iOS or Windows operating system.

A timing chart of the interaction of the modules in the above technical architecture provided in the embodiment of the present application will be described below by using fig. 5; the intelligent perception application, the framework layer module, the intelligent perception interface and the intelligent perception service module (collectively referred to as the intelligent perception interface and the intelligent perception service module), the intelligent perception client, the intelligent perception algorithm platform and the front-facing camera are used for carrying out information interaction, and the process that the electronic equipment realizes AON intelligent perception through the front-facing camera is exemplarily described.

S101, after the electronic equipment is started, the intelligent perception service module interacts with the intelligent perception client to start a camera provider process.

S102, the intelligent perception service module applies for registering to the intelligent perception client.

S103, the intelligent perception application receives the operation of starting the intelligent perception function by the user.

It should be noted that, here, the intelligent sensing function may be any of the following: the intelligent automatic vision system comprises a space gesture function, an intelligent code recognition function, a gaze non-screen extinction function, a gaze volume reduction function, an intelligent horizontal and vertical screen function, an auxiliary photographing function and an intelligent AOD function.

Illustratively, FIG. 6 shows a schematic diagram of an interactive interface for a user to turn on a smart awareness function. As shown in (a) and (b) of fig. 6, the electronic device displays a setting main interface in response to an operation of clicking a setting application icon in a desktop by a user. Auxiliary function options are displayed in the setting main interface. As shown in (b) and (c) of fig. 6, the electronic device displays an auxiliary function interface in response to an operation of clicking an auxiliary function option by the user. A smart sense option is displayed in the auxiliary function interface. As shown in (c) and (d) of fig. 6, the electronic device displays a smart perception setting interface in response to an operation of clicking a smart perception option by a user.

As shown in fig. 6 (d), an intelligent gaze setting field, a space gesture setting field, and other setting fields are displayed in the intelligent perception setting interface. The intelligent watching setting column comprises a switch for watching the screen without extinguishing the screen and a switch for watching the screen to weaken the volume. The space gesture setting column comprises a switch of a space sliding screen, a switch of a space screen capture, a switch of a space pressing and a switch of intelligent payment. Other setting columns comprise a switch of an intelligent AOD, a switch of an intelligent horizontal and vertical screen and a switch for assisting photographing.

The user can turn on or off one or more of the above functions according to actual use requirements. For ease of explanation, the following is exemplary of a user triggering the open space screen capture function.

S104, the intelligent sensing application responds to the operation of triggering and starting the blank screen capturing function by the user, and inquires the capability information of the blank screen capturing service from the intelligent sensing client.

The intelligent perception application sends a message for inquiring the service capability to the framework layer module, the framework layer module sends the message to the intelligent perception interface and the service module, and then the intelligent perception interface and the service module transmit the message to the intelligent perception client. The intelligent perception client can call a capability management module to inquire the capability information of the screen capturing service.

Wherein the capability values of different services are different. For example, the capability value of the screen capturing function is 1-1, the capability value of the screen sliding function is 1-2, and the capability value of the screen pressing function is 1-3.

S105, the intelligent perception client feeds the queried business capability information back to the intelligent perception application.

S106, the intelligent perception application registers the AON service event to the intelligent perception algorithm platform according to the business capability value.

The registered data flow starts from the intelligent perception application, sequentially passes through the framework layer module, the intelligent perception interface, the service module and the intelligent perception client, and reaches the intelligent perception algorithm platform.

S107, the intelligent perception algorithm platform subscribes to the AON service event according to the business capability value.

For example, assuming that the business capability value is 1-1, as described above, the capability value of the screen capturing function is 1-1, then the AON service event subscribed to by the intelligent awareness algorithm platform is the screen capturing.

Where the AON service event is subscribed, the intelligent awareness algorithm platform may employ a lower resolution image specification, such as QQVGA (160×120), in some cases, and a higher resolution image specification, such as QVGA (320×240), in other cases, as desired. And the intelligent perception algorithm platform can use the TCM memory to perform data caching under some conditions according to requirements, and use the DDR memory and the TCM memory to perform data storage under other conditions.

The following S108-S117 are loop execution procedures of AON intelligent sensing.

S108, the front camera collects continuous multi-frame images when the equipment is on and off, and transmits the collected continuous multi-frame images to the intelligent perception algorithm platform.

Wherein, leading camera refers to AON camera.

S109, the intelligent perception algorithm platform acquires continuous multi-frame images of QQVGA specifications, stores the images in the TCM memory and analyzes the continuous multi-frame images.

Illustratively, an AON camera (front camera) outputs a lower resolution image in accordance with the QQVGA (160×120) specification. Correspondingly, the intelligent perception algorithm platform acquires a lower resolution image of QQVGA (160 multiplied by 120) specification, and performs image analysis.

S110, the intelligent perception algorithm platform judges whether a first preset trigger condition is met according to the analysis result.

Illustratively, the first preset trigger condition may include: the image comprises hand features, or the image comprises two-dimensional code features, or the image comprises human eye features, or the image comprises human face features. It should be noted that, the first preset trigger condition is described as an example, and may be specifically set according to actual use requirements, which is not limited in this embodiment of the present application.

For example, assuming that the analysis result indicates that there is a hand feature in the image, and the first preset trigger condition is satisfied, it may be determined that the spaced gesture service is triggered.

For another example, if the analysis result indicates that there is a human eye feature in the image, and the first preset triggering condition is met, it may be determined that the intelligent gaze service is triggered.

And S111, when the first preset trigger condition is judged not to be met according to the analysis result, the intelligent perception client does not respond and releases the invalid frame.

S112-S113 are continued to be performed after S110 determines that the first preset trigger condition is satisfied, and S114 is continued to be performed after S110 determines that the first preset trigger condition is satisfied.

S112, the intelligent perception client reports the message triggered by the service to the intelligent perception application.

S113, the intelligent perception application displays prompt information corresponding to the triggered service on a screen.

Illustratively, fig. 7 shows a schematic diagram of an interactive interface when a space-efficient gesture service is triggered. As shown in fig. 7 (a), the front camera collects an image of palm expansion of the user, and after image analysis, it can be determined that the spaced gesture service is triggered, and at this time, the palm expansion image 11 can be displayed on the mobile phone screen. As shown in fig. 7 (b), the front camera collects the folded and unfolded image of the user, and after image analysis, it can be determined that the space gesture service triggered by the user is specifically a space screen capture service, and accordingly, the palm folded image 12 can be displayed on the mobile phone screen, and the prompt message of "screen capture in progress" is displayed. Therefore, after the user sees the prompt information displayed on the mobile phone, the user can know that the space gesture made by the user is being recognized by the mobile phone, and the interaction experience of the user is improved.

S114, the intelligent perception algorithm platform acquires continuous multi-frame images of QVGA specifications, stores the images in the DDR memory and the TCM memory, and analyzes the continuous multi-frame images to obtain an image analysis result.

Illustratively, an AON camera (front camera) outputs a higher resolution image in accordance with the QVGA (320×240) specification. Correspondingly, the intelligent perception algorithm platform acquires a higher-resolution image with the QVGA (320 multiplied by 240) specification, and performs image analysis.

It should be noted that the DDR memory and the TCM memory operate in different hardware buffers, respectively. The DDR memory can access data in the TCM memory, and the TCM memory does not support access to the data in the DDR memory.

S115, the intelligent perception algorithm platform reports the image analysis result to the intelligent perception application.

S116, the intelligent perception application responds according to a preset strategy corresponding to the image analysis result.

For example, after analyzing the continuous multi-frame images, it is found that the user gesture is specifically a grasp gesture, that is, the analysis result of the service 1 is a grasp gesture, and the preset policy corresponding to the grasp gesture is a blank screen capturing. Thus, the smart aware application responds according to a preset strategy corresponding to the grip gesture, such as a blank screen capture.

The above describes the process of the user triggering the turning on of the intelligent awareness function and performing AON intelligent awareness. In some embodiments, the user may trigger the turning off of the smart awareness function according to the use requirement, see S117-S119 below for a specific procedure.

S117, the intelligent perception application receives the operation of turning off the intelligent perception function by the user.

Here, the smart sensor function may be any of the following: the intelligent automatic vision system comprises a space gesture function, an intelligent code recognition function, a gaze non-screen extinction function, a gaze volume reduction function, an intelligent horizontal and vertical screen function, an auxiliary photographing function and an intelligent AOD function. The user can turn off one or more of the above functions according to actual use requirements. For ease of explanation, the following is exemplified by a user triggering a close-spaced screen capture function.

S118, responding to the user operation, and sending a message of logging off the AON service event to the intelligent perception algorithm platform by the intelligent perception application.

S119, the intelligent perception algorithm platform stops subscribing to the AON service event.

Illustratively, the AON service event is a screen shot that is not triggered after the subscription to the screen shot is stopped, that is, the handset no longer responds to the screen shot gesture.

The above description is based on the software architecture shown in fig. 4, and the timing diagram shown in fig. 5 is combined to describe the service sensing and recognition in the image processing method provided in the embodiment of the present application from the local perspective of the newly added intelligent algorithm platform. The image processing method provided in the embodiment of the present application is described in stages from the overall point of view of the native AON algorithm platform and the newly added intelligent algorithm platform with reference to the flowchart shown in fig. 8.

Firstly, it should be noted that the services supported by the native AON algorithm platform include a face unlocking screen service, and the services supported by the newly added intelligent algorithm platform include a blank gesture service, an intelligent code recognition service, a watching screen non-extinguishing screen service, a watching screen volume reducing service, an intelligent horizontal and vertical screen service, an auxiliary photographing service and an intelligent extinguishing screen display service.

S201, continuous multi-frame images are collected in real time through an AON camera (front-facing camera) of the electronic equipment.

S202, acquiring continuous multi-frame images with lower resolution (for example, QQVGA specification), performing data caching by adopting a low-power consumption TCM memory, and analyzing the continuous multi-frame images.

Among them, an AON camera (front camera) outputs a higher resolution image according to QVGA (320×240) specifications. Correspondingly, the intelligent perception algorithm platform acquires a higher-resolution image with the QVGA (320 multiplied by 240) specification, and performs image analysis.

S203, judging whether a first preset triggering condition is met according to the image analysis result.

Whether the service is triggered or not can be judged by judging whether the analysis result meets a first preset triggering condition or not.

Illustratively, the first preset trigger condition may be any one of the following: the image comprises a first preset feature, and the image is acquired when the screen is on and unlocked; or the image contains the face characteristics, and the image is acquired when the screen is off. The first preset feature may include any one of a face feature, an eye feature, a gesture feature, and a two-dimensional code feature.

On the one hand, when S204 determines that the first preset trigger condition is satisfied, S204 to S207 described below are continuously performed. And processing the business data through a newly-added intelligent perception algorithm platform. As described above, the preset services supported by the added intelligent perception algorithm platform may include a space gesture service, an intelligent code recognition service, a gazing non-screen-off service, a gazing volume reduction service, an intelligent horizontal and vertical screen service, an auxiliary photographing service, and an intelligent AOD service capability.

S204, acquiring continuous multi-frame images with higher resolution (such as QVGA specification), and performing data caching by adopting a DDR memory and a TCM memory.

S205, analyzing continuous multi-frame images through a newly added intelligent perception algorithm platform to obtain an image analysis result 1.

S206, reporting the image analysis result 1 to the application layer by the newly added intelligent perception algorithm platform.

S207, responding according to a preset strategy corresponding to the image analysis result 1.

For example, multiple continuous images collected by the front-end camera are analyzed, and at this time, the multiple continuous images adopt low-resolution images and are cached by adopting a low-power-consumption memory. If the multi-frame continuous images are found to meet the first preset triggering condition, the triggered service can be judged, for example, the multi-frame continuous images comprise two-dimensional code characteristics, and therefore the triggered service can be judged to be the intelligent code identification service. According to the judgment, the intelligent code recognition service is a preset service in the newly added intelligent perception algorithm platform, so that the service analysis task is distributed to the newly added intelligent perception algorithm platform, the newly added intelligent perception algorithm platform performs image analysis on multiple continuous images, and at the moment, the multiple continuous images adopt high-resolution images and are cached by adopting normal power consumption memories and low power consumption memories.

On the other hand, when S204 determines that the first preset trigger condition is not satisfied, S208 to S212 described below are continued to be executed. And processing the service data through a native AON algorithm platform.

S208, judging whether a second preset triggering condition is met or not according to the image analysis result.

The second preset trigger condition is that the image contains a face feature, and the image is an image acquired when the screen is on and locked.

S209, when judging that the second preset trigger condition is met, acquiring continuous multi-frame images with higher resolution (for example, VGA specification), and adopting a DDR memory to perform data caching.

S210, analyzing continuous multi-frame images through a native AON algorithm platform to obtain an image analysis result 2.

S211, reporting an image analysis result 2 to an application layer by the original AON algorithm platform.

S212, responding according to a preset strategy corresponding to the image analysis result 2.

For example, multiple continuous images collected by the front-end camera are analyzed, and at this time, the multiple continuous images adopt low-resolution images and are cached by adopting a low-power-consumption memory. If the multi-frame continuous images are found to meet the first preset triggering condition, it can be judged that the service is triggered, for example, the multi-frame continuous images collected when the electronic equipment is on and locked include the face features, and therefore the triggered service can be judged to be the face unlocking screen service. According to the judgment, the face unlocking screen service is a preset service in the original AON algorithm platform, so that the service analysis task is distributed to the original AON algorithm platform, the original AON algorithm platform performs image analysis on a plurality of continuous images, and at the moment, the continuous images of the plurality of frames adopt high-resolution images and are cached by adopting a normal power consumption memory and a low power consumption memory.

It should be noted that, in the embodiment of the present application, when it is determined that the first preset trigger condition is not satisfied, and the second preset trigger condition is not satisfied, that is, when no service is triggered, the acquired continuous multi-frame images are discarded, and then the above-described S201 is executed in a return manner.

The embodiment of the application realizes AON intelligent sensing through the newly added intelligent sensing algorithm platform and the original AON algorithm platform, wherein the image processing process finished through the intelligent sensing algorithm platform can be divided into two stages:

the first stage: in the step S201-S203, coarse analysis is performed on the multi-frame continuous images collected by the front camera, and it is determined whether the multi-frame continuous images meet a first preset trigger condition, so as to determine whether a service is triggered. In the first stage, the multi-frame continuous images adopt low-resolution images, and the data caching is carried out by adopting a low-power-consumption memory.

And a second stage: as described in S204-S207 above, the multi-frame continuous images collected by the front camera are subjected to fine analysis to identify the specific content of the triggered service. In the second stage, the multi-frame continuous images adopt high-resolution images, and the normal power consumption memory and the low power consumption memory are adopted to carry out data caching together.

The image processing method provided in the embodiment of the present application is described in stages with reference to the flowchart shown in fig. 8. The following describes a service identification procedure in the image processing method provided in the embodiment of the present application, in conjunction with the flowchart shown in fig. 9, according to the judgment of the on/off scene.

It should be noted that, it is assumed that services such as a space gesture service, an intelligent code recognition service, a watching non-screen-off service, a watching volume reduction service, an intelligent horizontal-vertical screen service, an auxiliary photographing service, an intelligent AOD and the like supported by the newly added intelligent perception algorithm platform are all started, wherein the intelligent AOD service is applied to a screen-off scene, and other services are applied to a screen-on scene.

As shown in fig. 9, the service identification process includes S301 to S316.

S301, continuous multi-frame images are collected in real time through a front-facing camera of the electronic equipment.

S302, acquiring continuous multi-frame images with lower resolution, performing data caching by adopting a low-power consumption TCM memory, and analyzing the continuous multi-frame images.

Illustratively, as described above, the continuous multi-frame image may be analyzed by a smart perception algorithm platform.

S303, judging whether a first preset trigger condition is met according to the analysis result.

Similarly to S203 described above, whether or not a service is triggered is determined by determining whether or not the analysis result satisfies the first preset trigger condition.

When it is determined in S303 that the first preset trigger condition is not satisfied in the image, the acquired image frame is discarded, and the execution returns to S301.

S304, when judging that the first preset triggering condition is met, judging whether the service is triggered in the bright screen scene.

When it is determined that the service is triggered in the bright screen scene, S305 described below is continued to be executed. When it is judged that the trigger is not in the bright screen state, execution of S307 described below is continued.

S305, judging whether the screen is locked.

If the screen is locked, continuing to execute S306-S307; if the screen is not locked, i.e., the screen is in the unlocked state, execution continues with S308-S3311.

S306, when the screen is on and the service is triggered during unlocking, acquiring continuous multi-frame images with higher resolution, performing data caching by adopting the DDR memory and the TCM memory, and analyzing the images through the intelligent perception algorithm platform.

S307, responding according to an execution strategy corresponding to the image analysis result, such as screen capturing/page turning/incoming call answering at intervals, intelligent code recognition, screen blackout fixation, volume reduction fixation, intelligent horizontal and vertical screen fixation, and/or auxiliary photographing.

Note that, the specific method of processing an image when the screen is on and unlocked will be described in detail in fig. 11.

S308, when the service is triggered when the screen is on and locked, judging whether the face features exist in the image.

S309, when judging that the face features exist in the image, determining that the triggered service is a face unlocking screen service.

S310, analyzing the image through the intelligent perception algorithm platform.

S311, responding according to the execution strategy corresponding to the image analysis result, namely unlocking the screen.

The image processing method in the off-screen scene is further seen below.

S312, judging whether the service is triggered in the off-screen scene.

S313, when judging that the service is triggered in the screen-off scene, judging whether the face features exist in the image.

When it is determined that there is no face feature in the image, the execution returns to S301 without responding.

And S314, when judging that the face features exist in the image, determining that the triggered service is an intelligent AOD service.

S315, acquiring continuous multi-frame images with higher resolution, performing data caching by adopting a DDR memory and a TCM memory, and analyzing the images through an intelligent perception algorithm platform.

S316, responding according to the execution strategy corresponding to the image analysis result, namely the intelligent AOD.

Illustratively, FIG. 10 shows an interface schematic of a smart AOD scenario applied by embodiments of the present application.

By combining the above-mentioned fig. 9, through the branch judgment of the bright screen scene and the branch judgment of the off screen scene, the services under different scenes can be distinguished and identified.

Fig. 9 illustrates an image analysis process in a bright screen scene and an off screen scene according to an embodiment of the present application. The following describes an exemplary image analysis process under a bright screen and unlock scene provided in the embodiment of the present application from the perspective of a newly added smart perception algorithm platform in combination with the flowchart shown in fig. 11.

S401, continuous multi-frame images are collected in real time through a front-facing camera of the electronic equipment.

S402, acquiring continuous multi-frame images with lower resolution, performing data caching by adopting a low-power consumption TCM memory, and analyzing the continuous multi-frame images.

S403, judging whether the first preset triggering condition is met according to the analysis result.

And judging whether the service is triggered or not by judging whether the analysis result meets a first preset triggering condition or not.

When S403 determines that the first preset trigger condition is not satisfied in the image, the acquired image frame is discarded, and the execution returns to S401.

S404, when judging that the first preset trigger condition is met, judging whether the screen is on or not and unlocking.

When it is judged that the screen is on and unlocked, the following S406 to S415 are continued.

And S405, when the trigger is judged to be in the bright screen state, acquiring continuous multi-frame images with higher resolution, performing data caching by adopting the DDR memory and the TCM memory, and analyzing the images through the intelligent perception algorithm platform.

S406, judging whether the image contains the face features.

S407, when S406 judges that the image contains the face feature, judging whether the camera is started.

S408, when S407 determines that the camera is turned on, it is determined that the image analysis result is: the triggered services include an auxiliary photographing service and an intelligent landscape/portrait screen service.

S409, when S407 determines that the camera is not turned on, determining that the image analysis result is: the triggered services include a gazing screen non-extinguishing service, a gazing screen volume reducing service and an intelligent horizontal and vertical screen service.

S410, when S406 judges that the image does not contain the face feature, judging whether the image contains the two-dimensional code.

S411, when S410 determines that the image includes the two-dimensional code, determining that the image analysis result is: the triggered service is an intelligent code identification service.

S412, when S410 judges that the image does not contain the two-dimensional code, judging whether the image contains the preset gesture.

S413, when the image is judged to contain the preset gesture in S412, determining that the image analysis result is: the triggered service is a space-time gesture service.

When S412 determines that the image does not include the preset gesture, the method returns to continue S401 without responding.

After the above-described S408, S411, S413, S414 to S415 are continued.

S414, obtaining an image analysis result.

S415, responding according to a preset strategy corresponding to the image analysis result.

Illustratively, if the image analysis results in: the triggered service comprises an auxiliary photographing service and an intelligent horizontal and vertical screen service, and the corresponding preset strategy comprises the auxiliary photographing and the intelligent horizontal and vertical screen based on the face direction.

Illustratively, if the image analysis results in: the triggered service comprises a watching screen non-extinguishing service, a watching screen volume reducing service and an intelligent horizontal and vertical screen service, and the corresponding preset strategy comprises watching screen non-extinguishing service, watching screen volume reducing service and intelligent horizontal and vertical screen service.

Illustratively, if the image analysis results in: the triggered service is an intelligent code identification service, and the corresponding preset strategy is code scanning and code identification, and jumps to a payment code interface or a riding code interface.

Illustratively, if the image analysis results in: the triggered service is a blank gesture service and is a grasp gesture, and the corresponding preset strategy is screen capturing; if the image analysis result is: the triggered service is a space gesture service and is an up/down sliding gesture, and the corresponding preset strategy is page turning; if the image analysis result is: the triggered service is a blank gesture service and a blank pressing gesture, and the corresponding preset strategy is to confirm answering the incoming call when receiving an incoming signal or to quickly jump to a payment code interface when the electronic device displays a desktop. Illustratively, FIG. 12 shows an interface schematic of a blank screen capture scenario employed by embodiments of the present application.

According to the scheme, the triggered service is distributed to the newly-added intelligent perception algorithm platform in the bright screen scene, and the newly-added intelligent perception algorithm platform performs image analysis to obtain an image analysis result, so that response can be made according to a preset strategy corresponding to the image analysis result. Therefore, the image processing method provided by the embodiment of the application can support more AON intelligent perception functions and services, and user experience is improved.

Table 1 below shows the individual power consumption and the total power consumption in the case where the present application scheme is applied to different services or characteristics. As shown in table 1, assuming that the graph resolution is 320×240, the frame rate is 10, the total power consumption in the intelligent code scanning scene is 19.03mA, the total power consumption in the space gesture scene is 19.03mA, and the total power consumption in the intelligent horizontal/vertical screen/face auxiliary photographing scene is 11.03mA. And the total power consumption of each application scene is about 100mA when the related technology is adopted. Therefore, the scheme of the application is lower in power consumption under the condition of being applied to different services or characteristics, and the low-power consumption AON intelligent sensing function and service are realized.

TABLE 1

In this embodiment, the "greater than" may be replaced with "greater than or equal to" and "less than or equal to" may be replaced with "less than" or "greater than or equal to" may be replaced with "greater than" and "less than" may be replaced with "less than or equal to".

The various embodiments described herein may be separate solutions or may be combined according to inherent logic, which fall within the scope of the present application.

The solutions provided in the embodiments of the present application are mainly described above from the perspective of method steps. It will be appreciated that, in order to implement the above-described functions, an electronic device implementing the method includes corresponding hardware structures and/or software modules that perform the respective functions. Those of skill in the art will appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as hardware or combinations of hardware and computer software. Whether a function is implemented as hardware or computer software driven hardware depends upon the particular application and design constraints imposed on the solution. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.

The present application also provides a chip coupled to a memory for reading and executing a computer program or instructions stored in the memory to perform the methods of the embodiments described above.

The present application also provides an electronic device comprising a chip for reading and executing a computer program or instructions stored in a memory, such that the methods in the embodiments are performed.

The present embodiment also provides a computer-readable storage medium having stored therein computer instructions which, when executed on an electronic device, cause the electronic device to perform the above-described related method steps to implement the image processing method in the above-described embodiments.

The present embodiment also provides a computer program product, the computer readable storage medium storing a program code which, when run on a computer, causes the computer to perform the above-described related steps to implement the image processing method in the above-described embodiments.

In addition, embodiments of the present application also provide an apparatus, which may be specifically a chip, a component, or a module, and may include a processor and a memory connected to each other; the memory is used for storing computer-executable instructions, and when the device is running, the processor can execute the computer-executable instructions stored in the memory, so that the chip executes the image processing method in each method embodiment.

The electronic device, the computer readable storage medium, the computer program product or the chip provided in this embodiment are used to execute the corresponding method provided above, so that the beneficial effects thereof can be referred to the beneficial effects in the corresponding method provided above, and will not be described herein.

In the several embodiments provided in this application, it should be understood that the disclosed apparatus and method may be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative, e.g., the division of modules or units is merely a logical function division, and there may be additional divisions when actually implemented, e.g., multiple units or components may be combined or integrated into another apparatus, or some features may be omitted or not performed. Alternatively, the coupling or direct coupling or communication connection shown or discussed with each other may be an indirect coupling or communication connection via some interfaces, devices or units, which may be in electrical, mechanical or other form.

The foregoing is merely specific embodiments of the present application, but the scope of the present application is not limited thereto, and any person skilled in the art can easily think about changes or substitutions within the technical scope of the present application, and the changes and substitutions are intended to be covered by the scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. The image processing method is characterized by being applied to a system architecture of an electronic device, wherein the system architecture comprises a first algorithm platform and a second algorithm platform, the first algorithm platform supports processing services different from the second algorithm platform supports processing services, and the second algorithm platform supports processing services which are face unlocking screen services, and the method comprises the following steps:

the electronic equipment starts a first function, and after the first function is started, a camera module of the electronic equipment is in a normally-open state;

invoking the first algorithm platform to perform image analysis on the multi-frame first image to obtain a first image analysis result;

outputting a multi-frame second image with a second size specification through the camera module when the first image analysis result meets a first preset trigger condition; invoking the first algorithm platform to perform image analysis on the multi-frame second image to obtain a second image analysis result; responding according to a preset strategy corresponding to the second image analysis result;

outputting a multi-frame third image of a third size specification through the camera module when the first image analysis result meets a second preset trigger condition; invoking the second algorithm platform to perform image analysis on the multi-frame third image to obtain a third image analysis result; responding according to a preset strategy corresponding to the third image analysis result;

2. The method according to claim 1, wherein the first preset trigger condition is any one of the following:

the image comprises a first preset feature, and the image is acquired when the screen is on and unlocked; the first preset features comprise any one of face features, eye features, gesture features and two-dimensional code features;

the image comprises human face characteristics, and the image is acquired when the screen is turned off;

the second preset triggering condition is that the image contains face characteristics, and the image is acquired when the screen is on and locked.

3. The method according to any one of claim 1 to 2, wherein,

the first algorithm platform supports the processing services including a space gesture service, an intelligent code recognition service, a screen watching non-screen-off service, a screen watching volume reducing service, an intelligent horizontal and vertical screen service, an auxiliary photographing service and an intelligent screen-off display service.

4. A method according to any one of claims 1 to 3, further comprising:

after outputting a multi-frame first image of a first size specification by a camera module of the electronic device, storing the multi-frame first image in a first memory;

after outputting a multi-frame second image of a second size specification by the camera module, storing the multi-frame second image in the first memory and a second memory;

after outputting a multi-frame third image of a third size specification by the camera module, storing the multi-frame third image in the second memory;

wherein, in the case of storing the same image, the power consumption caused by the first memory is smaller than the power consumption caused by the second memory, and the power consumption caused by the first memory and the second memory together is smaller than the power consumption caused by the second memory.

5. The method of claim 4, wherein the first memory is a tightly coupled memory TCM and the second memory is a double rate synchronous dynamic random access memory DDR.

6. The method according to any one of claim 1 to 5, wherein,

The third dimension specification is video graphic array VGA, and the corresponding resolution is 640 multiplied by 480;

the second dimension specification is QVGA, and the corresponding resolution is 320 multiplied by 240;

the first size specification is QQVGA, and the corresponding resolution is 160×120.

7. The method of any one of claims 1 to 6, wherein the camera module is a front-facing camera.

8. The method according to any one of claims 1 to 7, wherein the system framework further comprises a smart aware application, the first function being a function provided by the smart aware application;

after the electronic device turns on the first function, the method further includes:

the intelligent perception application receives a first operation of starting a first service by a user, wherein the first service is a service supported by the intelligent perception application;

responsive to the first operation, the smart aware application issues a first message to the first algorithm platform, the first message being for indicating subscription to the first service;

the first algorithm platform is used for subscribing the first service in response to the first message;

the first service is at least one of a blank gesture, intelligent code recognition, screen fixation, volume reduction, intelligent horizontal and vertical screen fixation, auxiliary photographing and intelligent screen fixation display.

9. The method of claim 8, wherein after the subscribing to the first service, the method further comprises:

the first algorithm platform sends a second message to the camera module requesting the camera module to output an image according to the first size specification.

10. The method according to claim 8 or 9, wherein after the obtaining of the first image analysis result, the method further comprises:

the first algorithm platform judges that the first image analysis result meets the first preset trigger condition;

the first algorithm platform sends a third message to the camera module requesting the camera module to output an image according to the second dimensional specification.

11. The method of claim 10, wherein after the first algorithm platform determines that the first image analysis result meets the first preset trigger condition, the method further comprises:

the first algorithm platform reports a fourth message to the intelligent perception application, wherein the fourth message is used for indicating that a first service supported by the intelligent perception application is triggered;

And the intelligent perception application responds to the fourth message and triggers the electronic equipment to display the prompt information of the first service at a preset area of the screen.

12. The method according to any one of claims 8 to 11, wherein before said responding according to the preset policy corresponding to the second image analysis result, the method further comprises:

the first algorithm platform reports the second image analysis result to the intelligent perception application;

the responding according to the preset strategy corresponding to the second image analysis result comprises the following steps:

the intelligent perception application responds according to a preset strategy corresponding to the second image analysis result;

the preset strategy comprises at least one of screen capturing or page turning at intervals or incoming call answering, intelligent code recognition, screen non-extinguishing of a watching screen, volume reduction of the watching screen, intelligent screen transverse and vertical screen, auxiliary photographing and intelligent screen extinguishing display.

13. The method according to any one of claims 8 to 12, wherein the system framework further comprises a smart awareness interface;

and the intelligent perception application and the first algorithm platform conduct data interaction through the intelligent perception interface.

14. The method of any one of claims 1 to 13, wherein the system framework further comprises a traffic management module, a distributor, a smart awareness client, and a native platform client;

when the first image analysis result meets the first preset trigger condition, the service management module determines a first processing task and sends the first processing task to the intelligent perception client through the distributor, and the intelligent perception client forwards the first processing task to the first algorithm platform;

when the first image analysis result meets the second preset trigger condition, the service management module determines a second processing task and sends the second processing task to the native platform client through the distributor, and the native platform client forwards the second processing task to the second algorithm platform;

the first processing task is a task for processing the multi-frame second image, and the second processing task is a task for processing the multi-frame third image.

15. The method of any one of claims 1 to 14, wherein the system framework further comprises an algorithm library comprising a first image processing algorithm and a second image processing algorithm;

The image analysis is performed on the multi-frame first image through the first algorithm platform to obtain a first image analysis result, which comprises the following steps: the first algorithm platform calls the first image processing algorithm in the algorithm library, and performs image analysis on the multi-frame first image to obtain a first image analysis result;

the image analysis is performed on the multi-frame second image through the first algorithm platform to obtain a second image analysis result, which comprises the following steps: the first algorithm platform calls the second image processing algorithm in the algorithm library, and performs image analysis on the multi-frame second image to obtain a second image analysis result;

wherein the first image processing algorithm requires less image resolution than the second image processing algorithm.

16. The method according to any one of claims 1 to 15, wherein,

the second algorithm platform is a framework supporting a camera always-on AON algorithm provided by a native chip, and the first algorithm platform is an AON algorithm integration framework supporting multiple services and created based on the second algorithm platform.

17. An electronic device comprising a processor, a memory, and a computer program stored on the memory, the processor configured to execute the computer program to cause the electronic device to implement the method of any one of claims 1-16.

18. A computer readable storage medium, characterized in that the computer readable storage medium stores a computer program which, when run on an electronic device, causes the electronic device to perform the method of any one of claims 1 to 16.