WO2024055817A1 - 一种扫码方法及电子设备 - Google Patents

一种扫码方法及电子设备 Download PDF

Info

Publication number
WO2024055817A1
WO2024055817A1 PCT/CN2023/114075 CN2023114075W WO2024055817A1 WO 2024055817 A1 WO2024055817 A1 WO 2024055817A1 CN 2023114075 W CN2023114075 W CN 2023114075W WO 2024055817 A1 WO2024055817 A1 WO 2024055817A1
Authority
WO
WIPO (PCT)
Prior art keywords
recognition
frame
frame image
camera
code
Prior art date
Application number
PCT/CN2023/114075
Other languages
English (en)
French (fr)
Inventor
沈涛
马晓伟
王国英
Original Assignee
荣耀终端有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 荣耀终端有限公司 filed Critical 荣耀终端有限公司
Publication of WO2024055817A1 publication Critical patent/WO2024055817A1/zh

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06K—GRAPHICAL DATA READING; PRESENTATION OF DATA; RECORD CARRIERS; HANDLING RECORD CARRIERS
    • G06K7/00—Methods or arrangements for sensing record carriers, e.g. for reading patterns
    • G06K7/10—Methods or arrangements for sensing record carriers, e.g. for reading patterns by electromagnetic radiation, e.g. optical sensing; by corpuscular radiation
    • G06K7/14—Methods or arrangements for sensing record carriers, e.g. for reading patterns by electromagnetic radiation, e.g. optical sensing; by corpuscular radiation using light without selection of wavelength, e.g. sensing reflected white light
    • G06K7/1404—Methods for optical code recognition
    • G06K7/1408—Methods for optical code recognition the method being specifically adapted for the type of code
    • G06K7/1417—2D bar codes
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/54—Interprogram communication
    • G06F9/545—Interprogram communication where tasks reside in different layers, e.g. user- and kernel-space

Definitions

  • the present application relates to the field of terminal technology, and in particular, to a code scanning method and electronic equipment.
  • QR code scanning scenarios has become increasingly widespread. For example, you need to scan the payment code/receipt code when making payment transactions through payment applications, you need to scan personal QR codes when adding friends through social applications, you need to scan pass codes when entering and exiting specific places, etc.
  • code scanning speed is one factor that affects the user’s code scanning experience.
  • the current method of identifying the scanned code image collected by the camera during the code scanning process is slow in scanning code, which in turn causes difficulties in scanning codes for users and affects the user experience.
  • Embodiments of the present application provide a code scanning method and an electronic device, which can reduce the number of invalid recognition frame images recognized by the electronic device, thereby reducing the number of threads created for recognition frame images, the memory occupied by the recognition process, and the recognition time. Therefore, the recognition efficiency is improved.
  • the present application provides a code scanning method, which can be applied to electronic devices, where the electronic device includes a camera, and the method includes: the electronic device displays a code scanning interface. Then, the recognition frame image and the preview frame image are obtained, and the recognition frame image and the preview frame image are obtained according to the scanning code image collected by the camera. The electronic device displays a preview frame image on the code scanning interface. If the ambient light brightness of the recognition frame image is greater than or equal to the threshold, the electronic device performs code scanning and recognition on the recognition frame image to determine the code scanning recognition result.
  • the electronic device does not recognize the recognition frame images collected when the ambient light brightness is less than the preset value, but only recognizes the recognition frame images collected when the ambient light brightness is greater than the preset value. Therefore, the number of recognition frame images required to be recognized is less. And compared with the solution of not discarding the identification frame images, this application discards the invalid identification frame images, that is, discards the identification frame images for which the identification code cannot be recognized. Therefore, this application not only does not increase the code scanning time, but also avoids the target application's recognition of invalid frames, which causes a waste of device computing resources and reduces the code scanning speed, thus improving the code scanning efficiency.
  • the electronic device only selectively recognizes the recognition frame images, and the number of preview frame images displayed on the code scanning interface is consistent with the code scanning images collected by the camera, which does not affect the normal preview of the images collected by the camera on the code scanning interface. That is to say, discarding the recognition frame image will not have a visual perception for the user, and the preview frame image displayed on the scanning interface will not affect the user's viewing experience due to frame dropping. Therefore, this application does not affect the camera acquisition.
  • the image is previewed normally on the code scanning interface, it can reduce the number of recognition frame images to be recognized by the target application, so as to It reduces the number of threads created for recognizing frame images, the memory footprint of the recognition process, and the recognition time, thus improving recognition efficiency.
  • scanning the code for recognition of the recognition frame image includes: if the ambient light brightness of the recognition frame image is greater than or equal to threshold, and the exposure convergence operation and focus convergence operation of the camera are completed, then the recognition frame image is scanned and recognized.
  • the method also includes: if the ambient light brightness of the recognition frame image is greater than or equal to the threshold, and the exposure convergence operation or focus convergence operation of the camera is detected, then delete the camera according to the scan code during the execution of the exposure convergence operation or focus convergence operation.
  • the electronic device deletes the recognition frame image obtained by scanning the code image during the camera exposure convergence process or the focus convergence process.
  • the recognition frame image obtained during the camera exposure process or the convergence process is not scanned and recognized to reduce the risk.
  • the number of processing of invalid recognition frame images with poor brightness and poor clarity Specifically, when the camera is in the automatic exposure convergence state, the collected scanned code images will be underexposed or overexposed, resulting in poor brightness of the recognition frame image obtained from the scanned code images, thus causing difficulty in scanning code recognition. . If an electronic device scans and recognizes a recognition frame image with poor brightness, it will not only waste the computing power resources of the device, but also reduce the scanning speed.
  • recognition frame images with poor brightness are discarded, which improves recognition efficiency.
  • the camera when the camera is in the autofocus convergence state, the camera automatically adjusts the focus, and the focus does not reach the target value during the adjustment process.
  • the clarity of the scanned code image collected during the focus adjustment process is not good, so the recognition obtained by the scanned code image
  • the clarity of the frame image is also poor, making scanning and recognition difficult. Based on this, the electronic device discards identification frame images with poor definition, thereby improving identification efficiency.
  • scanning the code for recognition of the recognition frame image includes: if the ambient light brightness of the recognition frame image is greater than or equal to The threshold, the identification frame image includes the identification code and the size of the identification code is increased, and the identification frame image corresponding to the increased size identification code is scanned and recognized.
  • the electronic device only scans and recognizes the identification frame image when the identification code is included in the identification frame image and the size of the identification code is increased. In this way, the number of recognition frame images can be reduced to improve recognition efficiency.
  • scanning the code for recognition of the recognition frame image includes: if the ambient light brightness of the recognition frame image is greater than or equal to threshold, and the image amplification operation performed on the code scanning interface is detected, the code scanning recognition is performed on the recognition frame image obtained based on the code scanning image after the image amplification operation is performed.
  • the electronic device detects an image magnification operation, where the image magnification operation indicates that the user adjusts the display size of the identification code. Then the size of the adjusted identification code is larger and the image quality of the identification frame image is better. Therefore, after identifying the recognition frame image, the probability of successful recognition is greater and the recognition efficiency is higher.
  • the method before performing code scanning recognition based on the recognition frame image obtained by scanning the code image after performing the image magnification operation, the method further includes: determining that the recognition frame image includes the recognition code.
  • the method before displaying the code scanning interface, further includes: starting the target application. If the ambient light brightness of the recognition frame image is greater than or equal to the threshold, the recognition frame image will be processed. Scanning code recognition includes: if the ambient light brightness of the recognition frame image is greater than or equal to the threshold and the application logo of the target application is a preset logo, then scanning the code recognition frame image is performed.
  • the preset identifier can be set in advance to distinguish target applications that do not require interval frame loss.
  • the application identifier is the preset identifier, it means that the target application only recognizes the latest recognition frame image after each thread recognition is completed, which is equivalent to reducing the number of recognition frame images that the application needs to recognize.
  • Targets carrying the preset identifier In application, the electronic device does not perform the process of deleting recognition frame images to avoid the problem of slowing down the scanning speed caused by deleting too many recognition frame images.
  • scanning the code for recognition of the recognition frame image includes: if the ambient light brightness of the recognition frame image is greater than or equal to If the threshold value and the recognition frame image do not include the identification code, and if the drag operation on the preview frame image is detected on the code scanning interface, then the recognition frame image obtained by processing the same scan code image as the preview frame image is scanned and recognized.
  • the recognition frame image is not deleted, that is, the recognition frame image after the drag operation is performed is scanned and recognized.
  • the user can adjust the shooting angle of the camera or enlarge the image on the code scanning interface to realize that the identification code is included in the code scanning interface.
  • users can also adjust the position of the identification code on the code scanning interface by dragging the image as described in this design method.
  • the target application detects the operation of dragging the image, it means that the code scanning interface after the operation is more likely to include the identification code. Therefore, the continuous identification frame images collected by the camera after the operation are identified to avoid missed identification.
  • the frame image causes the scanning time to be too long, thus improving the scanning efficiency.
  • the method further includes: obtaining a metadata frame, where the metadata frame includes collection parameters of the scanned code image used to process the identification frame image. Obtain the light intensity parameter from the collection parameters, and determine the ambient light brightness of the recognition frame image based on the light intensity parameter.
  • the ambient light brightness and light intensity parameters are represented, and the light intensity parameters include: Luxindex.
  • the ambient light brightness is the reciprocal of Luxindex, that is, 1/Luxindex.
  • the larger the value of Luxindex the smaller the ambient light brightness, and the darker the light of the code scanning scene.
  • the smaller the value of Luxindex the greater the ambient light brightness, and the brighter the light of the code scanning scene.
  • the electronic device can obtain the light intensity parameter from the metadata frame, thereby determining the ambient light brightness of the recognition frame image and preview frame image processed by the scan code image relative to the metadata frame, and based on the ambient light brightness Determine the quality of the lighting environment. When the lighting environment is poor, code scanning and recognition will not be performed; when the lighting environment is good, code scanning and recognition will be performed. This reduces the recognition of invalid recognition frame images obtained under poor lighting conditions and improves code scanning efficiency.
  • the method further includes: detecting the exposure of the camera if the ambient light brightness of the recognition frame image is less than the threshold, or detecting the exposure of the camera when the ambient light brightness of the recognition frame image is greater than or equal to the threshold. Convergence operation or focus convergence operation, or when the ambient light brightness of the recognition frame image is greater than or equal to the threshold, the size of the identification code in the recognition frame image is less than or equal to the preset size, or when the ambient light brightness of the recognition frame image is greater than or equal to If the recognition frame image does not include the identification code under the threshold value, the frame drop flag is set for the recognition frame image.
  • the recognition frame image is scanned and recognized, including: scanning the code for recognition of the recognition frame image that does not contain the frame drop mark, wherein the recognition frame image does not contain the frame drop mark.
  • the ambient light brightness of the frame image is greater than or equal to the threshold.
  • the electronic device sets a frame drop flag for the recognition frame image that needs to be deleted to indicate that the recognition frame image is not used for code scanning recognition. In this way, the electronic device can scan the identification frame image that does not contain the frame loss flag. Processing to reduce the number of recognition frames in the image.
  • setting the frame drop flag for the recognition frame image includes: setting the frame drop flag for at least one recognition frame image in each preset frame drop period.
  • the electronic device uses an interval frame dropping strategy to set a frame dropping flag for the recognition frame image. That is, taking T as a frame dropping period, N frames of identification frame images are discarded in each frame dropping period. Then according to the frame loss strategy, the electronic device will set the frame loss flag for N frame recognition frame images within the frame loss period T.
  • the electronic device may set a frame drop flag for all identified frame images that meet the above frame drop conditions.
  • the electronic device may set a frame drop flag for the first M frame recognition frame images that meet the above frame drop condition.
  • electronic devices can selectively delete invalid recognition frame images to achieve the beneficial effect of fast code scanning.
  • the present application provides an electronic device, which includes: a memory and one or more processors; and the memory is coupled to the processor.
  • the memory is used to store computer program code, and the computer program code includes computer instructions; when the computer instructions are executed by the processor, the electronic device is caused to execute the method described in the first aspect and any possible design manner thereof.
  • the present application provides a computer-readable storage medium.
  • the computer-readable storage medium may include computer instructions.
  • the electronic device causes the electronic device to execute the first aspect and its possible design methods. Methods.
  • the present application provides a computer program product.
  • the computer program product When the computer program product is run on a computer, it causes the computer to execute the method of the first aspect and its possible design methods.
  • the electronic device of the second aspect, the computer storage medium of the third aspect, and the computer program product of the fourth aspect provided above are all used to execute the corresponding methods provided above. Therefore, what they can achieve For beneficial effects, please refer to the beneficial effects in the corresponding methods provided above and will not be described again here.
  • Figure 1 is a schematic diagram of the code scanning recognition principles for two types of applications provided by the embodiment of the present application
  • Figure 2 is a schematic diagram of the hardware structure of an electronic device provided by an embodiment of the present application.
  • Figure 3 is a software structure block diagram of an electronic device provided by an embodiment of the present application.
  • Figure 4 is a schematic diagram of a set of code scanning scenarios provided by the embodiment of this application.
  • Figure 5 is a schematic diagram of another set of code scanning scenarios provided by the embodiment of the present application.
  • Figure 6 is an interaction diagram of a code scanning method provided by an embodiment of the present application.
  • FIG. 6A is a schematic diagram of an operation of opening a camera according to an embodiment of the present application.
  • Figure 6B is a schematic diagram of another operation of opening the camera provided by the embodiment of the present application.
  • Figure 7A is an interaction diagram of yet another code scanning method provided by an embodiment of the present application.
  • Figure 7B is a schematic diagram of the effect corresponding to yet another code scanning method provided by the embodiment of the present application.
  • Figure 8 is a schematic diagram of the corresponding effects of another code scanning method provided by the embodiment of the present application.
  • Figure 9 is an interaction diagram of yet another code scanning method provided by an embodiment of the present application.
  • Figure 10 is a schematic diagram of the effect corresponding to another code scanning method provided by the embodiment of the present application.
  • Figure 11 is an interaction diagram of another code scanning method provided by the embodiment of the present application.
  • Figure 12 is a schematic diagram of the effect corresponding to another code scanning method provided by the embodiment of the present application.
  • Figure 13 is a flow chart corresponding to a code scanning method provided by an embodiment of the present application.
  • first and second are used for descriptive purposes only and cannot be understood as indicating or implying relative importance or implicitly indicating the quantity of indicated technical features. Therefore, features defined as “first” and “second” may explicitly or implicitly include one or more of these features. In the description of this embodiment, unless otherwise specified, “plurality” means two or more.
  • the recognition speed will be slow when the equipment load pressure is high, the available memory capacity is small, the environment is dark, and the identification code picture is small; but when the equipment load pressure is low, the available memory capacity is large, and the light is bright And when the identification code picture is large, the recognition speed is fast.
  • the recognition speed is fast, the user can perceive that the scanning process takes a short time, so the user has a good scanning experience; once the device is in the above scenario where the recognition speed is slow, the user can perceive that the scanning process is The process will take longer, so the user’s QR code scanning experience will be poor.
  • the current code scanning technology cannot quickly identify the identification code when the device is in a scenario where the recognition speed is slow, so it brings difficulty in scanning the code to users.
  • code scanning speed is also affected by differences in the applied code scanning recognition principles.
  • the following describes the identification process of QR code scanning using two types of applications with different identification algorithms.
  • Figure 1 is a schematic diagram of the code scanning recognition principles for two types of applications provided by embodiments of the present application.
  • the first type of application recognizes each recognition frame (also called recognition frame image), so the recognition speed is slow.
  • the camera returns 30fps (Frames Per Second, the number of frames per second) to the abstraction layer, that is, after the camera is started, it will return every 33ms (millisecond, milliseconds) to return one frame of data, which includes a preview frame (also called a preview frame image, not shown in Figure 1) and a recognition frame.
  • the preview frame is used to display on the code scanning interface
  • the identification frame is used to obtain identification code information.
  • Recognition frames are stored in a cache in the abstraction layer so that they can be processed before they are rendered to the screen.
  • the first type of application creates a thread for each identification frame returned to the algorithm processing module 101 of the first type application to identify the information of the frame identification frame. Mutex locks are used between each thread to ensure the continuity between threads. Execute. As shown in (a) of Figure 1, when the first thread 102 recognizes the 0th frame recognition frame, although the 1st frame recognition frame has been returned to the algorithm processing module 101 of the first type of application, the second thread 103 does not recognize the 1st frame. The recognition of the recognition frame needs to wait until the first thread 102 completes the recognition of the 0th frame recognition frame.
  • the first thread 102 recognizes the 0th frame recognition frame
  • the first thread 102 occupies the mutex lock.
  • the second thread 103 is locked (also called a lock) and cannot perform the recognition operation. It is not until the first thread 102 has finished identifying the 0th frame identification frame that the second thread 103 occupies the mutex lock and starts to identify the 1st frame identification frame (also called a process).
  • the second type of application recognizes the latest recognition frame after the end of each thread recognition, so the recognition speed is fast.
  • the camera also returns a frame of data every 33ms.
  • the data includes a preview frame and a recognition frame.
  • the algorithm processing module 101 of the second type of application completes the identification of a frame, if the identification code is not recognized, a thread is created to capture the latest frame and return it to the identification frame of the algorithm processing module 101, and Identify the latest frame of identification.
  • the first type of application identifies each frame, so there are at least two problems in using the first type of application to scan codes.
  • a large number of threads are created, which takes up a large amount of memory space.
  • the computing resources allocated to the thread are insufficient, so the recognition speed is affected, which in turn affects the user's scanning speed.
  • the first type of application will identify the image frames collected under poor lighting environment and image quality. This not only fails to recognize the identification code, but also wastes thread resources and memory space.
  • the second type of application only recognizes part of the recognition frames, it also starts processing from the first recognition frame collected by the camera. In situations where the lighting environment and image quality are poor, the second type of applications will also perform a large number of invalid recognitions, resulting in a waste of equipment computing resources.
  • embodiments of the present application provide a code scanning method, which can be applied to electronic devices.
  • an electronic device scans a code in a scene with poor lighting environment and image quality, it will only recognize a part of the recognition frames collected by the camera in the above scene, thereby reducing the number of threads created for the recognition frame, the memory occupied by the recognition process, and the number of recognition frames. time, thus improving recognition efficiency.
  • the electronic device in the embodiment of the present application may be a mobile phone, a tablet computer, a desktop, a laptop, a handheld computer, a notebook computer, an ultra-mobile personal computer (UMPC), a netbook, or an augmented reality computer.
  • UMPC ultra-mobile personal computer
  • UMPC ultra-mobile personal computer
  • UMPC ultra-mobile personal computer
  • netbook or an augmented reality computer.
  • AR augmented reality
  • VR virtual reality
  • FIG. 2 is a schematic diagram of the hardware structure of an electronic device provided by an embodiment of the present application.
  • the mobile phone 200 may include a processor 210, an external memory interface 220, an internal memory 221, a universal serial bus (USB) interface 230, and a charging management module 240 , power management module 241, battery 242, antenna 1, antenna 2, mobile communication module 250, wireless communication module 260, audio module 270, speaker 270A, receiver 270B, microphone 270C, headphone interface 270D, sensor module 280, touch sensor 280A, Ambient light sensor 280B, button 290, motor 291, indicator 292, camera 293, display screen 294, and subscriber identification module (subscriber identification module, SIM) card interface 295, etc.
  • SIM subscriber identification module
  • the structure illustrated in this embodiment does not constitute a specific limitation on the mobile phone 200 .
  • the mobile phone 200 may include more or fewer components than shown, or some components may be combined, some components may be separated, or some components may be arranged differently.
  • the components illustrated may be implemented in hardware, software, or a combination of software and hardware.
  • the processor 210 may include one or more processing units.
  • the processor 210 may include an application processor (application processor, AP), a modem processor, a graphics processing unit (GPU), and an image signal processor. (image signal processor, ISP), controller, memory, video editor Decoder, digital signal processor (DSP), baseband processor, and/or neural network processing unit (NPU), etc.
  • application processor application processor, AP
  • modem processor graphics processing unit
  • GPU graphics processing unit
  • image signal processor image signal processor
  • ISP image signal processor
  • controller memory
  • DSP digital signal processor
  • NPU neural network processing unit
  • different processing units can be independent devices or integrated in one or more processors.
  • the controller may be the nerve center and command center of the mobile phone 200.
  • the controller can generate operation control signals based on the instruction operation code and timing signals to complete the control of fetching and executing instructions.
  • the processor 210 may also be provided with a memory for storing instructions and data.
  • the memory in processor 210 is cache memory. This memory may hold instructions or data that have been recently used or recycled by processor 210 . If the processor 210 needs to use the instructions or data again, it can be called directly from the memory. Repeated access is avoided and the waiting time of the processor 210 is reduced, thus improving the efficiency of the system.
  • the memory may store executable programs for identifying lighting environments and image magnification operations, as well as user data, application data, etc. generated by executing the above operations.
  • the wireless communication function of the mobile phone 200 can be realized through the antenna 1, the antenna 2, the mobile communication module 250, the wireless communication module 260, the modem processor and the baseband processor.
  • the mobile phone 200 can send a signal to start the target application, a signal to call the camera driver, etc. through the above-mentioned wireless communication module 260.
  • the mobile phone 200 can also perform corresponding operations based on the recognized identification code through the mobile communication module 250 or the wireless communication module 260, such as opening a web page or sending an unlocking instruction to a shared vehicle.
  • Touch sensor 280A is also called a "touch device”.
  • the touch sensor 280A can be disposed on the display screen 294.
  • the touch sensor 280A and the display screen 294 form a touch screen, which is also called a "touch screen”.
  • Touch sensor 280A is used to detect touch operations on or near it.
  • Touch sensor 280A may pass the detected touch operation to the application processor to determine the touch event type.
  • Visual output related to the touch operation may be provided through display screen 294.
  • the touch sensor 280A may be disposed on the surface of the mobile phone 200 at a different location than the display screen 294 .
  • the mobile phone 200 can detect the user's touch operation on the icon of the target application through the touch sensor 280A, and in response to the touch operation, the mobile phone 200 starts the target application.
  • the mobile phone 200 can detect through the touch sensor 280A that the user performs an image magnification operation on the code scanning preview interface.
  • the mobile phone 200 ends the operation of identifying partial recognition frames and begins to recognize self-images frame by frame. All recognition frames collected by the camera 293 after performing the image magnification operation.
  • the ambient light sensor 280B is used to sense ambient light brightness (also called lighting environment).
  • the mobile phone 200 determines whether to perform identification code recognition processing on the currently collected identification frame according to the ambient light brightness. Specifically, if the ambient light brightness is less than the threshold, it means that the lighting environment is dark, and the mobile phone 200 does not process the recognition frames collected under the ambient light brightness; if the ambient light brightness is greater than or equal to the threshold, it means that the lighting environment is bright, and the mobile phone 200 does not process the recognition frames collected under the ambient light brightness.
  • the recognition frames collected under light brightness are processed. In this way, the mobile phone 200 can effectively avoid the problem of wasting device computing resources caused by processing invalid recognition frames in scenes with poor lighting conditions, thereby improving recognition efficiency.
  • the mobile phone 200 implements display functions through the GPU, the display screen 294, and the application processor.
  • the GPU is an image processing microprocessor and is connected to the display screen 294 and the application processor. GPUs are used to perform mathematical and geometric calculations for graphics rendering.
  • Processor 210 may include one or more GPUs that execute program instructions to generate or alter display information.
  • the display screen 294 is used to display images, videos, etc.
  • Display 294 includes a display panel.
  • the display panel can be With liquid crystal display (LCD), organic light-emitting diode (OLED), active matrix organic light emitting diode or active matrix organic light emitting diode (active-matrix organic light emitting diode), AMOLED), flexible light-emitting diodes (FLED), Miniled, MicroLed, Micro-oLed, quantum dot light emitting diodes (QLED), etc.
  • LCD liquid crystal display
  • OLED organic light-emitting diode
  • active matrix organic light emitting diode or active matrix organic light emitting diode active-matrix organic light emitting diode
  • AMOLED active-matrix organic light emitting diode
  • FLED flexible light-emitting diodes
  • Miniled MicroLed, Micro-oLed, quantum dot light emitting diodes (QLED), etc.
  • the mobile phone 200 can realize the shooting function through the ISP, camera 293, video codec, GPU, display screen 294 and APP processor.
  • the ISP is used to process the data fed back by the camera 293. For example, when taking a photo, the shutter is opened, the light is transmitted through the lens to the camera sensor, the optical signal is converted into an electrical signal, the camera sensor passes the electrical signal to the ISP for processing, and the electrical signal is converted into an image visible to the naked eye, in which the image can be It may be an image including an identification code or an image not including an identification code.
  • ISP can also perform algorithm optimization on image noise, brightness, and skin color. ISP can also optimize the exposure, color temperature and other parameters of the shooting scene.
  • the ISP may be provided in the camera 293.
  • Camera 293 is used to capture still images or video.
  • the object passes through the lens to produce an optical image that is projected onto the photosensitive element.
  • the photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor.
  • CMOS complementary metal-oxide-semiconductor
  • the photosensitive element converts the optical signal into an electrical signal, and then passes the electrical signal to the ISP to convert it into a digital image signal.
  • ISP outputs digital image signals to DSP for processing.
  • DSP converts digital image signals into standard RGB, YUV and other format image signals.
  • the mobile phone 200 may include 1 or N cameras 293, where N is a positive integer greater than 1.
  • the mobile phone 200 can obtain preview frames, recognition frames, and metadata frames through the camera 293.
  • the camera collects scanned code images, and the scanned code images are also called raw data.
  • the ISP will Raw data is converted into YUV data (also known as brightness, color, and saturation maps), and according to the needs of the target application, one channel of YUV data is output as a preview stream and a preview frame, and the metadata frame of each frame of YUV data is output at the same time. , used to describe the light intensity, automatic focus parameters, automatic exposure parameters, automatic white balance parameters, etc. of YUV data.
  • Figure 3 is a software structure block diagram of an electronic device provided by an embodiment of the present application.
  • Software systems of electronic devices can adopt layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture.
  • the embodiment of this application takes the Android system with a layered architecture as an example to illustrate the software structure of the mobile phone 200 .
  • the layered architecture can divide the software into several layers, and each layer has clear roles and division of labor.
  • the layers communicate through software interfaces.
  • the Android system is divided into four layers. From top to bottom, they are the application layer (referred to as the application layer) 310, the application framework layer (referred to as the framework layer) 320, and the hardware abstract layer (HAL). ) layer 330, and the kernel layer (Kernel, also called the driver layer).
  • the kernel layer is used to provide simple access hardware logic, and functional actions that embody the hardware access logic (such as the frame skipping identification logic detailed in the method embodiment below) are placed in the HAL layer , and the HAL layer can directly communicate with the camera of the hardware layer without going through the kernel layer, so the kernel layer is not shown in Figure 3.
  • the application layer (Application) 310 may include a series of application packages.
  • the multiple application packages can be used for camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, short message And desktop launcher (Launcher) and other applications.
  • the application layer 310 may include a camera system application (also referred to as a camera application) 311.
  • the operation of the camera application 311, for example, taking pictures, recording videos, etc., requires the use of a camera.
  • the application layer 310 may also include other applications (also called third-party applications) 312 that require the use of cameras.
  • applications also called third-party applications
  • the QR code scanning function of WeChat TM the QR code scanning function of Alipay TM
  • the QR code scanning function of Douyin TM all require the use of cameras
  • the other third-party applications 312 can be WeChat TM , Alipay TM , Douyin TM and other applications.
  • the camera application 311 and the third-party application 312 can be used to display the preview image (also called a preview frame) reported by the underlying layer in the preview interface.
  • the mobile phone 200 may include multiple cameras, and each camera may be used to collect preview frames.
  • the cameras appearing below are understood to be the cameras currently collecting preview frames.
  • the application layer 310 can instruct the HAL layer 330 to start the camera according to the user's operation.
  • the application layer 310 can also instruct the HAL layer 330 to perform an amplification operation on the image collected by the camera according to the user's operation, or instruct the HAL layer 330 to start/ Turn off the flashlight function.
  • the framework layer (Framework) 320 provides an application programming interface (API) and programming framework for the application layer 310.
  • the framework layer 320 includes some predefined functions. As shown in Figure 3, the framework layer 320 can provide camera APIs, such as Camera API (API 1
  • API 2) 321 serves as the interface for the bottom layer (such as HAL layer 330) to interact with the application layer 310.
  • API 2) 321 can receive a request from the upper layer (such as the application layer 310) to open the specified camera, and process the request through the camera service 322.
  • the camera service 322 turns on the corresponding camera through the HAL layer 330, and obtains the preview frame, identification frame, metadata frame, etc. reported by the camera through the HAL layer 330.
  • the camera service 322 may be used to obtain a policy for discarding identification frames based on requests from the application layer 310 and/or based on identification frames reported from lower layers (such as the HAL layer 330).
  • the strategy of discarding recognition frames includes: the camera service 322 receives the recognition frames reported by the HAL layer 330, in which the status of discarding (can be understood as deleting) the recognition frames that are not recognized is error (error refers to exceptions that the application does not need to handle), and the camera The service 322 determines the status of the recognition frame.
  • the camera service 322 does not return the recognition frame to the application; if the status of the recognition frame is not error, if the status of the recognition frame is not set, the camera service 322 322 returns the recognition frame to the application layer 310 for recognition processing.
  • the HAL layer 330 is used to connect the framework layer 320 and the kernel layer.
  • the HAL layer 330 can perform data transmission between the framework layer 320 and the kernel layer.
  • the HAL layer 330 can also process data from the bottom layer (ie, the kernel layer), and then transmit it to the framework layer 320.
  • the HAL layer 330 can process the preview frames, recognition frames, and metadata frames reported by the camera of the hardware layer, and return the recognition frames and preview frames to the camera service 322.
  • the HAL layer 330 may include a HAL interface definition language (HAL interface definition language, HIDL) interface 331, a camera hardware abstraction layer (Camera Provider) 332 and a HAL interface.
  • HAL interface definition language HAL interface definition language
  • HIDL HAL interface definition language
  • Camera hardware abstraction layer Camera Provider
  • HAL interface a standard HIDL interface for calls to the upper layer (such as the camera service 322).
  • the camera hardware abstraction layer 332 controls the camera downward through a standard HAL interface, such as HAL3.0, and obtains preview frames, recognition frames, metadata frames and other data reported by the camera.
  • the camera hardware abstraction layer 332 can report the obtained preview frame and the identification frame of the set status to the upper layer (such as the camera service 322).
  • the camera hardware abstraction layer 332 includes a camera driver. Different from the camera driver in the kernel layer, the camera driver in the camera hardware abstraction layer 332 is used to set the status of the recognition frame reported by the camera according to the lighting environment and image quality. determined, and report the set state recognition frame together with the preview frame to the camera service 322.
  • the code scanning method provided by the embodiment of the present application can be implemented in the mobile phone 200 having the above hardware structure and software structure.
  • the following is an example of the workflow of the mobile phone 200 software and hardware based on the QR code scanning scenario.
  • a corresponding hardware interrupt is sent to the kernel layer.
  • the kernel layer processes touch operations into raw input events (including touch coordinates, timestamps of touch operations, and other information).
  • Raw input events are stored in the kernel layer.
  • the framework layer 320 obtains the original input event from the kernel layer and identifies the control corresponding to the input event. Take the touch operation as a touch click operation, and the control corresponding to the click operation is a control that implements the code scanning function on the interface of the third-party application 312.
  • the third-party application 312 calls the camera service 322 of the framework layer 320, and then uses the camera service 322 calls the camera driver, starts the camera 293 through the camera driver to collect the scanned code image, and processes the scanned code route into a preview frame and a recognition frame. Then the camera 293 reports data such as preview frames, recognition frames, and metadata frames to the camera driver of the HAL layer 330 .
  • the camera driver sets a frame drop flag (also called status error, error or status) for the recognition frame reported by the camera 293 according to the lighting environment or image quality, and reports the recognition frame with the frame drop flag set to the camera service 322 together with the preview frame.
  • a frame drop flag also called status error, error or status
  • the camera service 322 receives the recognition frame reported by the HAL layer 330, determines whether to return the recognition frame to the third-party application 312 according to whether the recognition frame has a frame loss flag, and returns a preview frame at the same time.
  • the third-party application 312 displays the preview frame in the preview frame and processes the information in the identification frame at the same time. If the identification code is recognized, the third-party application 312 issues an instruction to close the camera, the camera stops collecting data, and the third-party application 312 instructs the mobile phone 200 according to The recognized identification code performs corresponding operations, such as opening a web page or sending an unlocking instruction to a shared vehicle.
  • Figure 4 shows a schematic diagram of a set of code scanning scenarios provided by the embodiment of the present application.
  • the user performs a code scanning operation under poor lighting conditions.
  • the camera 293 reports the recognition frame, preview frame and metadata frame to the camera driver.
  • the camera driver determines the ambient light based on the metadata frame. If the lighting conditions are poor, the camera driver will set the status of the recognition frame to error and report it to the camera service together with the preview frame.
  • the camera service determines not to report the recognition frame to the third-party application based on the error status of the recognition frame, so the camera service only reports the preview frame to the third-party application.
  • Third-party applications display preview frames in the preview interface.
  • the third-party application Since the third-party application has not received the identification frame, it has not performed identification processing, so it will not perform invalid identification.
  • the user turns on the flashlight function of the mobile phone.
  • the brightness of the viewing environment increases, and the corresponding camera driver identifies the ambient light brightness as good lighting conditions based on the metadata frame.
  • the camera driver does not set an error for the recognition frame, but uploads the recognition frame and preview frame to the camera service together.
  • the camera service determines that the recognition frame needs to be reported to the third-party application based on the fact that error is not set in the recognition frame. Therefore, the camera service reports the recognition frame and preview frame to the third-party application together.
  • the third-party application displays the preview frame on the preview interface and performs recognition processing on the identification frame until the identification code is recognized.
  • the third-party application only recognizes the recognition frames collected in the scene with good lighting conditions, thus greatly reducing the number of invalid recognition frames that need to be recognized (that is, recognition frames with poor lighting conditions), thus reducing the number of recognition frames to recognition frames.
  • the number of threads created, the memory footprint of the recognition process and the recognition time are therefore improved, thus improving recognition efficiency.
  • FIG. 5 shows another set of schematic diagrams of code scanning scenarios provided by the embodiment of the present application.
  • (a) of Figure 5 use The user performs a code scanning operation, but the identification code displayed in the preview interface is small, which means that the size of the identification code in the image collected by the camera is too small and the image quality is poor. Therefore, the camera driver marks the status of the recognition frame by not marking the recognition frame with an interval of one recognition frame, and marking the error status of two consecutive recognition frames.
  • the camera service adopts a frame dropping strategy of discarding two recognition frames and reporting one recognition frame to the third-party application.
  • the user performs an image magnification operation, and the third-party application sends an image magnification request to the camera driver.
  • the camera driver stops marking the status of the recognition frame and directly uploads the recognition frame and preview frame to the camera service. And returned to the third-party application by the camera service.
  • the third-party application performs recognition processing on consecutive recognition frames.
  • the third-party application first performs frame skipping recognition on the recognition frame to avoid identifying recognition frames with poor image quality and wasting computing resources.
  • the camera driver After detecting that the user performs an image magnification operation, the camera driver confirms that the image quality has improved (the identification code in the image has changed from a small size to a large size), and then stops marking the status of the identification frame.
  • Third-party applications perform continuous frame recognition on preview frames.
  • the third-party application only continuously recognizes the recognition frames collected in scenes with good image quality, thus greatly reducing the number of invalid recognition frames (that is, recognition frames with poor image quality) that need to be recognized, thus reducing them to recognition frames.
  • the number of threads created, the memory footprint of the recognition process and the recognition time are therefore improved, thus improving recognition efficiency.
  • the target application In response to the user's operation of opening the camera, the target application displays a code scanning interface and sends a request to start the camera to the camera service.
  • the target application can receive the user's operation of opening the camera.
  • the target application can be a camera application or a third-party application.
  • the operation of opening the camera includes an operation of triggering the mobile phone to run the camera application in the background.
  • the operation of opening the camera is the QR code scanning interface 603a that the user enters after clicking on the icon 602a in the main interface 601a of the mobile phone shown in (a) of FIG. 6A.
  • the operation of turning on the camera includes triggering the phone to use the camera to start the code scanning function when a third-party application is running in the background.
  • the function of using the camera can be an operation of clicking the "Scan 602b" icon in the main interface 601b of the third-party application shown in (a) in Figure 6B.
  • the mobile phone detects this operation, starts the code scanning function of the third-party application, and displays the code scanning interface 603b shown in (b) in Figure 6B.
  • the operation of turning on the camera includes a switching operation of the shooting mode.
  • the functions of the camera used in the camera application or third-party applications can provide a variety of shooting modes, such as shooting, video recording, panorama, code scanning, object recognition, etc.
  • shooting modes such as shooting, video recording, panorama, code scanning, object recognition, etc.
  • select the operation of scanning the code 604a icon on the main interface of the camera application shown in (b) in Figure 6A, select the operation of scanning the code 604a icon, and the camera application can Switch to scanning mode.
  • the icon 602a is the application icon of the camera application
  • the scan code 604a is the function icon of the code scanning function in the camera application.
  • the application can call the camera interface (such as the open Camera interface) to request to start the camera, that is, send a request to start the camera to the camera service.
  • the request to start the camera carries functional parameters for indicating the camera type and shooting mode, so that the bottom layer can be clearly instructed to start the corresponding camera to meet the requirements of the corresponding function.
  • the request to start the camera also includes an application identifier, which is used by the camera driver to determine whether the application that starts the camera is a preset target application.
  • the camera service In response to receiving the request to start the camera, the camera service sends a request to call the camera driver to the camera driver.
  • the camera service may receive a request from the target application to start the camera.
  • the camera service after the camera service receives the request to start the camera, it sends a message calling the camera driver to the underlying camera driver to instruct the camera driver to drive the camera to start.
  • the camera driver In response to receiving a request to call the camera driver, the camera driver starts the camera and obtains the metadata frame, recognition frame and preview frame.
  • the camera driver may receive a request from the camera service to call the camera driver.
  • the camera driver can control the camera to turn on. After the camera is turned on, it can continuously collect scanned code images, process the scanned code images into recognition frames and preview frames, and output metadata frames.
  • the preview image in the code scanning interface can be realized by reporting the preview frame to the upper layer through the camera; the code scanning recognition result (also called the recognition result) can be realized by recognizing the identification code.
  • the camera reports the metadata frame, recognition frame and preview frame to the camera driver.
  • the camera reports to the camera driver is the metadata stream, recognition stream and preview stream.
  • This step uses a frame of metadata, recognition image and preview image collected by the camera at a certain moment as an example. Then the collection The received frame of metadata is the metadata frame, the frame of the recognition image is the recognition frame, and the frame of the preview image is the preview frame.
  • the metadata frame is used to describe the values of the image collected by the camera at the above-mentioned moment.
  • the metadata frame includes frame rate, image size, encoding, color bit depth, ambient light brightness, etc.
  • Both the recognition frame and the preview frame are image data (equivalent to the aforementioned YUV data).
  • the image data can be displayed on the display screen of the mobile phone, or can be used for recognition processing to obtain the identification code in the image data.
  • the value of the image data can be obtained from the metadata frame that describes the camera collection parameters under the image data collection timestamp.
  • the mobile phone can determine the lighting environment when the camera collects preview frames and recognition frames based on the light intensity in the metadata frame. (Also known as ambient light brightness).
  • the metadata frame, recognition frame and preview frame are cached in the memory block so that the recognition frame or preview frame can be processed before being rendered to the screen. Therefore, metadata frames, recognition frames, and preview frames will occupy memory space, which will affect the recognition speed when the available memory capacity is small.
  • S601 to S604 can be processed according to the process of turning on the camera to collect raw data and reporting the preview image, recognition image and metadata in the existing technology, and no redundant explanation will be given here.
  • the camera driver identifies the ambient light brightness based on the metadata frame.
  • the ambient light brightness and illumination intensity parameters are represented, and the illumination intensity parameters include: Luxindex (also called illumination intensity, referred to as illuminance).
  • Luxindex also called illumination intensity, referred to as illuminance
  • the ambient light brightness is the reciprocal of Luxindex, that is, 1/Luxindex.
  • the larger the value of Luxindex the smaller the ambient light brightness, and the darker the light of the code scanning scene.
  • the smaller the value of Luxindex the greater the ambient light brightness, and the brighter the light of the code scanning scene.
  • the camera driver can obtain the light intensity from the metadata frame to determine the ambient light brightness of the recognition frame and preview frame processed by the scanned image relative to the metadata frame, where the ambient light brightness is used to determine the lighting environment. bad.
  • the camera driver determines that the ambient light brightness of the metadata frame is less than the threshold, and sets a frame loss flag for the recognition frame collected at the same timestamp as the metadata frame.
  • the camera driver sets a frame drop flag for the recognition frame collected at the same timestamp as the metadata frame to indicate the relative status of the upper layer.
  • the host service discards the identification frame.
  • the recognition frames discarded by the camera service will not be recognized by the upper-layer target application. Therefore, the upper-layer target application does not need to allocate threads for invalid frames to identify the QR codes in invalid frames. In this way, it can avoid device calculation caused by a large number of invalid recognitions. Waste of human resources.
  • the frame loss flag may be the status of the identification frame. For example, if the status is error, it means that the frame identification frame is an invalid identification frame that needs to be discarded. If the status is not error, for example, the status is NULL or the status is not set, it means that the frame recognition frame is a recognition frame that needs to be recognized. It should be understood that the advantage of the camera driver setting the frame drop flag for invalid recognition frames is that the upper-layer camera service can determine whether to upload the recognition frame to the target application based on the frame drop flag. Therefore, in some embodiments, the identification frame flag may also be a pair of flags representing different meanings, such as (Y/N), (1/0), etc., where the frame loss flag may be N, or 0, then no The flag of frame loss is Y,1.
  • the camera driver reports the recognition frame and preview frame to the camera service.
  • the reported identification frames include identification frames with a frame loss flag and identification frames without a frame loss flag. That is to say, the recognition frame obtained by the camera service is an image obtained by the camera driver based on ambient light analysis and processing. Therefore, after receiving the recognition frame, the camera service can detect whether the recognition frame is marked, that is, perform S608.
  • the camera service detects whether the recognition frame is marked, and returns the preview frame and the recognition frame that does not contain the mark to the target application.
  • the preview frame is used to display on the code scanning interface, and the identification frame is used to obtain identification code information. That is to say, the user is aware of whether the preview frame has dropped frames, but whether the identification frame has dropped frames will not affect the user's scanning experience. Therefore, when scanning codes in a dark environment, in this embodiment, the number of preview frames uploaded by the camera service to the target application is consistent with the number of preview frames reported by the camera, and the number of uploaded recognition frames is smaller than the number of recognition frames reported by the camera.
  • the number of recognition frames to be recognized by the target application can be reduced, thereby reducing the number of threads created for recognition frames, the memory occupied by the recognition process, and the recognition time. Therefore, Improved recognition efficiency.
  • the camera service detecting whether the recognition frame is marked includes: the camera service detects whether the status of the recognition frame is error. If the status of the recognition frame is error, it means that the recognition frame is marked. Then the identification frame without error is returned to the target application.
  • the camera service detecting whether the recognition frame is marked includes: the camera service detects whether the recognition frame is marked "N (or 0)". If the recognition frame is marked "N (or 0)", it means the recognition frame It is marked. If the recognition frame is not marked “N (or 0)", if the recognition frame is marked "Y (or 1)", it means that the recognition frame is not marked. Then the camera service will return the identification frame that does not contain the tag "N (or 0)" to the target application.
  • the target application After the camera service returns the preview frame and the recognition frame without the mark, the target application receives the preview frame and the recognition frame without the mark. Then the target application executes S609 to realize the recognition of the QR code.
  • the target application displays the preview frame on the code scanning interface, identifies the recognition frame, and determines the recognition result.
  • the target application continues to receive the preview frame of the next frame and the identification frame of the next frame that does not contain the mark, and performs the identification frame on the next frame. Recognize to determine whether the QR code is recognized.
  • the target application recognizes the QR code from the recognition frame, the target application sends a request to turn off the camera to the camera service.
  • the target application calls a camera interface (such as the close Camera interface) to request to close the camera, and sends a request to close the camera to the camera service.
  • the camera service After the camera service receives the request to turn off the camera, it transmits the request to turn off the camera to the lower layer, turns off the corresponding camera, and finally stops in the scanning area. Preview in the page.
  • the camera driver can set the frame drop flag on the recognition frames collected in the dark environment, thereby discarding the recognition frames in the camera service.
  • the camera service The recognition frames reported to the target application do not include invalid recognition frames recognized in dark environments. In this way, the problem of wasting device computing resources and reducing the code scanning speed due to the target application's recognition of invalid frames can be avoided, thus improving the code scanning efficiency.
  • the embodiment of the present application also provides another code scanning method, which can effectively reduce the identification of invalid frames by the target application and improve the code scanning speed based on the embodiment shown in FIG. 6 .
  • the code scanning method provided by the embodiment of the present application may include:
  • the camera driver determines that the ambient light brightness of the metadata frame is greater than or equal to the threshold, and obtains the first convergence result of automatic exposure and the second convergence result of automatic focus from the metadata frame.
  • the camera uses automatic exposure to make the brightness of the photographed object normal.
  • the camera's aperture, shutter, sensitivity (international standards organization, ISO) and other parameters are gradually adjusted. That is to say, there is a certain adjustment time for the camera's aperture, shutter, ISO and other parameters, which is called adjustment. process.
  • the camera's automatic exposure is in a convergence state.
  • the camera's aperture, shutter, ISO and other parameters remain unchanged, so the automatic exposure is in a convergence end state.
  • the camera uses autofocus to make the object captured normal sharpness.
  • the camera if the brightness of the object changes from dark to bright, the camera will use automatic focus to change the image captured by the camera from blur to clear.
  • the camera adjusts the focus, the focus has not yet reached the target value, so the autofocus is in a convergent state.
  • the focus reaches the target value, so the autofocus is in the convergence end state.
  • the metadata frame is also used to provide camera parameters (also called camera parameters), so that the camera driver determines the first convergence result of automatic exposure and the second convergence result of automatic focus.
  • camera parameters also called camera parameters
  • the camera parameters include: AEdone, AFdone.
  • AE is also called automatic exposure (auto exposure)
  • AEdone means that the automatic exposure is in the convergence end state.
  • AF is also called auto focus.
  • AFdone means that the auto focus is in the convergence end state. That is, in this embodiment, the camera driver determines that the ambient light brightness of the metadata frame is greater than or equal to the threshold, and obtains AEdone and AFdone from the metadata frame. If AEdone is obtained, it means that the automatic exposure is in the convergence end state. If AFdone is obtained, it means that the autofocus is in the focusing end state. Therefore, the camera driver can determine the first convergence result of automatic exposure and the second convergence result of automatic focus based on whether AEdone and AFdone are recognized.
  • the camera parameters include aperture, shutter, ISO, focal length and other parameters of the camera.
  • the camera driver obtains the camera parameters of at least two metadata frames, and determines whether the automatic exposure or automatic focus is in a convergence state by determining whether the camera parameters change, that is, determining the first convergence result of the automatic exposure and the second convergence result of the automatic focus.
  • the camera driver determines that the automatic exposure and/or the automatic focus are in a converged state based on the first convergence result and the second convergence result, and sets a frame drop flag for the identification frame collected at the same timestamp as the metadata frame according to the frame drop policy.
  • the ambient light brightness value is greater than or equal to the threshold, but the target application still cannot recognize the QR code from the recognition frame. Therefore, identifying the recognition frames collected during the process of automatic exposure and automatic focus will not only fail to recognize the QR code from the invalid frames, but also occupy thread resources. Based on this, the recognition frames collected during the automatic exposure and autofocus processes are discarded in order to reduce the number of threads created for the recognition frames, the memory occupied by the recognition process, and the recognition time, thus improving the recognition efficiency.
  • the frame dropping strategy includes: discarding all recognition frames collected when the auto-exposure and/or auto-focus are in a converged state.
  • the camera driver sets the frame loss flag for the recognition frame where the automatic exposure and/or autofocus are in a converged state.
  • the frame loss strategy includes: taking T as a frame loss period, discarding N frames of identification frames in each frame loss period.
  • the camera driver sets the frame loss flag for N frame identification frames within the frame loss period T.
  • T is 3 frames and N is 2, that is, two recognition frames are discarded out of every three recognition frames, and only one recognition frame is retained for target application recognition.
  • the frame dropping strategy includes: recognizing frames every other frame, discarding two consecutive recognition frames, and repeating this cycle.
  • the frame dropping strategy includes: identifying the first M frames collected when automatic exposure and/or automatic focus are in a converged state.
  • M is 3, 6, 10, 15, 20, etc.
  • the camera driver sets the frame loss flag for the first M frame recognition frames when the automatic exposure and/or autofocus are in the convergence state.
  • the camera driver determines that both the automatic exposure and the automatic focus are in the convergence end state based on the first convergence result and the second convergence result, and ends setting the frame drop flag for the identified frame according to the frame drop policy.
  • the automatic exposure and automatic focus are both in the convergence end state.
  • the camera parameters in the metadata frame include AEdone and AFdone.
  • both auto-exposure and auto-focus are in the convergence end state, which means that the camera parameters in the metadata frame remain unchanged compared to the previous frame.
  • the camera driver may end the operation of setting the frame drop flag for the recognition frame according to the frame drop policy multiple times. Each completed operation can be called "the camera driver ends setting the frame drop flag for the identified frame according to the frame drop policy.”
  • the camera driver stops setting the frame loss flag for the recognition frame according to the frame loss strategy, so that the camera driver no longer sets the frame loss flag for all subsequent recognition frames received. Then the camera driver only performs the operation of stopping setting the frame loss flag for the recognition frame according to the frame loss strategy once.
  • the camera driver after the camera driver ends setting the frame drop flag for the identified frame according to the frame drop policy, the camera The driver no longer sets the frame drop flag for subsequent received recognition frames. Until the recognition frame meets the preset conditions, the camera driver continues to set the frame loss flag for subsequent received recognition frames.
  • the preset conditions include: Luxindex is less than the threshold. For example, when the user scans the code in a dark environment, when the user activates the flashlight function of the mobile phone and the Luxindex is greater than or equal to the threshold, the camera automatically exposes and auto-focuses. During the auto-exposure and auto-focus process, the camera driver uses the frame loss strategy to identify Frame sets the frame drop flag.
  • the camera driver ends setting the frame drop flag for the identified frame according to the frame drop policy.
  • the user accidentally touches the mobile phone, causing the flashlight function to be turned off, and the Luxindex rises to greater than the preset value.
  • the camera driver continues to set the frame loss flag for the recognition frame corresponding to the Luxindex greater than the preset value. That is to say, in this scenario, the frame dropping process executed by the camera driver is: first discard all recognition frames with a Luxindex greater than the preset value; and then discard two consecutive recognition frames according to the frame dropping strategy (such as recognizing frames every other frame). ) discards the recognition frame; then after the automatic exposure and autofocus are completed, the frame dropping ends; finally, when the Luxindex rises to greater than the preset value, continue to discard all recognition frames with a Luxindex greater than the preset value.
  • the camera driver After the camera driver sets the frame drop flag for the recognition frame, it uploads the recognition frame and preview frame to the camera service so that the camera service can perform the frame drop operation based on the frame drop flag.
  • the frame dropping operation performed by the camera service please refer to the previous description of S607. Based on this, after the mobile phone executes S703, it executes the above S607.
  • the camera driver can not only set the frame drop flag for the recognition frame corresponding to the ambient light brightness less than the threshold, but also set the frame drop flag for the recognition frame in the convergence state when the ambient light brightness is greater than or equal to the threshold.
  • Setting the frame drop flag can reduce the number of recognition frames uploaded by the camera service to the target application, thereby reducing the number of threads created by the target application for recognition frames, the memory occupied by the recognition process, and the recognition time, thus improving recognition efficiency.
  • the metadata frame does not need to be processed by the hardware, but the preview frame and the recognition frame need to be processed by the hardware.
  • the hardware needs to crop and beautify the image. etc., so the metadata frame is three more frames than the identification frame.
  • the camera driver After the camera driver obtains the metadata frame, it discards the first three data frames and identifies the ambient light brightness of each metadata frame starting from the fourth data frame. Then, the camera driver determines the metadata information of the first identification frame based on the ambient light brightness in the fourth metadata frame. That is to say, in this embodiment, the i-th frame identification frame corresponds to the i+3-th frame metadata frame. The camera driver obtains the metadata information corresponding to the recognition frame based on the corresponding relationship to determine the ambient light brightness.
  • the first three metadata frames are discarded, and metadata information such as Luxindex is obtained starting from the fourth metadata frame, and the ambient light is determined based on the metadata information. brightness.
  • the ambient light brightness of the first six recognition frames is less than the threshold, so the recognition frames are discarded.
  • Starting from the seventh frame it is detected that the camera is in the AE convergence state or the AF convergence state based on the metadata frame, so frames are dropped at intervals.
  • the quality of the image captured by the camera when scanning the code with the mobile phone is not good, such as the size of the identification code in the image. too small.
  • the tilt of the scanning posture causes the identification code area in the image to become a non-rectangular area, affecting the image quality.
  • the embodiment of the present application also provides another code scanning method, which can effectively reduce the recognition time when scanning the code when the image quality is poor based on the embodiment of FIG. 6 .
  • the code scanning method provided by the embodiment of the present application may include:
  • the camera driver determines that the ambient light brightness of the metadata frame is greater than or equal to the threshold, and sets a frame drop flag for the recognition frame collected at the same timestamp as the metadata frame according to the frame drop policy.
  • the frame dropping strategy in S901 and the frame dropping strategy in S702 are two different strategies.
  • the interval frame dropping method is used to set the frame drop flag for the recognition frame
  • the frame drop flag is set for all recognition frames.
  • the frame dropping strategy in S901 and the frame dropping strategy in S702 are two identical strategies.
  • both S702 and S901 use interval frame dropping to set the frame drop flag for identification frames.
  • the target application In response to the image magnification operation performed on the code scanning interface, the target application sends a request to end frame loss to the camera driver.
  • the target application can receive the image magnification operation performed by the user on the QR code scanning interface.
  • the image magnification operation may be a double-click operation or a two-finger swipe operation.
  • the image magnification operation includes the user double-clicking the image in the code scanning frame of the code scanning interface. For example, after double-clicking the image in the scanning box, the image is enlarged and displayed in the scanning box.
  • the image magnification operation includes the user performing a two-finger swipe operation on the code scanning interface.
  • the user's thumb and index finger touch the touch screen at the same time, forming two touch points.
  • One touch point moves in the first direction
  • the other touch point moves in the second direction.
  • the first direction and the second direction are opposite.
  • the target application determines that the user performs a two-finger swipe operation based on the touch trajectory of the touch point, and then the target application sends a request to end frame loss to the camera driver.
  • the user performs an image magnification operation to shorten the shooting distance and enlarge the photographed object. Then, the objects in the recognition frame and preview frame are enlarged, thereby improving the image quality. Therefore, in response to the image magnification operation, the target application sends a request to end frame dropping to the camera driver to end the frame dropping processing of recognition frames when the image quality is good, thereby avoiding the long scanning time caused by discarding valid recognition frames. question.
  • the camera driver may determine whether the recognition frame includes the identification code. If the recognition frame includes the identification code, the camera driver confirms the image magnification operation performed by the user. Used to enlarge the size of the identification code.
  • S902 may be replaced by S902a.
  • the target application In response to the dragging operation of the image performed on the code scanning interface, the target application sends a request to end frame dropping to the camera driver.
  • the preview frame image displayed on the code scanning interface does not include the identification code.
  • the preview frame image displayed on the code scanning interface includes the identification code.
  • the user can adjust the shooting angle of the camera so that the identification code is included in the code scanning interface.
  • users can also adjust the position of the identification code on the code scanning interface by dragging the image.
  • the target application detects the operation of dragging the image, it means that the code scanning interface after the operation includes recognition There is a high possibility of aliasing, so the continuous recognition frames collected by the camera after this operation are recognized to avoid missing recognition frames and causing the scanning time to be too long.
  • S902 may be replaced by S902b.
  • the target application In response to the image reduction operation performed on the code scanning interface, the target application sends a request to end frame loss to the camera driver.
  • the preview frame image displayed in the code scanning interface does not include the complete identification code.
  • the identification code is completely displayed in the code scanning interface. Then when the target application detects the image reduction operation, it means that the identification code can be completely displayed in the code scanning interface after this operation. Therefore, the continuous identification frames collected by the camera after this operation are identified to avoid missing identification frames and causing code scanning. Too long.
  • the camera driver ends setting the frame loss flag for the identified frame according to the frame loss policy.
  • the camera driver may receive a request from the target application to end frame dropping.
  • the camera driver after the camera driver receives the request to end frame dropping, it can finish setting the frame dropping flag for the recognition frame. Then, the recognition frame uploaded by the camera driver to the camera service is the recognition frame with no frame dropping flag set. After the camera service receives the recognition frame, it uploads it to the target application for recognition. Among them, the operations performed by the camera service can be found in the previous description of S607. Based on this, after the mobile phone executes S903, it executes the above-mentioned S607.
  • the camera driver can set the frame drop flag for the recognition frames collected when the image quality is poor, so as to reduce the number of recognition frames uploaded to the target application by the camera service, and when the image quality is good, The frame dropping operation of the identification frame is ended to avoid the problem of long scanning time caused by discarding the valid identification frame.
  • the camera driver when the camera driver obtains the recognition frame, the camera driver sets the frame loss flag for two consecutive needle recognition frames every other frame. The camera driver no longer discards recognition frames when receiving an image zoom operation from the user. In this way, when the image quality is poor, the number of recognition frames recognized by the target application is much smaller than the number of recognition frames collected by the camera. Therefore, the recognition speed is fast, the user can perceive the scanning process to be short, and the user's Good scanning experience.
  • the frame dropping operation during the code scanning process was mainly explained from the perspective of lighting environment and image quality.
  • the principles of code scanning recognition used by different applications are different.
  • the second type of application introduced above only recognizes the latest recognition frame after the end of each thread recognition, which is equivalent to reducing the number of applications.
  • the number of recognition frames required to be recognized. Therefore, for this type of application, the embodiment of the present application also provides a code scanning method. See Figure 11.
  • the frame loss strategy can be determined according to the application type through S1101-S1102, S603-S606, S1103 and S607-S608.
  • the target application In response to the user's operation of opening the camera, the target application displays a code scanning interface and sends a request to start the camera and an application identification to the camera service.
  • the target application also uploads an application identifier to the camera service to distinguish the type of application.
  • the application identifier is used to distinguish the above-mentioned first type of application from the second type of application.
  • the camera service In response to receiving the request to start the camera, the camera service sends a request to call the camera driver and an application identifier to the camera driver.
  • the camera service also sends the application identifier sent by the target application to the camera driver, so that the camera driver determines whether to set the frame drop flag for consecutive multi-frame recognition frames based on the application identifier.
  • the mobile phone After the mobile phone executes S1102, it can execute S602-S606 above to set the frame drop flag for the recognition frame whose ambient light brightness is less than the threshold, and then the mobile phone executes S1103.
  • the camera driver responds that the obtained application identifier is a non-target identifier, and when it is determined that the ambient light brightness of the metadata frame is greater than or equal to the threshold, it ends setting the frame drop flag for consecutive multi-frame recognition frames.
  • the non-target identifier is used to indicate the second type of application mentioned above.
  • the target representation is then used to indicate the first type of application described above.
  • the non-target identifier is also called a preset identifier.
  • the non-target identifier can be set in advance to distinguish target applications that do not require interval frame loss.
  • the application identifier is a non-target identifier, it means that the target application only recognizes the latest recognition frame after the end of each thread recognition, which is equivalent to reducing the number of recognition frames that the application needs to recognize. Therefore, for the second type of application, the camera The driver ends by setting the frame drop flag for consecutive multi-frame recognition frames to avoid the problem of slow scanning speed caused by discarding too many recognition frames, thereby improving the target application's ability to recognize codes in dark environments without reducing the scanning speed. recognition speed.
  • the camera driver then reports the recognition frame and preview frame to the camera driver, so that the camera driver can perform frame loss processing based on the frame loss flag of the recognition frame. Based on this, after S1103, the mobile phone executes the aforementioned S607-S609 to complete the QR code scanning process.
  • the camera driver determines whether the target application is a non-target application. If it is a non-target application, the camera driver will directly report all subsequent recognition frames to the camera service, and the camera service will The identification frame is reported to the target application for identification. In this way, when the user uses the second type of application to scan the code, the camera driver will not discard too many recognition frames, which can improve the recognition speed of the target application in the dark environment without reducing the scanning speed. Therefore, the recognition speed is fast, the QR code scanning process perceived by the user is short, and the user's QR code scanning experience is good.
  • the application identifier can also distinguish the third type of application from the fourth type of application.
  • the preview stream is used as the video stream of the real-time preview screen and identification code.
  • the third type of application only delivers one stream configuration, and the preview stream is used both for display on the QR code scanning interface and for obtaining identification code information.
  • the fourth type of application delivers two stream configurations, using the preview stream as the video stream for real-time preview images, and the recognition stream as the recognition stream for identifying identification codes.
  • the target application sends the application identification to the QR code scanning service, so that the underlying camera service determines whether the target application only delivers one flow configuration. If it is determined that only one stream configuration is delivered, the camera driver sets the frame drop flag for the preview stream. The camera service will copy one preview stream, upload one preview stream to the target application, discard the preview frame with the frame drop flag set in the other preview stream, and then upload it to the target application. In this way, this embodiment can avoid the problem that the preview frame cannot be displayed normally on the code scanning interface after the preview stream is dropped due to the target application only delivering one stream configuration. This will not affect the normal display of the image captured by the camera on the code scanning interface. In the case of preview, it can reduce the number of preview frames to be recognized by the target application.
  • Figure 13 is a flow chart of a code scanning method in a specific example of this application.
  • the code scanning method provided by the embodiment of the present application may include:
  • Start the camera service When the phone is turned on, the camera service will be started, or after the camera service is started, the camera service will be automatically restarted.
  • Opening the camera application Opening the camera application, entering the function of using the camera in a third-party application, and switching the shooting mode can trigger opening the camera and entering the QR code scanning function. It should be understood that opening the camera means opening the camera and entering the QR code scanning interface.
  • Connecting the camera can also be understood as requesting to start the camera: after receiving the user's operation to open the camera, the target program can send a request to start the camera and an application identification to the camera service, where the application identification is used to distinguish the type of the target application. .
  • S1304 Collect metadata frames, recognition frames, and preview frames; after receiving a request to call the camera driver, the camera driver can control the camera to turn on. After the camera is turned on, it can continuously collect raw data, process the raw data to obtain YUV data, and divide the YUV data into two video streams. One video stream provides identification frames, and the other video stream provides preview frames, and at the same time, the elements of the YUV data are reported. Data Frame.
  • the ambient light brightness of the metadata frame is less than the threshold, which means that the lighting environment of the recognition frame and preview frame is a dark scene. Then the camera driver sets the frame drop flag for the recognition frame collected at the same time stamp as the metadata frame, so as to Instructs the upper-layer camera service to discard the recognition frame. If the ambient light brightness is greater than or equal to the threshold, it means that the lighting environment is bright. Then the camera driver does not set the frame drop flag for the recognition frame corresponding to the metadata frame, and directly uploads the recognition frame to the camera service to instruct the camera service to reduce the ambient light brightness. The collected recognition frames are uploaded to the target application for recognition processing.
  • the application identifier is a target identifier, it means that the target application is the first type of application mentioned above, so it is necessary to perform frame dropping processing on the recognition frame after the lighting environment is bright.
  • the application identifier is a non-target identifier, indicating that the target application is the second type of application mentioned above, and the recognition frame after the lighting environment is bright is continuously recognized and processed.
  • Interval frame dropping perform interval frame dropping on the recognition frames collected by the camera in the AE convergence state and AF convergence state.
  • an electronic device which may include: the above-mentioned display screen (such as a touch screen), a memory, and one or more processors.
  • the display, memory and processor are coupled.
  • the memory is used to store computer program code, which includes computer instructions.
  • the processor executes the computer instructions, the electronic device can perform each function or step performed by the mobile phone in the above method embodiment.
  • the structure of the electronic device may refer to the structure of the mobile phone 200 shown in FIG. 2 .
  • Embodiments of the present application also provide a computer-readable storage medium.
  • the computer storage medium includes computer instructions.
  • the electronic device When the computer instructions are run on the above-mentioned electronic device, the electronic device causes the electronic device to perform various functions performed by the mobile phone in the above-mentioned method embodiment. Or steps.
  • Embodiments of the present application also provide a computer program product.
  • the computer program product When the computer program product is run on a computer, it causes the computer to perform various functions or steps performed by the mobile phone in the above method embodiments.
  • the disclosed devices and methods can be implemented in other ways.
  • the device embodiments described above are only illustrative.
  • the division of modules or units is only a logical function division.
  • there may be other division methods for example, multiple units or components may be The combination can either be integrated into another device, or some features can be omitted, or not implemented.
  • the coupling or direct coupling or communication connection between each other shown or discussed may be through some interfaces, and the indirect coupling or communication connection of the devices or units may be in electrical, mechanical or other forms.
  • the units described as separate components may or may not be physically separated.
  • the components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place, or they may be distributed to multiple different places. . Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.
  • each functional unit in each embodiment of the present application can be integrated into one processing unit, each unit can exist physically alone, or two or more units can be integrated into one unit.
  • the above integrated units can be implemented in the form of hardware or software functional units.
  • the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it may be stored in a readable storage medium.
  • the technical solutions of the embodiments of the present application are essentially or contribute to the existing technology, or all or part of the technical solution can be embodied in the form of a software product, and the software product is stored in a storage medium , including several instructions to cause a device (which can be a microcontroller, a chip, etc.) or a processor to execute all or part of the steps of the methods described in various embodiments of this application.
  • the aforementioned storage media include: U disk, mobile hard disk, read only memory (ROM), random access memory (RAM), magnetic disk or optical disk and other media that can store program code.

Abstract

一种扫码方法及电子设备,可应用于电子设备，其中，电子设备包括摄像头，该方法包括：电子设备显示扫码界面。然后获取识别帧图像和预览帧图像，识别帧图像和预览帧图像根据摄像头采集的扫码图像处理获得。电子设备在扫码界面显示预览帧图像。若识别帧图像的环境光亮度大于或等于阈值，则电子设备对识别帧图像进行扫码识别，确定扫码识别结果。该方法通过删除无效识别帧图像提高了扫码效率。

Description

一种扫码方法及电子设备
本申请要求于2022年9月15日提交国家知识产权局、申请号为202211124961.4、发明名称为“一种扫码方法及电子设备”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本申请涉及终端技术领域，尤其涉及一种扫码方法及电子设备。
背景技术
随着智能终端的普及，以及图像识别技术的发展和广泛应用，扫码场景的应用愈加广泛。例如在通过支付应用进行支付交易时需要扫描付款码/收款码，在通过社交应用互加好友时需要扫描个人二维码，在进出特定场所时需要扫描通行码等等。
其中，扫码速度是影响用户扫码体验的一个因素。目前在扫码过程中对摄像头采集的扫码图像进行识别的方式扫码速度慢，进而为用户带来扫码困难的问题，影响用户体验。
针对扫码速度慢问题，目前尚未得到有效解决方案。
发明内容
本申请实施例提供一种扫码方法及电子设备，可以减少电子设备识别的无效识别帧图像的数量，从而减少为识别帧图像创建的线程的数量、识别过程对内存的占用空间以及识别时间，因此提高了识别效率。
为达到上述目的，本申请的实施例采用如下技术方案：
第一方面，本申请提供一种扫码方法，该方法可应用于电子设备，其中，电子设备包括摄像头，该方法包括：电子设备显示扫码界面。然后获取识别帧图像和预览帧图像，识别帧图像和预览帧图像根据摄像头采集的扫码图像处理获得。电子设备在扫码界面显示预览帧图像。若识别帧图像的环境光亮度大于或等于阈值，则电子设备对识别帧图像进行扫码识别，确定扫码识别结果。
本申请中，电子设备不对环境光亮度小于预设值时采集的识别帧图像进行识别，只对环境光亮度大于预设值时采集的识别帧图像进行识别，因此所需识别的识别帧图像数量更少。且相比于不丢弃识别帧图像的方案，本申请丢弃的无效识别帧图像，即丢弃了无法识别出识别码的识别帧图像。故本申请不仅不会增加扫码时长，还可以避免目标应用对无效帧的识别而导致浪费设备算力资源和降低扫码速度的问题，因此提高了扫码效率。
此外，电子设备只对识别帧图像进行有选择的识别，而预览帧图像显示在扫码界面的数量与摄像头采集的扫码图像一致，从而不影响摄像头采集的图像在扫码界面正常预览。也就是说，丢弃识别帧图像对用户来说是不会有视觉上的感知的，扫码界面显示的预览帧图像不会因掉帧而影响用户观看感受，故本申请在不影响摄像头采集的图像在扫码界面正常预览的情况下能够减少目标应用所要识别的识别帧图像数量，以 减少为识别帧图像创建的线程的数量、识别过程对内存的占用空间以及识别时间，因此提高了识别效率。
在第一方面的一种可能的设计方式中，上述若识别帧图像的环境光亮度大于或等于阈值，则对识别帧图像进行扫码识别，包括：若识别帧图像的环境光亮度大于或等于阈值，且摄像头的曝光收敛操作和对焦收敛操作完成，则对识别帧图像进行扫码识别。
此外，该方法还包括：若识别帧图像的环境光亮度大于或等于阈值，且检测到摄像头的曝光收敛操作或对焦收敛操作，则删除摄像头在曝光收敛操作或对焦收敛操作执行过程中根据扫码图像获得的识别帧图像，其中，被删除的识别帧图像不用于扫码识别。
该设计方式下，电子设备删除在摄像头曝光收敛过程或对焦收敛过程中由扫码图像获得的识别帧图像，这样可以不对摄像头曝光过程或收敛过程下获得的识别帧图像进行扫码识别，以减少对亮度不佳以及清晰度不佳的无效识别帧图像的处理数量。具体的，当摄像头处于自动曝光收敛状态时，采集的扫码图像会有曝光不足或过曝的问题，从而导致由扫码图像获得的识别帧图像的亮度不佳，因此对扫码识别造成难度。若电子设备对亮度不佳的识别帧图像进行扫码识别不仅浪费设备算力资源，还降低扫码速度。故该设计方式下丢弃亮度不佳的识别帧图像，提高了识别效率。同样的，当摄像头处于自动对焦收敛状态时，相机自动调整焦距，并且调整过程中焦距未达到目标值，焦距调整过程中采集的扫码图像的清晰度不佳，因此由扫码图像获得的识别帧图像的清晰度也不佳，对扫码识别造成难度。基于此，电子设备丢弃清晰度不佳的识别帧图像，提高了识别效率。
在第一方面的一种可能的设计方式中，上述若识别帧图像的环境光亮度大于或等于阈值，则对识别帧图像进行扫码识别，包括：若识别帧图像的环境光亮度大于或等于阈值、识别帧图像包括识别码且识别码的尺寸增大，对尺寸增大后的识别码所对应的识别帧图像进行扫码识别。
该设计方式下，电子设备在识别帧图像中包括识别码且识别码的尺寸增大的情况下，才对识别帧图像进行扫码识别。这样便可减少对识别帧图像的识别数量，以提高识别效率。
在第一方面的一种可能的设计方式中，上述若识别帧图像的环境光亮度大于或等于阈值，则对识别帧图像进行扫码识别，包括：若识别帧图像的环境光亮度大于或等于阈值，且检测到在扫码界面执行的图像放大操作，则对图像放大操作执行后根据扫码图像获得的识别帧图像进行扫码识别。
该设计方式下，电子设备检测到图像放大操作，其中，图像放大操作表示用户调整识别码的显示尺寸。那么调整后的识别码的尺寸更大，识别帧图像的图像质量更好。因此对识别帧图像进行识别后识别成功的几率更大，识别的效率更高。
在第一方面的一种可能的设计方式中，在上述对图像放大操作执行后根据扫码图像获得的识别帧图像进行扫码识别之前，该方法还包括：确定识别帧图像包括识别码。
在第一方面的一种可能的设计方式中，在上述显示扫码界面之前，该方法还包括：启动目标应用。上述若识别帧图像的环境光亮度大于或等于阈值，则对识别帧图像进 行扫码识别，包括：若识别帧图像的环境光亮度大于或等于阈值且目标应用的应用标识为预设标识，则对识别帧图像进行扫码识别。
其中，预设标识可以预先设置，用于区分不需要间隔丢帧的目标应用。当应用标识为预设标识时，表示目标应用只对每个线程识别结束后的最新一帧识别帧图像进行识别，相当于减少了应用所需识别的识别帧图像数量，携带预设标识的目标应用，电子设备不进行删除识别帧图像的处理，以避免删除过多识别帧图像造成的扫码速度变慢的问题。
在第一方面的一种可能的设计方式中，上述若识别帧图像的环境光亮度大于或等于阈值，则对识别帧图像进行扫码识别，包括：若识别帧图像的环境光亮度大于或等于阈值、识别帧图像未包括识别码，且检测在扫码界面对预览帧图像的拖动操作，则对与预览帧图像为同一扫码图像处理得到的识别帧图像进行扫码识别。
该设计方式中，检测到拖动图像的操作，便不删除识别帧图像，即对拖动操作执行后的识别帧图像进行扫码识别。具体的，在扫码过程中，当扫码界面不包括识别码时，一方面用户可以调整相机的拍摄角度或者在扫码界面放大图像以实现识别码在扫码界面中。此外，用户也可以通过该设计方式所说的拖动图像来调整识别码在扫码界面的位置。当目标应用检测到拖动图像的操作，表示在该操作后扫码界面中包括识别码的可能性较大，因此对摄像头在该操作后采集的连续识别帧图像进行识别，以避免漏检识别帧图像造成扫码时间过长，从而提高了扫码效率。
在第一方面的一种可能的设计方式中，该方法还包括：获取元数据帧，其中，元数据帧包括用于处理得到识别帧图像的扫码图像的采集参数。从采集参数中获取光照强度参数，根据光照强度参数确定识别帧图像的环境光亮度。
该设计方式中，环境光亮度光照强度参数表示，光照强度参数包括：Luxindex。例如，环境光亮度为Luxindex的倒数，即1/Luxindex。Luxindex的数值越大，表示环境光亮度越小，那么扫码场景的光线越暗，Luxindex的数值越小，表示环境光亮度越大，那么扫码场景的光线越亮。基于此，电子设备可以从元数据帧中获取光照强度参数，从而确定与该元数据帧相对于的扫码图像所处理得到的识别帧图像和预览帧图像的环境光亮度，并根据环境光亮度确定光照环境好坏。在光照环境不好的情况下，不进行扫码识别；在光照环境好的情况下，进行扫码识别。从而减少对光照环境不好的情况下获取的无效识别帧图像的识别，以减提高扫码效率。
在第一方面的一种可能的设计方式中，该方法还包括：若识别帧图像的环境光亮度小于阈值，或在识别帧图像的环境光亮度大于或等于阈值的情况下检测到摄像头的曝光收敛操作或对焦收敛操作，或在识别帧图像的环境光亮度大于或等于阈值的情况下识别帧图像中识别码的尺寸小于或等于预设尺寸，或在识别帧图像的环境光亮度大于或等于阈值的情况下识别帧图像未包括识别码，则为识别帧图像设置丢帧标志。上述若识别帧图像的环境光亮度大于或等于阈值，则对识别帧图像进行扫码识别，包括：对未包含丢帧标志的识别帧图像进行扫码识别，其中，未包含丢帧标志的识别帧图像的环境光亮度大于或等于阈值。
该设计方式下，电子设备为需要删除的识别帧图像设置丢帧标志，以指示该识别帧图像不用于扫码识别。这样电子设备便可对未包含丢帧标志的识别帧图像进行扫码 处理，以减少识别帧图像的识别数量。
在第一方面的一种可能的设计方式中，上述为识别帧图像设置丢帧标志，包括：在每一预设丢帧周期内为至少一帧识别帧图像设置丢帧标志。
该设计方式中，电子设备采用间隔丢帧的策略为识别帧图像设置丢帧标志。即以T为一个丢帧周期，在每一丢帧周期内丢弃N帧识别帧图像。那么根据丢帧策略，电子设备会为丢帧周期T内的N帧识别帧图像设置丢帧标志。
在一些实施例中，电子设备可以为满足上述丢帧条件的所有识别帧图像设置丢帧标志。
在另一些实施例中，电子设备可以为满足上述丢帧条件的前M帧识别帧图像设置丢帧标志。
通过不同的丢帧策略，电子设备便可有选择的删除无效识别帧图像，以便于实现快速扫码的有益效果。
第二方面，本申请提供一种电子设备，该电子设备包括：存储器和一个或多个处理器；存储器和处理器耦合。其中，存储器用于存储计算机程序代码，计算机程序代码包括计算机指令；当该计算机指令被处理器执行时，使得电子设备执行如第一方面及其任一种可能的设计方式所述的方法。
第三方面，本申请提供一种计算机可读存储介质，该计算机可读存储介质可以包括计算机指令，当计算机指令在电子设备上运行时，使得电子设备执行如第一方面及其可能的设计方式的方法。
第四方面，本申请提供一种计算机程序产品，当计算机程序产品在计算机上运行时，使得计算机执行如第一方面及其可能的设计方式的方法。
可以理解地，上述提供的第二方面的电子设备，第三方面的计算机存储介质，以及第四方面的计算机程序产品均用于执行上文所提供的对应的方法，因此，其所能达到的有益效果可参考上文所提供的对应的方法中的有益效果，此处不再赘述。
附图说明
图1为本申请实施例提供的两类应用的扫码识别原理的示意图；
图2为本申请实施例提供的一种电子设备的硬件结构示意图；
图3为本申请实施例提供的一种电子设备的软件结构框图；
图4为本申请实施例提供的一组扫码场景示意图；
图5为本申请实施例提供的另一组扫码场景示意图；
图6为本申请实施例提供的一种扫码方法的交互图；
图6A为本申请实施例提供的一种打开相机的操作的示意图；
图6B为本申请实施例提供的另一种打开相机的操作的示意图；
图7A为本申请实施例提供的再一种扫码方法的交互图；
图7B为本申请实施例提供的再一种扫码方法对应的效果示意图；
图8为本申请实施例提供的另一种扫码方法对应的效果示意图；
图9为本申请实施例提供的又一种扫码方法的交互图；
图10为本申请实施例提供的又一种扫码方法对应的效果示意图；
图11为本申请实施例提供的又一种扫码方法的交互图；
图12为本申请实施例提供的又一种扫码方法对应的效果示意图；
图13为本申请实施例提供的一种扫码方法对应的流程图。
具体实施方式
以下，术语“第一”、“第二”仅用于描述目的，而不能理解为指示或暗示相对重要性或者隐含指明所指示的技术特征的数量。由此，限定有“第一”、“第二”的特征可以明示或者隐含地包括一个或者更多个该特征。在本实施例的描述中，除非另有说明，“多个”的含义是两个或两个以上。
目前，尽管扫码场景的应用渐趋广泛，但由于设备负载、内存、光照环境以及图像质量等因素的差异，识别码的识别速度受到影响。
例如：采用同一应用扫描同一识别码，在设备负载压力大，内存可用容量小、黑暗环境、识别码图片小的情况下，识别速度慢；而在设备负载压力小、内存可用容量大、光线明亮且识别码图片大的情况下，识别速度快。在上述识别速度快的场景下，用户所能感知到的扫码过程的耗时短，因此用户的扫码体验好；一旦设备处于上述识别速度慢的场景下，用户所能感知到的扫码过程的耗时会加长，因此用户的扫码体验就差。而目前扫码技术无法在设备处于识别速度慢的场景下较快地识别出识别码，因此给用户带来了扫码困难的问题。
此外，扫码速度还受到应用的扫码识别原理的差异影响。下面以不同识别算法的两类应用进行扫码的识别流程进行说明。
图1为本申请实施例提供的两类应用的扫码识别原理的示意图。如图1所示，第一类应用对每一帧识别帧(又称识别帧图像)都进行识别，因此识别速度慢。具体地，如图1的(a)所示，设备运行第一类应用后，摄像头返回30fps(Frames Per Second，每秒帧数目)给抽象层，即摄像头启动后，便每隔33ms(millisecond,毫秒)返回一帧数据，数据包括一帧预览帧(又称预览帧图像，图1中未示出)和一帧识别帧。其中预览帧用于在扫码界面上显示，识别帧用于获取识别码的信息。在抽象层中识别帧以缓存形式存储，以使得识别帧在渲染到屏幕之前能够对其进行处理。第一类应用为每一帧返回到第一类应用的算法处理模块101的识别帧都创建一个线程来识别该帧识别帧的信息，每个线程之间通过互斥锁来保证线程之间串行执行。如图1的(a)中，在第一线程102识别第0帧识别帧时，虽然第1帧识别帧已经返回到第一类应用的算法处理模块101，但第二线程103对第1帧识别帧的识别需等到第一线程102对第0帧识别帧识别结束后才开始。也就是说，第一线程102识别第0帧识别帧时，第一线程102占据互斥锁，此时第二线程103被锁住(也称lock)，并不能执行识别操作。直到第一线程102识别完第0帧识别帧时，第二线程103才占据互斥锁，并开始识别第1帧识别帧(也称process)。
第二类应用对每个线程识别结束后的最新一帧识别帧进行识别，因此识别速度快。如图1的(b)所示，设备运行第二类应用后，摄像头同样每隔33ms返回一帧数据，数据包括一帧预览帧和一帧识别帧。但是，第二类应用的算法处理模块101每次对一帧识别帧识别完成后，若没有识别到识别码，则创建一个线程来抓取最新一帧返回到算法处理模块101的识别帧，并对最新一帧的识别帧进行识别。如图1的(b)中，在第三线程104识别第0帧识别帧时，虽然第1帧识别帧已经返回到第二类应用的算法 处理模块101，但算法处理模块101等待第三线程104将第0帧识别帧识别结束后才创建第四线程105。然后算法处理模块101为第四线程105抓取最新一帧识别帧，即图1的(b)所示的第2帧识别帧被返回到算法处理模块101，并由第四线程105开始识别第2帧识别帧。也就是说，虽然在创建第四线程105时，算法处理模块101收到两帧识别帧，但第二类应用并不会对每一帧识别帧都进行识别，而是抓取最新一帧识别帧来识别是否存在识别码。
上述第一类应用和第二类应用的识别流程的说明中，第一类应用对每帧识别帧都进行识别，因此采用第一类应用进行扫码会存在至少两点问题。其一，创建的线程数量多，占用的内存空间大。在设备负载压力大、内存可用容量小的场景下，线程所分配到的计算资源不足，故识别速度受到影响，进而影响用户扫码的速度。其二，第一类应用会对光照环境和图像质量不好情况下采集的图像帧进行识别，这样不仅无法识别出识别码，还浪费了线程资源和内存空间。
第二类应用虽然只对部分识别帧进行识别，但同样从摄像头采集的第一帧识别帧开始处理。在光照环境和图像质量不好的情况下，第二类应用同样会做大量无效识别，导致浪费设备算力资源。
为减少设备对无效识别帧进行识别，本申请实施例提供一种扫码方法，可以应用于电子设备中。电子设备在光照环境和图像质量不好的场景下进行扫码时，只识别一部分上述场景下摄像头采集的识别帧，从而减少为识别帧创建的线程的数量、识别过程对内存的占用空间以及识别时间，因此提高了识别效率。
示例性的，本申请实施例中的电子设备可以是手机、平板电脑、桌面型、膝上型、手持计算机、笔记本电脑、超级移动个人计算机(ultra-mobile personal computer，UMPC)、上网本、增强现实(augmented reality，AR)或者虚拟现实(virtual reality，VR)设备等支持相机功能的设备，本申请实施例对该电子设备的具体形态不作特殊限制。
在详细说明本申请实施例提供的扫码方法之前，先在此介绍电子设备的结构。
图2为本申请实施例提供的一种电子设备的硬件结构示意图。如图2所示，以电子设备是手机200为例，手机200可以包括处理器210，外部存储器接口220，内部存储器221，通用串行总线(universal serial bus，USB)接口230，充电管理模块240，电源管理模块241，电池242，天线1，天线2，移动通信模块250，无线通信模块260，音频模块270，扬声器270A，受话器270B，麦克风270C，耳机接口270D，传感器模块280，触摸传感器280A，环境光传感器280B，按键290，马达291，指示器292，摄像头293，显示屏294，以及用户标识模块(subscriber identification module，SIM)卡接口295等。
可以理解的是，本实施例示意的结构并不构成对手机200的具体限定。在另一些实施例中，手机200可以包括比图示更多或更少的部件，或者组合某些部件，或者拆分某些部件，或者不同的部件布置。图示的部件可以以硬件，软件或软件和硬件的组合实现。
处理器210可以包括一个或多个处理单元，例如：处理器210可以包括应用处理器(application processor，AP)，调制解调处理器，图形处理器(graphics processing unit，GPU)，图像信号处理器(image signal processor，ISP)，控制器，存储器，视频编 解码器，数字信号处理器(digital signal processor，DSP)，基带处理器，和/或神经网络处理器(neural-network processing unit，NPU)等。其中，不同的处理单元可以是独立的器件，也可以集成在一个或多个处理器中。
控制器可以是手机200的神经中枢和指挥中心。控制器可以根据指令操作码和时序信号，产生操作控制信号，完成取指令和执行指令的控制。
处理器210中还可以设置存储器，用于存储指令和数据。在一些实施例中，处理器210中的存储器为高速缓冲存储器。该存储器可以保存处理器210刚用过或循环使用的指令或数据。如果处理器210需要再次使用该指令或数据，可从所述存储器中直接调用。避免了重复存取，减少了处理器210的等待时间，因而提高了系统的效率。在本申请一些实施例中，存储器可以存储有识别光照环境和图像放大操作的可执行程序，以及执行上述操作生成的用户数据、应用数据等。
手机200的无线通信功能可以通过天线1，天线2，移动通信模块250，无线通信模块260，调制解调处理器以及基带处理器等实现。在本申请一些实施例中，手机200可以通过上述无线通信模块260发送启动目标应用的信号、调用相机驱动的信号等。手机200还可以在识别出识别码后，通过上述移动通信模块250或上述无线通信模块260根据识别出的识别码执行对应的操作，如打开某个网页或向共享交通工具发送解锁指令等。
触摸传感器280A，也称“触摸器件”。触摸传感器280A可以设置于显示屏294上，由触摸传感器280A和显示屏294组成触摸屏，也称“触控屏”。触摸传感器280A用于检测作用于其上或附近的触摸操作。触摸传感器280A可以将检测到的触摸操作传递给应用处理器，以确定触摸事件类型。可以通过显示屏294提供与触摸操作相关的视觉输出。在一些实施例中，触摸传感器280A可以设置于手机200的表面，与显示屏294所处的位置不同。在本申请实施例中，手机200可以通过触摸传感器280A检测到用户作用于目标应用的图标上的触摸操作，响应于该触摸操作，手机200启动目标应用。在本申请一些实施例中，手机200可以通过触摸传感器280A检测到用户在扫码预览界面执行图像放大操作，响应于该图像放大操作，手机200结束识别部分识别帧的操作，开始逐帧识别自执行图像放大操作后摄像头293采集到的所有识别帧。
环境光传感器280B用于感知环境光亮度(也称光照环境)。本申请一些实施例中，手机200根据环境光亮度确定是否对当下采集到的识别帧进行识别码识别处理。具体地，若环境光亮度小于阈值，表示光照环境黑暗，手机200不对该环境光亮度下采集到的识别帧进行处理；若环境光亮度大于或等于阈值，表示光照环境明亮，手机200对该环境光亮度下采集到的识别帧进行处理。如此，手机200便可有效避免在光照环境不好场景下对无效识别帧进行处理而造成的浪费设备算力资源的问题，因此提高了识别效率。
手机200通过GPU，显示屏294，以及应用处理器等实现显示功能。GPU为图像处理的微处理器，连接显示屏294和应用处理器。GPU用于执行数学和几何计算，用于图形渲染。处理器210可包括一个或多个GPU，其执行程序指令以生成或改变显示信息。
显示屏294用于显示图像，视频等。显示屏294包括显示面板。显示面板可以采 用液晶显示屏(liquid crystal display，LCD)，有机发光二极管(organic light-emitting diode，OLED)，有源矩阵有机发光二极体或主动矩阵有机发光二极体(active-matrix organic light emitting diode，AMOLED)，柔性发光二极管(flex light-emitting diode，FLED)，Miniled，MicroLed，Micro-oLed，量子点发光二极管(quantum dot light emitting diodes，QLED)等。
手机200可以通过ISP，摄像头293，视频编解码器，GPU，显示屏294以及APP处理器等实现拍摄功能。
ISP用于处理摄像头293反馈的数据。例如，拍照时，打开快门，光线通过镜头被传递到摄像头感光元件上，光信号转换为电信号，摄像头感光元件将电信号传递给ISP处理，电信号被转换为肉眼可见的图像，其中图像可以是包括识别码的图像，也可以是不包括识别码的图像。ISP还可以对图像的噪点、亮度、肤色进行算法优化。ISP还可以对拍摄场景的曝光、色温等参数优化。在一些实施例中，ISP可以设置在摄像头293中。
摄像头293用于捕获静态图像或视频。物体通过镜头生成光学图像投射到感光元件。感光元件可以是电荷耦合器件(charge coupled device，CCD)或互补金属氧化物半导体(complementary metal-oxide-semiconductor，CMOS)光电晶体管。感光元件把光信号转换成电信号，之后将电信号传递给ISP转换成数字图像信号。ISP将数字图像信号输出到DSP加工处理。DSP将数字图像信号转换成标准的RGB，YUV等格式的图像信号。在一些实施例中，手机200可以包括1个或N个摄像头293，N为大于1的正整数。
在本申请实施例中，手机200可以通过摄像头293来获取预览帧、识别帧、元数据帧，具体的，摄像头采集扫码图像，扫码图像又被称为原始数据(raw数据)，ISP将raw数据转换成YUV数据(又称亮度色度饱和度图)，并且根据目标应用的需要，将一路YUV数据输出为一路预览流和一路预览帧，并同时输出每一帧YUV数据的元数据帧，用于描述YUV数据的光照强度、自动对焦参数、自动曝光参数、自动白平衡参数等。
图3为本申请实施例提供的一种电子设备的软件结构框图。电子设备的软件系统可以采用分层架构，事件驱动架构，微核架构，微服务架构，或云架构。本申请实施例以分层架构的Android系统为例，示例性说明手机200的软件结构。
如图3所示，分层架构可将软件分成若干个层，每一层都有清晰的角色和分工。层与层之间通过软件接口通信。在一些实施例中，将Android系统分为四层，从上至下分别为应用程序层(简称应用层)310，应用程序框架层(简称框架层)320，硬件抽象层(hardware abstract layer，HAL)层330，以及内核层(Kernel，也称为驱动层)。需要说明的是，本申请实施例中，内核层用于提供简单的访问硬件逻辑，而体现硬件访问逻辑的功能性动作(如下文方法实施例中详细介绍的跳帧识别逻辑)放在HAL层，且HAL层可不经过内核层就直接与硬件层的摄像头进行通信，因此内核层在图3中未示出。
其中，应用层(Application)310可以包括一系列应用程序包。该多个应用程序包可以为相机，图库，日历，通话，地图，导航，WLAN，蓝牙，音乐，视频，短信息 以及桌面启动(Launcher)等应用程序。
本申请实施例中，该应用层310可以包括相机系统应用(也称为相机应用)311。相机应用311的运行，例如，拍照、录像等，都需要使用摄像头。
本申请实施例中，该应用层310还可以包括其它需要使用摄像头的应用(也称为三方应用)312。例如，微信TM的扫码功能、支付宝TM的扫码功能、抖音TM的扫码功能等都需要使用摄像头，则该其他三方应用312可以是微信TM、支付宝TM、抖音TM等应用。
如图3所示，相机应用311和三方应用312可以用于在预览界面中展示底层上报的预览图像(也称预览帧)。其中，手机200可以包括多个摄像头，每个摄像头都可以用于采集预览帧。为了方便说明，在下文中出现的摄像头理解为当前采集预览帧的摄像头。
在本申请实施例中，应用层310可以根据用户操作指示HAL层330启动摄像头，应用层310还可以根据用户的操作指示HAL层330对摄像头采集的图像执行放大操作，或者指示HAL层330启动/关闭手电筒功能。
框架层(Framework)320为应用层310提供应用编程接口(application programming interface，API)和编程框架。框架层320包括一些预先定义的函数。如图3所示，框架层320可以提供相机API，如Camera API(API 1|API 2)321、相机服务(Camera Service)322。
其中，Camera API(API 1|API 2)321作为底层(如HAL层330)与应用层310交互的接口。具体的，Camera API(API 1|API 2)321可以接收来自上层(如应用层310)的开启指定摄像头的请求，并通过相机服务322处理该请求。本申请实施例中，相机服务322再通过HAL层330开启相应的摄像头，并通过HAL层330获得摄像头上报的预览帧、识别帧、元数据帧等。
本申请实施例中，相机服务322可用于根据来自应用层310的请求和/或根据来自下层(如HAL层330)上报的识别帧来得到丢弃识别帧的策略。例如，丢弃识别帧的策略包括：相机服务322接收HAL层330上报的识别帧，其中丢弃(可以理解为删除)不作识别的识别帧的状态为error(error指应用程序不必处理的异常)，相机服务322确定识别帧的状态，若识别帧的状态为error，则相机服务322不将该识别帧返回给应用程序；若识别帧的状态不为error，如识别帧未被设置状态，则相机服务322将该识别帧返回给应用层310进行识别处理。
HAL层330用于连接框架层320和内核层。例如，HAL层330可以在框架层320和内核层之间进行数据传输。当然，HAL层330也可以对来自底层(即内核层)的数据进行处理，然后再传输给框架层320。本申请实施例中，HAL层330可以对硬件层的摄像头上报的预览帧、识别帧、元数据帧进行处理，并返回识别帧以及预览帧给相机服务322。
如图3所示，HAL层330可以包括HAL接口定义语言(HAL interface definition language，HIDL)接口331、相机硬件抽象层(Camera Provider)332和HAL接口。其中，相机硬件抽象层332通过提供标准的HIDL接口给上层(如相机服务322)进行调用，保持与上层的正常通信。以及，相机硬件抽象层332通过标准的HAL接口，如HAL3.0，向下控制摄像头，并获取摄像头上报的预览帧、识别帧、元数据帧等数据。
本申请实施例中，相机硬件抽象层332可以将获取的预览帧、设置状态的识别帧上报给上层(如相机服务322)。
本申请实施例中，相机硬件抽象层332包括相机驱动，不同于内核层中的相机驱动，相机硬件抽象层332中的相机驱动用于根据光照环境和图像质量对摄像头上报的识别帧进行状态设定，并将设置好状态的识别帧连同预览帧一起上报给相机服务322。
本申请实施例提供的扫码方法，可以在具有上述硬件结构和软件结构的手机200中实现。
下面结合扫码场景，示例性说明手机200软件及硬件的工作流程。
当触摸传感器280A接收到触摸操作，相应的硬件中断被发给内核层。内核层将触摸操作加工成原始输入事件(包括触摸坐标，触摸操作的时间戳等信息)。原始输入事件被存储在内核层。框架层320从内核层获取原始输入事件，识别该输入事件所对应的控件。以该触摸操作是触摸单击操作，该单击操作所对应的控件为三方应用312的界面上实现扫码功能的控件为例，三方应用312调用框架层320的相机服务322，进而通过相机服务322调用相机驱动，通过相机驱动启动摄像头293采集扫码图像，并将扫码路线处理为预览帧、识别帧。然后摄像头293将预览帧、识别帧和元数据帧等数据上报给HAL层330的相机驱动。相机驱动根据光照环境或图像质量为摄像头293上报的识别帧设置丢帧标志(又称状态error、error或状态)，并将设置好丢帧标志的识别帧连同预览帧一起上报给相机服务322。相机服务322接收HAL层330上报的识别帧，根据识别帧是否有丢帧标志确定是否将识别帧返回给三方应用312，并同时返回预览帧。三方应用312将预览帧显示在预览框中，并同时处理识别帧中的信息，若识别到识别码，则三方应用312发出关闭相机的指令，摄像头停止采集数据，且三方应用312指示手机200根据识别出的识别码执行对应的操作，如打开某个网页或向共享交通工具发送解锁指令等。
图4示出了本申请实施例提供的一组扫码场景示意图。如图4的(a)所示，用户在光照条件不好的情况下执行扫码操作，摄像头293将识别帧、预览帧和元数据帧上报给相机驱动，相机驱动根据元数据帧确定环境光亮度为光照条件不好，相机驱动便将识别帧的状态设为error，并连同预览帧一起上报给相机服务。相机服务根据识别帧的状态error确定不将该识别帧上报给三方应用，因此相机服务只将预览帧上报给三方应用。三方应用将预览帧显示在预览界面中。由于三方应用未接收到到识别帧，则并未进行识别处理，因此不会做无效识别。如图4的(b)所示，用户开启手机的手电筒功能，此时取景环境的亮度上升，对应的相机驱动根据元数据帧识别环境光亮度为光照条件好。相机驱动便不为识别帧设置error，而是将识别帧和预览帧一起上传给相机服务。相机服务根据识别帧未设置error确定需要将该识别帧上报给三方应用，因此相机服务将识别帧和预览帧一起上报给三方应用。三方应用将预览帧显示在预览界面，并对识别帧进行识别处理，直至识别到识别码。该场景下，三方应用只对光照条件好的场景下采集的识别帧进行识别，因此大大减少了所需识别的无效识别帧(即光照条件不好的识别帧)的数量，从而减少为识别帧创建的线程的数量、识别过程对内存的占用空间以及识别时间，因此提高了识别效率。
图5示出了本申请实施例提供的另一组扫码场景示意图。如图5的(a)所示，用 户执行扫码操作，但预览界面中显示的识别码较小，也就是说摄像头采集到的图像中识别码尺寸太小，图像质量较差。因此相机驱动对识别帧采用间隔一帧识别帧不做标记，对连续两帧识别帧进行error状态标记的方式来标记识别帧的状态。那么相应地，相机服务便采取丢弃两帧识别帧，将一帧识别帧上报给三方应用的丢帧策略。如图5的(b)所示，用户执行图像放大操作，三方应用下发图像放大请求至相机驱动，相机驱动停止对识别帧进行状态的标记，将识别帧及预览帧直接上传给相机服务，并由相机服务返回给三方应用。三方应用对连续的识别帧进行识别处理。该场景下，三方应用对识别帧先进行跳帧识别，以避免对图像质量不好的识别帧进行识别，浪费算力资源。在检测到用户执行图像放大操作后，相机驱动确认图像质量变好(图像中的识别码由小尺寸变为大尺寸)，便停止对识别帧进行状态的标记。三方应用对预览帧进行连续帧识别。基于此，三方应用只对图像质量好的场景下采集的识别帧进行连续识别，因此大大减少了所需识别的无效识别帧(即图像质量不好的识别帧)的数量，从而减少为识别帧创建的线程的数量、识别过程对内存的占用空间以及识别时间，因此提高了识别效率。
下文中，将结合图6所示的方法流程图，来详细介绍本申请实施例提供的一种扫码方法。
S601、目标应用响应于用户打开相机的操作，显示扫码界面并向相机服务发送启动摄像头的请求。
在S601之前，目标应用可接收用户打开相机的操作。其中，目标应用可以是相机应用或者三方应用。
在一些实施例中，打开相机的操作包括触发手机在后台运行相机应用的操作。例如，打开相机的操作是用户对图6A中的(a)示出的手机主界面601a中图标602a进行点击操作后，进入的扫码界面603a。
或者，打开相机的操作包括触发手机在后台运行三方应用时使用摄像头来启动扫码功能的操作。以三方应用是支付宝TM为例，则使用摄像头的功能可以是对图6B中的(a)示出的三方应用主界面601b中点击“扫一扫602b”图标的操作。手机检测到该操作，启动三方应用的扫码功能，手机显示图6B中的(b)示出的扫码界面603b。
在另一些实施例中，打开相机的操作包括拍摄模式的切换操作。相机应用或三方应用中使用的摄像头的功能中，可以提供多种拍摄模式，如拍摄、录像、全景、扫码、识物等。以打开相机的操作是切换相机应用的拍摄模式为扫码模式为例，在进入的图6A中的(b)示出的相机应用的主界面上选择扫码604a图标的操作，相机应用便可切换至扫码模式。其中图标602a是相机应用的应用图标，扫码604a是相机应用中扫码功能的功能图标。
在应用程序接收到上述用户打开相机的操作后，应用程序可调用相机接口(如open Camera接口)，以请求启动摄像头，即向相机服务发送启动摄像头的请求。其中，该启动摄像头的请求中携带有用于指示摄像头类型、拍摄模式的功能参数，从而可以明确指示底层启动对应的摄像头来满足相应功能的需求。在一些实施例中，启动摄像头的请求中还包括应用标识，该应用标识用于相机驱动确定启动摄像头的应用是否为预设的目标应用。
S602、相机服务响应于接收到启动摄像头的请求，向相机驱动发送调用相机驱动的请求。
在S602之前，相机服务可接收来自目标应用的启动摄像头的请求。
本实施例中，相机服务接收启动摄像头的请求后，则向底层的相机驱动发送调用相机驱动的消息，以指示相机驱动去驱动摄像头启动。
S603、相机驱动响应于接收到调用相机驱动的请求，启动摄像头并获取元数据帧、识别帧和预览帧。
在S603之前，相机驱动可接收来自相机服务的调用相机驱动的请求。
本实施例中，相机驱动在接收到调用相机驱动的请求后，则可控制摄像头开启。摄像头开启后，便可持续采集扫码图像，将扫码图像处理为识别帧和预览帧，并输出元数据帧。其中，扫码界面中的预览图像可通过摄像头向上层上报预览帧实现；扫码识别结果(又称识别结果)可通过对识别码进行识别实现。
S604、摄像头向相机驱动上报元数据帧、识别帧和预览帧。
需要说明的是，实际应用中摄像头向相机驱动上报的是元数据流、识别流和预览流，本步骤以某一时刻摄像头采集到的一帧元数据、识别图像和预览图像进行举例，则采集到的一帧元数据为元数据帧，一帧识别图像为识别帧，一帧预览图像为预览帧。
其中，元数据帧用于描述在上述某一时刻摄像头采集到的图像的数值，例如元数据帧包括帧速率、图像大小、编码、色彩位深、环境光亮度等。识别帧和预览帧均为图像数据(相当于前述YUV数据)，图像数据可以显示于手机的显示屏上，或者可以被用于识别处理，从而得到图像数据中的识别码。而图像数据的数值可以由描述该图像数据采集时间戳下的摄像头采集参数的元数据帧中获取，例如，手机可以根据元数据帧中的光照强度确定摄像头采集预览帧和识别帧时的光照环境(又称环境光亮度)。
其中，元数据帧、识别帧和预览帧均在内存块中缓存，以使得识别帧或预览帧在渲染到屏幕之前能够对其进行处理。由此，元数据帧、识别帧和预览帧会占用内存空间，在内存可用容量小的情况下，会对识别速度造成影响。
上述S601至S604，可以按照现有技术中开启摄像头采集raw数据并上报预览图像、识别图像和元数据的流程来处理，此处不作累赘说明。
S605、相机驱动根据元数据帧识别环境光亮度。
该实施例中，环境光亮度光照强度参数表示，光照强度参数包括：Luxindex(又称光照强度，简称照度)。例如，环境光亮度为Luxindex的倒数，即1/Luxindex。Luxindex的数值越大，表示环境光亮度越小，那么扫码场景的光线越暗，Luxindex的数值越小，表示环境光亮度越大，那么扫码场景的光线越亮。
相机驱动可以从元数据帧中获取光照强度，从而确定与该元数据帧相对于的扫码图像所处理得到的识别帧和预览帧的环境光亮度，其中，环境光亮度用于确定光照环境好坏。
S606、相机驱动确定元数据帧的环境光亮度小于阈值，为与元数据帧同一时间戳采集的识别帧设置丢帧标志。
元数据帧的环境光亮度小于阈值，表示识别帧和预览帧的光照环境为黑暗场景，那么相机驱动为与元数据帧同一时间戳采集的识别帧设置丢帧标志，以指示上层的相 机服务丢弃该识别帧。相机服务丢弃的识别帧不会被上层的目标应用所识别，因此上层的目标应用不需要为无效帧分配线程来识别无效帧中的二维码，如此，可以避免因做大量无效识别导致设备算力资源浪费。
进一步的，丢帧标志可以是识别帧的状态，例如，状态为error，则表示该帧识别帧为需要丢弃的无效识别帧。若状态不为error，例如状态为空(NULL)或未被设置状态，则表示该帧识别帧为需要识别的识别帧。应理解，相机驱动为无效识别帧设置丢帧标志的好处在于，上层的相机服务可以根据丢帧标志确定是否将识别帧上传给目标应用。因此，在一些实施例中，识别帧标志也可以是一对表示不同含义的标识，如(Y/N)，(1/0)等，其中，丢帧标志可以是N，或者0，那么不丢帧的标志为Y，1。
S607、相机驱动向相机服务上报识别帧和预览帧。
其中，上报的识别帧中包括设有丢帧标志的识别帧以及未设有丢帧标志的识别帧。也就是说，相机服务获取的识别帧是经相机驱动基于环境光亮度分析处理后得到的图像，因此相机服务接收到识别帧后可以检测识别帧是否被标记，即执行S608。
S608、相机服务检测识别帧是否被标记，将预览帧以及不含有标记的识别帧返回给目标应用。
基于前文可知，预览帧用于在扫码界面上显示，识别帧用于获取识别码的信息。也就是说预览帧是否有丢帧对用户来说是有感知的，而识别帧是否有丢帧不会影响用户的扫码体验。因此黑暗环境下扫码时，本实施例中相机服务向目标应用上传的预览帧的数量与摄像头上报的预览帧的数量一致，而上传的识别帧的数量小于摄像头上报的识别帧的数量，从而在不影响摄像头采集的图像在扫码界面正常预览的情况下能够减少目标应用所要识别的识别帧数量，以减少为识别帧创建的线程的数量、识别过程对内存的占用空间以及识别时间，因此提高了识别效率。
在一些实施例中，相机服务检测识别帧是否被标记包括：相机服务检测识别帧的状态是否为error，若识别帧的状态为error，表示识别帧被标记。那么将不含有error的识别帧返回给目标应用。
在另一些实施例中，相机服务检测识别帧是否被标记包括：相机服务检测识别帧识别被标记“N(或者0)”，若识别帧被标记“N(或者0)”，则表示识别帧被标记。若识别帧未被标记“N(或者0)”，如识别帧被标记为“Y(或者1)”，则表示识别帧未被标记。那么相机服务将不含有标记“N(或者0)”的识别帧返回给目标应用。
相机服务返回预览帧和不含有标记的识别帧之后，目标应用接收预览帧以及不含有标记的识别帧。然后目标应用执行S609，以实现对二维码的识别。
S609、目标应用在扫码界面显示预览帧，并对识别帧进行识别，确定识别结果。
在S609中，若当前帧的识别结果为未识别出二维码，则目标应用继续接收下一帧的预览帧及下一帧的不含有标记的识别帧，并对下一帧的识别帧进行识别，以确定是否识别出二维码。当目标应用从识别帧中识别出二维码，目标应用向相机服务发送关闭摄像头的请求。示例性的，目标应用调用相机接口(如close Camera接口)，以请求关闭摄像头，以向相机服务发送关闭摄像头的请求。相机服务接收关闭摄像头的请求后，向下层传输关闭摄像头的请求，关闭相应的摄像头，并最终实现停止在扫码界 面中的预览。
综上所述，采用本实施例的方法在黑暗环境下扫码时，相机驱动可通过对黑暗环境下采集的识别帧设置丢帧标志，从而实现在相机服务对识别帧进行丢弃，这样相机服务上报给目标应用的识别帧中便不包括黑暗环境下识别的无效识别帧。如此，可以避免目标应用对无效帧的识别而导致浪费设备算力资源和降低扫码速度的问题，因此提高了扫码效率。
上述图6所示的实施例，只对环境光亮度大于或等于阈值的识别帧进行识别。而实际扫码场景下，当用户在黑暗环境下扫码时，相机会自动曝光和自动对焦，使所采集的图像的亮度和清晰度趋于正常。在摄像头自动曝光和自动对焦的过程中，可能出现环境光亮度值大于或等于阈值，但目标应用仍无法从识别帧中识别出二维码的情况。但针对该情况，上述实施例将识别帧上报给目标应用进行识别，导致目标应用对无效帧做了大量识别，造成设备算力资源浪费。
基于此，本申请实施例还提供了另一种扫码方法，可以在图6所示的实施例的基础上，有效减少目标应用对无效帧的识别，提高扫码速度。
参照图7A，在S606之后，本申请实施例提供的扫码方法可以包括：
S701、相机驱动确定元数据帧的环境光亮度大于或等于阈值，从元数据帧中获取自动曝光的第一收敛结果和自动对焦的第二收敛结果。
基于前文可知，摄像头通过自动曝光使所摄物体亮度正常。自动曝光的过程中，相机的光圈、快门、感光度(international standards organization，ISO)等参数是逐步调整的，也就是说相机的光圈、快门、ISO等参数存在一定的调整时间，被称为调整过程。在调整过程中相机的自动曝光处于收敛状态，而当调整结束时，相机的光圈、快门、ISO等参数维持不变，因此自动曝光处于收敛结束状态。
同样的，摄像头通过自动对焦使所摄物体清晰度正常。相机曝光过程中，物体亮度由暗转亮，那么相机通过自动对焦实现相机拍摄的图像由模糊转清晰。在相机调整焦距的过程中，焦距还未达到目标值，因此自动对焦处于收敛状态。而当相机结束调整焦距，焦距达到目标值，因此自动对焦处于收敛结束状态。
基于此，元数据帧还用于提供摄像头参数(又称相机参数)，以便于相机驱动确定自动曝光的第一收敛结果和自动对焦的第二收敛结果。
在一些实施例中，相机参数包括：AEdone、AFdone。其中，AE又称自动曝光(auto exposure)，AEdone表示自动曝光处于收敛结束状态。AF又称自动对焦(auto focus)，AFdone表示自动对焦处于收敛结束状态。即该实施例中，相机驱动确定元数据帧的环境光亮度大于或等于阈值，从元数据帧中获取AEdone和AFdone，若获取到AEdone，表示自动曝光处于收敛结束状态。若获取到AFdone，表示为自动对焦处于对焦结束状态。因此相机驱动可以通过是否识别到AEdone、AFdone，以确定自动曝光的第一收敛结果和自动对焦的第二收敛结果。
在另一些实施例中，相机参数包括相机的光圈、快门、ISO、焦距等参数。相机驱动获取至少两帧元数据帧的相机参数，通过判断相机参数是否变化，从而确定自动曝光或自动对焦是否处于收敛状态，即确定自动曝光的第一收敛结果和自动对焦的第二收敛结果。
S702、相机驱动根据第一收敛结果和第二收敛结果确定自动曝光和/或自动对焦处于收敛状态，根据丢帧策略为与元数据帧同一时间戳采集的识别帧设置丢帧标志。
其中，关于丢帧标志的设置方式，可参考上述S606中对丢帧标志的介绍，在此不做累赘说明。
下面着重介绍本实施例的丢帧策略。
在摄像头自动曝光和自动对焦的过程中，可能出现环境光亮度值大于或等于阈值，但目标应用仍无法从识别帧中识别出二维码的情况。因此对自动曝光和自动对焦的过程中采集的识别帧进行识别，会导致不仅无法从无效帧无法中识别出二维码，还占用线程资源的问题。基于此，对自动曝光和自动对焦过程中采集的识别帧进行丢弃，以便于减少为识别帧创建的线程的数量、识别过程对内存的占用空间以及识别时间，因此提高了识别效率。
应理解，对摄像头自动曝光和自动对焦过程中的任一或多帧识别帧进行丢弃，都能实现减少为识别帧创建的线程的数量的有益效果。在本实施例中，提出三种丢帧策略。
其一，丢帧策略包括：丢弃所有在自动曝光和/或自动对焦处于收敛状态下采集的识别帧。
那么根据丢帧策略，相机驱动为自动曝光和/或自动对焦处于收敛状态的识别帧设置丢帧标志。
其二，丢帧策略包括：以T为一个丢帧周期，在每一丢帧周期内丢弃N帧识别帧。
那么根据丢帧策略，相机驱动为丢帧周期T内的N帧识别帧设置丢帧标志。
示例性的，T为3帧，N为2，即每三帧识别帧中丢弃两帧识别帧，只保留一帧识别帧用于目标应用识别。
例如，丢帧策略包括：每间隔一帧识别帧，丢弃连续两帧识别帧，并以此循环。
其三，丢帧策略包括：在自动曝光和/或自动对焦处于收敛状态下采集的前M帧识别帧。例如，M为3、6、10、15、20等。
那么根据丢帧策略，相机驱动为自动曝光和/或自动对焦处于收敛状态下的前M帧识别帧设置丢帧标志。
S703、相机驱动根据第一收敛结果和第二收敛结果确定自动曝光和自动对焦都处于收敛结束状态，结束根据丢帧策略为识别帧设置丢帧标志。
其中，自动曝光和自动对焦都处于收敛结束状态可以是元数据帧中的相机参数包括AEdone和AFdone。或者，自动曝光和自动对焦都处于收敛结束状态可以是元数据帧中的相机参数相比于前一帧保持不变。
应理解，在扫码过程中，相机驱动可以多次结束根据丢帧策略为识别帧设置丢帧标志的操作。每次结束的操作都可称为“相机驱动结束根据丢帧策略为识别帧设置丢帧标志”。
在一些实施例中，相机驱动结束根据丢帧策略为识别帧设置丢帧标志，以使相机驱动不再为后续接收到的所有识别帧设置丢帧标志。那么相机驱动只进行一次结束根据丢帧策略为识别帧设置丢帧标志的操作。
在另一些实施例中，相机驱动结束根据丢帧策略为识别帧设置丢帧标志后，相机 驱动不再为后续接收到的识别帧设置丢帧标志。直至识别帧满足预设条件，相机驱动继续为后续接收到的识别帧设置丢帧标志，例如预设条件包括：Luxindex小于阈值。示例性的，用户在黑暗环境下扫码，当用户启动手机的手电筒功能，Luxindex大于或等于阈值，摄像头自动曝光和自动对焦，在自动曝光和自动对焦过程中，相机驱动根据丢帧策略为识别帧设置丢帧标志。当自动曝光和自动对焦都处于收敛结束状态，相机驱动结束根据丢帧策略为识别帧设置丢帧标志。用户误触手机导致手电筒功能被关闭，Luxindex上升至大于预设值，则相机驱动继续为Luxindex大于预设值对应的识别帧设置丢帧标志。也就是说，在该场景下，相机驱动执行的丢帧过程为：先丢弃所有Luxindex大于预设值的识别帧；然后根据丢帧策略(如每间隔一帧识别帧，丢弃连续两帧识别帧)丢弃识别帧；再在自动曝光和自动对焦完成后，结束丢帧；最后在Luxindex上升至大于预设值时，继续丢弃所有Luxindex大于预设值的识别帧。
相机驱动为识别帧设置丢帧标志后，将识别帧和预览帧上传给相机服务，以便于相机服务根据丢帧标志执行丢帧操作。其中，相机服务执行的丢帧操作可参见前文S607的说明。基于此，手机执行S703之后，执行上述S607。
综上所述，采用本实施例的方法，相机驱动不仅可以为环境光亮度小于阈值所对应的识别帧设置丢帧标志，还可以为环境光亮度大于或等于阈值时，收敛状态下的识别帧设置丢帧标志，可以减少相机服务上传给目标应用的识别帧的数量，从而减少目标应用为识别帧创建的线程的数量、识别过程对内存的占用空间以及识别时间，因此提高了识别效率。
示例性的，如图7B所示，在采用本实施例的方法后，在黑暗环境下进行扫码，摄像头在屏幕亮度暗(即Luxindex大)的场景下采集的多帧识别帧全部被丢弃；当屏幕亮度变亮(即Luxindex小)时，相机自动曝光和自动对焦的过程中，相机驱动间隔一帧标识两帧需丢弃的识别帧。当相机结束自动曝光和自动对焦，相机驱动不再标识识别帧。这样目标应用所识别的识别帧的数量远小于摄像头采集的识别帧的数量，因此识别速度快、用户所能感知到的扫码过程的耗时短，用户的扫码体验好。
在前述的图6所示的实施例中，相机启动后，元数据帧不需要被硬件做任何处理，而预览帧和识别帧是需要经过硬件处理的，如硬件需要对图像进行裁剪、美颜等处理，因此元数据帧比识别帧多三帧。
基于此，相机驱动获取元数据帧后，丢弃前三帧数据帧，从第四帧数据帧开始识别每帧元数据帧的环境光亮度。而后，相机驱动根据第四帧元数据帧中的环境光亮度确定第一帧识别帧的元数据信息。也就是说，该实施例中，第i帧识别帧与第i+3帧元数据帧相对应。相机驱动根据对应关系获取识别帧所对应的元数据信息，以便于确定环境光亮度。
示例性的，如图8所示，在采用本实施例的方法后，前三帧元数据帧被丢弃，从第四帧元数据帧开始获取元数据信息如Luxindex，根据元数据信息确定环境光亮度。前六帧识别帧的环境光亮度小于阈值，因此识别帧被丢弃。自第七帧起，基于元数据帧检测到相机处于AE收敛状态或AF收敛状态，因此间隔丢帧。当基于元数据帧检测到相机结束AE收敛和AF收敛，停止间隔丢帧，将每一帧识别帧都上传给目标应用。
在另一场景中，手机扫码时摄像头采集到的图像质量不好，如图像中识别码尺寸 太小。或者扫码姿势倾斜造成图像中识别码区域变成非矩形的区域，影响图像质量。
基于此，本申请实施例还提供了另一种扫码方法，可以在图6的实施例的基础上，有效减少图像质量不好的情况下扫码时的识别时间。
参照图9，在S606之后，本申请实施例提供的扫码方法可以包括：
S901、相机驱动确定元数据帧的环境光亮度大于或等于阈值，根据丢帧策略为与元数据帧同一时间戳采集的识别帧设置丢帧标志。
其中，丢帧策略可参考上述S702中的丢帧策略的介绍，在此不做累赘说明。
在一些实施例中，S901中的丢帧策略与S702中的丢帧策略为两种不同策略。例如，S702中采用间隔丢帧的方式为识别帧设置丢帧标志，在S901中为所有识别帧设置丢帧标志。
在另一些实施例中，S901中的丢帧策略与S702中的丢帧策略为两种相同策略。例如，S702和S901都采用间隔丢帧的方式为识别帧设置丢帧标志。
S902、目标应用响应于在扫码界面执行的图像放大操作，向相机驱动发送结束丢帧的请求。
在S902之前，目标应用可接收用户在扫码界面执行的图像放大操作。其中，图像放大操作可以是双击操作或者双指外划操作。
在一些实施例中，图像放大操作包括用户双击扫码界面的扫码框内图像。例如，双击扫码框内图像后，图像被放大显示在扫码框内。
在另一些实施例中，图像放大操作包括用户在扫码界面执行双指外划操作。该实施例中，用户的拇指和食指同时接触触摸屏，形成两个触摸点，其中一个触摸点向第一方向移动，另一触摸点向第二方向移动，第一方向和第二方向相反。则目标应用根据触摸点的触摸轨迹确定用户执行的是双指外划操作，那么目标应用向相机驱动发送结束丢帧的请求。
该步骤中，用户执行图像放大操作，以缩短拍摄距离，使所摄物体放大，那么识别帧和预览帧中的物体被放大，从而提高图像质量。因此，目标应用响应于图像放大操作，向相机驱动发送结束丢帧的请求，以结束在图像质量好的情况下对识别帧进行丢帧处理，从而避免丢弃有效识别帧造成的扫码时间长的问题。
在一些实施例中，在接收用户在扫码界面执行的图像放大操作之前，相机驱动可确定识别帧是否包括识别码，在识别帧包括识别码的情况下，相机驱动确认用户执行的图像放大操作用于放大识别码的尺寸。
在一些实施例中，S902可以被S902a替代。
S902a、目标应用响应于在扫码界面执行的拖动图像的操作，向相机驱动发送结束丢帧的请求。
在S902a之前，扫码界面显示的预览帧图像不包括识别码，执行S902a后，扫码界面显示的预览帧图像包括识别码。
具体的，在扫码过程中，当扫码界面不包括识别码时，用户可以调整相机的拍摄角度以实现识别码在扫码界面中。此外，用户也可以通过拖动图像来调整识别码在扫码界面的位置。
基于此，当目标应用检测到拖动图像的操作，表示在该操作后扫码界面中包括识 别码的可能性较大，因此对摄像头在该操作后采集的连续识别帧进行识别，以避免漏检识别帧造成扫码时间过长。
在另一些实施例中，S902可以被S902b替代。
S902b、目标应用响应于在扫码界面执行的图像缩小操作，向相机驱动发送结束丢帧的请求。
在S902b之前，扫码界面显示的预览帧图像不包括完整的识别码，执行S902b之后，由于扫码界面中的图像被缩小，因此识别码完整的显示在扫码界面内。那么当目标应用检测图像缩小操作，表示在该操作后识别码可被完整显示在扫码界面内，因此对摄像头在该操作后采集的连续识别帧进行识别，以避免漏检识别帧造成扫码时间过长。
S903、相机驱动响应于接收到结束丢帧的请求，结束根据丢帧策略为识别帧设置丢帧标志。
在S903之前，相机驱动可接收来自目标应用的结束丢帧的请求。
本实施例中，相机驱动接收结束丢帧的请求后，则可结束为识别帧设置丢帧标志，那么相机驱动上传给相机服务的识别帧为未设置丢帧标志的识别帧。相机服务接收到识别帧后，上传给目标应用进行识别。其中，相机服务执行的操作可参见前文S607的说明。基于此，手机执行S903之后，执行上述S607。
综上所述，采用本实施例的方法，相机驱动可以为图像质量不好时采集的识别帧设置丢帧标志，以减少相机服务上传给目标应用的识别帧的数量，并且在图像质量好情况下结束对识别帧的丢帧操作，从而避免因丢弃有效识别帧造成的扫码时间长的问题。
示例性的，如图10所示，在采用本实施例的方法后，当相机驱动获取识别帧，相机驱动便每间隔一帧便为连续两针识别帧设置丢帧标志。当接收到用户的图像放大操作，相机驱动不再丢弃识别帧。这样在图像质量不好的情况下，目标应用所识别的识别帧的数量远小于摄像头采集的识别帧的数量，因此识别速度快、用户所能感知到的扫码过程的耗时短，用户的扫码体验好。
前文实施例中，主要以光照环境和图像质量的角度说明了扫码过程中的丢帧操作。而在实际实施时，不同应用所采用的扫码识别原理是不同的，如前述介绍的第二类应用，只对每个线程识别结束后的最新一帧识别帧进行识别，相当于减少了应用所需识别的识别帧数量。因此针对该类应用，本申请实施例还提供一种扫码方法，可参见图11，可以通过S1101-S1102，S603-S606，S1103以及S607-S608来根据应用类型确定丢帧策略。
S1101、目标应用响应于用户打开相机的操作，显示扫码界面并向相机服务发送启动摄像头的请求以及应用标识。
区别于S601，该步骤中目标应用还向相机服务上传应用标识，以区分应用程序的类型。例如，应用标识用于区分上述第一类应用和第二类应用。
S1102、相机服务响应于接收到启动摄像头的请求，向相机驱动发送调用相机驱动的请求以及应用标识。
关于该步骤的介绍，可参考前文S602的说明。
除此之外，相机服务将目标应用发送的应用标识也发送给相机驱动，以使得相机驱动根据应用标识确定是否为连续多帧识别帧设置丢帧标志。
手机执行S1102之后，可以执行上文中的S602-S606，从而为环境光亮度小于阈值的识别帧设置丢帧标志，然后手机执行S1103。
S1103、相机驱动响应于获取的应用标识为非目标标识，在确定元数据帧的环境光亮度大于或等于阈值时，结束为连续多帧识别帧设置丢帧标志。
其中，非目标标识用于指示前文的第二类应用。那么目标表示用于指示前文的第一类应用。
其中，非目标标识又称预设标识，非目标标识可以预先设置，用于区分不需要间隔丢帧的目标应用。当应用标识为非目标标识时，表示目标应用只对每个线程识别结束后的最新一帧识别帧进行识别，相当于减少了应用所需识别的识别帧数量，因此针对第二类应用，相机驱动结束为连续多帧识别帧设置丢帧标志，以避免丢弃过多识别帧造成的扫码速度变慢的问题，从而可以在不降低扫码速度的情况下提高目标应用对黑暗环境下识别码的识别速度。然后相机驱动将识别帧和预览帧上报给相机驱动，以便于相机驱动根据识别帧的丢帧标志进行丢帧处理。基于此，在S1103之后，手机执行前文的S607-S609，以完成扫码过程。
示例性的，如图12所示，在采用本实施例的方法后，在黑暗环境下进行扫码，摄像头在屏幕亮度暗(即Luxindex大)的场景下采集的多帧识别帧全部被丢弃；当屏幕亮度变亮(即Luxindex小)时，相机驱动确定目标应用是否为非目标应用，若为非目标应用，则相机驱动将后续的所有识别帧都直接上报给相机服务，并由相机服务将识别帧上报给目标应用进行识别。这样在用户采用第二类应用进行扫码时，相机驱动不会丢弃过多识别帧，从而可以在不降低扫码速度的情况下提高目标应用对黑暗环境下识别码的识别速度。因此识别速度快、用户所能感知到的扫码过程的耗时短，用户的扫码体验好。
在一些实施例中，应用标识还可区分第三类应用和第四类应用，其中第三类应用中，以预览流作为实时预览画面及识别识别码的视频流。也就是说，第三类应用只下发一路流配置，预览流既用于在扫码界面上显示，又用于获取识别码的信息。第四类应用下发两路流配置，以预览流作为实时预览画面的视频流，以识别流作为识别识别码的识别流。
目标应用向扫码服务发送应用标识，从而使底层的相机服务确定目标应用是否只下发一路流配置。若确定只下发一路流配置，则相机驱动为预览流设置丢帧标志。相机服务将复制一路预览流，将一路预览流上传给目标应用，并将另一路预览流中设置丢帧标志的预览帧丢弃，然后上传给目标应用。这样本实施例便可避免因目标应用只下发一路流配置造成的对预览流进行丢帧后，扫码界面无法正常显示预览帧的问题，从而在不影响摄像头采集的图像在扫码界面正常预览的情况下能够减少目标应用所要识别的预览帧数量。
至此，为方便对前文中各个实施例的理解，下面以一个完整的流程示例来说明本申请方案及其效果。图13为本申请一个具体示例中的扫码方法的流程图。
参照图13，本申请实施例提供的扫码方法可以包括：
S1301、启动相机服务：在手机开机时，会启动相机服务，或者，在相机服务启动后，会自动重启相机服务。
S1302、打开相机：其中，打开相机应用、进入三方应用中使用摄像头的功能、切换拍摄模式都可以触发打开相机并进入扫码功能，应理解，打开相机是指打开相机并进入扫码界面。
S1303、连接相机，也可以理解为请求启动摄像头：目标程序在接收到用户打开相机的操作后，则可以向相机服务发送启动摄像头的请求以及应用标识，其中，应用标识用于区分目标应用的类型。
S1304、采集元数据帧、识别帧、预览帧；相机驱动在接收到调用相机驱动的请求后，则可控制摄像头开启。摄像头开启后，便可持续采集raw数据，对raw数据处理得到YUV数据，并将YUV数据分为两路视频流，一路视频流提供识别帧，一路视频流提供预览帧，同时上报YUV数据的元数据帧。
S1305、识别环境光亮度；相机驱动可以从元数据帧中获取光照强度，从而确定识别帧和预览帧的光照环境。
S1306、是否小于阈值；元数据帧的环境光亮度小于阈值，表示识别帧和预览帧的光照环境为黑暗场景，那么相机驱动为与元数据帧同一时间戳采集的识别帧设置丢帧标志，以指示上层的相机服务丢弃该识别帧。环境光亮度大于或等于阈值，表示光照环境明亮，那么相机驱动不为元数据帧所对应的识别帧设置丢帧标志，直接将识别帧上传给相机服务，以指示相机服务将该环境光亮度下采集到的识别帧上传给目标应用进行识别处理。
S1307、是否为目标标识；应用标识为目标标识，表示目标应用为前文的第一类应用，那么需要对光照环境明亮之后的识别帧进行丢帧处理。而应用标识为非目标标识，表示目标应用为前文的第二类应用，对光照环境明亮之后的识别帧进行持续地识别处理。
S1308、是否为收敛状态；AE或AF为收敛状态，表示曝光度或焦距在调整过程中。AE和AF不为收敛状态，表示AEdone以及AFdone。
S1308a、是否是图像放大操作；当接收到用户的图像放大操作，相机驱动不再丢弃识别帧。
S1309、间隔丢帧；对AE收敛状态和AF收敛状态下摄像头采集的识别帧进行间隔丢帧。
S1310、不处理；不作间隔丢帧，将每一帧识别帧都上传给目标应用。
S1311、上报识别帧和预览帧；其中预览帧用于在扫码界面上显示，识别帧用于获取识别码的信息。
S1312、显示预览帧；其中，丢弃识别帧不会影响预览帧在扫码界面正常显示，因此不影响摄像头采集的图像在扫码界面正常预览。
S1313、确定识别结果；在应用程序层对框架层上报的识别帧进行识别，并跳转至识别结果对应的界面。
由此可见，上述图13所示的完整示例，通过在抽象层对识别帧进行丢帧处理，可以减少应用程序层识别的无效识别帧的数量，从而减少为识别帧创建的线程的数量、 识别过程对内存的占用空间以及识别时间，因此提高了识别效率。
本申请另一些实施例提供了一种电子设备，该电子设备可以包括：上述显示屏(如触摸屏)、存储器和一个或多个处理器。该显示屏、存储器和处理器耦合。该存储器用于存储计算机程序代码，该计算机程序代码包括计算机指令。当处理器执行计算机指令时，电子设备可执行上述方法实施例中手机执行的各个功能或者步骤。该电子设备的结构可以参考图2所示的手机200的结构。
本申请实施例还提供一种计算机可读存储介质，该计算机存储介质包括计算机指令，当所述计算机指令在上述电子设备上运行时，使得该电子设备执行上述方法实施例中手机执行的各个功能或者步骤。
本申请实施例还提供一种计算机程序产品，当所述计算机程序产品在计算机上运行时，使得所述计算机执行上述方法实施例中手机执行的各个功能或者步骤。
通过以上实施方式的描述，所属领域的技术人员可以清楚地了解到，为描述的方便和简洁，仅以上述各功能模块的划分进行举例说明，实际应用中，可以根据需要而将上述功能分配由不同的功能模块完成，即将装置的内部结构划分成不同的功能模块，以完成以上描述的全部或者部分功能。
在本申请所提供的几个实施例中，应该理解到，所揭露的装置和方法，可以通过其它的方式实现。例如，以上所描述的装置实施例仅仅是示意性的，例如，所述模块或单元的划分，仅仅为一种逻辑功能划分，实际实现时可以有另外的划分方式，例如多个单元或组件可以结合或者可以集成到另一个装置，或一些特征可以忽略，或不执行。另一点，所显示或讨论的相互之间的耦合或直接耦合或通信连接可以是通过一些接口，装置或单元的间接耦合或通信连接，可以是电性，机械或其它的形式。
所述作为分离部件说明的单元可以是或者也可以不是物理上分开的，作为单元显示的部件可以是一个物理单元或多个物理单元，即可以位于一个地方，或者也可以分布到多个不同地方。可以根据实际的需要选择其中的部分或者全部单元来实现本实施例方案的目的。
另外，在本申请各个实施例中的各功能单元可以集成在一个处理单元中，也可以是各个单元单独物理存在，也可以两个或两个以上单元集成在一个单元中。上述集成的单元既可以采用硬件的形式实现，也可以采用软件功能单元的形式实现。
所述集成的单元如果以软件功能单元的形式实现并作为独立的产品销售或使用时，可以存储在一个可读取存储介质中。基于这样的理解，本申请实施例的技术方案本质上或者说对现有技术做出贡献的部分或者该技术方案的全部或部分可以以软件产品的形式体现出来，该软件产品存储在一个存储介质中，包括若干指令用以使得一个设备(可以是单片机，芯片等)或处理器(processor)执行本申请各个实施例所述方法的全部或部分步骤。而前述的存储介质包括：U盘、移动硬盘、只读存储器(read only memory，ROM)、随机存取存储器(random access memory，RAM)、磁碟或者光盘等各种可以存储程序代码的介质。
以上内容，仅为本申请的具体实施方式，但本申请的保护范围并不局限于此，任何在本申请揭露的技术范围内的变化或替换，都应涵盖在本申请的保护范围之内。因此，本申请的保护范围应以所述权利要求的保护范围为准。

Claims (12)

  1. 一种扫码方法，其特征在于，应用于电子设备，所述电子设备包括摄像头，所述方法包括：
    显示扫码界面；
    获取识别帧图像和预览帧图像，所述识别帧图像和所述预览帧图像根据所述摄像头采集的扫码图像处理获得；
    在所述扫码界面显示所述预览帧图像；
    若所述识别帧图像的环境光亮度大于或等于阈值，则对所述识别帧图像进行扫码识别，确定扫码识别结果。
  2. 根据权利要求1所述的方法，其特征在于，所述若所述识别帧图像的环境光亮度大于或等于阈值，则对所述识别帧图像进行扫码识别，包括：
    若所述识别帧图像的环境光亮度大于或等于阈值，且所述摄像头的曝光收敛操作和对焦收敛操作完成，则对所述识别帧图像进行扫码识别；
    所述方法还包括：
    若所述识别帧图像的所述环境光亮度大于或等于所述阈值，且检测到所述摄像头的曝光收敛操作或对焦收敛操作，则删除所述摄像头在所述曝光收敛操作或所述对焦收敛操作执行过程中根据所述扫码图像获得的识别帧图像，其中，被删除的所述识别帧图像不用于扫码识别。
  3. 根据权利要求1所述的方法，其特征在于，所述若所述识别帧图像的环境光亮度大于或等于阈值，则对所述识别帧图像进行扫码识别，包括：
    若所述识别帧图像的所述环境光亮度大于或等于阈值、所述识别帧图像包括识别码且所述识别码的尺寸增大，对尺寸增大后的所述识别码所对应的所述识别帧图像进行扫码识别。
  4. 根据权利要求1或3所述的方法，其特征在于，所述若所述识别帧图像的环境光亮度大于或等于阈值，则对所述识别帧图像进行扫码识别，包括：
    若所述识别帧图像的所述环境光亮度大于或等于阈值，且检测到在所述扫码界面执行的图像放大操作，则对所述图像放大操作执行后根据所述扫码图像获得的所述识别帧图像进行扫码识别。
  5. 根据权利要求4所述的方法，其特征在于，在所述则对所述图像放大操作执行后根据所述扫码图像获得的所述识别帧图像进行扫码识别之前，所述方法还包括：
    确定所述识别帧图像包括识别码。
  6. 根据权利要求1-5中任一项所述的方法，其特征在于，在所述显示扫码界面之前，所述方法还包括：
    启动目标应用；
    所述若所述识别帧图像的环境光亮度大于或等于阈值，则对所述识别帧图像进行扫码识别，包括：
    若所述识别帧图像的环境光亮度大于或等于阈值且所述目标应用的应用标识为预设标识，则对所述识别帧图像进行扫码识别。
  7. 根据权利要求1所述的方法，其特征在于，所述若所述识别帧图像的环境光亮 度大于或等于阈值，则对所述识别帧图像进行扫码识别，包括：
    若所述识别帧图像的环境光亮度大于或等于阈值、所述识别帧图像未包括识别码，且检测在所述扫码界面对所述预览帧图像的拖动操作，则对与所述预览帧图像为同一扫码图像处理得到的所述识别帧图像进行扫码识别。
  8. 根据权利要求1-7中任一项所述的方法，其特征在于，所述方法还包括：
    获取元数据帧，其中，所述元数据帧包括用于处理得到所述识别帧图像的扫码图像的采集参数；
    从所述采集参数中获取光照强度参数，根据所述光照强度参数确定所述识别帧图像的所述环境光亮度。
  9. 根据权利要求1-8中任一项所述的方法，其特征在于，所述方法还包括：若所述识别帧图像的环境光亮度小于阈值，
    或在所述识别帧图像的环境光亮度大于或等于阈值的情况下检测到所述摄像头的曝光收敛操作或对焦收敛操作，
    或在所述识别帧图像的环境光亮度大于或等于阈值的情况下所述识别帧图像中识别码的尺寸小于或等于预设尺寸，
    或在所述识别帧图像的环境光亮度大于或等于阈值的情况下所述识别帧图像未包括识别码，
    则为所述识别帧图像设置丢帧标志；
    所述若所述识别帧图像的环境光亮度大于或等于阈值，则对所述识别帧图像进行扫码识别，包括：
    对未包含所述丢帧标志的所述识别帧图像进行扫码识别，其中，未包含所述丢帧标志的所述识别帧图像的环境光亮度大于或等于阈值。
  10. 根据权利要求9所述的方法，其特征在于，所述为所述识别帧图像设置丢帧标志，包括：
    在每一预设丢帧周期内为至少一帧所述识别帧图像设置所述丢帧标志。
  11. 一种电子设备，其特征在于，包括：存储器和一个或多个处理器；所述存储器和所述处理器耦合；所述存储器用于存储计算机程序代码，所述计算机程序代码包括计算机指令；
    当所述处理器执行所述计算机指令时，所述电子设备执行如权利要求1-10中任一项所述的方法。
  12. 一种计算机可读存储介质，其特征在于，包括计算机指令，当所述计算机指令在电子设备上运行时，使得所述电子设备执行如权利要求1-10中任一项所述的方法。
PCT/CN2023/114075 2022-09-15 2023-08-21 一种扫码方法及电子设备 WO2024055817A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202211124961.4A CN116739014A (zh) 2022-09-15 2022-09-15 一种扫码方法及电子设备
CN202211124961.4 2022-09-15

Publications (1)

Publication Number Publication Date
WO2024055817A1 true WO2024055817A1 (zh) 2024-03-21

Family

ID=87908544

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2023/114075 WO2024055817A1 (zh) 2022-09-15 2023-08-21 一种扫码方法及电子设备

Country Status (2)

Country Link
CN (1) CN116739014A (zh)
WO (1) WO2024055817A1 (zh)

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20110155808A1 (en) * 2009-12-31 2011-06-30 Samsung Electrônica da Amazônia Ltda. System and automatic method for capture, reading and decoding barcode images for portable devices having digital cameras.
CN110830730A (zh) * 2018-08-07 2020-02-21 三星电子株式会社 电子装置中用于生成移动图像数据的设备和方法
WO2021167230A1 (ko) * 2020-02-19 2021-08-26 삼성전자(주) 전자장치 및 그 제어방법
CN113408314A (zh) * 2020-03-16 2021-09-17 华为技术有限公司 一种条码识别方法以及相关设备
CN113919382A (zh) * 2021-04-29 2022-01-11 荣耀终端有限公司 一种扫码的方法及装置
CN113935352A (zh) * 2020-06-29 2022-01-14 比亚迪股份有限公司 乘车二维码的扫描解码方法、装置、设备及存储介质

Family Cites Families (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105243350A (zh) * 2014-05-28 2016-01-13 腾讯科技（深圳）有限公司 扫码方法及装置
JP2019091209A (ja) * 2017-11-14 2019-06-13 株式会社キーエンス 光学情報読取装置
CN108399349B (zh) * 2018-03-22 2020-11-10 腾讯科技（深圳）有限公司 图像识别方法及装置
CN108763999B (zh) * 2018-04-17 2021-08-17 维沃移动通信有限公司 条形码识别方法以及终端设备
CN110363036B (zh) * 2019-07-17 2021-02-26 珠海格力电器股份有限公司 基于线控器的扫码方法及装置、扫码系统
KR20210049682A (ko) * 2019-10-25 2021-05-06 삼성전자주식회사 전자 장치의 슬로우 모션 모드에서 프리뷰 프레임들로 피크 강도 프레임들을 유지하는 방법
CN112333397B (zh) * 2020-03-26 2022-05-13 华为技术有限公司 图像处理方法和电子设备
CN113496133B (zh) * 2020-04-07 2024-03-29 深圳爱根斯通科技有限公司 二维码识别方法、装置、电子设备及存储介质
CN112749575B (zh) * 2020-12-31 2023-02-17 支付宝(杭州)信息技术有限公司 扫码设备及扫码设备的工作模式识别方法、装置、设备

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20110155808A1 (en) * 2009-12-31 2011-06-30 Samsung Electrônica da Amazônia Ltda. System and automatic method for capture, reading and decoding barcode images for portable devices having digital cameras.
CN110830730A (zh) * 2018-08-07 2020-02-21 三星电子株式会社 电子装置中用于生成移动图像数据的设备和方法
WO2021167230A1 (ko) * 2020-02-19 2021-08-26 삼성전자(주) 전자장치 및 그 제어방법
CN113408314A (zh) * 2020-03-16 2021-09-17 华为技术有限公司 一种条码识别方法以及相关设备
CN113935352A (zh) * 2020-06-29 2022-01-14 比亚迪股份有限公司 乘车二维码的扫描解码方法、装置、设备及存储介质
CN113919382A (zh) * 2021-04-29 2022-01-11 荣耀终端有限公司 一种扫码的方法及装置

Also Published As

Publication number Publication date
CN116739014A (zh) 2023-09-12

Similar Documents

Publication Publication Date Title
WO2021147482A1 (zh) 一种长焦拍摄的方法及电子设备
WO2021244455A1 (zh) 一种图像内容的去除方法及相关装置
WO2021129198A1 (zh) 一种长焦场景下的拍摄方法及终端
US20220417416A1 (en) Photographing method in telephoto scenario and mobile terminal
WO2021223500A1 (zh) 一种拍摄方法及设备
WO2022022726A1 (zh) 一种拍摄方法及设备
CN113630558B (zh) 一种摄像曝光方法及电子设备
CN115484403B (zh) 录像方法和相关装置
WO2021185374A1 (zh) 一种拍摄图像的方法及电子设备
CN113918766B (zh) 视频缩略图的显示方法、设备和存储介质
EP4109879A1 (en) Image color retention method and device
EP4060603A1 (en) Image processing method and related apparatus
WO2023035921A1 (zh) 一种录像中抓拍图像的方法及电子设备
WO2024055817A1 (zh) 一种扫码方法及电子设备
WO2023011302A1 (zh) 拍摄方法及相关装置
WO2023231696A1 (zh) 一种拍摄方法及相关设备
WO2023160230A1 (zh) 一种拍摄方法及相关设备
WO2022228010A1 (zh) 一种生成封面的方法及电子设备
CN117119285A (zh) 一种拍摄方法
CN115883958A (zh) 一种人像拍摄方法
CN117560574A (zh) 一种拍摄方法、电子设备和可读存储介质
CN115460343A (zh) 图像处理方法、设备及存储介质