WO2023016207A1

WO2023016207A1 - Method for focusing on specified area by means of ai action recognition, and terminal and storage medium

Info

Publication number: WO2023016207A1
Application number: PCT/CN2022/106696
Authority: WO
Inventors: 张虹
Original assignee: 惠州Tcl云创科技有限公司
Priority date: 2021-08-12
Filing date: 2022-07-20
Publication date: 2023-02-16
Also published as: CN113810599A

Abstract

Provided in the present application are a method for focusing on a specified area by means of AI action recognition, and a terminal and a storage medium. The method comprises: acquiring a preview image; performing recognition on the preview image, so as to obtain a target in the preview image; and when the target is a display item, performing focusing processing on the area where the target is located, so as to obtain an updated preview image. By means of the present application, a product can be better displayed, and the goal of rapidly focusing a picture on a product is achieved, thereby achieving the effect of presenting the product in a clear state in the picture.

Description

Method, terminal and storage medium for focusing on designated area by using AI action recognition

This application claims the priority of a Chinese patent application with an application date of August 12, 2021, an application number of 2021109225369, and an invention titled "A Method for Using AI Action Recognition to Focus on Specified Areas, a Mobile Terminal, and a Storage Medium". The entire contents are incorporated by reference in this application.

technical field

The present application relates to the field of cameras, and in particular to a method, a terminal and a storage medium for focusing on a designated area by using AI action recognition.

Background technique

At present, live broadcasting is very popular, and mobile phone products that support front-facing camera focusing have begun to appear on the market. When the anchor shows the product in hand to the user, if the item is small or slender, it is often difficult for the camera to focus on the product. The anchor needs to get closer to make the product occupy a larger proportion of the screen, or on the same plane as the display object Only by placing a larger background on the display can the camera focus on the exhibit.

In the prior art, the focus mode generally only has the default full-screen calculation focus, face area focus, and the user manually clicks to focus, and does not deal with the special case of live broadcast display. The plane where the display area is located occupies a small proportion of the screen. Focusing on the face prevents the camera from focusing on the exhibits, and AI motion recognition may misjudgment, resulting in the camera not being able to focus on the exhibits well, resulting in blurred exhibits displayed on the live broadcast screen, affecting users viewing the display product visual effects.

Therefore, the prior art still needs to be improved and developed.

technical problem

When displaying products live, the picture is not easy to focus on the technical problems of the exhibits.

technical solution

The technical scheme of the application is as follows:

The first embodiment of the present application provides a method for using AI action recognition to focus on a designated area, including:

Get the preview image;

Identifying the preview image to obtain a target in the preview image;

When the target is a display item, focus processing is performed on the area where the target is located to obtain an updated preview image.

Optionally, after identifying the preview image and obtaining the target in the preview image, the method further includes:

When the target is not a display item, focus processing is performed on the preview image to obtain an updated preview image.

Optionally, when the target is not a display item, performing focus processing on the preview image to obtain an updated preview image includes:

When the target is not a display item, determine an area of interest in the preview image according to the preview image;

Perform image processing on the region of interest to obtain a focus position of the region of interest; perform focus on the focus position to obtain an updated preview image.

Optionally, the region of interest is the entire image of the preview image.

Optionally, when the target is a display item, the step of performing focus processing on the area where the target is located, and obtaining an updated preview image includes:

When the target is a display item, the area where the target is located is used as an area of interest;

performing image processing on the region of interest to obtain a focus position of the region of interest;

Focus on the focus position to obtain an updated preview image.

Optionally, the displayed item is an item in which the hand is displayed forward in the preview image, and the area where the target is located is an area of the hand in the preview image.

Optionally, the ISP is an image signal processing unit, which performs post-processing on the output signal of the front-end image sensor.

The second embodiment disclosed in this application is a system for focusing on a specified area by using AI motion recognition, which includes:

Acquisition module, obtain preview image;

The AI action recognition module recognizes the preview image to obtain the target in the preview image;

Select the focus module, and when the target is a display item, focus on the area where the target is located to obtain an updated preview image; when the target is not a display item, perform focus processing on the preview image to obtain an updated preview image of .

The third embodiment disclosed in the present application is a mobile terminal, including: a processor, a memory, and a control program for focusing on a specified area stored on the memory and operable on the processor, wherein the control of focusing on the specified area When the program is executed by the processor, the steps of the above-mentioned method for using AI motion recognition to focus on a specified area are realized.

The fourth embodiment of the present application is a computer-readable storage medium. The computer-readable storage medium stores a control program for specifying area focus. When the control program for specifying area focus is executed by a processor, the above-mentioned Steps of a method for focusing on a specified area by using AI motion recognition.

Beneficial effect

The present application provides a method, a mobile terminal, and a storage medium for focusing on a designated area by using AI action recognition, including acquiring a preview image; identifying the preview image to obtain a target in the preview image; Focus on the area to get an updated preview image. The principle of the method provided in this application is to increase the proportion of the background plane of the display item in the screen, so that the camera image can focus on the display item, and by changing the focusing logic, the product can be displayed better, and the screen can be quickly focused on the product. The purpose is to achieve the effect that the product presents a clear state on the screen.

Description of drawings

FIG. 1 is a flow chart of steps in a method for using AI action recognition to focus on a specified area provided by an embodiment of the present application.

Embodiment of this application

This application provides a method, terminal, and storage medium for focusing on a specified area by using AI motion recognition. In order to make the purpose, technical solution, and effect of this application clearer and clearer, the following further describes this application in detail. It should be understood that the specific embodiments described here are only used to explain the present application, not to limit the present application.

The method for focusing on a specified area by using AI action recognition, the terminal and the terminal in the storage medium may include a mobile terminal.

It should be noted that when a component is referred to as being “fixed on” or “disposed on” another component, it may be directly on the other component or indirectly on the other component. When an element is referred to as being "connected to" another element, it can be directly connected to the other element or indirectly connected to the other element.

It should also be noted that the same or similar symbols in the drawings of the embodiments of the present application correspond to the same or similar components; The orientation or positional relationship indicated by "left", "right", etc. is based on the orientation or positional relationship shown in the drawings, and is only for the convenience of describing the application and simplifying the description, rather than indicating or implying that the referred device or element must have a specific Orientation, construction and operation in a specific orientation, therefore, the terms describing the positional relationship in the drawings are only for illustrative purposes, and should not be understood as limitations on this patent. For those of ordinary skill in the art, the Understand the specific meaning of the above terms.

At present, live broadcasting is very popular, and mobile phone products that support front-facing camera focusing have begun to appear on the market. When the anchor shows the product in hand to the user, if the item is small or slender, it is often difficult for the camera to focus on the product. The anchor needs to get closer to make the product occupy a larger proportion of the screen, or on the same plane as the display object Only by placing a larger background on the display can the camera focus on the exhibit. In the existing technology, the focus mode generally only has the default full-screen calculation focus, face area focus, and the user manually clicks to focus, and does not deal with the special situation of live broadcast display. AI motion recognition may misjudgment, resulting in the camera not being able to Focusing on the exhibits well leads to blurred exhibits displayed on the live broadcast screen, which affects the visual effect of users watching the exhibits.

The first embodiment of the present application provides a method of using AI action recognition to focus on a specified area, which is applied to a live broadcast scene, as shown in Figure 1, which includes:

S1: Obtain a preview image;

Display items are in front of the camera as a target, without clicking the image on the display screen to focus, start the camera work, make the portrait within the working range of the camera, the camera is collecting preview images, the display screen is connected to a microprocessor, and the display screen accepts microprocessor processing The controller controls and displays the data output by the microprocessor, processes the collected preview image through the microprocessor and displays the output data on the display screen.

Further, the mobile terminal is specifically a mobile phone, the mobile phone is fixed, and the live broadcaster is facing the front camera of the mobile phone. When the live broadcast starts, the camera is started to work so that the portrait is within the working range of the camera. Process and display the output data on the display screen, the image information acquired by the camera is displayed on the display screen, the display screen is connected to a microprocessor, and the display screen accepts the control of the microprocessor and displays the The data output by the microprocessor, the preview picture displayed on the display screen can include portraits, space backgrounds, exhibits and combined pictures.

S2: Identify the preview image to obtain a target in the preview image;

Further, after identifying the preview image and obtaining the target in the preview image, the method further includes: when the target is not a display item, performing focus processing on the preview image to obtain an updated preview image of .

Optionally, the preview screen is sent to the AI motion recognition module, and the AI motion recognition module judges and collects the motion in the preview screen, and the AI motion recognition judges whether it recognizes the action of the live broadcaster reaching out to the camera in the preview screen, thereby identifying Preview the actions in the screen to choose the focus logic.

It should be noted that AI action recognition detects the actions of people in the screen in real time, and detects actions such as making calls, raising hands, and smoking.

S3: When the target is a display item, focus on the area where the target is located to obtain an updated preview image.

When the target is a display item, performing focus processing on the area where the target is located to obtain an updated preview image includes: when the target is a display item, taking the area where the target is located as an area of interest; Perform image processing on the region of interest to obtain a focus position of the region of interest; perform focus on the focus position to obtain an updated preview image.

Further, the displayed item is an item in which the hand is displayed forward in the preview image, and the area where the target is located is the area of the hand in the preview image.

Further, when the AI motion recognition detects that the hands of the live broadcaster are stretched forward to display the items, the focus logic can be changed. The focus logic can include full-screen calculation focus, face focus, click focus, etc., and narrow the focus ROI to the hand display area . The image signal processing ISP calculates the focus position of the displayed items and focuses on the displayed items.

It should be noted that ROI (region of interest) is the region of interest. In machine vision and image processing, the region to be processed is outlined from the processed image in the form of a box, circle, ellipse, irregular polygon, etc., which is called the region of interest. ROI is used to calculate clarity Focusing area; ISP (Image Signal Processing) is an image signal processing unit that is mainly used to process the output signal of the front-end image sensor.

Optionally, when the target is not a display item, determine the region of interest of the preview image according to the preview image; perform image processing on the region of interest to obtain the focus position of the region of interest; Focusing is performed on the focusing position to obtain an updated preview image. Further, the region of interest is the entire image of the preview image.

Further, when the AI motion recognition does not detect that the hands of the live broadcaster are stretched forward to display the item, the focus logic will be restored, and the focus ROI will be restored to the entire screen. Image signal processing ISP calculates the focus position of the displayed items and focuses on the entire screen.

The above-mentioned method disclosed in the present application will be described in more detail below with reference to FIG. 1 .

K1. The mobile terminal is specifically a mobile phone. Fix the mobile phone, and the live broadcaster is facing the front camera of the mobile phone. When the live broadcast starts, the camera is started to work so that the portrait is within the working range of the camera.

K2, the camera is collecting the preview picture, and it is processed by the microprocessor and the output data is displayed on the display screen. The image information obtained by the camera is displayed on the display screen, and the display screen is connected to the microprocessor. The display screen accepts the control of the microprocessor and displays the data output by the microprocessor. The preview images displayed and collected on the display screen may include portraits, space backgrounds, exhibits and combined images.

K3. Send the preview screen to the AI motion recognition module, and judge and collect the motions in the preview screen through the AI motion recognition module. The action of the live broadcaster reaching out to the camera in the preview screen was not recognized, so the motion in the preview screen was recognized.

K4, judging whether to stretch out the hand to display the item forward.

K411. When the AI motion recognition detects that the live broadcaster’s hand stretches forward to display the item, change the focus logic. The focus logic can include full-screen calculation focus, face focus, click focus, etc., and narrow the focus ROI to the hand display area .

K412. The image signal processing ISP calculates the focus position of the displayed items, and focuses on the displayed items.

K421. When the AI motion recognition does not detect that the live broadcaster’s hand is stretched forward to display the item, the focus logic will be restored, and the focus ROI will be restored to the entire screen.

K422, image signal processing ISP calculates the focus position of the displayed items, and focuses on the entire screen.

K5. Determine whether to end the live broadcast, and return to K2 to repeat the steps if the live broadcast continues.

K6. If K5 judges to end the live broadcast, then end.

The second embodiment disclosed in this application is a system for focusing on a specified area by using AI action recognition, including:

Acquisition module, obtain preview image;

The third embodiment disclosed in this application is a mobile terminal, which includes: a processor, a memory, and a control program for focusing on a designated area stored on the memory and operable on the processor, wherein the focusing on the designated area When the control program is executed by the processor, the above steps of the method for using AI motion recognition to focus on a specified area are realized.

Optionally, the mobile terminal may include a radio frequency (Radio Frequency, RF) circuit, a memory including one or more computer-readable storage media, an input unit, a display unit, a sensor, an audio circuit, a Wireless Fidelity (Wireless Fidelity, Wi‑Fi) module, including a processor with one or more processing cores, and power supplies and other components.

Furthermore, the RF circuit can be used for receiving and sending signals during information sending or talking, especially, after receiving the downlink information of the base station, it will be processed by one or more processors; in addition, the uplink data will be sent to the base station . Typically, RF circuitry includes, but is not limited to, an antenna, at least one amplifier, a tuner, one or more oscillators, a Subscriber Identity Module, SIM) card, transceiver, coupler, low noise amplifier (LowNoise Amplifier, LNA), duplexer, etc. In addition, RF circuits can also communicate with networks and other devices through wireless communication. The wireless communication can use any communication standard or protocol, including but not limited to Global System for Mobile Communications (Global System of Mobile communication, GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (Code Division Multiple Access, CDMA), Wideband Code Division Multiple Access (WCDMA), Long Term Evolution (Long Term Evolution, LTE), e-mail, Short Message Service (Short Messaging Service, SMS), etc.

Optionally, the memory can be used to store software programs and modules, and the processor executes various functional applications and data processing by running the software programs and modules stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application program required by a function (such as a sound playback function, an image playback function, etc.); Data created by the use of the terminal (such as audio data, phonebook, etc.) and the like. In addition, the memory may include high-speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other volatile solid-state storage devices. Correspondingly, the memory may also include a memory controller to provide access to the memory by the processor and the input unit.

Optionally, the input unit can be used to receive input numbers or character information, and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control. Specifically, in a specific embodiment, the input unit may include a touch-sensitive surface as well as other input devices. A touch-sensitive surface, also known as a touch display or trackpad, collects user touch operations on or near it (for example, the user uses a finger, stylus, etc. any suitable object or accessory on the touch-sensitive surface or on the touch-sensitive Operation near the surface), and drive the corresponding connection device according to the preset program. Optionally, the touch-sensitive surface may include two parts: a touch detection device and a touch controller. Among them, the touch detection device detects the user's touch orientation, and detects the signal brought by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device, converts it into contact coordinates, and sends it to the To the processor, and can receive the command sent by the processor and execute it. In addition, touch-sensitive surfaces can be implemented in various types such as resistive, capacitive, infrared, and surface acoustic wave. In addition to the touch-sensitive surface, the input unit may also include other input devices. Specifically, other input devices may include, but are not limited to, one or more of physical keyboards, function keys (such as volume control keys, switch keys, etc.), trackballs, mice, joysticks, and the like.

Optionally, the display unit can be used to display information input by or provided to the user and various graphical user interfaces of the mobile terminal. These graphical user interfaces can be composed of graphics, text, icons, videos and any combination thereof. The display unit can include a display screen, optionally, a liquid crystal display (Liquid Crystal Display, LCD), Organic Light-Emitting Diode (Organic Light-Emitting Diode, OLED) and other forms to configure the display screen. Further, the touch-sensitive surface can cover the display screen, and when the touch-sensitive surface detects a touch operation on or near it, it is sent to the processor to determine the type of the touch event, and then the processor displays a touch event on the display screen according to the type of the touch event. Provide corresponding visual output. In the embodiment of the present application, the display unit is a display device, and the display device includes a side light source, a light guide plate, photosynthetic glue, a display panel, and a reflective sheet, and the side light source is arranged on the first side of the light guide plate. The photosynthetic glue is used to bond the second end surface of the light guide plate and the first end surface of the display panel, the reflective sheet is fixed on the second side of the light guide plate, and the first side of the light guide plate is opposite to the second side of the light guide plate , the light guide plate includes a first end surface of the light guide plate and a second end surface of the light guide plate, the first end surface of the light guide plate is provided with a plurality of groove-like structures, the groove-like structures are used to refract the light emitted by the side light source, and the second end surface of the light guide plate The end surface is provided with a plurality of prisms, and the prisms are used to refract the light emitted by the side light source. The groove structure, the prisms and the light guide plate are an integrated structure, and the first end surface of the light guide plate and the light guide plate The second end surface of the light plate is the opposite surface, the second end surface of the light guide plate is close to the display panel, and the display panel is a reflective liquid crystal display.

Optionally, the mobile terminal may further include at least one sensor, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor, wherein the ambient light sensor may adjust the brightness of the display screen according to the brightness of the ambient light, and the proximity sensor may turn off the display screen and/or backlight. As a kind of motion sensor, the gravitational acceleration sensor can detect the magnitude of acceleration at various positions (generally three axes), and can detect the magnitude and position of gravity when it is stationary, and can be used for applications that recognize the attitude of mobile phones (such as horizontal and vertical screen switching, related Games, magnetometer attitude calibration), vibration recognition related functions (such as pedometer, tap), etc.; as for mobile terminals, other sensors such as gyroscope, barometer, hygrometer, thermometer, infrared sensor, etc. can also be configured.

Furthermore, the audio circuit, speaker, and microphone can provide an audio interface between the user and the mobile terminal. The audio circuit can transmit the electrical signal converted from the received audio data to the speaker, and the speaker converts it into a sound signal output; on the other hand, the microphone converts the collected sound signal into an electrical signal, which is converted into an audio signal after being received by the audio circuit After the audio data is output to the processor for processing, the audio data is sent to another mobile terminal through the RF circuit, or the audio data is output to the memory for further processing. The audio circuit may also include an earphone jack to provide communication between peripheral earphones and the mobile terminal. Wi-Fi is a short-distance wireless transmission technology. Mobile terminals can help users send and receive emails, browse web pages, and access streaming media through Wi‑Fi modules. It provides users with wireless broadband Internet access.

Further, the processor is the control center of the mobile terminal, using various interfaces and lines to connect various parts of the entire mobile phone, by running or executing software programs and/or modules stored in the memory, and calling data stored in the memory, Execute various functions of the mobile terminal and process data, so as to monitor the mobile phone as a whole. Optionally, the processor may include one or more processing cores; preferably, the processor may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface and application programs, etc., and the modulation The demodulation processor mainly handles wireless communication. It can be understood that the foregoing modem processor may not be integrated into the processor.

Optionally, the mobile terminal also includes a power supply (such as a battery) for supplying power to various components. Preferably, the power supply can be connected to the processor logic through the power management system, so as to manage charging, discharging, and power consumption management through the power management system. Function. The power supply may also include one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, power status indicators, and any other components.

The present application provides a method, a terminal and a storage medium for focusing on a specified area by using AI action recognition, including acquiring a preview image; identifying the preview image to obtain the target in the preview image; Perform focus processing to obtain an updated preview image. The principle of the method provided by this application is to increase the proportion of the background plane of the display item in the screen, so that the camera image is focused on the display item. When performing a live display of the display item, there is no need to manually click to focus, as long as the display personnel stretch out their hands to display the product, Using AI motion recognition, when it is detected that the host has reached out to show the product, change the focus logic so that the product can be displayed better, and the screen can be quickly focused on the product. This application can quickly focus in this scenario Display items in the hands of the live broadcast, so that the product can be clearly displayed on the screen and improve the user experience.

It should be understood that the application of the present application is not limited to the above examples, and those skilled in the art can make improvements or changes based on the above descriptions, and all these improvements and changes should belong to the protection scope of the appended claims of the present application.

Claims

A method for using AI action recognition to focus on a designated area, including:

Get the preview image;

Identifying the preview image to obtain a target in the preview image;

When the target is a display item, focus processing is performed on the area where the target is located to obtain an updated preview image.
The method for using AI action recognition to focus on a specified area according to claim 1, wherein after identifying the preview image and obtaining the target in the preview image, the method further includes:

When the target is not a display item, focus processing is performed on the preview image to obtain an updated preview image.
The method for focusing on a specified area by using AI action recognition according to claim 2, wherein, when the target is not a display item, the step of performing focus processing on the preview image to obtain an updated preview image includes :

When the target is not a display item, determine an area of interest in the preview image according to the preview image;

performing image processing on the region of interest to obtain a focus position of the region of interest;

Focus on the focus position to obtain an updated preview image.
The method for focusing on a specified area by using AI action recognition according to claim 3, wherein the area of interest is the entire image of the preview image.
The method for focusing on a specified area by using AI action recognition according to claim 1, wherein, when the target is a display item, the step of performing focus processing on the area where the target is located to obtain an updated preview image includes :

When the target is a display item, the area where the target is located is used as an area of interest;

performing image processing on the region of interest to obtain a focus position of the region of interest;

Focus on the focus position to obtain an updated preview image.
The method for focusing on a specified area by using AI action recognition according to claim 5, wherein the display item is an item with hands displayed forward in the preview image, and the area where the target is located is the hand in the preview image. area of the Ministry.
The method for focusing on a specified area by using AI action recognition according to claim 6, wherein said focusing on said focus position to obtain an updated preview image comprises:

Focusing on the focus position to increase the proportion of the target in the preview image to obtain an updated preview image.
The method for focusing on a specified area by using AI action recognition according to claim 1, wherein said identifying said preview image to obtain a target in said preview image comprises:

The preview image is recognized by an AI action recognition module to obtain the target in the preview image.
The method for focusing on a specified area by using AI action recognition according to any one of claims 1 to 6, wherein the performing image processing on the area of interest to obtain the focus position of the area of interest includes:

The image signal processing unit performs image processing on the region of interest to obtain the focus position of the region of interest.
A system for using AI action recognition to focus on a designated area, including:

Acquisition module, obtain preview image;

The AI action recognition module recognizes the preview image to obtain the target in the preview image;

Select the focus module, and when the target is a display item, focus on the area where the target is located to obtain an updated preview image; when the target is not a display item, perform focus processing on the preview image to obtain an updated preview image of .
A mobile terminal, including: a processor, a memory, and a control program for focusing on a designated area stored on the memory and operable on the processor, wherein the control program for focusing on a designated area is executed by the processor Realized when:

Get the preview image;

Identifying the preview image to obtain a target in the preview image;

When the target is a display item, focus processing is performed on the area where the target is located to obtain an updated preview image.
The mobile terminal according to claim 11, wherein when the control program for specifying area focus is executed by the processor, it further implements:

When the target is not a display item, focus processing is performed on the preview image to obtain an updated preview image.
The mobile terminal according to claim 12, wherein when the control program for specifying area focus is executed by the processor, it further implements:

When the target is not a display item, determine an area of interest in the preview image according to the preview image;

performing image processing on the region of interest to obtain a focus position of the region of interest;

Focus on the focus position to obtain an updated preview image.
The mobile terminal according to claim 13, wherein the region of interest is the entire image of the preview image.
The mobile terminal according to claim 11, wherein when the control program for specifying area focus is executed by the processor, it further implements:

When the target is a display item, the area where the target is located is used as an area of interest;

performing image processing on the region of interest to obtain a focus position of the region of interest;

Focus on the focus position to obtain an updated preview image.
The mobile terminal according to claim 15, wherein the displayed item is an item in which the hand is shown forward in the preview image, and the area where the target is located is an area of the hand in the preview image.
The mobile terminal according to claim 16, wherein when the control program for specifying area focus is executed by the processor, it further implements:

Focusing on the focus position to increase the proportion of the target in the preview image to obtain an updated preview image.
The mobile terminal according to claim 11, wherein when the control program for specifying area focus is executed by the processor, it further implements:

The preview image is recognized by an AI action recognition module to obtain the target in the preview image.
The mobile terminal according to any one of claims 11-16, wherein when the control program for specifying area focus is executed by the processor, it further implements:

The image signal processing unit performs image processing on the region of interest to obtain the focus position of the region of interest.
A computer-readable storage medium, wherein, the computer-readable storage medium stores a control program for specifying focus in an area, and when the control program for specifying focus in a specified area is executed by a processor, any one of claims 1-9 can be realized. The steps of the method for using AI action recognition to focus on a designated area described in the item.