CN112835506B

CN112835506B - Display device and control method thereof

Info

Publication number: CN112835506B
Application number: CN202110184228.0A
Authority: CN
Inventors: 孙娜; 杨鲁明; 王大勇; 程晋
Original assignee: Hisense Visual Technology Co Ltd
Current assignee: Hisense Visual Technology Co Ltd
Priority date: 2021-02-08
Filing date: 2021-02-08
Publication date: 2022-08-12
Anticipated expiration: 2041-02-08
Also published as: CN114780010A; CN115525203A; CN115291792A; CN112835506A

Abstract

After a global gesture switch is turned on, if an image collector is not occupied by a designated application which needs to start the image collector, a global gesture detection function is turned on, and a user image including a user gesture collected by the image collector is obtained. When the user gesture in the user image is detected to be a play pause control gesture, the play mode broadcasts a full-screen play mode, and then a control instruction generated by the play pause control gesture is responded to perform play operation or pause operation on the video file; and when the detected user gesture in the user image is a mute control gesture, responding to a control instruction corresponding to the mute control gesture, and executing mute operation or anti-mute operation. Therefore, the method and the display device provided by the embodiment of the invention realize efficient control of the display device based on gesture recognition, and a user can conveniently control the display device even without a remote controller, thereby improving user experience.

Description

Display device and control method thereof

Technical Field

The application relates to the technical field of intelligent television interaction, in particular to a display device and a control method thereof.

Background

Along with the rapid development of display equipment, the functions of the display equipment are more and more abundant, and the performance is also more and more powerful, and at present, the display equipment comprises an intelligent television, an intelligent set top box, an intelligent box, other products with an intelligent display screen and the like. Taking the smart television as an example, the smart television provides a traditional television function and can play different television programs.

During the process of using the display device, a user can carry out global mute and anti-mute control on the display device based on the use requirement, and control the playing and the pause of the video when playing the video file. At present, the operations of muting, playing or pausing a video and the like of a display device are generally realized by using a remote controller configured with the display device, and the operations of muting, playing or pausing a video and the like are executed by controlling the display device by triggering corresponding keys on the remote controller.

However, in this control method, a remote controller is needed, and if the remote controller is lost or cannot be operated in time in an emergency, the control of the display device is not convenient enough, which affects the user experience.

Disclosure of Invention

The application provides a display device and a control method thereof, which are used for solving the problems that the existing control mode is not convenient enough to control the display device and influences the user experience.

In a first aspect, the present application provides a display device comprising:

a display configured to present a user interface;

an image collector or a user input interface connectable to an image collector, the image collector configured to collect a user image;

a controller connected to the display and the image collector, respectively, the controller configured to:

responding to a switch opening instruction generated when a global gesture switch in the user interface is triggered, and detecting whether the image collector is occupied by a specified application needing to start the image collector;

if the image collector is occupied by the appointed application which needs to start the image collector, the global gesture detection function is not started;

and if the image collector is not occupied by the appointed application which needs to start the image collector, starting a global gesture detection function, wherein the global gesture detection function is used for detecting the user image comprising the user gesture, and correspondingly controlling the display equipment based on the user gesture.

In some embodiments of the application, the controller, when performing the detecting whether the image collector is occupied by a specified application that needs to start the image collector, is further configured to:

acquiring an attribute state value of the image collector;

if the attribute state value is a first numerical value, determining that the image collector is occupied by a specified application which needs to start the image collector;

and if the attribute state value is a second numerical value, determining that the image collector is not occupied by the specified application which needs to start the image collector.

In a second aspect, the present application also provides a display device, including:

a display configured to present a user interface with a video file displayed thereon;

under the condition that the global gesture detection function is started, acquiring a user image which is acquired by an image acquisition device and comprises a user gesture;

when detecting that the user gesture in the user image is a play pause control gesture, acquiring a play mode broadcast for playing the video file;

if the broadcast form broadcast represents a full-screen broadcast form, responding to a control instruction generated by the broadcast pause control gesture, and performing play operation or pause operation on the video file;

and if the playing form broadcast represents the small window playing form, not responding to the control instruction generated by the playing pause control gesture.

In some embodiments of the present application, before performing the obtaining of the broadcast form broadcast of the broadcast video file, the controller is further configured to:

acquiring a signal source ID for identifying a channel type;

if the signal source ID represents a first channel type, acquiring a playing form broadcast of a playing video file, and determining whether to respond to a control instruction generated by the playing pause control gesture based on the playing form represented by the playing form broadcast;

and if the signal source ID represents a second channel type, not responding to a control instruction generated by the playing and pausing control gesture.

In a third aspect, the present application further provides a display device, including:

a display configured to present a user interface;

if the user gesture is a playing and pausing control gesture, responding to a control instruction corresponding to the playing and pausing control gesture to execute playing operation or pausing operation;

and if the user gesture is a mute control gesture, responding to a control instruction corresponding to the mute control gesture to execute mute operation or anti-mute operation.

In some embodiments of the present application, in executing that if the user gesture is a play/pause control gesture, the controller is further configured to execute a play operation or a pause operation in response to a control instruction corresponding to the play/pause control gesture, and to:

recognizing a user gesture in the user image, and judging whether the user gesture is a play pause control gesture;

when the user gesture is a playing pause control gesture, calculating a first effective successful frame rate of the playing pause control gesture;

and if the first effective successful frame rate of the playing and pausing control gesture exceeds a first effective threshold value, responding to a control instruction corresponding to the playing and pausing control gesture to execute playing operation or pausing operation.

In some embodiments of the present application, the controller, in performing the determining whether the user gesture is a play pause control gesture, is further configured to:

matching the user gesture with a preset control gesture, wherein the preset control gesture comprises a play pause control gesture and a mute control gesture;

if the user gesture is matched with the playing and pausing control gesture in a consistent manner, calculating the playing and pausing gesture confidence degrees of the user gesture and the playing and pausing control gesture;

and if the confidence of the playing pause gesture exceeds a gesture success threshold, determining that the user gesture is a playing pause control gesture.

In some embodiments of the present application, when the controller performs the operation of calculating the first effective successful frame rate of the play/pause control gesture when the user gesture is a play/pause control gesture, the controller is further configured to:

when the user gesture is a play pause control gesture, acquiring a total gesture recognition frame number of the user image acquired in a first detection duration and a gesture success frame number of the corresponding user image when the user gesture is the play pause control gesture;

and calculating the ratio of the gesture success frame number and the total gesture recognition frame number, and determining the ratio as a first effective success frame rate for playing the pause control gesture.

In some embodiments of the present application, the controller, when executing the control instruction corresponding to the play/pause control gesture to execute the play operation or the pause operation, is further configured to:

when the first effective successful frame rate of the playing pause control gesture exceeds a first effective threshold, acquiring a user image collected within a second detection duration;

calculating a second effective successful frame rate for playing the pause control gesture in the user image collected in the second detection duration;

if the second effective successful frame rate of the playing and pausing control gesture exceeds a third effective threshold value, responding to a control instruction corresponding to the playing and pausing control gesture to execute playing operation or pausing operation;

and if the second effective successful frame rate of the playing and pausing control gesture does not exceed a third effective threshold value, canceling the execution of playing operation or pausing operation in response to the control instruction corresponding to the playing and pausing control gesture.

responding to a play pause control instruction generated when a second effective successful frame rate of the play pause control gesture exceeds a third effective threshold value, and acquiring a play form broadcast of a play video file;

if the broadcast form broadcast represents a full-screen broadcast form, responding to the play pause control instruction, and executing play operation or pause operation on the video file;

and if the playing form broadcast represents a small window playing form, not responding to the playing pause control instruction.

In some embodiments of the present application, in executing that if the broadcast of the play form characterizes a full-screen play form, the controller is further configured to, in response to the play pause control instruction, perform a play operation or a pause operation on the video file:

broadcasting a full-screen playing mode in the playing mode, and acquiring the current playing state of the video file;

if the current playing state is a video playing state, responding to the playing pause control instruction, and executing pause operation on the video file;

and if the current playing state is a video pause state, responding to the play pause control instruction, and executing play operation on the video file.

In some embodiments of the present application, in executing that if the user gesture is a mute control gesture, the controller is further configured to, in response to a control instruction corresponding to the mute control gesture, execute a mute operation or an anti-mute operation:

recognizing a user gesture in the user image, and judging whether the user gesture is a mute control gesture;

when the user gesture is a mute control gesture, calculating a first effective successful frame rate of the mute control gesture;

and if the first effective successful frame rate of the mute control gesture exceeds a second effective threshold, responding to a control instruction corresponding to the mute control gesture to execute mute operation or anti-mute operation.

In some embodiments of the application, the controller, in performing the determining whether the user gesture is a mute control gesture, is further configured to:

if the user gesture is matched with the mute control gesture in a consistent manner, calculating the mute gesture confidence of the user gesture and the mute control gesture;

if the mute gesture confidence exceeds a gesture success threshold, determining that the user gesture is a mute control gesture.

In some embodiments of the present application, when the controller performs the calculating of the first effective successful frame rate of the mute control gesture when the user gesture is the mute control gesture, the controller is further configured to:

when the user gesture is a mute control gesture, acquiring a total gesture recognition frame number of the user image acquired in a first detection duration and a gesture success frame number of the corresponding user image when the user gesture is the mute control gesture;

and calculating the ratio of the gesture success frame number and the total gesture recognition frame number, and determining the ratio as the first effective success frame rate of the mute control gesture.

In some embodiments of the present application, the controller, when executing the control instruction corresponding to the mute control gesture to execute a mute operation or an anti-mute operation, is further configured to:

when the first effective successful frame rate of the mute control gesture exceeds a second effective threshold, acquiring a user image collected within a second detection duration;

calculating a second effective successful frame rate of a mute control gesture in the user image acquired within the second detection duration;

if the second effective successful frame rate of the mute control gesture exceeds a third effective threshold, responding to a control instruction corresponding to the mute control gesture to execute mute operation or anti-mute operation;

and if the second effective successful frame rate of the mute control gesture does not exceed a third effective threshold, canceling the execution of mute operation or anti-mute operation in response to the control instruction corresponding to the mute control gesture.

responding to a mute control instruction generated when a second effective successful frame rate of the mute control gesture exceeds a third effective threshold value, and acquiring a current mute zone bit of the system;

if the current mute flag bit is True, responding to the mute control instruction to execute anti-mute operation;

and if the current mute flag bit is False, performing mute operation in response to the mute control instruction.

In some embodiments of the present application, the controller is further configured to:

when the user gesture is determined to be a play pause control gesture or the user gesture is determined to be a mute control gesture, presenting a gesture detection prompt interface in the user interface, wherein gesture recognition success prompt information and gesture patterns are presented in the gesture detection prompt interface;

and when the user image collected in the second detection duration is obtained, displaying the gesture pattern in a buffering animation mode, wherein the displaying duration of the buffering animation is the second detection duration.

after a control process of one-time playing operation or pause operation, or mute operation or anti-mute operation is executed, acquiring a next frame of user image which is acquired by an image acquisition device and comprises user gestures at preset time intervals;

when the user gesture in the next frame of user image is a playing and pausing control gesture, responding to a control instruction corresponding to the playing and pausing control gesture to execute the next playing operation or pausing operation;

and when the user gesture in the next frame of user image is a mute control gesture, responding to a control instruction corresponding to the mute control gesture to execute the next mute operation or anti-mute operation.

and after a control process of one-time playing operation or pause operation, or mute operation or anti-mute operation is executed, presenting an operation completion prompt interface in the user interface, wherein an operation completion pattern and operation completion prompt information are displayed in the operation completion prompt interface.

In a fourth aspect, the present application further provides a method for controlling a display device, the method including:

In a fifth aspect, the present application further provides a method for controlling a display device, the method including:

In a sixth aspect, the present application further provides a method for controlling a display device, the method including:

In a seventh aspect, the present application further provides a storage medium, where the computer storage medium may store a program, and the program may implement, when executed, some or all of the steps in the embodiments of the control method for a display device provided in the present application.

According to the technical scheme, after the global gesture switch is turned on, if the image collector is not occupied by the appointed application which needs to start the image collector, the global gesture detection function is turned on, and the user image which is collected by the image collector and comprises the user gesture is obtained. When the user gesture in the user image is detected to be a play pause control gesture, the play mode broadcasts a full-screen play mode, and then a control instruction generated by the play pause control gesture is responded to perform play operation or pause operation on the video file; and when the detected user gesture in the user image is a mute control gesture, responding to a control instruction corresponding to the mute control gesture, and executing mute operation or anti-mute operation. Therefore, the method and the display device provided by the embodiment of the invention realize efficient control of the display device based on gesture recognition, and a user can conveniently control the display device even without a remote controller, thereby improving user experience.

Drawings

In order to more clearly explain the technical solution of the present application, the drawings needed to be used in the embodiments will be briefly described below, and it is obvious to those skilled in the art that other drawings can be obtained according to the drawings without any creative effort.

FIG. 1 illustrates a usage scenario of a display device according to some embodiments;

fig. 2 illustrates a block diagram of a hardware configuration of the control apparatus 100 according to some embodiments;

fig. 3 illustrates a hardware configuration block diagram of the display apparatus 200 according to some embodiments;

FIG. 4 illustrates a software configuration diagram in the display device 200 according to some embodiments;

FIG. 5 illustrates an icon control interface display of an application in the display device 200, in accordance with some embodiments;

FIG. 6 illustrates a first flowchart of a method of controlling a display device according to some embodiments;

FIG. 7 illustrates an interface diagram showing a global gesture switch in a user interface, in accordance with some embodiments;

FIG. 8 illustrates a second flowchart of a method of controlling a display device according to some embodiments;

FIG. 9 illustrates a third flowchart of a control method of a display device according to some embodiments;

FIG. 10 illustrates a general flow diagram of a control method of a display device according to some embodiments;

FIG. 11 illustrates a flow diagram of a method of controlling a display device to perform a play operation or a pause operation according to some embodiments;

FIG. 12 illustrates a data flow diagram for controlling a display device to perform a play operation or a pause operation according to some embodiments;

FIG. 13 illustrates a schematic diagram of a gesture detection prompt interface in a play pause scenario, in accordance with some embodiments;

FIG. 14 illustrates another schematic diagram of a gesture detection prompt interface in a play pause scenario, in accordance with some embodiments;

FIG. 15 illustrates a schematic diagram of presenting an operation completion prompt interface in a user interface, in accordance with some embodiments;

FIG. 16 illustrates a flow diagram of a method of controlling a display device to perform a mute operation or an anti-mute operation, according to some embodiments;

FIG. 17 illustrates a data flow diagram for controlling a display device to perform a mute operation or an anti-mute operation according to some embodiments;

FIG. 18 illustrates a schematic diagram of a gesture detection prompt interface in a silent scenario, in accordance with some embodiments;

FIG. 19 illustrates another schematic diagram of a gesture detection prompt interface in a silent scenario, in accordance with some embodiments;

FIG. 20 illustrates a flow chart of a method of volume adjustment according to some embodiments;

fig. 21 shows a data flow diagram of a method of volume adjustment according to some embodiments;

FIG. 22 illustrates a flow diagram of a method for a user interface to display a volume bar, according to some embodiments;

FIG. 23 illustrates a schematic diagram that shows a volume adjustment gesture prompt interface in a user interface, in accordance with some embodiments;

FIG. 24 illustrates a schematic diagram of a user interface displaying a volume adjustment interface, in accordance with some embodiments;

FIG. 25 illustrates a flow chart of a method of adjusting a volume corresponding to a volume bar according to some embodiments;

FIG. 26 illustrates a schematic diagram of calculating an amount of abscissa variation, according to some embodiments;

FIG. 27 illustrates a schematic diagram of displaying volume adjustment status prompt information in a user interface, in accordance with some embodiments;

FIG. 28 illustrates a schematic diagram of a user interface displaying a volume adjustment complete interface, in accordance with some embodiments;

fig. 29 illustrates another flow diagram of a method of volume adjustment according to some embodiments.

Detailed Description

To make the purpose and embodiments of the present application clearer, the following will clearly and completely describe the exemplary embodiments of the present application with reference to the attached drawings in the exemplary embodiments of the present application, and it is obvious that the described exemplary embodiments are only a part of the embodiments of the present application, and not all of the embodiments.

It should be noted that the brief descriptions of the terms in the present application are only for the convenience of understanding the embodiments described below, and are not intended to limit the embodiments of the present application. These terms should be understood in their ordinary and customary meaning unless otherwise indicated.

The terms "first," "second," "third," and the like in the description and claims of this application and in the above-described drawings are used for distinguishing between similar or analogous objects or entities and not necessarily for describing a particular sequential or chronological order, unless otherwise indicated. It is to be understood that the terms so used are interchangeable under appropriate circumstances.

The terms "comprises" and "comprising," as well as any variations thereof, are intended to cover a non-exclusive inclusion, such that a product or device that comprises a list of elements is not necessarily limited to all of the elements explicitly listed, but may include other elements not expressly listed or inherent to such product or device.

The term "module" refers to any known or later developed hardware, software, firmware, artificial intelligence, fuzzy logic, or combination of hardware or/and software code that is capable of performing the functionality associated with that element.

FIG. 1 illustrates a usage scenario of a display device according to some embodiments. As shown in fig. 1, the display apparatus 200 is also in data communication with a server 400, and a user can operate the display apparatus 200 through the smart device 300 or the control device 100.

In some embodiments, the control apparatus 100 may be a remote controller, and the communication between the remote controller and the display device includes at least one of an infrared protocol communication or a bluetooth protocol communication, and other short-distance communication methods, and controls the display device 200 in a wireless or wired manner. The user may control the display apparatus 200 by inputting a user instruction through at least one of a key on a remote controller, a voice input, a control panel input, and the like.

In some embodiments, the smart device 300 may include any of a mobile terminal, a tablet, a computer, a laptop, an AR/VR device, and the like.

In some embodiments, the smart device 300 may also be used to control the display device 200. For example, the display device 200 is controlled using an application program running on the smart device.

In some embodiments, the smart device 300 and the display device may also be used for communication of data.

In some embodiments, the display device 200 may also be controlled in a manner other than the control apparatus 100 and the smart device 300, for example, the voice instruction control of the user may be directly received by a module configured inside the display device 200 to obtain a voice instruction, or may be received by a voice control apparatus provided outside the display device 200.

In some embodiments, the display device 200 is also in data communication with a server 400. The display device 200 may be allowed to be communicatively connected through a Local Area Network (LAN), a Wireless Local Area Network (WLAN), and other networks. The server 400 may provide various contents and interactions to the display apparatus 200. The server 400 may be a cluster or a plurality of clusters, and may include one or more types of servers.

In some embodiments, software steps executed by one step execution agent may be migrated on demand to another step execution agent in data communication therewith for execution. Illustratively, software steps performed by the server may be migrated to be performed on a display device in data communication therewith, and vice versa, as desired.

Fig. 2 illustrates a block diagram of a hardware configuration of the control apparatus 100 according to some embodiments. As shown in fig. 2, the control device 100 includes a controller 110, a communication interface 130, a user input/output interface 140, a memory, and a power supply. The control apparatus 100 may receive an input operation instruction of a user and convert the operation instruction into an instruction recognizable and responsive to the display device 200, serving as an interaction intermediary between the user and the display device 200.

In some embodiments, the communication interface 130 is used for external communication, and includes at least one of a WIFI chip, a bluetooth module, NFC, or an alternative module.

In some embodiments, the user input/output interface 140 includes at least one of a microphone, a touchpad, a sensor, a key, or an alternative module.

Fig. 3 illustrates a hardware configuration block diagram of a display device 200 according to some embodiments. Referring to fig. 3, in some embodiments, the display apparatus 200 includes at least one of a tuner demodulator 210, a communicator 220, a detector 230, an external device interface 240, a controller 250, a display 260, an audio output interface 270, a memory, a power supply, and a user interface.

In some embodiments the controller comprises a central processor, a video processor, an audio processor, a graphics processor, a RAM, a ROM, a first interface to an nth interface for input/output.

In some embodiments, the display 260 includes a display screen component for displaying pictures, and a driving component for driving image display, and is used for receiving image signals from the controller output, displaying video content, image content, and components of a menu manipulation interface, and a user manipulation UI interface, etc.

In some embodiments, the display 260 may be at least one of a liquid crystal display, an OLED display, and a projection display, and may also be a projection device and a projection screen.

In some embodiments, the tuner demodulator 210 receives broadcast television signals via wired or wireless reception and demodulates audio/video signals, such as EPG data signals, from a plurality of wireless or wired broadcast television signals.

In some embodiments, communicator 220 is a component for communicating with external devices or servers according to various communication protocol types. For example: the communicator may include at least one of a Wifi module, a bluetooth module, a wired ethernet module, and other network communication protocol chips or near field communication protocol chips, and an infrared receiver. The display apparatus 200 may establish transmission and reception of control signals and data signals with the control device 100 or the server 400 through the communicator 220.

In some embodiments, the detector 230 is used to collect signals of the external environment or interaction with the outside. For example, detector 230 includes a light receiver, a sensor for collecting ambient light intensity; alternatively, the detector 230 includes an image collector, such as a camera, which may be used to collect external environment scenes, attributes of the user, or user interaction gestures, or the detector 230 includes a sound collector, such as a microphone, which is used to receive external sounds.

In some embodiments, the external device interface 240 may include, but is not limited to, the following: high Definition Multimedia Interface (HDMI), analog or data high definition component input interface (component), composite video input interface (CVBS), USB input interface (USB), RGB port, and the like. The interface may be a composite input/output interface formed by the plurality of interfaces.

In some embodiments, the controller 250 and the modem 210 may be located in different separate devices, that is, the modem 210 may also be located in an external device of the main device where the controller 250 is located, such as an external set-top box.

In some embodiments, the controller 250 controls the operation of the display device and responds to user operations through various software control programs stored in memory. The controller 250 controls the overall operation of the display apparatus 200. For example: in response to receiving a user command for selecting a UI object displayed on the display 260, the controller 250 may perform an operation related to the object selected by the user command.

In some embodiments, the object may be any one of selectable objects, such as a hyperlink, an icon, or other actionable control. The operations related to the selected object are: displaying an operation connected to a hyperlink page, document, image, or the like, or performing an operation of a program corresponding to the icon.

In some embodiments the controller comprises at least one of a Central Processing Unit (CPU), a video processor, an audio processor, a Graphics Processing Unit (GPU), a RAM Random Access Memory (RAM), a ROM (Read-Only Memory), a first to nth interface for input/output, a communication Bus (Bus), and the like.

And the CPU is used for executing the operating system and the application program instructions stored in the memory and executing various application programs, data and contents according to various interaction instructions for receiving external input so as to finally display and play various audio and video contents. The CPU processor may include a plurality of processors. E.g. comprising a main processor and one or more sub-processors.

In some embodiments, a graphics processor for generating various graphics objects, such as: at least one of an icon, an operation menu, and a user input instruction display figure. The graphic processor comprises an arithmetic unit, which performs operation by receiving various interactive instructions input by a user and displays various objects according to display attributes; the system also comprises a renderer which renders various objects obtained based on the arithmetic unit, and the rendered objects are used for being displayed on a display.

In some embodiments, the video processor is configured to receive an external video signal, and perform at least one of video processing such as decompression, decoding, scaling, noise reduction, frame rate conversion, resolution conversion, and image synthesis according to a standard codec protocol of the input signal, so as to obtain a signal displayed or played on the direct display device 200.

In some embodiments, the video processor includes at least one of a demultiplexing module, a video decoding module, an image composition module, a frame rate conversion module, a display formatting module, and the like. The demultiplexing module is used for demultiplexing the input audio and video data stream. And the video decoding module is used for processing the demultiplexed video signal, including decoding, scaling and the like. And the image synthesis module is used for carrying out superposition mixing processing on the GUI signal input by the user or generated by the user and the video image after the zooming processing by the graphic generator so as to generate an image signal for display. And the frame rate conversion module is used for converting the frame rate of the input video. And the display formatting module is used for converting the received video output signal after the frame rate conversion, and changing the signal to be in accordance with the signal of the display format, such as an output RGB data signal.

In some embodiments, the audio processor is configured to receive an external audio signal, decompress and decode the received audio signal according to a standard codec protocol of the input signal, and perform at least one of noise reduction, digital-to-analog conversion, and amplification processing to obtain a sound signal that can be played in the speaker.

In some embodiments, a user may enter user commands on a Graphical User Interface (GUI) displayed on display 260, and the user input interface receives the user input commands through the Graphical User Interface (GUI). Alternatively, the user may input the user command by inputting a specific sound or gesture, and the user input interface receives the user input command by recognizing the sound or gesture through the sensor.

In some embodiments, a "user interface" is a media interface for interaction and information exchange between an application or operating system and a user that enables conversion between an internal form of information and a form that is acceptable to the user. A commonly used presentation form of the User Interface is a Graphical User Interface (GUI), which refers to a User Interface related to computer operations and displayed in a graphical manner. It may be an interface element such as an icon, a window, a control, etc. displayed in the display screen of the electronic device, where the control may include at least one of an icon, a button, a menu, a tab, a text box, a dialog box, a status bar, a navigation bar, a Widget, etc. visual interface elements.

In some embodiments, user interface 280 is an interface that may be used to receive control inputs (e.g., physical buttons on the body of the display device, or the like).

In some embodiments, a system of a display device may include a Kernel (Kernel), a command parser (shell), a file system, and an application program. The kernel, shell, and file system together make up the basic operating system structure that allows users to manage files, run programs, and use the system. After power-on, the kernel is started, kernel space is activated, hardware is abstracted, hardware parameters are initialized, and virtual memory, a scheduler, signals and interprocess communication (IPC) are operated and maintained. And after the kernel is started, loading the Shell and the user application program. The application program is compiled into machine code after being started, and a process is formed.

Fig. 4 illustrates a software configuration diagram in the display device 200 according to some embodiments. Referring to fig. 4, in some embodiments, the system is divided into four layers, which are an Application (Applications) layer (abbreviated as "Application layer"), an Application Framework (Application Framework) layer (abbreviated as "Framework layer"), an Android runtime (Android runtime) and system library layer (abbreviated as "system runtime library layer"), and a kernel layer from top to bottom.

In some embodiments, at least one application program runs in the application program layer, and the application programs may be windows (windows) programs carried by an operating system, system setting programs, clock programs or the like; or an application developed by a third party developer. In particular implementations, the application packages in the application layer are not limited to the above examples.

The framework layer provides an Application Programming Interface (API) and a programming framework for the application. The application framework layer includes a number of predefined functions. The application framework layer acts as a processing center that decides to let the applications in the application layer act. The application program can access the resources in the system and obtain the services of the system in execution through the API interface.

As shown in fig. 4, in the embodiment of the present application, the application framework layer includes Managers (Managers), providers (Content providers), a network management system, and the like, where the Managers include at least one of the following modules: an Activity Manager (Activity Manager) is used for interacting with all activities running in the system; the Location Manager (Location Manager) is used for providing the system service or application with the access of the system Location service; a Package Manager (Package Manager) for retrieving various information related to an application Package currently installed on the device; a Notification Manager (Notification Manager) for controlling display and clearing of Notification messages; a Window Manager (Window Manager) is used to manage the icons, windows, toolbars, wallpapers, and desktop components on a user interface.

In some embodiments, the activity manager is used to manage the lifecycle of the various applications and the usual navigation fallback functions, such as controlling exit, opening, fallback, etc. of the applications. The window manager is used for managing all window programs, such as obtaining the size of a display screen, judging whether a status bar exists, locking the screen, intercepting the screen, controlling the change of the display window (for example, reducing the display window, displaying a shake, displaying a distortion deformation, and the like), and the like.

In some embodiments, the system runtime layer provides support for the upper layer, i.e., the framework layer, and when the framework layer is used, the android operating system runs the C/C + + library included in the system runtime layer to implement the functions to be implemented by the framework layer.

In some embodiments, the kernel layer is a layer between hardware and software. As shown in fig. 4, the core layer includes at least one of the following drivers: audio drive, display driver, bluetooth drive, camera drive, WIFI drive, USB drive, HDMI drive, sensor drive (like fingerprint sensor, temperature sensor, pressure sensor etc.) and power drive etc..

FIG. 5 illustrates an icon control interface display of an application in display device 200, according to some embodiments. In some embodiments, the display device may directly enter the interface of the preset vod program after being activated, and the interface of the vod program may include at least a navigation bar 510 and a content display area located below the navigation bar 510, as shown in fig. 5, where the content displayed in the content display area may change according to the change of the selected control in the navigation bar. The programs in the application program layer can be integrated in the video-on-demand program and displayed through one control of the navigation bar, and can also be further displayed after the application control in the navigation bar is selected.

In some embodiments, the display device may directly enter a display interface of a signal source selected last time after being started, or a signal source selection interface, where the signal source may be a preset video-on-demand program, or may be at least one of an HDMI interface, a live tv interface, and the like, and after a user selects different signal sources, the display may display contents obtained from different signal sources.

In some embodiments, when the display device is used to implement a smart tv function or a video playing function, different tv programs or different video files, audio files, etc. may be played in the display device. During use of the display device, a user may control the display device, such as playing a video, pausing a video, muting or anti-muting an operation, and so forth. The anti-mute operation refers to an operation of canceling mute, that is, switching the mute state to the output volume state.

At present, the control mode of the display equipment is usually realized by using a configured remote controller, and if the remote controller is lost or cannot be operated correspondingly through the remote controller in time under emergency, the control on the display equipment is not convenient enough, and the user experience is influenced.

External camera has been supported in a large number to current smart TV, and built-in self-carrying camera, this provides the basis for shooting the image detection gesture through the camera. Therefore, in order to improve the control efficiency of the display device, the embodiment of the invention provides a set of gesture recognition method for recognizing gestures through a camera to control muting and pause video playing.

The intelligent method for realizing global silence and anti-silence based on gesture recognition and controlling video playing pause in a full-screen scene, provided by the embodiment of the invention, can realize high-efficiency control on display equipment, and can judge whether to cancel or continue to execute corresponding operations by calculating an effective frame rate in gesture detection.

Fig. 6 illustrates a first flowchart of a control method of a display device according to some embodiments. An embodiment of the present invention provides a display device, including: a display configured to present a user interface; the image collector or the user input interface 140 which can be connected with the image collector, the image collector is configured to collect user images, the user input interface 140 is used for realizing a user instruction input function through actions such as voice, touch, gestures, pressing and the like, the input interface converts received analog signals into digital signals, and the digital signals are converted into corresponding instruction signals and sent to the image collector; a controller connected to the display and the image collector, respectively, the controller being configured to perform the following steps when executing the control method of the display apparatus shown in fig. 6:

and S11, responding to a switch opening instruction generated when a global gesture switch in the user interface is triggered, and detecting whether the image collector is occupied by a specified application which needs to start the image collector.

When the display device is controlled based on gesture recognition, a global gesture detection function needs to be configured in the display device, and the implementation manner of the global gesture detection function can be implemented by configuring a global gesture switch of the display device. If the global gesture switch is turned on, the global gesture detection function of the display device can be turned on, so that the display device can be correspondingly controlled according to the recognized user gesture; if the global gesture switch is turned off, the global gesture detection function of the display device may be turned off.

FIG. 7 illustrates an interface diagram showing a global gesture switch in a user interface according to some embodiments. Referring to fig. 7 (a), if the global gesture detection function is configured in the display device, an AI setting button is displayed in the user interface, the AI setting button is triggered, an AI setting interface is displayed in the user interface, and the global gesture on-off control button is presented in the AI setting interface. And (3) clicking the global gesture switch control button, displaying a global gesture setting interface shown as (b) in fig. 7 in the user interface, wherein the gesture control switch is presented in the global gesture setting interface, namely the global gesture switch. And the global gesture switch of the display equipment can be turned on by clicking the gesture control switch. To enable control of the display device based on gesture recognition, a global gesture switch may be turned on.

After the global gesture switch is turned on, the display device may perform a global gesture detection function. However, since the global gesture detection function needs to call an image collector (camera) to collect a user image including a user gesture in real time, the display device is controlled by identifying the type of the user gesture in the user image, such as playing, pausing, muting, anti-muting, and the like. Therefore, it is necessary to ensure that the image collector is in an unoccupied state when the global gesture detection function is implemented.

The application of the image collector can be called as a designated application needing to start the image collector, and the designated application refers to an application needing to utilize a camera to realize corresponding functions during operation, such as mirror viewing application. If the image collector is occupied by the appointed application which needs to start the image collector, the image collector cannot collect images in real time, and therefore, the global gesture detection function can be realized only when the image collector is not occupied by the appointed application which needs to start the image collector.

In order to accurately judge whether the display device can realize the global gesture detection function after the global gesture switch is turned on, whether the image collector is occupied by a specific application which needs to start the image collector needs to be judged. Specifically, in some embodiments, the controller, in performing detecting whether the image collector is occupied by a specified application that needs to start the image collector, is further configured to perform the following steps:

and step 111, acquiring the attribute state value of the image collector.

Step 112, if the attribute state value is the first value, determining that the image collector is occupied by the specified application which needs to start the image collector.

And 113, if the attribute state value is the second numerical value, determining that the image collector is not occupied by the specified application which needs to start the image collector.

Whether the image collector is occupied by the appointed application needing to start the image collector can be judged according to the attribute state value, and the attribute state value of the image collector is obtained under the condition that the global gesture switch is in the starting state. The attribute state value may include two values, a first value and a second value, depending on whether it is occupied. In some embodiments, the first value may be 1 and the second value may be 0.

And if the attribute state value is a first numerical value, namely 1, determining that the image collector is occupied by the specified application needing to start the image collector. And if the attribute state value is a second numerical value, namely 0, determining that the image collector is not occupied by the specified application needing to start the image collector.

And S12, if the image collector is occupied by the appointed application which needs to start the image collector, not starting the global gesture detection function.

And S13, if the image collector is not occupied by the appointed application which needs to start the image collector, starting a global gesture detection function, wherein the global gesture detection function is used for detecting the user image comprising the user gesture, and correspondingly controlling the display device based on the user gesture.

When the image collector is occupied by the appointed application which needs to start the image collector, the image collector cannot collect the user image, and at the moment, the global gesture detection function is not started, namely, the global gesture detection process is not carried out. When the image collector is not occupied by the appointed application which needs to start the image collector, the image collector can collect the user image in real time, and at the moment, the global gesture detection function is started, namely, the global gesture detection process is carried out.

When the display equipment performs global gesture detection, the user image comprising the user gesture can be collected in real time through the image collector, gesture recognition is performed on the user gesture in the user image, and corresponding control over the display equipment is achieved according to the recognized designated gesture.

In some embodiments, to enable control of a display device based on gesture recognition, the control gesture corresponding to each control may be preconfigured. For example, when implementing mute and anti-mute operations for controlling a display device, the control gesture may be preset to be one-handed compared to number 1 (mute control gesture), such as representing number 1 with the index finger; when the playing and pausing operations of the display device are controlled, the control gesture can be preset to be a single-hand five-finger gesture (playing and pausing control gesture), such as a palm of a single hand.

For example, under the normal operation of the display device, the global mute is realized by comparing the single hand with the number 1, and the anti-mute operation is realized by comparing the single hand with the number 1 again; when the video is played in a full-screen playing mode except the physical channel, the current playing is paused through the gesture of five fingers compared with a single hand, and the video is continuously played again compared with the gesture of five fingers compared with the single hand.

Therefore, the control of the display equipment is realized through gesture recognition, the participation of a remote controller is not needed, and the control of the display equipment can be realized by making corresponding gestures by the user with hands, so that the control efficiency of the display equipment is improved, and the user experience is good.

In some embodiments, when the video file is played by using the display device, two playing forms, namely a full-screen playing form and a small-window playing form, can be adopted. The full-screen playing mode refers to displaying a video picture (full-screen interface) in the user interface in a global mode, and the small-window playing mode refers to generating a small window in the user interface and displaying the video picture in a local mode.

In order to facilitate accurate control over playing and pausing of the video file, the display device can be set to be in a full-screen playing mode, and playing or pausing operation can be performed on the display device based on a gesture recognition mode. Therefore, when the user gesture in the user image is recognized as the play pause control gesture, the play mode of the video file in the current display device needs to be determined.

Fig. 8 illustrates a second flowchart of a control method of a display apparatus according to some embodiments. An embodiment of the present invention provides a display device, including: a display configured to present a user interface with a video file displayed thereon; the image collector or the user input interface 140 which can be connected with the image collector, the image collector is configured to collect user images, the user input interface 140 is used for realizing a user instruction input function through actions such as voice, touch, gestures, pressing and the like, the input interface converts received analog signals into digital signals, and the digital signals are converted into corresponding instruction signals and sent to the image collector; a controller connected to the display and the image collector, respectively, the controller being configured to perform the following steps when executing the control method of the display apparatus shown in fig. 8:

and S21, acquiring the user image including the user gesture collected by the image collector under the condition that the global gesture detection function is started.

When the control of the display equipment is realized based on gesture recognition, the global gesture switch needs to be started first, and the global gesture detection process is started under the condition that the image collector is not occupied. At this time, the image collector collects the user image in the detection area in real time.

In the process of using the display device by a user, if the display device needs to be controlled by using a gesture, the user can stand in a detection area of an image collector (camera) and swing out the gesture by using fingers. The image collector collects user images in the detection area, the user images comprise user gestures, the controller conducts gesture recognition on the user images comprising the user gestures, and whether the user gestures are play pause control gestures or mute control gestures is judged.

The play-pause control gesture is used to implement video play or pause operation on the display device, and in some embodiments, the play-pause control gesture can be set to be a one-hand-to-five-finger gesture. The mute control gesture is used to enable mute and anti-mute operations for the display device, and in some embodiments, the mute control gesture may be set to a one-handed to digital 1 gesture. The play pause control gesture or the mute control gesture can also be customized according to the use habits of the user, and the embodiment is not particularly limited.

The image collector collects user images in the detection area in real time according to a preset frequency and sequentially sends the user images to the controller. The controller performs gesture recognition on each frame of user image. In some embodiments, the preset frequency of the image collector is 30-40 ms/frame, i.e. the time taken for the image collector to collect one frame of user image is 30-40 ms.

And S22, when the user gesture in the user image is detected to be a play pause control gesture, acquiring a play mode broadcast of the play video file.

In some embodiments, the device for recognizing the user gesture in the user image may be an image collector, and when detecting that the user gesture in the user image is a play/pause control gesture, the image collector may send a detection result to the controller, and the controller controls the display device to implement a corresponding operation based on the play/pause control gesture.

In some embodiments, the device for recognizing the user gesture in the user image may also be a controller, and the controller performs gesture recognition on the user image after receiving the user image sent by the image collector. If the user gesture in the user image is detected to be the playing and pausing control gesture, the display device can be controlled to realize corresponding operation based on the playing and pausing control gesture.

Due to the fact that the video playing or pausing operation of the display device is controlled based on the gestures, the full-screen playing scene is needed. Therefore, when the user gesture in the user image is determined to be the play/pause control gesture, it indicates that the user needs to perform play or pause operation on the video file played in the display device, and at this time, the play mode adopted by the display device when playing the video file needs to be acquired, so as to determine whether the play operation or pause operation can be performed on the video file based on the control instruction generated by the play/pause control gesture.

The playing form of the video file can be obtained in a broadcasting form, the controller obtains the playing form broadcast of the playing video file, and if the received playing form broadcast is as follows: com, show, five finger menu, which shows that the video file is played in full screen mode. If the received broadcast form broadcast is: com, high, five finger menu, which shows that the video file is played in the form of a small window.

And S23, if the broadcast of the play mode represents the full-screen play mode, responding to a control instruction generated by the play pause control gesture, and performing play operation or pause operation on the video file.

And S24, if the broadcast of the play form indicates the small window play form, not responding to the control instruction generated by the play pause control gesture.

Because the playing pause operation scene is only limited under the condition of full-screen playing, the application scene needs to be judged if a single-hand five-finger gesture (playing pause control gesture) is detected in the user image acquired by the image acquisition device.

And when the playing form of the video file is judged to represent the full-screen playing form, the control of the display equipment based on the gesture can be realized in the scene, namely, the playing operation or the pause operation can be performed on the video file in response to the control instruction generated by the playing pause control gesture. And when the playing form of the video file is judged to broadcast the representation small window playing form, the control of the display equipment can not be realized based on the gesture in the scene, and at the moment, the control instruction generated by the playing pause control gesture is not responded, so that the playing operation or the pause operation can not be carried out on the video file.

In some embodiments, the mode of determining whether the current playing mode is the full-screen playing interface may be determined by obtaining a playing mode broadcast and a channel monitoring mode, determining whether the current mode is a physical channel or an android channel by monitoring the channel, and determining whether the current mode is the full-screen window playing or the small-window playing by the playing mode broadcast.

The physical channel refers to a channel for connecting a signal source when the display device implements picture output, for example, an HDMI interface, a television box, or the like. The android channel is a channel for connecting an android system to realize picture output.

If the display equipment adopts an android channel when playing the video file and is under the condition of full screen broadcasting, the full screen playing interface is judged, and at the moment, a single hand acts compared with a five-finger gesture, namely a control instruction generated by a playing pause control gesture can be responded, and the playing operation or pause operation is carried out on the video file.

If a small window broadcast is received or a channel is monitored to be switched to a physical channel, the single-hand five-finger gesture is not effective, namely, a control instruction generated by the play pause control gesture is not responded, the video file cannot be played or paused, and only global mute non-mute operation can be executed.

In some embodiments, the controller, prior to performing the retrieving the broadcast form broadcast of the broadcast video file, is further configured to perform the steps of:

step 221, acquiring a signal source ID for identifying a channel type.

Step 222, if the signal source ID represents the first channel type, acquiring a play mode broadcast of the play video file, and determining whether to respond to a control instruction generated by the play pause control gesture based on the play mode represented by the play mode broadcast.

And 223, if the signal source ID represents the second channel type, not responding to the control instruction generated by the playing and pausing control gesture.

When the user gesture in the user image is detected to be a playing and pausing control gesture, the fact that the current application scene needs to carry out video playing or pausing control on the display device is indicated. At this time, it is necessary to determine whether the display device plays the video file for the full screen interface.

In some embodiments, whether a scene is played in a full screen mode is determined by acquiring a broadcast mode broadcast and a channel monitoring mode, that is, a channel type of a display device is determined first, and then a broadcast mode broadcast of a video file is determined.

When determining the channel type of the display device, the signal source ID for identifying the channel type in the system can be directly acquired. The channel types include a physical channel and an android channel, the signal source ID of the android channel is 2562, and the signal source ID of the physical channel is 64 or other numerical values except 2562.

If the acquired signal source ID represents the first channel type, for example, 2562, it indicates that the channel type of the current display device is an android channel, and at this time, it may be continuously determined whether the video file playing mode is a full-screen playing mode or a small-window playing mode. Specifically, a playing form broadcast for playing the video file is acquired, and whether a control instruction generated by a playing pause control gesture is responded is determined based on the playing form represented by the playing form broadcast. The specific implementation process of determining the playing form can refer to the foregoing embodiments, and details are not repeated here.

If the video file is played in the android channel in a full-screen playing mode, the playing operation or the pause operation can be performed on the video file in response to a control instruction generated by the playing pause control gesture. If the video file is played in the android channel in a small window playing mode, the control instruction generated by the playing pause control gesture cannot be responded, and the playing operation or the pause operation cannot be performed on the video file.

If the acquired signal source ID represents a second channel type, for example, 64 or other numerical values except 2562, which indicates that the channel type of the current display device is a physical channel, at this time, it is not necessary to determine whether the mode for playing the video file is a full-screen playing mode or a small-window playing mode, and the control instruction generated by the play/pause control gesture may not be responded directly, and the play operation or the pause operation cannot be performed on the video file.

Therefore, when the display device is controlled by using the gesture, if the user gesture is identified as the play pause control gesture based on the user image acquired by the image acquirer, the signal type is monitored and the play form broadcast of the played video is acquired, so as to judge whether the play form of the video file is the full-screen play form or the small-window play form. Under the condition of full-screen broadcasting in an android channel, playing or pausing a video file in response to a control instruction generated by a playing and pausing control gesture; if the playing mode is a small window playing mode or under a physical channel, the control instruction generated by the playing pause control gesture is not responded. Therefore, under the scene of controlling the playing or pausing of the display device, whether the control instruction of the playing and pausing control gesture can be responded or not is judged based on the channel type and the playing form, so that the display device is efficiently controlled, and the user experience is improved.

When the display equipment is controlled based on the gesture recognition method, if the gesture recognition is carried out on the collected user image and the recognized user gesture is a playing and pausing control gesture, the display equipment is controlled to carry out playing operation or pausing operation; and if the recognized user gesture is a mute control gesture, controlling the display equipment to perform mute operation or anti-mute operation. The gesture recognition method can realize efficient control over the display device, and a user can conveniently control the display device even without a remote controller, so that user experience is improved.

FIG. 9 illustrates a third flowchart of a control method of a display device according to some embodiments; fig. 10 illustrates a general flowchart of a control method of a display apparatus according to some embodiments. In order to accurately perform corresponding control on a display device based on recognized user gestures, an embodiment of the present invention provides a display device, including: a display configured to present a user interface; the image collector or the user input interface 140 which can be connected with the image collector, the image collector is configured to collect user images, the user input interface 140 is used for realizing a user instruction input function through actions such as voice, touch, gestures, pressing and the like, the input interface converts received analog signals into digital signals, and the digital signals are converted into corresponding instruction signals and sent to the image collector; a controller connected to the display and the image collector, respectively, the controller being configured to perform the following steps when executing the control method of the display apparatus shown in fig. 9 and 10:

and S31, acquiring the user image including the user gesture collected by the image collector under the condition that the global gesture detection function is started.

And S32, if the user gesture is a play and pause control gesture, responding to a control instruction corresponding to the play and pause control gesture to execute play operation or pause operation.

And S33, if the user gesture is a mute control gesture, responding to a control instruction corresponding to the mute control gesture to execute mute operation or anti-mute operation.

In some embodiments, the device for recognizing the user gesture in the user image may be an image collector, and when detecting that the user gesture in the user image is a play/pause control gesture or a mute control gesture, the image collector may send a detected result to the controller, and the controller controls the display device to implement a corresponding operation based on the play/pause control gesture or the mute control gesture.

In some embodiments, the device for recognizing the user gesture in the user image may also be a controller, and the controller performs gesture recognition on the user image after receiving the user image sent by the image collector. If the user gesture in the user image is detected to be a play pause control gesture or a mute control gesture, the display device can be controlled to realize corresponding operation based on the play pause control gesture or the mute control gesture.

Therefore, when the user gesture in the user image is recognized as the playing and pausing control gesture, it indicates that the user needs to perform video playing or pausing operation on the display device by using the gesture, and at this time, the display device may respond to the control instruction corresponding to the playing and pausing control gesture and execute the playing operation or the pausing operation. When the user gesture in the user image is recognized as the mute control gesture, it is indicated that the user needs to perform mute or anti-mute operation on the display device by using the gesture, and at this time, the display device can respond to the control instruction corresponding to the mute control gesture and execute the mute operation or the anti-mute operation.

Therefore, in the display device provided by the embodiment of the invention, under the condition that the global gesture detection function is started, the image collector collects the user image in real time, and the controller acquires the user image which is collected by the image collector and comprises the user gesture and performs gesture recognition on the user image; if the gesture is identified as the playing and pausing control gesture, executing corresponding playing operation or pausing operation; and if the gesture is recognized as the mute control gesture, executing corresponding mute operation or anti-mute operation. Therefore, the embodiment can realize efficient control of the display device based on the gesture recognition method, and the user can conveniently control the display device even without a remote controller, so that the user experience is improved.

When the control of the display device is realized, the control gestures responded are different based on different control results, so that the gesture recognition modes adopted when different control gestures (a play pause control gesture and a mute control gesture) are recognized based on the user image are different.

FIG. 11 illustrates a flow diagram of a method of controlling a display device to perform a play operation or a pause operation according to some embodiments; fig. 12 illustrates a data flow diagram for controlling a display device to perform a play operation or a pause operation according to some embodiments. Referring to fig. 11 and 12, in some embodiments, in step S32, that is, if the user gesture is a play-pause control gesture, the controller is further configured to perform a play operation or a pause operation in response to a control instruction corresponding to the play-pause control gesture, and further configured to:

s321, recognizing the user gesture in the user image, and judging whether the user gesture is a play pause control gesture.

S322, when the user gesture is the playing pause control gesture, calculating a first effective successful frame rate of the playing pause control gesture.

And S323, if the first effective successful frame rate of the playing and pausing control gesture exceeds a first effective threshold, responding to a control instruction corresponding to the playing and pausing control gesture to execute playing operation or pausing operation.

Since the user is located within the detection area of the image collector, for example, the user stands in front of the display device, the image collector collects the user image including the user gesture for AI intelligent detection. However, if the user gesture that the user puts out is not the gesture to realize the control of the display device but other gestures that are put out carelessly, the gesture detection process for controlling the display device does not need to be started at this time.

And performing gesture recognition on each frame of collected user image, wherein the operation for controlling the display equipment comprises playing (pause) operation and muting (anti-mute) operation, and the corresponding preset control gesture is playing pause control gesture and muting gesture. Therefore, it is necessary to determine whether the user gesture is a play pause control gesture or a mute control gesture according to the gesture result of the user image. In the playback pause scene, it is necessary to determine whether a user gesture recognized from the user image is a playback pause control gesture.

In some embodiments, the controller, in executing step S321, namely executing to determine whether the user gesture is a play/pause control gesture, is further configured to execute the following steps:

step 3211, matching the user gesture with a preset control gesture, where the preset control gesture includes a play pause control gesture and a mute control gesture.

And 3212, if the user gesture matches the play/pause control gesture, calculating a play/pause gesture confidence of the user gesture and the play/pause control gesture.

In step 3213, if the confidence of the play/pause gesture exceeds the gesture success threshold, it is determined that the user gesture is a play/pause control gesture.

The main body capable of performing gesture recognition on the user image can be an image collector or a controller. Therefore, when the image collector or the controller recognizes the user gesture in each frame of user image, the user gesture in each frame of user image is respectively matched with the preset control gesture, and the preset control gesture comprises a play-pause control gesture and a mute control gesture, so as to determine that the user gesture in the current frame of user image is to control the display device to perform play (pause) operation or mute (anti-mute) operation.

If the gesture of the user is matched with the playing and pausing control gesture in a matching mode, the fact that the user wants to control the display device to execute playing or pausing operation is indicated. However, if the effect of the user gesture is not in place, for example, if the playback pause control gesture is a single-hand five-finger gesture, at least one finger of the user is bent, so that the gesture made by the user is not similar to the single-hand five-finger gesture. At this time, the ambiguous user gesture will cause that whether the user gesture is recognized as the play/pause control gesture cannot be accurately determined, and the misrecognition will cause that the control of the display device cannot be performed in time.

Therefore, in order to ensure that the display device can be effectively controlled in time, it is necessary to determine whether the user image in which the user gesture is recognized is a gesture success frame of the play/pause control gesture, that is, when the similarity of the user gesture as the play/pause control gesture exceeds a threshold, the user image is determined as the play/pause control gesture.

And calculating the confidence degrees, namely the similarity, of the user gesture in each frame of user image and the playing and pausing gesture of the playing and pausing control gesture. Determining whether each play-pause gesture confidence exceeds a gesture success threshold, which in some embodiments may be set to 99%, and if a play-pause gesture confidence does not exceed 99%, determining that the user gesture is not a play-pause control gesture; and if the confidence coefficient of a certain play pause gesture exceeds 99%, determining the user gesture corresponding to the confidence coefficient of the play pause gesture as a play pause control gesture so as to avoid generating misrecognition.

And meanwhile, determining the user image to which the user gesture with the playing pause gesture confidence coefficient exceeding the gesture success threshold belongs as a gesture success frame. In the gesture detection process, the frame number of the gesture success frame and the total gesture recognition frame number of all the collected user images are counted.

In the gesture detection process, if the user gesture in a certain frame of user image is not recognized as the playing pause control gesture, the gesture recognition is regarded as successful, namely, a corresponding control instruction is executed, and the display device is controlled to execute corresponding operation. But after a plurality of frames of user images are required to be detected continuously, the gesture recognition success can be judged only after the ratio of the gesture success frames corresponding to the playing pause control gesture exceeds a certain threshold value, and then the corresponding control instruction is executed to control the display equipment to execute the corresponding operation.

When the gesture recognition is judged to be successful, the calculation is carried out by using the continuously collected frames of user images, so that the efficient control of the display equipment can be ensured, the situation that the user puts down the display equipment immediately after making a play pause control gesture, the display equipment responds to a corresponding control instruction to execute corresponding operation after recognizing the gesture is avoided, and the user experience is influenced because the display equipment does not need to be controlled in a scene that the user puts down the gesture immediately actually.

Therefore, when the user gesture is recognized to be the playing and pausing control gesture, the first effective success frame rate of the playing and pausing control gesture is calculated to judge whether the current scene can be regarded as the gesture recognition success or not, and therefore the display device can be accurately controlled.

In some embodiments, the controller in executing step S322, that is, when the user gesture is the play-pause control gesture, calculating a first effective successful frame rate of the play-pause control gesture, is further configured to:

step 3221, when the user gesture is a play/pause control gesture, acquiring a total gesture recognition frame number of the user images acquired in the first detection duration and a gesture success frame number of the corresponding user image when the user gesture is the play/pause control gesture.

Step 3222, calculating a ratio of the number of gesture success frames to the total number of gesture recognition frames, and determining the ratio as a first effective success frame rate for playing the pause control gesture.

When the confidence of the playing pause gesture corresponding to the user gesture exceeds 99%, the corresponding user gesture can be determined as the playing pause control gesture, the user image where the user gesture is located is a gesture success frame, and a first effective success frame rate of the playing pause control gesture in all user images collected by the image collector within a first detection time length from the moment when the gesture success frame is detected is calculated.

In some embodiments, a duration corresponding to a process of judging whether the gesture recognition is successful in the gesture detection process is a first detection duration, the starting time of the first detection duration is the time of detecting the gesture successful frame, and the first detection duration can be set to be 500 ms. The first detection time period may also be set to other time periods according to practical applications, and is not specifically limited herein.

The first detection duration is the maximum duration for continuously acquiring the user images including the user gestures in the gesture detection process by the image acquisition device, the image acquisition device sequentially sends the multi-frame user images acquired in the first detection duration to the controller, and the controller can count the total gesture recognition frame number of the multi-frame user images received in the first detection duration.

And calculating the ratio of the number of successful frames of the gesture and the total number of gesture recognition frames, namely calculating the ratio of the number of successful frames of the playing and pausing control gestures detected by the controller in the first detection time length to the total number of gesture recognition frames, and determining the ratio as the first effective successful frame rate of the playing and pausing control gestures.

For example, when the AI intelligent detection method detects that a user gesture in a certain frame of user image is a play pause control gesture, a counter of 500ms (a first detection duration) is started, each frame of user image within 500ms is detected, if the play pause control gesture is detected, a gesture success frame number (DetectedFrames) and a total gesture recognition frame number (TotalFrames) corresponding to the play pause control gesture are respectively added by one, if the detected gesture is not the play pause control gesture, only the total gesture recognition frame number (TotalFrames) is added by one, and when the 500ms time arrives, a ratio of the gesture success frame number to the total frame number (sucessrate/TotalFrames) is calculated, and the first effective frame rate for play pause control gesture is determined.

In some embodiments, to avoid the user from stopping for a short time after the play pause control gesture is put out, the controller cannot determine whether the corresponding operation needs to be performed. Therefore, in order to conveniently and accurately control the display device, a first effective threshold value can be set, so that in the multi-frame user images collected by the image collector within the first detection duration, only if the first effective successful frame rate of the playing and pausing control gestures exceeds the first effective threshold value, the gesture recognition can be regarded as successful.

If the first effective successful frame rate of the playing pause control gesture does not exceed the first effective threshold, the user only temporarily makes the playing pause control gesture and puts down, and at the moment, the gesture recognition is not successful, and subsequent control operation cannot be executed; and if the first effective successful frame rate of the playing and pausing control gesture exceeds a first effective threshold, indicating that the gesture recognition is successful, responding to a control instruction corresponding to the playing and pausing control gesture to execute playing operation or pausing operation, and simultaneously resetting the counts of the DetectedFrames and the TotalFrames to wait for the next detection.

In some embodiments, the first effective threshold may be set to 95%, and if the recognition success rate exceeds the first effective threshold by 95%, indicating that the gesture recognition is successful in the current gesture detection process (within the first detection duration), at this time, the play operation or the pause operation may be performed in response to the control instruction corresponding to the play/pause control gesture. The first effective threshold may also be set to other values according to practical application, and is not limited specifically herein.

In some embodiments, when the gesture recognition of the play/pause control gesture in the user image is successful, in order to prompt the user that the current gesture recognition is successful, please keep the play/pause control gesture, so as to avoid the play/pause operation being cancelled in the midway, a gesture detection prompt interface may be presented in the user interface.

FIG. 13 illustrates a schematic diagram of a gesture detection prompt interface in a play pause scenario, in accordance with some embodiments. At this time, referring to fig. 13, the controller is further configured to: and when the gesture of the user is determined to be the playing pause control gesture, presenting a gesture detection prompt interface in the user interface, and presenting gesture recognition success prompt information and gesture patterns in the gesture detection prompt interface.

And when the first effective successful frame rate of the playing pause control gesture in the user image acquired within the first detection duration exceeds a first effective threshold, indicating that the gesture recognition is successful, and at the moment, presenting a gesture detection prompt interface in the user interface. And presenting successful gesture recognition prompt information and gesture patterns in the gesture detection prompt interface, wherein the successful gesture recognition prompt information is used for prompting the user to be in the gesture detection process currently, and the prompt information content can be ' try-to-compare gesture ' and video playing is paused '. The prompt message indicating the success of the gesture recognition can be selectively configured or not displayed. The gesture pattern is used to identify the UI form of the play pause control gesture, which is typically the same UI form as the user's gesture with a finger. For example, if the play pause control gesture is a single-hand five-finger gesture, the play pause control gesture pattern is a "palm" pattern.

In some embodiments, after the gesture recognition is successful, the play operation or the pause operation may be executed in response to the control instruction corresponding to the play/pause control gesture. However, in order to further ensure whether the user consistently maintains the play-pause control gesture without putting down halfway after the gesture recognition is successful, the second effective success frame rate of the play-pause control gesture within 500ms may be calculated again after the gesture recognition is successful.

Specifically, the controller, in executing step S323, namely executing the playing operation or the pause operation in response to the control instruction corresponding to the playing pause control gesture, is further configured to execute the following steps:

step 3241, when the first effective successful frame rate of the play pause control gesture exceeds a first effective threshold, acquiring a user image acquired within a second detection duration.

Step 3242, calculating a second effective successful frame rate for playing the pause control gesture in the user image collected within the second detection duration.

Step 3243, if the second effective successful frame rate of the play/pause control gesture exceeds the third effective threshold, execute a play operation or a pause operation in response to the control instruction corresponding to the play/pause control gesture.

Step 3244, if the second effective successful frame rate of the play/pause control gesture does not exceed the third effective threshold, canceling the execution of the play operation or the pause operation in response to the control instruction corresponding to the play/pause control gesture.

And when the gesture recognition is successful, namely when the first effective successful frame rate of the playing pause control gesture is detected to exceed the first effective threshold value, recalculating a second effective successful frame rate of the playing pause control gesture in a second detection duration, and acquiring the user image acquired by the image acquirer in the second detection duration. The calculation method of the second successful frame rate is the same as the calculation method of the first successful frame rate, and the details are not repeated here.

The starting time of the second detection duration is the time when the gesture recognition is detected successfully, and the second detection duration may be set to 500 ms. The second detection time period may also be set to other time periods according to practical applications, and is not specifically limited herein.

In the process of calculating the effective successful frame rate for the second time, in order to conveniently and accurately control the display device, a third effective threshold value may be set, so that, in the multi-frame user image acquired by the image acquirer within the second detection duration, only if the second effective successful frame rate of the play/pause control gesture exceeds the third effective threshold value, the play operation or the pause operation may be executed in response to the control instruction corresponding to the play/pause control gesture.

In some embodiments, the third effective threshold may be set to 70%, and if the recognition success rate exceeds the third effective threshold by 70%, it indicates that the gesture recognition is successful in the current gesture detection process (within the second detection duration), at this time, the play operation or the pause operation may be performed in response to the control instruction corresponding to the play/pause control gesture. The third effective threshold may also be set to other values according to practical application, and is not specifically limited herein.

Therefore, when it is determined that the second effective successful frame rate of the play/pause control gesture exceeds the third effective threshold, the play operation or the pause operation is executed in response to the control instruction corresponding to the play/pause control gesture. And when the second effective successful frame rate of the playing and pausing control gesture does not exceed the third effective threshold, which indicates that the user puts down the gesture midway, determining that the user cancels the corresponding operation, and at the moment, canceling the execution of the playing operation or the pausing operation in response to the control instruction corresponding to the playing and pausing control gesture.

In the process of calculating the effective successful frame rate for the second time, in order to prompt the user that gesture recognition is currently performed, a buffer animation can be presented in the gesture detection prompt interface. After the gesture recognition is successful, a 500ms buffering animation is immediately executed after a gesture detection prompt interface is presented in the user interface, and the display duration of the buffering animation is the duration for judging the gesture recognition success for the second time, namely the second detection duration.

FIG. 14 illustrates another schematic diagram of a gesture detection prompt interface in a play pause scenario, according to some embodiments. At this time, referring to fig. 14, the controller is configured to perform: and when the user image acquired within the second detection duration is acquired, displaying the gesture pattern in a buffering animation mode, wherein the display duration of the buffering animation is the second detection duration.

In this scenario, the gesture recognition success prompt information in the gesture detection prompt interface is used to inform the user that the designated gesture recognition that can currently realize the play/pause operation control is successful, the content of the prompt information may be "… in gesture recognition", and after seeing the prompt information, the user can know that the controllable display device executes the play operation or pause operation based on the play/pause control gesture.

The gesture pattern is displayed in a buffering animation mode, the displaying mode can be a circle turning mode, the gesture pattern starts from the top point, the gesture pattern turns clockwise for one circle, and the duration of the buffering animation turning for one circle is the second detection duration.

It should be noted that the gesture detection prompt interface shown in fig. 13 and the gesture detection prompt interface shown in fig. 14 are in a continuous display state, when the first gesture recognition is successful, the gesture detection prompt interface shown in fig. 13 is displayed in the user interface, and after the first gesture recognition is successful, the gesture detection prompt interface shown in fig. 14 is immediately displayed in the user interface, that is, after a gesture pattern appears in the user interface, the circle turning effect of the cache animation is immediately performed. The gesture detection prompt interface shown in fig. 14 is shown as a process of performing gesture recognition for the second time.

In some embodiments, when it is determined that the second effective successful frame rate of the play/pause control gesture does not exceed the third effective threshold, which indicates that the user puts down the gesture halfway, it is determined that the user cancels the corresponding operation, at this time, the buffer animation of the gesture pattern is not executed any more, the gesture detection prompt interface cancels display, and subsequently does not execute the corresponding operation any more, that is, cancels execution of the play operation or the pause operation in response to the control instruction corresponding to the play/pause control gesture.

In some embodiments, after the gesture recognition for the second time is successful, the display device may respond to the control instruction corresponding to the play/pause control gesture to perform the play operation or the pause operation. In the playback pause scene, the playback is limited to the full-screen playback, and therefore, whether a control instruction corresponding to the playback pause control gesture can be responded to is determined according to the playback mode of the current video file.

Specifically, the controller is further configured to, when executing the control instruction corresponding to the play/pause control gesture to execute the play operation or the pause operation:

step 3251, in response to a play pause control instruction generated when the second effective successful frame rate of the play pause control gesture exceeds a third effective threshold, acquiring a play format broadcast of a play video file.

And 3252, if the broadcast mode represents a full-screen broadcast mode, responding to a play/pause control instruction, and executing a play operation or a pause operation on the video file.

Step 3253, if the broadcast form indicates the small window broadcast form, not responding to the play pause control command.

If the gesture recognition for the second time is successful, that is, when the second effective successful frame rate of the playing and pausing control gesture exceeds the third effective threshold value, it is indicated that the playing operation or the pausing operation can be performed on the display device, and at this time, a playing and pausing control instruction is generated. The playing pause control instruction is sent in the form of a virtual key of a remote controller.

The controller responds to the playing pause control instruction to acquire the playing form broadcast of the playing video file. And when the playing form of the video file is judged to represent the full-screen playing form, the control of the display equipment based on the gesture can be realized in the scene, namely, the playing operation or the pause operation can be performed on the video file in response to the control instruction generated by the playing pause control gesture. And when the playing form of the video file is judged to broadcast the representation small window playing form, the control of the display equipment can not be realized based on the gesture in the scene, and at the moment, the control instruction generated by the playing pause control gesture is not responded, so that the playing operation or the pause operation can not be carried out on the video file.

If the video file is played in the android channel in a full-screen playing mode, the playing operation or the pause operation can be performed on the video file in response to a control instruction generated by the playing pause control gesture. If the video file is played in the android channel in a small window playing mode, the control instruction generated by the playing pause control gesture cannot be responded, and the playing operation or the pause operation cannot be performed on the video file. If the channel type of the display device is a physical channel, at this time, it is not necessary to judge whether the mode of playing the video file is a full-screen playing mode or a small-window playing mode, and the control instruction generated by the playing pause control gesture can be directly not responded, so that the video file cannot be played or paused. For a specific implementation process, reference may be made to steps S21 to S23 and related contents provided in the foregoing embodiments, and details are not described herein again.

To specifically determine whether to perform a play operation or a pause operation, the current state of the display device needs to be determined. If the current display equipment is in a playing state, responding to a control instruction corresponding to the playing pause control gesture, and executing pause operation; and if the current display equipment is in the pause state, responding to a control instruction corresponding to the playing pause control gesture, and executing playing operation.

In some embodiments, the controller, at execution step 3252, that is, if the play form broadcast characterizes a full-screen play form, performs a play operation or a pause operation on the video file in response to the play pause control instruction, is further configured to perform the steps of:

step 3261, broadcasting the full-screen playing mode in the playing mode, and acquiring the current playing state of the video file.

Step 3262, if the current playing status is the video playing status, responding to the playing pause control command, and performing a pause operation on the video file.

Step 3263, if the current playing status is the video pause status, responding to the play pause control command, and executing the playing operation on the video file.

And when the playing mode of the video file played by the current display equipment is judged to be the full-screen playing mode, acquiring the current playing state of the video file played by the display equipment. If the current playing state is the video playing state, the gesture made by the current user can be used for carrying out pause operation, namely, the pause operation can be carried out on the video file in response to the playing pause control instruction. If the current playing state is the video pause state, the gesture made by the current user can be used for playing operation, namely, the playing operation can be executed on the video file in response to the playing pause control instruction.

The current playing state of the display device can be determined by simulating the transmission of the remote controller center key value. When full-screen video is played, a user makes a play pause control gesture, such as a five-finger gesture, and after detection is successful, a remote controller Center key value is sent in a simulated mode, and when the system receives the Center key, whether the current video is in a play state or not can be judged according to whether interface video playing is true or not.

If the video playing is true, the video playing state is indicated, and at the moment, the system can respond to a playing pause control instruction and call a video playing interface to execute pause operation again; if the video playing is false, it indicates that the current video is in a video pause state, and at this time, the system will respond to the play pause control instruction and call the video playing to execute the play operation.

In some embodiments, after completing one process of controlling the display device to perform the play operation or the pause operation based on the foregoing manner, in order to prompt the user that the control is completed, an operation completion prompt interface may be presented in the user interface.

FIG. 15 illustrates a schematic diagram of presenting an operation completion prompt interface in a user interface, in accordance with some embodiments. Referring to fig. 15, after completing one display device control, an operation completion prompt interface is presented in the user interface. The operation completion prompting interface displays an operation completion pattern and operation completion prompting information, the operation completion pattern can be in a 'number matching' UI form, and the content of the operation completion prompting information can be 'operation success' and the like.

As can be seen, when the display device is controlled based on the gesture recognition method, if the gesture recognition is performed on the collected user image and the recognized user gesture is the play/pause control gesture, the display device is controlled to perform the play operation or the pause operation. And during control operation, the confidence of the pause gesture to be played exceeds a gesture success threshold, a first effective success frame rate of the pause control gesture to be played in the first detection duration exceeds a first effective threshold, a second effective success frame rate of the pause control gesture to be played in the second detection duration exceeds a third effective threshold, and in a scene of a full-screen playing mode, a control instruction corresponding to the pause control gesture to be played or paused can be responded, so that the control operation is more accurate. The gesture recognition method can realize efficient control over the display device, and a user can conveniently control the display device even without a remote controller, so that user experience is improved.

FIG. 16 illustrates a flow diagram of a method of controlling a display device to perform a mute operation or an anti-mute operation, in accordance with some embodiments; fig. 17 illustrates a data flow diagram for controlling a display device to perform a mute operation or an anti-mute operation according to some embodiments. Referring to fig. 16 and 17, in some embodiments, in step S33, that is, if the user gesture is a mute control gesture, the controller is further configured to perform the following steps in response to a control instruction corresponding to the mute control gesture to perform a mute operation or an anti-mute operation:

s331, recognizing the user gesture in the user image, and judging whether the user gesture is a mute control gesture.

S332, when the user gesture is a mute control gesture, calculating a first effective success frame rate of the mute control gesture.

And S333, if the first effective successful frame rate of the mute control gesture exceeds a second effective threshold, responding to a control instruction corresponding to the mute control gesture to execute mute operation or anti-mute operation.

And performing gesture recognition on each frame of collected user image, wherein the operation for controlling the display equipment comprises playing (pause) operation and muting (anti-mute) operation, and the corresponding preset control gesture is playing pause control gesture and muting gesture. Therefore, it is necessary to determine whether the user gesture is a play pause control gesture or a mute control gesture according to the gesture result of the user image. In a mute scene, it is necessary to determine whether a user gesture recognized from a user image is a mute control gesture.

In some embodiments, in step S331, namely performing the determination of whether the user gesture is the mute control gesture, the controller is further configured to perform the following steps:

step 3311, match user gestures with preset control gestures, the preset control gestures including a play pause control gesture and a mute control gesture.

Step 3312, if the user gesture matches the mute control gesture consistently, calculate a mute gesture confidence of the user gesture and the mute control gesture.

Step 3313, if the mute gesture confidence exceeds the gesture success threshold, then the user gesture is determined to be a mute control gesture.

When the image collector or the controller identifies the user gesture in each frame of user image, the user gesture in each frame of user image is respectively matched with a preset control gesture, and the preset control gesture comprises a play pause control gesture and a mute control gesture so as to determine that the user gesture in the current frame of user image is to control the display device to perform play (pause) operation or mute (anti-mute) operation.

If the user gesture is identified to be matched with the mute control gesture in a consistent way, the user wants to control the display device to execute mute operation or anti-mute operation. However, if the effect of the user gesture is not good, for example, if the mute control gesture is a one-hand-to-digital-1 gesture, if the user's finger is bent, the gesture made by the user is not similar to the one-hand-to-digital-1 gesture. At this time, the ambiguous user gesture may cause that whether the user gesture is recognized as a mute control gesture cannot be accurately determined, and a misrecognition may occur, which may cause that a display device may not be controlled in time.

Therefore, in order to ensure that the display device can be effectively controlled in time, it is necessary to determine whether the user image in which the user gesture is recognized is a gesture success frame of the mute control gesture, that is, when the similarity that the user gesture is the mute control gesture exceeds a threshold value, the user image is determined as the mute control gesture.

And calculating the mute gesture confidence coefficient, namely the similarity, of the user gesture and the mute control gesture in each frame of user image. Determining whether each mute gesture confidence exceeds a gesture success threshold, which in some embodiments may be set to 99%, and if a mute gesture confidence does not exceed 99%, determining that the user gesture is not a mute control gesture; and if the confidence coefficient of a certain mute gesture exceeds 99%, determining the user gesture corresponding to the confidence coefficient of the mute gesture as a mute control gesture so as to avoid generating false recognition.

Meanwhile, determining the user image to which the user gesture with the mute gesture confidence coefficient exceeding the gesture success threshold belongs as a gesture success frame. In the gesture detection process, the frame number of the gesture success frame and the total gesture recognition frame number of all the collected user images are counted.

In the gesture detection process, if the user gesture in a certain frame of user image is not recognized as the mute control gesture, the gesture recognition is regarded as successful, namely, a corresponding control instruction is executed, and the display device is controlled to execute corresponding operation. And after a plurality of frames of user images need to be continuously detected, judging that the gesture recognition is successful only after the ratio of the gesture success frames corresponding to the mute control gesture exceeds a certain threshold value, and then executing a corresponding control instruction to control the display equipment to execute corresponding operation.

When judging whether the gesture recognition is successful, the calculation is carried out by using the continuously collected frames of user images, so that the efficient control of the display equipment can be ensured, the situation that the user puts down the display equipment immediately after making a mute control gesture and the display equipment responds to a corresponding control instruction to execute corresponding operation after recognizing the gesture is avoided, and the user experience is influenced because the display equipment does not need to be controlled in a scene that the user puts down the gesture immediately actually.

Therefore, when the user gesture is recognized to be the mute control gesture, the first effective success frame rate of the mute control gesture is calculated to judge whether the current scene can be regarded as the gesture recognition success or not, so that the display device can be accurately controlled.

In some embodiments, the controller in executing step S332, namely executing that when the user gesture is a mute control gesture, the first effective success frame rate of the mute control gesture is calculated, and the controller is further configured to:

step 3321, when the user gesture is a mute control gesture, acquiring a total gesture recognition frame number of the user image acquired in the first detection duration and a gesture success frame number of the corresponding user image when the user gesture is the mute control gesture;

step 3322, calculating the ratio of the number of successful frames of the gesture and the number of the total gesture recognition frames, and determining the ratio as the first effective successful frame rate of the mute control gesture.

When the confidence of the mute gesture corresponding to the user gesture exceeds 99%, the corresponding user gesture can be determined as the mute control gesture, the user image where the user gesture is located is a gesture success frame, and the first effective success frame rate of the mute control gesture in all the user images collected by the image collector within the first detection duration from the moment when the gesture success frame is detected is calculated.

In some embodiments, a duration corresponding to a process of judging whether the gesture recognition is successful in the gesture detection process is a first detection duration, the starting time of the first detection duration is the time of detecting the gesture success frame, and the first detection duration can be set to be 500 ms. The first detection time period may also be set to other time periods according to practical applications, and is not specifically limited herein.

And calculating the ratio of the number of successful frames of the gesture to the number of total gesture recognition frames, namely calculating the ratio of the number of successful frames of the mute control gesture detected by the controller to the number of total gesture recognition frames in the first detection duration, and determining the ratio as the first effective successful frame rate of the mute control gesture.

For example, when the AI intelligent detection method detects that a user gesture in a certain frame of user image is a mute control gesture, a counter of 500ms (a first detection duration) is started, each frame of user image within 500ms is detected, if the mute control gesture is detected, a gesture success frame number (detectframes) and a total gesture recognition frame number (TotalFrames) corresponding to the mute control gesture are respectively added by one, if the detected gesture is not the mute control gesture, only the total gesture recognition frame number (TotalFrames) is added by one, and when the 500ms time arrives, a ratio of the gesture success frame number to the total frame number (sucessrate ═ detectframes/TotalFrames) is calculated, and the first effective success frame rate of the mute control gesture is determined.

In some embodiments, to avoid the user from staying for only a short time after the mute control gesture is put down, the controller cannot determine whether the corresponding operation needs to be performed. Therefore, in order to conveniently and accurately control the display device, a second effective threshold value can be set, so that in the multi-frame user images collected by the image collector within the first detection duration, only if the first effective successful frame rate of the mute control gesture exceeds the second effective threshold value, the gesture recognition can be regarded as successful.

If the first effective successful frame rate of the mute control gesture does not exceed the second effective threshold, the user only makes the mute control gesture for a short time and puts down, and at the moment, the gesture recognition is not successful, and subsequent control operation cannot be executed; and if the first effective successful frame rate of the mute control gesture exceeds a second effective threshold, indicating that the gesture recognition is successful, responding to a control instruction corresponding to the mute control gesture to execute mute operation or anti-mute operation, and clearing the counts of DetectedFrames and TotalFrames to wait for the next detection.

In some embodiments, the second effective threshold may be set to 85%, if the recognition success rate exceeds the second effective threshold by 85%, which indicates that the gesture recognition is successful in the current gesture detection process (within the first detection duration), and at this time, a mute operation or an anti-mute operation may be performed in response to the control instruction corresponding to the mute control gesture. The second effective threshold may also be set to other values according to practical application situations, and is not limited specifically here.

In some embodiments, when the gesture recognition of the mute control gesture in the user image is successful, in order to prompt the user that the current gesture recognition is successful, please keep the mute control gesture, so as to avoid that the mute operation is cancelled in the midway, a gesture detection prompt interface may be presented in the user interface.

FIG. 18 illustrates a schematic diagram of a gesture detection prompt interface in a silent scenario, in accordance with some embodiments. At this time, referring to fig. 18, the controller is further configured to: and when the gesture of the user is determined to be the mute control gesture, presenting a gesture detection prompt interface in the user interface, and presenting gesture recognition success prompt information and a gesture pattern in the gesture detection prompt interface.

And when the first effective successful frame rate of the mute control gesture in the user image acquired within the first detection duration exceeds a second effective threshold, indicating that the gesture recognition is successful, and presenting a gesture detection prompt interface in the user interface. And presenting successful gesture recognition prompt information and gesture patterns in the gesture detection prompt interface, wherein the successful gesture recognition prompt information is used for prompting the user to be in the gesture detection process currently, and the prompt information content can be 'try-to-compare gesture' and mute. The prompt message indicating the success of the gesture recognition can be selectively configured or not displayed. The gesture pattern is used to identify the UI form of the mute control gesture, which is typically the same UI form as the user's gesture with a finger. For example, if the mute control gesture is a one-hand-to-digital-1 gesture, the mute control gesture pattern is an "index finger-to-1" pattern.

In some embodiments, after the gesture recognition is successful, a mute operation or an anti-mute operation may be performed in response to a control instruction corresponding to the mute control gesture. However, to further ensure whether the user consistently maintains the mute control gesture without dropping halfway after the gesture recognition is successful, the second effective success frame rate of the mute control gesture within 500ms may be calculated again after the gesture recognition is successful.

Specifically, in the step S333, the controller is further configured to perform a mute operation or an anti-mute operation in response to the control instruction corresponding to the mute control gesture, and further:

step 3341, when the first effective successful frame rate of the mute control gesture exceeds the second effective threshold, acquiring the user image acquired within the second detection duration.

Step 3342, calculating a second effective success frame rate of the mute control gesture in the user image collected within the second detection duration.

And step 3343, if the second effective successful frame rate of the mute control gesture exceeds a third effective threshold, executing a mute operation or an anti-mute operation in response to a control instruction corresponding to the mute control gesture.

And step 3344, if the second effective successful frame rate of the mute control gesture does not exceed the third effective threshold, canceling the execution of the mute operation or the anti-mute operation in response to the control instruction corresponding to the mute control gesture.

And when the gesture recognition is successful, namely when the first effective successful frame rate of the mute control gesture is detected to exceed a second effective threshold value, recalculating a second effective successful frame rate of the mute control gesture in a second detection duration, and at the moment, acquiring the user image acquired by the image acquirer in the second detection duration. The calculation method of the second effective successful frame rate is the same as the calculation method of the first effective successful frame rate, and is not described herein again.

In the process of calculating the effective successful frame rate for the second time, in order to conveniently and accurately control the display device, a third effective threshold value may be set, so that in the multi-frame user image acquired by the image acquirer within the second detection duration, only if the second effective successful frame rate of the mute control gesture exceeds the third effective threshold value, the mute operation or the anti-mute operation may be executed in response to the control instruction corresponding to the mute control gesture.

In some embodiments, the third effective threshold may be set to 70%, and if the recognition success rate exceeds the third effective threshold by 70%, indicating that the gesture recognition is successful in the current gesture detection process (within the second detection duration), at this time, a mute operation or an anti-mute operation may be performed in response to the control instruction corresponding to the mute control gesture. The third effective threshold may also be set to other values according to practical application, and is not specifically limited herein.

Therefore, when it is determined that the second effective successful frame rate of the mute control gesture exceeds the third effective threshold, a mute operation or an anti-mute operation is performed in response to the control instruction corresponding to the mute control gesture. And when the second effective successful frame rate of the mute control gesture does not exceed the third effective threshold, which indicates that the user puts down the gesture midway, determining that the user cancels the corresponding operation, and at the moment, canceling the execution of the mute operation or the anti-mute operation in response to the control instruction corresponding to the mute control gesture.

FIG. 19 illustrates another schematic diagram of a gesture detection prompt interface in a mute scenario, in accordance with some embodiments. At this time, referring to fig. 19, the controller is configured to perform: and when the user image acquired within the second detection duration is acquired, displaying the gesture pattern in a buffering animation mode, wherein the display duration of the buffering animation is the second detection duration.

In this scenario, the gesture recognition success prompt information in the gesture detection prompt interface is used to inform the user that the specified gesture recognition that can currently implement mute operation control is successful, the content of the prompt information may be "… in gesture recognition", and after seeing the prompt information, the user can know that the controllable display device executes mute operation or anti-mute operation based on the mute control gesture.

It should be noted that the gesture detection prompt interface shown in fig. 18 and the gesture detection prompt interface shown in fig. 19 are in a continuous display state, when the gesture recognition is successful for the first time, the gesture detection prompt interface shown in fig. 18 is displayed in the user interface, and after the gesture recognition is successful for the first time, the gesture detection prompt interface shown in fig. 19 is immediately displayed in the user interface, that is, after a gesture pattern appears in the user interface, the circling effect of the cache animation is immediately performed. The gesture detection prompt interface shown in fig. 19 is shown as a process of performing the second gesture recognition.

In some embodiments, when it is determined that the second effective successful frame rate of the mute control gesture does not exceed the third effective threshold, which indicates that the user puts down the gesture halfway, it is determined that the user cancels the corresponding operation, at this time, the buffer animation of the gesture pattern is not executed any more, the gesture detection prompt interface cancels the display, and subsequently does not execute the corresponding operation any more, that is, the execution of the mute operation or the anti-mute operation in response to the control instruction corresponding to the mute control gesture is cancelled.

To specifically determine whether to perform a mute operation or an anti-mute operation, the current state of the display device needs to be determined first. If the current display equipment is in a mute state, responding to a control instruction corresponding to the mute control gesture, and executing anti-mute operation; and if the current display equipment is in the anti-mute state, responding to a control instruction corresponding to the mute control gesture, and executing mute operation.

In some embodiments, the controller, in executing the control instruction corresponding to the mute control gesture, is further configured to:

step 3351, in response to a mute control instruction generated when the second effective successful frame rate of the mute control gesture exceeds a third effective threshold, obtaining a current mute flag of the system.

And step 3352, if the current mute flag bit is True, performing an anti-mute operation in response to the mute control instruction.

And step 3353, if the current mute flag bit is False, executing a mute operation in response to the mute control instruction.

If the gesture recognition for the second time is successful, that is, when the second effective successful frame rate of the mute control gesture exceeds the third effective threshold value, it is indicated that the mute operation or the anti-mute operation can be performed on the display device, and at this time, a mute control instruction is generated. The mute control instruction is sent in the form of a virtual key of a remote controller.

The controller responds to the mute control instruction to acquire the current mute zone bit of the system. If the current mute flag bit is True, it indicates that the current system is in a mute state, so that a gesture made by a user can perform anti-mute operation, and at this time, the display device can respond to the mute control instruction to perform anti-mute operation. If the current mute flag is False, it indicates that the current system is in an anti-mute state, so that the gesture made by the user can perform a mute operation, and at this time, the display device can respond to the mute control instruction to perform a mute operation.

When the user gesture is a mute control gesture, such as a digital 1 gesture, after the gesture recognition is successful, a remote controller mute key (mute) value is sent in a simulating mode, after the system receives the mute key value of the remote controller, getmuteplag is called to obtain a current muteflag (mute flag bit), if the current muteflag is True, the display device is in a mute state, after the display device receives the mute key value, muting is cancelled, namely, anti-mute operation is executed, and muteflge is set to False. If the acquired current muteflag is False, the fact that the display device is in a non-mute state at the moment is indicated, the mute operation is executed after the display device receives the mutekey value, and the muteflag is set to True.

In some embodiments, after completing one process of controlling the display device to perform the mute operation or the anti-mute operation based on the foregoing manner, in order to prompt the user that the control is completed, an operation completion prompt interface may be presented in the user interface. The operation completion prompting interface may be as shown in fig. 15, where an operation completion pattern and operation completion prompting information are displayed in the operation completion prompting interface, the operation completion pattern may be in a "number matching" UI form, and the content of the operation completion prompting information may be "operation success" or the like.

As can be seen, when the display device is controlled based on the gesture recognition method, if the gesture recognition is performed on the collected user image and the recognized user gesture is a mute control gesture, the display device is controlled to perform a mute operation or an anti-mute operation. And in the control operation, if the confidence coefficient of the mute gesture exceeds a gesture success threshold, the first effective success frame rate of the mute control gesture exceeds a second effective threshold in the first detection duration, and the second effective success frame rate of the mute control gesture exceeds a third effective threshold in the second detection duration, the control instruction corresponding to the mute control gesture can be responded, and the mute operation or the anti-mute operation is executed, so that the control operation is more accurate. The gesture recognition method can realize efficient control over the display device, and a user can conveniently control the display device even without a remote controller, so that user experience is improved.

In some embodiments, to avoid continuously responding to the play pause control command or the mute control command, the control method executed by the present invention may execute the next operation after a preset time interval after executing one operation process.

Specifically, after completing one operation control, the controller is further configured to perform the steps of:

and step 41, after a control process of one-time playing operation or pause operation, or mute operation or anti-mute operation is executed, acquiring a next frame of user image including user gestures, which is acquired by the image acquisition device, at preset time intervals.

And 42, when the user gesture in the next frame of user image is a play/pause control gesture, responding to the control instruction corresponding to the play/pause control gesture to execute the next play operation or pause operation.

And 43, when the user gesture in the next frame of user image is a mute control gesture, responding to the control instruction corresponding to the mute control gesture to execute the next mute operation or anti-mute operation.

After the current playing operation or pause operation, or the mute operation or anti-mute operation process is completely finished, if the user makes the user gesture again in the detection area of the image collector, the image collector collects the next frame of user image including the user gesture and sends the next frame of user image to the controller. The controller judges the control gesture again after receiving a new frame of user image, and responds to a control instruction corresponding to the play/pause control gesture to execute the next play operation or pause operation when judging that the user gesture in the new frame of user image is the play/pause control gesture; and when the user gesture in the new user image is judged to be the mute control gesture, responding to a control instruction corresponding to the mute control gesture to execute the next mute operation or anti-mute operation. For the specific play operation or pause operation, or the mute operation or anti-mute operation control process, reference may be made to the contents of the foregoing embodiments, which are not described herein again.

In some embodiments, the preset duration of the interval may be set to 2 seconds, and may also be set to other values according to practical applications, which is not limited herein.

No matter the control operation is successfully completed once or the control operation is not successfully completed, namely the operation is cancelled midway, the next frame of user image is acquired after 2 seconds, and the phenomenon that the user frequently starts the operation based on the gesture control display device in a short time is avoided.

As can be seen from the foregoing technical solutions, according to the display device provided in the embodiments of the present invention, after the global gesture switch is turned on, if the image collector is not occupied by the specified application that needs to start the image collector, the global gesture detection function is turned on, and the user image including the user gesture collected by the image collector is acquired. When the user gesture in the user image is detected to be a play pause control gesture, the play mode broadcasts a full-screen play mode, and then a control instruction generated by the play pause control gesture is responded to perform play operation or pause operation on the video file; and when the detected user gesture in the user image is a mute control gesture, responding to a control instruction corresponding to the mute control gesture, and executing mute operation or anti-mute operation. Therefore, the display device provided by the embodiment of the invention realizes efficient control over the display device based on gesture recognition, and a user can conveniently control the display device even without a remote controller, so that the user experience is improved.

Fig. 6 illustrates a first flowchart of a control method of a display device according to some embodiments. Referring to fig. 6, the present application also provides a control method of a display device, which is executed by the controller in the display device provided in the foregoing embodiment, the method including:

s11, responding to a switch starting instruction generated when a global gesture switch in the user interface is triggered, and detecting whether the image collector is occupied by a specified application needing to start the image collector;

s12, if the image collector is occupied by the appointed application which needs to start the image collector, the global gesture detection function is not started;

s13, if the image collector is not occupied by the appointed application which needs to start the image collector, starting a global gesture detection function, wherein the global gesture detection function is used for detecting the user image comprising the user gesture, and correspondingly controlling the display device based on the user gesture.

Fig. 8 illustrates a second flowchart of a control method of a display apparatus according to some embodiments. Referring to fig. 8, the present application also provides a control method of a display device, which is executed by the controller in the display device provided in the foregoing embodiment, the method including:

s21, acquiring a user image including a user gesture collected by an image collector under the condition that the global gesture detection function is started;

s22, when detecting that the user gesture in the user image is a play pause control gesture, acquiring a play mode broadcast for playing the video file;

s23, if the broadcast form represents a full-screen broadcast form, responding to a control instruction generated by the broadcast pause control gesture, and performing play operation or pause operation on the video file;

and S24, if the playing form broadcast represents the small window playing form, not responding to the control instruction generated by the playing pause control gesture.

Fig. 9 illustrates a third flowchart of a control method of a display apparatus according to some embodiments. Referring to fig. 9, the present application also provides a control method of a display device, which is executed by the controller in the display device provided in the foregoing embodiment, the method including:

s31, acquiring a user image including a user gesture collected by an image collector under the condition that the global gesture detection function is started;

s32, if the user gesture is a playing and pausing control gesture, responding to a control instruction corresponding to the playing and pausing control gesture to execute playing operation or pausing operation;

s33, if the user gesture is a mute control gesture, responding to a control instruction corresponding to the mute control gesture to execute mute operation or anti-mute operation.

As can be seen from the foregoing technical solutions, according to the display device and the control method thereof provided in the embodiments of the present invention, after the global gesture switch is turned on, if the image acquirer is not occupied by the designated application that needs to start the image acquirer, the global gesture detection function is turned on, and the user image including the user gesture acquired by the image acquirer is acquired. When the user gesture in the user image is detected to be a play pause control gesture, the play mode broadcasts a full-screen play mode, and then a control instruction generated by the play pause control gesture is responded to perform play operation or pause operation on the video file; and when the detected user gesture in the user image is a mute control gesture, responding to a control instruction corresponding to the mute control gesture, and executing mute operation or anti-mute operation. Therefore, the method and the display device provided by the embodiment of the invention realize efficient control of the display device based on gesture recognition, and a user can conveniently control the display device even without a remote controller, thereby improving user experience.

In some embodiments, when the display device is used to implement a smart tv function or a video playing function, different tv programs or different video files, audio files, etc. may be played in the display device. During use of the display device, the user can adjust the output volume of the display device based on the usage requirements so that the user can be immersed in a television program or video.

Because the volume adjustment mode of the display device is usually realized by using a remote controller configured for the display device, the display device is not convenient enough and influences the user experience. Therefore, in order to improve the efficiency of adjusting the output volume of the display device, the embodiment of the invention provides a set of intelligent algorithms for adjusting the volume through gesture recognition, an AI image detection technology is used for recognizing the gestures of a person in an image and the position of the person in the image, and the volume is adjusted through the position change of the gestures of the person in the image. And current smart television has supported external camera in a large number, and built-in self-contained camera, this provides the basis for shooing the image detection gesture through the camera.

The intelligent algorithm for realizing volume adjustment based on gesture recognition provided by the embodiment of the invention can realize volume adjustment, can avoid the problem of unsmooth volume adjustment caused by frame loss, blocking and slow detection in the gesture detection process even if the load of a system CPU is overlarge, and can realize volume adjustment on display equipment smoothly by recognizing a sliding gesture.

FIG. 20 illustrates a flow chart of a method of volume adjustment according to some embodiments; fig. 21 illustrates a data flow diagram of a method of volume adjustment according to some embodiments. In order to realize gesture interactive recognition of a shot image by using a camera of a television and realize intelligent volume adjustment on the television by recognizing a sliding gesture, the display equipment provided by the embodiment of the invention comprises: a display configured to present a user interface; the image collector or the user input interface 140 which can be connected with the image collector, the image collector is configured to collect user images, the user input interface 140 is used for realizing a user instruction input function through actions such as voice, touch, gestures, pressing and the like, the input interface converts received analog signals into digital signals, and the digital signals are converted into corresponding instruction signals and sent to the image collector; a controller connected to the display and the image collector, respectively, the controller being configured to perform the following steps when performing the volume adjusting method shown in fig. 20 and 21:

and S41, acquiring the user image including the user gesture collected by the image collector.

In the process of using the display device by a user, if the volume of the display device needs to be adjusted, the user can stand in a detection area of an image collector (camera) and use fingers to swing out gestures. The image collector collects user images in the detection area, the user images comprise user gestures, the controller conducts gesture recognition on the user images comprising the user gestures, and whether the user gestures are volume adjustment gestures or not is judged.

The volume adjustment gesture is used to enable intelligent adjustment of the volume of the display device, and in some embodiments, the volume adjustment gesture may be set to an OK gesture. The volume adjustment gesture may also be customized according to the usage habit of the user, for example, the volume adjustment gesture may also be set to be a palm gesture, a finger bending gesture, or the like, and this embodiment is not particularly limited.

The image collector collects user images in the detection area in real time according to a preset frequency and sequentially sends the user images to the controller. The controller performs gesture recognition on each frame of user image, if the success rate of the recognized volume adjusting gesture within the first duration is greater than a first threshold, the gesture recognition is regarded as successful, and intelligent adjustment of the volume of the display device can be achieved according to the recognized volume adjusting gesture.

In some embodiments, the preset frequency of the image collector is 30-40 ms/frame, i.e. the time taken for the image collector to collect one frame of user image is 30-40 ms.

And S42, if the user gesture in the user image is recognized to be a volume adjusting gesture, displaying a volume bar in the user interface.

The controller identifies gestures of each received frame of user image, judges whether the user gestures in the user images are volume adjustment gestures or not, starts an intelligent volume adjustment function of the display device through sliding the volume adjustment gestures when the user gestures are identified as the volume adjustment gestures, and meanwhile, turns up the volume bar control to be displayed in a user interface, so that a user can sense the current visual effect of volume adjustment.

FIG. 22 illustrates a flow diagram for a method for a user interface to display a volume bar, according to some embodiments. Referring to fig. 22, in some embodiments, the controller, in performing recognizing that the user gesture in the user image is a volume adjustment gesture, displays a volume bar in the user interface, further configured to:

and S421, recognizing the user gesture in the user image.

S422, when the user gesture is a volume adjusting gesture, calculating the recognition success rate of the volume adjusting gesture.

And S423, if the recognition success rate of the volume adjusting gesture exceeds a first threshold, displaying a volume bar in the user interface.

Although the image collector collects the user image including the user gesture in the detection area, if the user gesture swung out by the user is not a gesture for realizing volume adjustment but other gestures which are carelessly swung out, the gesture detection process of volume adjustment does not need to be started, namely, the volume bar does not need to be turned up for volume adjustment.

In addition, if the gesture made by the user is a gesture for adjusting the volume, but the effect of the gesture made by the user is not good, for example, when the volume adjustment gesture is an OK gesture, the middle finger, the ring finger and the small finger of the user should be in a straight state, but the user makes the three fingers bent, so that the gesture made by the user is not like the OK gesture. At this time, the ambiguous user gesture will cause that it is unable to accurately determine whether to recognize the user gesture as a volume adjustment gesture, and cause that the volume adjustment cannot be performed in time due to the false recognition.

Therefore, when the controller performs gesture recognition on each frame of user image, it needs to first determine whether the user gesture in the user image is a volume adjustment gesture, that is, when the similarity that the user gesture is the volume adjustment gesture exceeds a threshold, the user gesture is determined to be the volume adjustment gesture. In the gesture detection process, the ratio of the user image to which the user gesture recognized as the volume adjusting gesture belongs to the total user image needs to exceed a threshold value, and then the fact that the user needs to start the volume adjusting function can be indicated, otherwise, the volume adjusting process is not carried out.

In some embodiments, the controller, when performing that the user gesture is a volume adjustment gesture, calculates a recognition success rate of the volume adjustment gesture, and is further configured to:

step 4221, matching the user gesture and the volume adjusting gesture, and calculating a gesture confidence of the user gesture and the volume adjusting gesture.

Step 4222, if the gesture confidence exceeds a second threshold, determining that the user gesture is a volume adjustment gesture.

Step 4223, acquiring a total number of gesture recognition frames of the user image acquired within the first time period and a number of gesture success frames of the corresponding user image when the user gesture is a volume adjustment gesture.

Step 4224, calculating the ratio of the number of successful gesture frames to the total number of gesture recognition frames, and determining the ratio as the recognition success rate of the volume adjustment gesture.

When the controller identifies the user gestures in each frame of user image, the user gestures in each frame of user image are respectively matched with the volume adjustment gestures to calculate the gesture confidence degrees, namely the similarity, of each user gesture and the volume adjustment gesture.

And determining whether each gesture confidence exceeds a second threshold, wherein in some embodiments, the second threshold may be set to 95%, and if a certain gesture confidence exceeds 95%, determining the user gesture corresponding to the gesture confidence as the volume adjustment gesture so as to avoid generating false recognition.

And meanwhile, determining the user image to which the user gesture with the gesture confidence coefficient exceeding a second threshold value belongs as a gesture success frame. In the gesture detection process, the frame number of the gesture success frame and the total gesture recognition frame number of all the collected user images are counted.

In some embodiments, the duration corresponding to the gesture detection process is a first duration, which may be set to 1 second. The first time period may also be set to other time periods according to practical applications, and is not limited specifically herein.

The first duration is the maximum duration for the image collector to continuously collect the user images including the user gestures, the image collector sequentially sends the multi-frame user images collected in the first duration to the controller, and the controller can count the total gesture recognition frame number of the multi-frame user images received in the first duration.

The controller respectively identifies gestures of each frame of user image, determines the user gesture as a volume adjustment gesture if the gesture confidence of the user gesture and the volume adjustment gesture of a certain frame of user image is identified to exceed a second threshold, and counts the gesture success frame number of the corresponding user image when the user gesture in the first time length is the volume adjustment gesture.

And calculating the ratio of the number of successful frames of the gesture and the number of total gesture recognition frames, namely calculating the ratio of the number of successful frames of the OK gesture detected by the controller in the total gesture recognition frames in the first time period, and determining the ratio as the recognition success rate of the volume adjustment gesture.

In some embodiments, to avoid the user from remaining for a short time after the user has posed the volume adjustment gesture and dropping the gesture, the controller cannot determine whether the volume bar needs to be turned out for volume adjustment. Therefore, for the convenience of timely calling the volume bar by the controller to adjust the volume, the first threshold value can be set, so that the gesture recognition can be regarded as successful only if the recognition success rate of the volume adjustment gesture exceeds the first threshold value in the multi-frame user image collected by the image collector in the first time length, and then the volume bar control is called up to perform the subsequent volume adjustment process.

Therefore, after determining the recognition success rate of the volume adjustment gesture corresponding to the first duration, it is determined whether the recognition success rate of the volume adjustment gesture exceeds a first threshold, in some embodiments, the first threshold may be set to 90%, and if the recognition success rate exceeds the first threshold by 90%, it indicates that the gesture recognition is successful in the current gesture detection process (within the first duration), and at this time, the volume bar may be displayed in the user interface. The first threshold may also be set to other values according to practical applications, and is not limited specifically herein.

If the recognition success rate of the volume adjustment gesture is smaller than the first threshold, the gesture recognition detection fails, the volume adjustment gesture of the same user detected currently may be lost, and the volume adjustment gesture is detected again immediately at the moment.

Before the controller calls the volume bar control to display, the controller needs to perform the step judgment of successful gesture recognition, so that the controller can display a volume adjustment gesture prompt interface in a user interface so that a user can clearly know that the current gesture recognition is successful.

In some embodiments, the controller, in performing if the recognition success rate of the volume adjustment gesture exceeds a first threshold, displays a volume bar in the user interface, further configured to:

step 4231, if the recognition success rate of the volume adjustment gesture exceeds a first threshold, presenting a volume adjustment gesture prompt interface in the user interface, and presenting gesture recognition success prompt information and a volume adjustment gesture pattern in the volume adjustment gesture prompt interface.

Step 4232, when the display duration of the volume adjustment gesture prompt interface exceeds the second duration, canceling the display of the volume adjustment gesture prompt interface, and displaying a volume adjustment interface in the user interface, where the volume adjustment interface includes a volume bar and volume adjustment operation prompt information.

When the recognition success rate of the volume adjustment gesture exceeds a first threshold value, the current gesture is successfully recognized, a subsequent process of triggering volume adjustment after sliding operation is performed based on the volume adjustment gesture can be performed, at the moment, feedback is provided for the volume adjustment gesture made by a user, a volume adjustment gesture prompt interface can be generated, and the volume adjustment gesture prompt interface is displayed in a user interface.

Fig. 23 illustrates a schematic diagram of a user interface displaying a volume adjustment gesture prompt interface, according to some embodiments. Referring to fig. 23, when the gesture recognition is successful, a gesture recognition success prompt message and a volume adjustment gesture pattern are presented in the volume adjustment gesture prompt interface, the gesture recognition success prompt message is used to inform a user that the designated gesture recognition that can currently implement the volume adjustment is successful, the content of the prompt message may be "… in gesture recognition", and after seeing the prompt message, the user can know that the subsequent volume adjustment process is triggered after performing a sliding operation based on the volume adjustment gesture.

The volume adjustment gesture pattern is used to identify the UI form of the volume adjustment gesture, which is typically the same UI form as the user's gesture with a finger. For example, if the volume adjustment gesture is an OK gesture, the volume adjustment gesture pattern is an "OK" pattern.

After seeing the prompt information of the successful gesture recognition and the volume adjusting gesture pattern, the user can know that the gesture recognition is successful as a result of the detection process of the volume adjusting gesture. At the moment, the controller presents a volume adjustment gesture prompt interface in the user interface and simultaneously informs the system to call out the volume bar control in a broadcast sending mode. A volume bar control is displayed in the user interface to prompt the user that the current volume adjustment process is to begin.

Because the volume adjustment gesture prompt interface is displayed in the user interface when the gesture recognition is successful, the display duration can be set for the volume adjustment gesture prompt interface so that the system can automatically display the volume adjustment gesture prompt interface after the volume bar is called. And when the display duration of the volume adjustment gesture prompt interface reaches a threshold value, canceling the display of the volume adjustment gesture prompt interface, and simultaneously displaying a volume bar in the user interface. In the implementation process, the content presented by the user interface is switched and displayed as a volume bar by the volume adjustment gesture prompt interface by the viewing effect of the user.

In some embodiments, the display duration of the volume adjustment gesture prompting interface is set to be a second duration, which may be 500ms, and after the display duration of the volume adjustment gesture prompting interface exceeds 500ms, the display of the volume adjustment gesture prompting interface may be cancelled, and the volume adjustment interface is displayed in the user interface, where the volume adjustment interface includes a volume bar and volume adjustment operation prompting information.

FIG. 24 illustrates a schematic diagram of displaying a volume adjustment interface in a user interface, according to some embodiments. Referring to fig. 24, when the display duration of the volume adjustment gesture prompt interface reaches the threshold, the volume adjustment interface is switched and displayed in the user interface, where the volume adjustment interface includes a volume bar and volume adjustment operation prompt information, the volume bar is used to present the output volume of the current display device, and the volume adjustment operation prompt information is used to prompt the user to perform an operation step of adjusting the volume of the display device by using the volume adjustment gesture, for example, the prompt information may be "move left and right to adjust the volume".

After seeing the volume adjustment operation prompt information in the user interface, the user can execute corresponding gesture operation according to the prompt content to start the volume adjustment process.

And S43, responding to a volume adjusting instruction generated when the user performs the specified action based on the volume adjusting gesture, and adjusting the volume value corresponding to the volume bar.

When the volume adjustment gesture made by the user meets the requirement of starting the volume adjustment process, the user can perform corresponding operation according to the volume adjustment operation prompt information presented in the user interface. That is, after the gesture recognition detection is successful, the user holds the volume adjustment gesture and adjusts the volume by performing a specified action, which in some embodiments may be a slide left or right gesture.

For example, the user can perform volume adjustment by holding the OK gesture with a finger and sliding the OK gesture in the horizontal direction in front of the display device (in the detection area of the image capture device).

When the user keeps the volume adjustment gesture to execute the specified action, such as sliding left and right in the horizontal direction, the finger of the user generates position change, and the effect presented in the user image generates abscissa change for the gesture. At the moment, when the position changes, a volume adjusting instruction can be generated, and the controller responds to the volume adjusting instruction, so that the volume of the volume bar can be adjusted in real time according to the change of the abscissa. At this time, the volume value displayed on the volume bar is changed, such as increased or decreased.

FIG. 25 illustrates a flow chart of a method of adjusting a volume corresponding to a volume bar, according to some embodiments. Referring to fig. 25, in some embodiments, the controller, in executing the volume adjustment instruction generated in response to the user performing the specified action based on the volume adjustment gesture, adjusts the volume value corresponding to the volume bar, and is further configured to:

and S431, receiving a volume adjustment instruction generated when the user executes a designated action based on the volume adjustment gesture, wherein the designated action is the action generated by the user based on the volume adjustment operation prompt information.

And S432, responding to the volume adjusting instruction, and acquiring a starting coordinate value and a stopping coordinate value when the user executes the specified action, which are presented in the user image in the third time length.

And S433, calculating the abscissa variation generated when the user executes the specified action based on the volume adjusting gesture based on the initial coordinate value and the termination coordinate value.

And S434, determining the volume adjustment value and the volume adjustment direction of the volume bar based on the variation of the abscissa.

And S435, adjusting the volume value corresponding to the volume bar based on the volume adjustment value and the volume adjustment direction of the volume bar.

The user executes a designated action according to the volume adjustment operation prompt information presented in the user interface, for example, the user holds an OK gesture with a finger, slides left and right in the horizontal direction in front of the display device (in a detection area of the image acquirer), generates a position change, and generates a volume adjustment instruction at this time.

Because the time for the image collector to collect one frame of user image is 30-40ms under normal conditions, in order to ensure the real-time performance of volume adjustment, when a gesture detection algorithm is carried out, the position change generated by the user keeping the volume adjustment gesture to execute the specified action is calculated according to every 100ms, so that the volume is linearly adjusted according to the position change information.

Therefore, after responding to the volume adjustment instruction, the controller can acquire the start coordinate value and the end coordinate value of the user performing the specified action presented in the user image in the third time length so as to calculate the abscissa variation generated when the user performs the specified action based on the volume adjustment gesture. In some embodiments, the third duration may be set to 100ms, and the change in position generated by the user slide gesture may be equivalent to the amount of change in abscissa generated in the user image.

And the image collector collects the user image at the initial moment when the user executes the specified action and the user image at the termination moment in the third duration in real time. And establishing a rectangular coordinate system by taking the upper left corner of the user image as the origin of coordinates, taking the X-axis forward direction from left to right and the Y-axis forward direction from top to bottom.

In the rectangular coordinate system, a pixel coordinate value of a volume adjustment gesture in a user image acquired at an initial time is an initial coordinate value, a pixel coordinate value of the volume adjustment gesture in the user image acquired at a termination time is a termination coordinate value, and an abscissa variation is calculated based on an abscissa value of the initial coordinate value and an abscissa value of the termination coordinate value. Wherein, the coordinate values are all expressed by pixel coordinates.

The termination time refers to a time corresponding to each time of image acquisition within a third duration, for example, after 30-40ms after the initial time, the image acquisition device acquires a frame of user image, and the time for completing the image acquisition is the termination time; and after 30-40ms, the image collector collects a frame of user image, and the time for completing the second image collection is another termination time. Because each frame of user image is collected, a termination time is generated, and therefore the final termination time is the corresponding time when the abscissa change rate meets the threshold condition.

FIG. 26 illustrates a schematic diagram of calculating an amount of abscissa variation, according to some embodiments. Referring to fig. 26, at the initial time when the user performs the designated action, the position of the volume adjustment gesture corresponding to the user is point a, and at this time, the initial coordinate value is a (x0, y 0); in the third time length, if the image acquisition processes are performed twice, the positions reached by the user sliding gesture are points B1 and B2, respectively, and at this time, the ending coordinate values are points B1(x1, y1) and point B2(x2, y2), respectively.

When B1 is the termination coordinate value, the abscissa change L1, which is generated when the corresponding user performs the specified operation based on the volume adjustment gesture, is x1-x 0. When B2 is the termination coordinate value, the abscissa change L2, which is generated when the corresponding user performs the specified operation based on the volume adjustment gesture, is x2-x 0.

In some embodiments, to avoid the phenomena of frame loss and pause during the process of capturing images by the image capturing device, a threshold condition may be set when determining the volume adjustment value and the volume adjustment direction of the volume bar based on the variation of the abscissa.

Specifically, the controller, in performing determining the volume adjustment value and the volume adjustment direction of the volume bar based on the amount of change in the abscissa, is further configured to perform the steps of:

step 4341, if the abscissa variation is greater than the third threshold, determining the volume adjustment value of the volume bar as a designated adjustment amount, and adjusting the direction as increasing the volume.

Step 4342, if the abscissa variation amount is smaller than the fourth threshold, determining the volume adjustment value of the volume bar as the designated adjustment amount, and adjusting the direction as decreasing the volume.

In some embodiments, a third threshold and a fourth threshold may be set according to the adjustment effect of increasing the volume and decreasing the volume, the third threshold may be 8, and the fourth threshold may be-8. Threshold 8 represents 8 pixels.

If the abscissa change rate is greater than the third threshold 8, it indicates that the user has left the volume adjust gesture to slide to the right, at which point the adjustment direction of the volume bar is determined to be increasing volume. If the abscissa change rate is less than the fourth threshold of-8, it indicates that the user has left the volume adjust gesture to slide to the left, at which point the adjustment direction of the volume bar is determined to be decreasing volume.

To achieve linear adjustment of the volume, the volume adjustment value, which is adjusted every volume, may be set to 3 volumes. For example, if the abscissa change amount is greater than the third threshold, the volume of the volume bar is increased by 3 volumes based on the current value; and if the abscissa change amount is smaller than the fourth threshold, decreasing the volume of the volume bar by 3 volumes based on the current value.

Taking the state shown in fig. 26 as an example, the user slides to the left based on the volume adjustment gesture, and the image capturing device captures two frames of user images in the third duration, at this time, the abscissa change amount generated when the user represented in the user image (corresponding to the position of B1 point) corresponding to the start time performs the specified action based on the volume adjustment gesture is calculated to be L1 ═ x1-x0, and the abscissa change amount generated when the user represented in the user image (corresponding to the position of B2) corresponding to the start time performs the specified action based on the volume adjustment gesture is calculated to be L2 ═ x2-x 0.

If L1> -8 (fourth threshold), the amount of change in the position indicating that the user holds the volume adjust gesture to slide from point A to point B1 is not enough to trigger the volume adjustment process, and the volume of the volume bar is not adjusted. If L2< -8 (the fourth threshold), the amount of change in the position indicating that the user holds the volume adjust gesture sliding from point A to point B2 is sufficient to trigger the volume adjustment process, at which point the volume of the volume bar may be adjusted.

After the volume adjustment value and the volume adjustment direction of the volume bar are determined, the adjustment of the volume corresponding to the volume bar can be realized, for example, when the volume adjustment value is 3 and the volume adjustment direction is volume increase, 3 volume values are increased on the basis of the current volume value of the volume bar. When the volume adjustment value is 3 and the volume adjustment direction is volume reduction, 3 volume values are reduced on the basis of the current volume value of the volume bar.

In some embodiments, when the controller linearly adjusts the volume of the volume bar according to the abscissa variation in the third duration, that is, when it is detected that the abscissa variation corresponding to the user during the sliding gesture generates a variation of 8 pixels within 100ms, the controller performs a volume adjustment process once, and adjusts 3 volume values at a time.

The adjustment direction of the volume can be determined according to the positive and negative numbers of the detected 8 pixel point changes, for example, if the numerical value corresponding to the detected 8 pixel points is a negative number, it indicates that the volume needs to be reduced; if the detected numerical value corresponding to the 8 pixel points is a positive number, the volume needs to be increased.

It should be noted that the set gesture detection time length (third time length), the position change threshold values (third threshold value and fourth threshold value) of 8 pixel points, and the adjustment values (specified adjustment amounts) of 3 volumes may also be set to other values according to the actual application, which are only used as examples herein and are not specifically limited.

In some embodiments, in the volume adjustment process, a phenomenon of frame loss and pause in gesture detection caused by an excessively large occupation of a display device CPU often occurs in the gesture sliding, so that the user gesture moves within a third duration (100ms) but does not correspondingly generate abscissa change in a user image, which causes an phenomenon of volume non-adjustment, and affects the accuracy of volume adjustment.

Therefore, to ensure accurate volume adjustment even in the presence of frame loss and a stuck phenomenon, in determining the start coordinate value and the end coordinate value, the controller is further configured to:

and 4321, if the abscissa variation is zero within the third time duration, prolonging the gesture detection time duration according to the third time duration.

4322, based on the total duration corresponding to the gesture detection duration, obtaining an initial coordinate value and a termination coordinate value when the user performs the specified action based on the volume adjustment gesture, where the total duration of the gesture detection duration refers to the total durations corresponding to the plurality of third durations, the initial coordinate value is the initial coordinate value corresponding to the first third duration, and the termination coordinate value is the termination coordinate value corresponding to the last third duration.

When the volume adjusting process is started, the controller performs gesture detection in real time according to the user image collected by the image collector so as to determine the abscissa variation generated when the gesture slides. If no position change is detected within the third duration (100ms), i.e., the abscissa change amount is zero, during the gesture detection process, the duration of the third duration (100ms) will continue to be delayed until the volume adjustment threshold is reached (the abscissa change amount satisfies the third threshold or the fourth threshold).

At this time, the total gesture detection time length is the time length corresponding to the prolonged third time length, at this time, the initial coordinate value generated when the user executes the specified action based on the volume adjustment gesture is still the initial coordinate value corresponding to the initial time, and the initial time is the time corresponding to the start of the gesture sliding in the first third time length; the termination coordinate value is a termination coordinate value corresponding to a termination time within the last third time period, and the termination time is a time corresponding to a time when the abscissa change rate within the last third time period meets the threshold condition.

For example, if the abscissa variation is not detected within the first 100ms during the gesture detection process, the time is delayed by 100ms again, and if the abscissa variation generated at the 150 th ms (within the second 100ms) meets the threshold condition, the pixel coordinate corresponding to the gesture position detected at the initial time when the gesture sliding is generated within the first 100ms is the start coordinate value, and the pixel coordinate corresponding to the gesture position detected at the 150ms (end time) is the end coordinate value.

Therefore, if the abscissa variation is not detected within the gesture detection time (third time), the time corresponding to the third time is continuously delayed until the abscissa variation reaches the volume adjustment threshold (third threshold or fourth threshold).

In some embodiments, when the user performs a sliding motion to adjust the volume of the display device based on the volume adjustment gesture, a volume adjustment status prompt may be displayed in the user interface to prompt the user that the user is currently in the process of volume adjustment.

Fig. 27 illustrates a schematic diagram of displaying volume adjustment status prompt information in a user interface, in accordance with some embodiments. Referring to fig. 27, in the volume adjustment process, the controller is further configured to: and responding to the volume adjustment instruction, and switching and displaying the volume adjustment operation prompt information presented in the user interface into volume adjustment state prompt information.

In the volume adjustment process, the volume adjustment interface is always displayed in the user interface, specifically, the volume bar is always displayed in the user interface, and a volume value corresponding to the volume bar is correspondingly changed, for example, increased or decreased, along with the sliding operation of the volume adjustment gesture of the user.

In order to prompt the user that the user is currently in the process of volume adjustment, the volume adjustment operation prompt message in the volume adjustment interface may be cancelled, instead of displaying the volume adjustment state prompt message, the prompt content of the volume adjustment state prompt message may be "… … in volume adjustment", and the like.

In some embodiments, after the user adjusts the volume of the display device to the volume required by the user, the user may put down the held volume adjustment gesture, and at this time, the image acquirer does not include the volume adjustment gesture any more in the next frame of user image acquired, which may indicate that the volume adjustment process of the user is completed.

Because of no gesture sliding, the volume of the volume bar does not slide left and right along with the gesture, namely the volume is not adjusted. Therefore, after the user completes the volume adjustment process, the volume adjustment interface (volume bar) displayed in the user interface is cancelled, and after the controller monitors that the volume bar disappears, the volume adjustment completion interface is generated and displayed in the user interface to prompt the user that the volume adjustment is completed.

Fig. 28 illustrates a schematic diagram of displaying a volume adjustment completion interface in a user interface, according to some embodiments. Referring to fig. 28, in order to prompt the user that the volume adjustment is completed, a volume adjustment completion pattern and volume adjustment completion prompt information are displayed in the volume adjustment completion interface, the volume adjustment completion pattern may be in the form of a "number matching" UI, and the content of the volume adjustment completion prompt information may be "volume adjustment was successful", and the like.

In some embodiments, when the user performs a sliding operation to adjust the volume based on the volume adjustment gesture, if the user keeps the volume adjustment gesture but does not perform the sliding operation, the abscissa variation is zero, which indicates that the volume of the volume bar is not increased or decreased. Because the volume adjustment is not carried out in the current state, the controller can identify the state as a volume adjustment completion state, and the influence on normal use of the display equipment by a user due to the fact that a volume adjustment interface is always displayed in a user interface is avoided.

To ensure that the controller can accurately detect whether the volume adjustment process is complete, the duration of the state with zero abscissa change rate may be defined, and in particular, the controller is further configured to: and within the fourth time period, when the abscissa variation generated when the user executes the designated action based on the volume adjustment gesture is zero, canceling the display of the volume bar, and presenting a volume adjustment completion interface in the user interface, wherein the volume adjustment completion interface comprises a volume adjustment completion pattern and volume adjustment completion prompt information.

In some embodiments, the duration of the state in which the abscissa change rate is zero is set to be a fourth duration, where the fourth duration refers to a duration from a time when the volume of the volume bar is changed for the last time to the current time, and also refers to a duration from a time corresponding to the time when the user stops sliding after keeping the volume adjustment gesture for sliding operation to the current time. The fourth time period may be set to 2 seconds, or may be other values, and may be set according to practical application conditions, and is not specifically limited herein.

If the abscissa variation is always zero after the fourth time period after the user performs the volume adjustment once, it indicates that the gesture of the user does not slide within 2 seconds, and further, the volume is not increased or decreased. Therefore, the volume adjusting interface (volume bar) displayed in the user interface is cancelled, and after the controller monitors that the volume bar disappears from broadcasting, a volume adjusting completion interface is generated and displayed in the user interface to prompt the user that the volume adjustment is completed. The volume adjustment completion interface is shown in fig. 28.

In some embodiments, if the user completes the volume adjustment process once, i.e., after the volume adjustment completion interface is displayed in the user interface, the controller may immediately respond to the sliding operation of the gesture to adjust the volume value of the display device if the user makes the volume adjustment gesture again. The one-time complete volume adjustment process is a process of successfully identifying a user image, recognizing a gesture, adjusting the volume through a sliding gesture, starting the gesture to adjust the volume, and finishing the volume adjustment.

Specifically, after completing one complete volume adjustment process, the controller is further configured to perform the following steps:

and step 441, after the current volume adjustment process is completed, acquiring the next frame of user image including the user gesture, which is acquired by the image acquisition device.

Step 442, when the user gesture in the next frame of user image is a volume adjustment gesture, executing a next volume adjustment process.

After the current volume adjustment process is completed completely, if the user makes a user gesture again in the detection area of the image collector, the image collector collects the next frame of user image including the user gesture and sends the next frame of user image to the controller. And after receiving a new frame of user image, the controller judges the volume adjusting gesture again, and when judging that the user gesture in the new user image is the volume adjusting gesture, the controller immediately responds to the volume adjusting instruction generated by executing the specified action based on the volume adjusting gesture according to the volume adjusting instruction so as to execute the next volume adjusting process. For the specific volume adjustment process, reference may be made to the contents of the foregoing embodiments, and details are not described herein.

In some embodiments, if in the gesture recognition stage, the user puts down immediately after putting a certain gesture, the duration of the user holding the gesture does not reach the first duration, but the user puts down the gesture immediately after putting the gesture, so that the controller frequently recognizes the user gesture in each frame of user image, and normal operation of the display device is affected. Therefore, when the process of starting the volume adjustment is forcibly terminated and there is a state where the user frequently turns on the volume adjustment function, the controller does not immediately perform gesture recognition on the user image, but performs gesture recognition after a period of time.

Specifically, in a case where there is a gesture frequently made by the user for a short time, causing the volume adjustment process to stop halfway, the controller is further configured to: and if the user gesture is not included in the acquired user image within the first time length, acquiring the next frame of user image acquired by the image acquisition device after a fifth time length.

The user puts down immediately after putting a certain gesture, at this time, the duration of the user holding the gesture does not reach the first duration, therefore, after the user puts the gesture for the first time, the controller does not detect whether the gesture is a volume adjustment gesture, the recognition success rate of the volume adjustment gesture may not exceed the first threshold, that is, the volume adjustment process is not started yet.

In order to avoid that the user frequently starts the volume adjustment process in a short time, the time of the fifth time interval can be set after the moment that the user immediately puts down after a certain gesture is put out, and then the image collector collects the next frame of user image, or when the image collector collects the user image in real time, the controller can obtain the corresponding user image after the fifth time interval.

In some embodiments, the fifth duration refers to a time interval from a moment when a user makes a certain gesture, and the image collector collects a user image including the gesture to a moment when a next frame of user image including the gesture is collected. The fifth time period may be set to 3 seconds, or may be set to other values according to practical applications, and is not limited specifically herein.

For example, a user makes a gesture at a time of 8:05, at which time the image capturer captures an image of the user that includes the gesture at a time of 8: 05. If the user immediately drops the gesture and lifts the gesture again, this time at an interval of 3 seconds, the image capturer may capture an image of the user including the gesture after time 8:08, or the controller may capture an image of the user including the gesture captured by the image capturer after time 8: 08.

In some embodiments, to ensure that the volume adjustment process is performed properly, the volume adjustment process is performed when a volume bar is displayed in the user interface, and the gesture detection process for starting the volume adjustment is not repeated. The controller performs gesture recognition on the user image acquired by the image acquisition device, and starts a process of adjusting the volume through a sliding gesture after the gesture recognition is successful. In the process, the volume bar is turned up and is always displayed in the user interface, and the subsequent collected user images including the volume adjusting gesture are not subjected to repeated gesture detection process, namely the process of adjusting the volume through the sliding gesture is not started repeatedly.

In some embodiments, if the volume adjustment process is completed once, and the volume bar is not displayed in the user interface, the gesture detection process of the volume adjustment is started to perform the volume adjustment process. The one-time complete volume adjustment process is a process of successfully identifying a user image, recognizing a gesture, adjusting the volume through a sliding gesture, starting the gesture to adjust the volume, and finishing the volume adjustment.

After the complete volume adjustment process is completed once, the display of the volume bar is cancelled in the user interface, which can indicate that the volume adjustment is completed this time, and at this time, the next volume adjustment process can be started, that is, the user image and the subsequent detection process are collected again.

Therefore, according to the display device provided by the embodiment of the invention, the image collector collects the user image in real time, and the controller performs gesture recognition on the user image after acquiring the user image which is collected by the image collector and comprises the user gesture; if the user gesture in the user image is recognized to be a volume adjusting gesture, starting an intelligent adjusting function of the volume of the display device through sliding the volume adjusting gesture, and displaying a volume bar in a user interface; and the user generates position change when executing the designated action based on the volume adjusting gesture, generates a corresponding volume adjusting instruction, and the controller responds to the volume adjusting instruction to start adjusting the volume value corresponding to the volume bar. Therefore, the display device provided by the embodiment of the invention identifies the person gesture in the image and the position in the image by using the AI image detection technology, and realizes the intelligent adjustment of the volume through the position change of the person gesture in the image.

In some embodiments, based on the foregoing, the completion of one volume adjustment process includes a complete volume adjustment process and a volume adjustment process that is stopped halfway, while the execution steps of the display device are different when the next volume adjustment is performed according to different forms of volume adjustment processes.

Fig. 29 illustrates another flow diagram of a method of volume adjustment according to some embodiments. An embodiment of the present invention provides a display device, including: a display configured to present a user interface; an image collector configured to collect a user image; a controller connected to the display and the image collector, respectively, the controller being configured to perform the following steps when executing the volume adjusting method shown in fig. 29:

and S51, receiving a volume adjustment termination notice generated when the user completes the previous volume adjustment process.

The display device generates a volume adjustment termination notification each time the volume adjustment process is performed, and the volume adjustment termination notification may be generated when the controller detects that there is no gesture scene in the user image or no gesture sliding scene.

The scene with no gesture in the user image may be a scene in which the user puts down the volume adjusting gesture after completing the volume adjusting process, or a scene in which the user has just lifted the gesture and quickly puts down the volume adjusting gesture (at this time, the volume adjusting process is not started yet).

The gesture no-slide scene may be a scene in which the user has finished adjusting the volume once sliding, or after the gesture recognition is successful, the volume adjustment function is started, but the user does not adjust the volume through the sliding gesture.

According to different scenes, the system can generate different volume adjustment termination notices and send the notice to the controller in a broadcasting mode, so that the controller can execute the next volume adjustment process by adopting different strategies according to the volume adjustment termination notices in different scenes.

And S52, responding to the notice of the termination of the volume adjustment, and acquiring the next frame of user image which is acquired by the image acquisition device and comprises the user gesture.

The volume adjustment termination notification is generated in different scenes, so that in different scenes, the volume adjustment termination notification comprises a volume adjustment completion notification and a volume adjustment non-proceeding notification, the volume adjustment completion notification is generated in the process that the display device completely executes a volume adjustment process, the volume adjustment non-proceeding notification is generated in the process that the display device disappears due to midway gestures, and the volume adjustment process is forcibly terminated.

In some embodiments, the volume adjustment termination notification is a volume adjustment completion notification when the volume adjustment process is completely completed once, and at this time, the controller is further configured to, in response to the volume adjustment termination notification, acquire a next frame of user image including a user gesture captured by the image capturing unit, perform the following steps:

step 5211, in the previous volume adjustment process, if the abscissa variation amount generated when the user performs the specified action based on the volume adjustment gesture is zero in the fourth time period, a volume adjustment completion notification is generated.

Step 5212, in response to the notification of volume adjustment completion, acquiring the next frame of user image including the user gesture collected by the image collector.

In the previous volume adjustment process, if the abscissa variation generated when the user executes the designated action based on the volume adjustment gesture is zero within the fourth time period, it indicates that the display device has normally completed a volume adjustment process, and at this time, the volume bar cancels the display and generates a volume adjustment completion notification at the same time.

The controller responds to the sound volume adjustment completion notice, can display a sound volume adjustment completion interface in the user interface, and immediately acquires the next frame of user image acquired by the image acquisition device so as to perform gesture recognition according to the next frame of user image, thereby being convenient for immediately starting the next sound volume adjustment process.

In some embodiments, when the volume adjustment process is not completed completely, the volume adjustment termination notification is a volume adjustment non-proceeding notification, and at this time, the controller is further configured to perform the following steps in response to the volume adjustment termination notification, acquiring a next frame of user image including a user gesture acquired by the image acquirer:

step 5221, in the previous volume adjustment process, if the next captured frame of user image does not include a volume adjustment gesture, generating a volume adjustment non-notification.

5222, in response to the fact that the volume adjustment is not performed, after a fifth time interval, acquiring a next frame of user image including the user gesture acquired by the image acquirer.

In the previous volume adjustment process, if the next frame of collected user image does not include a volume adjustment gesture, for example, in a gesture recognition stage, the user immediately puts down after swinging a certain gesture, at this time, the duration of the user holding the gesture does not reach the first duration, therefore, after the user swings the gesture for the first time, the controller does not detect whether the user is the volume adjustment gesture, the recognition success rate of the volume adjustment gesture may not exceed the first threshold, that is, the volume adjustment process is not started yet.

If the next frame of user image collected by the image collector does not comprise the volume adjusting gesture, the gesture disappears in the midway of the volume adjusting process, at the moment, the process of starting the volume adjusting is forcibly terminated, and a notification that the volume adjusting is not performed is generated.

And the controller responds to the fact that the volume adjustment is not carried out, and acquires the next frame of user image which is acquired by the image acquisition device and comprises the user gesture after a fifth time interval. The method can avoid the influence on the normal operation of the display equipment caused by the fact that the controller frequently identifies the user gestures in each frame of user image.

And S53, if the user gesture in the next frame of user image is a volume adjusting gesture, displaying a volume bar in the user interface.

And S54, responding to the volume adjusting instruction generated when the user executes the designated action based on the volume adjusting gesture, and adjusting the volume corresponding to the volume bar to execute the next volume adjusting process.

After the volume adjustment process is completed for one time, the next volume adjustment process is started, at this time, the controller executes a corresponding recognition gesture, that is, a subsequent process, according to the obtained next frame of user image, so as to adjust the volume of the volume bar according to the position change generated by the user through the sliding gesture, and for related specific implementation contents, reference may be made to steps S11 to S13 and contents of related parts in the foregoing embodiment, which are not described herein again.

It can be seen that, in the display device according to the embodiment of the present invention, after a volume adjustment process is completed, a volume adjustment termination notification generated when a user completes a previous volume adjustment process is received, when the volume adjustment termination notification is the volume adjustment completion notification, the controller may immediately respond to a next volume adjustment process, and when the volume adjustment termination notification is the volume adjustment non-notification, the controller may respond to the next volume adjustment process after a fifth time interval. Therefore, the display equipment can execute the corresponding processing process according to the different forms of the received volume adjustment termination notice when the next volume adjustment is carried out, so that the volume adjustment efficiency is improved, and the user experience is good.

Fig. 20 illustrates a flow diagram of a method of volume adjustment according to some embodiments. Referring to fig. 20, the present application also provides a volume adjustment method, which is executed by the controller in the display device provided in the foregoing embodiment, and the method includes:

s41, acquiring a user image including a user gesture, which is acquired by the image acquirer;

s42, if the user gesture in the user image is recognized to be a volume adjusting gesture, displaying a volume bar in the user interface;

and S43, responding to a volume adjusting instruction generated when the user performs a specified action based on the volume adjusting gesture, and adjusting the volume value corresponding to the volume bar.

Fig. 29 illustrates another flow diagram of a method of volume adjustment according to some embodiments. Referring to fig. 29, the present application also provides a volume adjustment method, which is executed by the controller in the display device provided in the foregoing embodiment, the method including:

s51, receiving a volume adjustment termination notice generated when the user completes the previous volume adjustment process;

s52, responding to the notification of the termination of the volume adjustment, and acquiring the next frame of user image which is acquired by the image acquisition device and comprises the user gesture;

s53, if the user gesture in the next frame of user image is a volume adjusting gesture, displaying a volume bar in the user interface;

and S54, responding to a volume adjusting instruction generated when the user executes the designated action based on the volume adjusting gesture, and adjusting the volume corresponding to the volume bar to execute the next volume adjusting process.

According to the technical scheme, the volume adjusting method and the display device provided by the embodiment of the invention have the advantages that the image collector collects the user image in real time, and the controller acquires the user image which is collected by the image collector and comprises the user gesture and performs gesture recognition on the user image; if the user gesture in the user image is recognized to be a volume adjusting gesture, starting an intelligent adjusting function of the volume of the display device through sliding the volume adjusting gesture, and displaying a volume bar in a user interface; and the user generates position change when executing the designated action based on the volume adjusting gesture, generates a corresponding volume adjusting instruction, and the controller responds to the volume adjusting instruction to start adjusting the volume value corresponding to the volume bar. Therefore, the method and the display device provided by the embodiment of the invention use the AI image detection technology to identify the person gesture in the image and the position in the image, and realize the intelligent adjustment of the volume through the position change of the person gesture in the image.

In specific implementation, the present invention further provides a computer storage medium, where the computer storage medium may store a program, and the program may include some or all of the steps in each embodiment of the control method and the volume adjustment method for a display device provided by the present invention when executed. The storage medium may be a magnetic disk, an optical disk, a read-only memory (ROM) or a Random Access Memory (RAM).

Those skilled in the art will readily appreciate that the techniques of the embodiments of the present invention may be implemented as software plus a required general purpose hardware platform. Based on such understanding, the technical solutions in the embodiments of the present invention may be essentially or partially implemented in the form of a software product, which may be stored in a storage medium, such as ROM/RAM, magnetic disk, optical disk, etc., and includes several instructions for enabling a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the method according to the embodiments or some parts of the embodiments.

The same and similar parts in the various embodiments in this specification may be referred to each other. In particular, as for the embodiments of the control method and the volume adjustment method of the display device, since they are basically similar to the embodiments of the display device, the description is simple, and the relevant points can be referred to the description in the embodiments of the display device.

Finally, it should be noted that: the above embodiments are only used for illustrating the technical solutions of the present application, and not for limiting the same; although the present application has been described in detail with reference to the foregoing embodiments, it should be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some or all of the technical features may be equivalently replaced; and the modifications or the substitutions do not make the essence of the corresponding technical solutions depart from the scope of the technical solutions of the embodiments of the present application.

The foregoing description, for purposes of explanation, has been presented in conjunction with specific embodiments. However, the illustrative discussions above are not intended to be exhaustive or to limit the embodiments to the precise forms disclosed above. Many modifications and variations are possible in light of the above teaching. The embodiments were chosen and described in order to best explain the principles and the practical application, to thereby enable others skilled in the art to best utilize the embodiments and various embodiments with various modifications as are suited to the particular use contemplated.

Claims

1. A display device, comprising:

a display configured to present a user interface;

responding to a switch opening instruction generated when a gesture detection function switch in the user interface is triggered, and detecting whether the image collector is occupied by a specified application needing to start the image collector or not based on the attribute state of the image collector;

if the attribute state is a first state, determining that the image collector is occupied by a specified application needing to start the image collector, and not executing a gesture detection function;

and if the attribute state is the second state, determining that the image collector is not occupied by the appointed application which needs to start the image collector, executing a gesture detection function, wherein the gesture detection function is used for detecting the user image comprising the user gesture, and correspondingly controlling the display equipment based on the user gesture.

2. A display device, comprising:

under the condition that the gesture detection function is started, acquiring a user image which is acquired by an image acquisition device and comprises a user gesture;

when detecting that the user gesture in the user image is a play pause control gesture, acquiring a play mode for playing the video file;

if the playing form of the video file is a full-screen playing form, responding to a control instruction generated by the playing pause control gesture, and performing playing operation or pause operation on the video file;

and if the playing form of the video file is a small window playing form, not responding to the control instruction generated by the playing pause control gesture.

3. The display device according to claim 2, wherein the controller, prior to performing the retrieving the playback style of playing the video file, is further configured to:

acquiring a signal source ID for identifying a channel type;

4. A display device, comprising:

a display configured to present a user interface;

if the user gesture represents a playing and pausing control gesture, calculating a first effective successful frame rate of the playing and pausing control gesture, and if the first effective successful frame rate of the playing and pausing control gesture exceeds a first effective threshold, responding to a control instruction corresponding to the playing and pausing control gesture to execute playing operation or pausing operation;

and if the user gesture represents a mute control gesture, responding to a control instruction corresponding to the mute control gesture to execute mute operation or anti-mute operation.

5. The display device of claim 4, wherein the controller, in determining whether the user gesture is a play pause control gesture, is further configured to:

6. The display device of claim 4, wherein the controller, in performing the calculating the first effective frame rate of success for the play-pause control gesture if the user gesture characterizes the play-pause control gesture, is further configured to:

7. The display device according to claim 4, wherein the controller, in executing the control instruction corresponding to the play/pause control gesture to execute a play operation or a pause operation, is further configured to:

8. The display device according to claim 7, wherein the controller, in executing the control instruction corresponding to the play/pause control gesture to execute a play operation or a pause operation, is further configured to:

9. The display device according to claim 8, wherein the controller, in performing the performing if the play-form broadcast characterizes a full-screen play form, performing a play operation or a pause operation on the video file in response to the play-pause control instruction, is further configured to:

10. The display device according to claim 4, wherein the controller, in executing the mute operation or the anti-mute operation in response to the control instruction corresponding to the mute control gesture if the user gesture is the mute control gesture, is further configured to:

11. The display device according to claim 10, wherein the controller, in performing the determining whether the user gesture is a mute control gesture, is further configured to:

determining that the user gesture is a mute control gesture if the mute gesture confidence exceeds a gesture success threshold.

12. The display device of claim 10, wherein the controller, in performing the calculating the first effective frame rate of success for the mute control gesture when the user gesture is the mute control gesture, is further configured to:

13. The display device according to claim 10, wherein the controller, in executing the control instruction corresponding to the mute control gesture to execute a mute operation or an anti-mute operation, is further configured to:

14. The display device according to claim 13, wherein the controller, in executing the control instruction corresponding to the mute control gesture to perform a mute operation or an anti-mute operation, is further configured to:

15. The display device of claim 4, wherein the controller is further configured to:

16. The display device of claim 4, wherein the controller is further configured to:

when the user gesture in the next frame of user image is a play pause control gesture, responding to a control instruction corresponding to the play pause control gesture to execute the next play operation or pause operation;

17. The display device of claim 4, wherein the controller is further configured to:

18. A method of controlling a display device, the method comprising:

responding to a switch opening instruction generated when a gesture detection function switch in a user interface is triggered, and detecting whether the image collector is occupied by a specified application needing to start the image collector or not based on the attribute state of the image collector;

19. A method of controlling a display device, the method comprising:

when detecting that the user gesture in the user image is a play pause control gesture, acquiring a play mode of playing a video file;

20. A method of controlling a display device, the method comprising: