WO2019228236A1 - 显示设备中人机交互方法、装置、计算机设备和存储介质 - Google Patents

显示设备中人机交互方法、装置、计算机设备和存储介质 Download PDF

Info

Publication number
WO2019228236A1
WO2019228236A1 PCT/CN2019/087901 CN2019087901W WO2019228236A1 WO 2019228236 A1 WO2019228236 A1 WO 2019228236A1 CN 2019087901 W CN2019087901 W CN 2019087901W WO 2019228236 A1 WO2019228236 A1 WO 2019228236A1
Authority
WO
WIPO (PCT)
Prior art keywords
user
different
data
display area
face
Prior art date
Application number
PCT/CN2019/087901
Other languages
English (en)
French (fr)
Inventor
刘国华
Original Assignee
Liu Guohua
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Liu Guohua filed Critical Liu Guohua
Priority to US17/059,091 priority Critical patent/US11416068B2/en
Priority to KR1020207037972A priority patent/KR20210013631A/ko
Publication of WO2019228236A1 publication Critical patent/WO2019228236A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G06F3/013—Eye tracking input arrangements
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/16—Human faces, e.g. facial parts, sketches or expressions
    • G06V40/161—Detection; Localisation; Normalisation
    • G06V40/166—Detection; Localisation; Normalisation using acquisition arrangements
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
    • G06F21/30—Authentication, i.e. establishing the identity or authorisation of security principals
    • G06F21/31—User authentication
    • G06F21/32—User authentication using biometric data, e.g. fingerprints, iris scans or voiceprints
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G06F3/012—Head tracking input arrangements
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/44—Arrangements for executing specific programs
    • G06F9/451—Execution arrangements for user interfaces
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/16—Human faces, e.g. facial parts, sketches or expressions
    • G06V40/161—Detection; Localisation; Normalisation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/16—Human faces, e.g. facial parts, sketches or expressions
    • G06V40/172—Classification, e.g. identification
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/16—Human faces, e.g. facial parts, sketches or expressions
    • G06V40/174—Facial expression recognition
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/18—Eye characteristics, e.g. of the iris
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/14—Digital output to display device ; Cooperation and interconnection of the display device with other functional units
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/16—Human faces, e.g. facial parts, sketches or expressions
    • G06V40/178—Human faces, e.g. facial parts, sketches or expressions estimating age from face image; using age information for improving recognition

Definitions

  • the present application relates to the field of human-computer interaction technology, and in particular, to a human-computer interaction method, device, computer device, and storage medium in a display device.
  • Human-computer interaction refers to the process of exchanging information between a person and a device in order to complete a certain task in a certain interaction manner using a certain dialogue language between the person and the device.
  • human-computer interaction technology With the development of science and technology, the application fields of human-computer interaction technology are becoming wider and wider, such as the radio play button, as large as the dashboard of an aircraft, or the control room of a power plant. Users can use the human-computer interaction interface. Communicate and operate with the system. At present, in the human-computer interaction technology, the human-computer interaction mode is single, and the user needs to issue specific voice instructions before the display device responds to perform further operations.
  • the traditional human-computer interaction method can realize the human-computer interaction function to a certain extent, due to the single interaction method, certain specific gestures need to be set in advance, and the interaction process is not very natural, which brings inconvenience to the user to a certain extent .
  • a human-computer interaction method in a display device includes:
  • the method further includes:
  • displaying a plurality of different candidate data in different display areas includes:
  • displaying a plurality of different candidate data in different display areas includes:
  • a recognition process is performed, and a plurality of different data to be selected are displayed in different display areas according to the recognition process result.
  • the recognition process includes at least one of face recognition, age recognition, expression recognition, and gender recognition.
  • collecting the user's face image includes:
  • the human-computer interaction method in the display device further includes:
  • a plurality of different candidate data associated with the identity of the user are displayed in different display areas.
  • the method further includes:
  • a plurality of different data to be selected are displayed in different display areas.
  • the target display area identifying the user's attention includes:
  • the candidate display area corresponding to the first accumulated dwell time exceeding the preset attention time threshold is selected as the target display area that the user pays attention to.
  • a human-computer interaction device in a display device includes:
  • An image data acquisition module configured to acquire a user's current image data
  • a display module configured to display a plurality of different candidate data in different display areas when the user is detected to be facing up;
  • a recognition module for identifying a target display area that is of interest to the user
  • the execution module is configured to read data to be selected corresponding to the target display area, and execute operations corresponding to the read data to be selected.
  • a computer device includes a memory and a processor.
  • the memory stores a computer program.
  • the processor executes the computer program, the steps of the method are implemented.
  • a computer-readable storage medium has stored thereon a computer program that, when executed by a processor, implements the steps of the method as described above.
  • the human-computer interaction method, device, computer device and storage medium in the above display device obtains the user's current image data.
  • a plurality of different candidate data are displayed in different display areas, and the user browses the candidate.
  • the display device recognizes the target display area of interest, reads the candidate data corresponding to the target display area, and performs an operation corresponding to the read candidate data.
  • the user's needs can be sensed without excessive operation by the user, and the human-computer interaction can be conveniently implemented, which brings convenience to the user.
  • FIG. 1 is an application environment diagram of a human-computer interaction method in a display device according to an embodiment
  • FIG. 2 is a schematic flowchart of a human-computer interaction method in a display device according to an embodiment
  • FIG. 3 is an application environment diagram of a human-computer interaction method in a display device according to another embodiment
  • FIG. 4 is a schematic flowchart of a human-computer interaction method in a display device according to another embodiment
  • FIG. 5 is an application environment diagram of a human-computer interaction method in a display device according to another embodiment
  • FIG. 6 is an application environment diagram of a human-computer interaction method in a display device according to another embodiment
  • FIG. 7 is an application environment diagram of a human-computer interaction method in a display device according to another embodiment
  • FIG. 8 is an application environment diagram of a human-machine interaction method in a display device according to another embodiment
  • FIG. 9 is a structural block diagram of a human-machine interaction device in a display device according to an embodiment
  • FIG. 10 is an internal structural diagram of a computer device in one embodiment.
  • the human-computer interaction method in a display device provided in this application can be applied to an application environment as shown in FIG. 1.
  • the display device (television) acquires the current image data of the user within the range of the viewing angle at which the user appears the display device.
  • a plurality of different candidate data is displayed in different display areas, and the user browses the display device
  • the display device recognizes the target display area that the user is interested in, the display device reads the candidate data corresponding to the target display area, and performs an operation corresponding to the read candidate data.
  • the display device may be a television, a tablet computer, a mosaic display device, or a vehicle display device.
  • a method for man-machine interaction in a display device is provided.
  • the method is applied to a display device as an example for description, and includes the following steps:
  • the display device acquires the current image data of the user, and the acquisition method may be obtained through a camera built in or external to the display device.
  • a camera can be set on the display device, and the camera can be rotated to shoot the user's image by 360 degrees to obtain the user's current image data.
  • a camera can be set directly in front of the display device, and the camera collects the current image data of the user who enters the viewing angle range of the display device.
  • Face recognition is performed on the user's current image data obtained in step S200.
  • a plurality of different candidate data are displayed in different display areas.
  • "front view recognition” here can be understood as a “switch” action.
  • the display device detects that the user is in the front view state, it immediately displays a plurality of different candidate data in different display areas. Furthermore, it can be detected by comparing the user's current image data with the historically acquired image data of the user in a front view state.
  • the front view image data of the display device in the front view state collected by the image acquisition device can be the data collected in the historical record or the data collected on the spot.
  • the image acquisition device may be a camera or other device.
  • the front view image data of the display device in the front view state is collected by the image acquisition device.
  • the image acquisition device may be set on the display device, or it may be set on the display device's auxiliary device or On the peripheral device, for example, when the device is a TV, the image acquisition device can be set on the TV, or can be set on a set-top box supporting the TV.
  • the front-facing image data of the display device captured by the camera in the front-facing state can be determined after image processing and image target coordinate conversion, and the relative position of the device and the user's face can be determined, that is, the user's Face image data. Judging that the display device is in a front view state may be implemented by using technologies such as head pose estimation (estimation) or gaze tracking (gaze tracking).
  • Adopting techniques such as head pose estimation and gaze tracking can accurately identify the target display area that the user is concerned about. Specifically, different data will be displayed in different display areas on the display device. When the user browses to these data, he directly pays attention to the target display area according to his own needs. In practical applications, it is possible to identify the location of the user's attention based on the gaze tracking technology, accumulate the time of the user's eyes to stay in each display area, and select the candidate display area corresponding to the first accumulated stay time exceeding the preset attention time threshold for the user's attention. region. For example, as shown in FIG. 3, the contents of making a call to granddaughter, eldest son, and youngest son are currently displayed in the three display areas on the left, middle, and right of the television, respectively. The user has been paying attention to the middle display area for the longest time, that is, identifying the user's attention.
  • the target display area is the middle display area.
  • S800 Read the candidate data corresponding to the target display area, and perform an operation corresponding to the read candidate data.
  • the display device stores the relationship between each display area and corresponding data. After the target display area is determined in step S600, the display device directly reads the stored candidate data corresponding to the target display area, runs the candidate data, and performs corresponding operations. Based on the embodiment shown in FIG. 3 as an example, after determining that the intermediate display area is the target display area in step S600, the display device reads the data of “calling the oldest son” and performs the operation of “calling the oldest son”. It is not necessary that when the display device does not have a communication function, it may also control the peripheral device (communication device) to perform an operation corresponding to the read candidate data.
  • the peripheral device communication device
  • the display device obtains the user's current image data.
  • a plurality of different candidate data are displayed in different display areas.
  • the user browses the candidate data, and the display device recognizes the attention.
  • the target display area reads the candidate data corresponding to the target display area, and performs operations corresponding to the read candidate data.
  • the user's needs can be sensed without excessive operation by the user, and the human-computer interaction can be conveniently implemented, which brings convenience to the user.
  • the method further includes:
  • step S900 may be performed within a limited time range. That is, when it is detected within a limited time range (for example, 10 seconds) that the user is not in the front view state, the operation corresponding to the read candidate data is interrupted. Avoid the need for users to stay face-to-face for a long time and bring convenience to users.
  • a limited time range for example, 10 seconds
  • the television detects that the user is not in the frontal view state
  • the current operation is interrupted.
  • the user currently wants to end the call with the elder son, the user turns his head, and the TV recognizes that the user is not in the state of facing up, that is, the call with the elder son is terminated.
  • S400 includes:
  • a requirement for a preset time threshold is further added for the face-to-face state, and only when the user is in the face-to-face state for more than the preset time threshold, a plurality of different candidate data will be displayed in different display areas.
  • the user looks at the television (in the face-to-face state) for a certain time (3 seconds), and the television displays a plurality of different data to be selected in different display areas.
  • displaying a plurality of different candidate data in different display areas includes: based on the user's current image data, using the front view recognition technology to initially detect whether the user is in the front view state; when initially When the detection result is that the user is in the face state, the user's face image is collected; according to the user's face image, recognition processing is performed, and according to the recognition processing result, a plurality of different candidate data is displayed in different display areas, and the recognition processing includes face recognition , At least one of age recognition, facial expression recognition, and gender recognition.
  • a combination of face recognition, face recognition, or age recognition, or expression recognition or gender recognition is used to more accurately display user interaction data. Specifically, firstly, based on the user's current image data, using the frontal recognition technology, a preliminary detection is made to determine whether the user is in the frontal state. When the user is in the frontal state, the user's face image is collected, and face recognition, age recognition, At least one of facial expression recognition and gender recognition processing. According to the processing result, a plurality of different candidate data is displayed in different display areas.
  • Candidate data associated with the user ’s identity such as the user ’s address book, avatars of the user ’s relatives, etc .
  • face recognition + face recognition + facial expression recognition is used to identify the user ’s identity and initially identify the user ’s happy expression
  • the candidate data associated with the identity and interested by the user can be displayed in different display areas, such as a user's favorite TV program, a user's favorite picture, photo, and the like.
  • the human-computer interaction method in the display device further includes: displaying a plurality of different data to be selected in different display areas when a user's face image size is greater than a preset face image size threshold.
  • the distance between the user and the display device is different, and the size of the user's face image in the user image collected by the display device is different.
  • the human-computer interaction is effective only when the human-computer interaction is within a certain distance from the display device. For example, the user and the television are in the process of human-computer interaction. If the user and the television are tens of meters away, the user and the television The human-computer interaction between them is obviously an invalid human-computer interaction action.
  • a plurality of different candidate data are displayed in different display areas.
  • the preset face image size threshold may be a face image collected when the user is preset to be away from the display device, for example, a face image collected when the user is 3 meters away from the television device, and the corresponding positive face image size is obtained as Preset facial image size threshold.
  • collecting the user's face image includes: when the preliminary detection result is that the user is facing up, performing face recognition on the user; when identifying the user as known to be legal When the user, the user's face image is collected.
  • the user When the user is detected to be facing up, the user is face-recognized to determine whether the user is a known legal user, and when the user is a known legal user, a frontal face image is collected.
  • a dual mechanism of face recognition + user identification is adopted to identify whether multiple different candidate data needs to be displayed in different display areas, to avoid misoperation, and to bring convenience to users.
  • the human-computer interaction method in the display device further includes:
  • a plurality of different candidate data associated with the identity of the user are displayed in different display areas.
  • the candidate data is associated with the user's identity. For example, when the user's identity is identified, multiple contact methods associated with the user can be displayed in different display areas. For example, the daughter, son, and mother can be displayed in the three display areas on the left, right, and right Contact information.
  • identifying the target display area that the user cares about includes: using head gesture recognition technology to confirm the user's head orientation and angle; identifying the display area browsed by the user based on the user's head orientation and angle, The display area is used as the display area to be selected; the user's staying time in different display areas is timed; the display area to be selected with the longest stay time is selected as the target display area that the user pays attention to.
  • steps in the flowcharts of FIG. 2 and FIG. 4 are sequentially displayed according to the directions of the arrows, these steps are not necessarily performed sequentially in the order indicated by the arrows. Unless explicitly stated in this document, the execution of these steps is not strictly limited, and these steps can be performed in other orders. Moreover, at least some of the steps in FIG. 2 and FIG. 4 may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily performed at the same time, but may be performed at different times. These sub-steps or The execution order of the phases is not necessarily sequential, but can be performed in turn or alternately with other steps or sub-steps of other steps or at least a part of the phases.
  • the human-computer interaction method in a display device of the present application includes steps:
  • the user faces the TV screen, and the TV collects user image data.
  • the TV judges whether the user is facing up according to the collected user image data, as shown in Figures 5 and 3.
  • the user is looking at the center of the TV, and the TV detects that the user is facing up, and the display is displayed in different display areas on the TV.
  • Different candidate data Specifically, the three contacts of the granddaughter, the eldest son, and the youngest son are displayed on the left, middle, and right of the TV, and other blank areas display narration or explanatory data (this part may not be realistic).
  • the user slightly shakes his head left and right, selects one of the contacts by watching, and pauses for 3 seconds to make a video call.
  • the television adopts a dual recognition mechanism of face recognition + user identification, and recognizes that the current user is a legitimate user, and the target display area that the user cares about is the left area.
  • the TV reads the data to be selected corresponding to the left area, and performs the corresponding operation. As shown in FIG. 7, the television reads data related to the granddaughter's video call and initiates a video call to the granddaughter.
  • the video call image is displayed on the TV.
  • the user needs to interrupt and end the video call, deflect his head and look sideways at the TV.
  • the TV detects that the current user is in a non-face-to-face state and ends the video call.
  • a human-machine interaction device in a display device includes:
  • An image data acquisition module 200 configured to acquire current image data of a user
  • a display module 400 configured to display a plurality of different data to be selected in different display areas when the user is detected to be facing up;
  • An identification module 600 configured to identify a target display area that is of interest to a user
  • the execution module 800 is configured to read the candidate data corresponding to the target display area, and perform operations corresponding to the read candidate data.
  • the image data acquisition module 200 acquires the current image data of the user.
  • the display module 400 displays a plurality of different candidate data in different display areas, and the user browses the candidate data.
  • the recognition module 600 recognizes the target display area of interest, the execution module 800 reads the data to be selected corresponding to the target display area, and performs operations corresponding to the read data to be selected.
  • the user's needs can be sensed without excessive operation by the user, and the human-computer interaction can be conveniently implemented, which brings convenience to the user.
  • the human-machine interaction device in the display device further includes:
  • An interruption module is configured to interrupt an operation corresponding to the read candidate data when it is detected that the user is not in the front view state.
  • the display module 400 is further configured to count the time when the user is in the front view state when the user is detected in the front view state; when the user is in the front view state exceeds a preset time threshold, a plurality of different displays are displayed in different display areas. Candidate data.
  • the display module 400 is further configured to initially detect whether the user is in the front view state based on the user's current image data and use the front view recognition technology; when the preliminary detection result is that the user is in the front state, collect the user's face image; according to The user's face image is subjected to recognition processing, and a plurality of different candidate data are displayed in different display areas according to the recognition processing result.
  • the recognition processing includes at least one of face recognition, age recognition, expression recognition, and gender recognition.
  • the display module 400 is further configured to perform face recognition on the user when the preliminary detection result indicates that the user is in the face-to-face state; and when the user is identified as a known legitimate user, collect a face image of the user.
  • the display module 400 is further configured to display a plurality of different candidate data associated with the user identity in different display areas when the user's face recognizes the user identity.
  • the recognition module 600 uses head gesture recognition technology to confirm the user's head orientation and angle; according to the user's head orientation and angle, identifies the display area browsed by the user, and uses the display area browsed by the user as a candidate display Area; accumulate the time spent by the user in different candidate display areas separately; select the candidate display area that first accumulates the residence time that exceeds the preset attention time threshold as the target display area that the user pays attention to.
  • Each module in the human-computer interaction device in the above display device may be implemented in whole or in part by software, hardware, and a combination thereof.
  • the above-mentioned modules may be embedded in the hardware in or independent of the processor in the computer device, or may be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.
  • a computer device is provided.
  • the computer device may be a server, and its internal structure diagram may be as shown in FIG. 10.
  • the computer device includes a processor, a memory, a network interface, and a database connected through a system bus.
  • the processor of the computer device is used to provide computing and control capabilities.
  • the memory of the computer device includes a non-volatile storage medium and an internal memory.
  • the non-volatile storage medium stores an operating system, a computer program, and a database.
  • the internal memory provides an environment for running the operating system and computer programs in a non-volatile storage medium.
  • the database of the computer equipment is used to store the image data of the user in a front view state.
  • the network interface of the computer device is used to communicate with an external terminal through a network connection.
  • the computer program is executed by a processor to implement a human-computer interaction method in a display device.
  • FIG. 10 is only a block diagram of a part of the structure related to the solution of the application, and does not constitute a limitation on the computer equipment to which the solution of the application is applied.
  • the specific computer equipment may be Include more or fewer parts than shown in the figure, or combine certain parts, or have a different arrangement of parts.
  • a computer device including a memory, a processor, and a computer program stored on the memory and executable on the processor.
  • the processor executes the computer program, the following steps are implemented:
  • the time when the user is in the front view state is counted; when the user is in the front view state exceeds a preset time threshold, a plurality of different candidate data is displayed in different display areas.
  • the user's current image data adopting face recognition technology to initially detect whether the user is in the face state; when the preliminary detection result is that the user is in the face state, collect the user's face image; according to the user's face image, perform recognition processing, and according to the recognition processing result Displaying a plurality of different candidate data in different display areas, and the recognition process includes at least one of face recognition, age recognition, expression recognition, and gender recognition.
  • a plurality of different candidate data associated with the identity of the user are displayed in different display areas.
  • the user is subjected to face recognition; when the user is identified as a known legitimate user, a face image of the user is collected.
  • Adopt head gesture recognition technology to confirm the user's head orientation and angle; according to the user's head orientation and angle, identify the display area browsed by the user, and use the display area browsed by the user as the display area to be selected; accumulate the timing of the user's time in the different The dwell time in the display area; the first display area corresponding to the first accumulated cumulative dwell time exceeding the preset attention time threshold is selected as the target display area that the user pays attention to.
  • a computer-readable storage medium on which a computer program is stored.
  • the computer program is executed by a processor, the following steps are implemented:
  • the computer program when executed by the processor further implements the following steps:
  • the computer program when executed by the processor further implements the following steps:
  • the time when the user is in the front view state is counted; when the user is in the front view state exceeds a preset time threshold, a plurality of different candidate data is displayed in different display areas.
  • the computer program when executed by the processor further implements the following steps:
  • the user's current image data adopting face recognition technology to initially detect whether the user is in the face state; when the preliminary detection result is that the user is in the face state, collect the user's face image; according to the user's face image, perform recognition processing, and according to the recognition processing result Displaying a plurality of different candidate data in different display areas, and the recognition process includes at least one of face recognition, age recognition, expression recognition, and gender recognition.
  • the computer program when executed by the processor further implements the following steps:
  • a plurality of different candidate data associated with the identity of the user are displayed in different display areas.
  • the computer program when executed by the processor further implements the following steps:
  • the user is subjected to face recognition; when the user is identified as a known legitimate user, a face image of the user is collected.
  • Adopt head gesture recognition technology to confirm the user's head orientation and angle; according to the user's head orientation and angle, identify the display area browsed by the user, and use the display area browsed by the user as the display area to be selected; accumulate the timing of the user's time in the different The dwell time in the display area; the first display area corresponding to the first accumulated cumulative dwell time exceeding the preset attention time threshold is selected as the target display area that the user pays attention to.
  • Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory.
  • Volatile memory can include random access memory (RAM) or external cache memory.
  • RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous chain (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Human Computer Interaction (AREA)
  • General Engineering & Computer Science (AREA)
  • Software Systems (AREA)
  • Health & Medical Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • Multimedia (AREA)
  • Oral & Maxillofacial Surgery (AREA)
  • Computer Security & Cryptography (AREA)
  • Computer Hardware Design (AREA)
  • Ophthalmology & Optometry (AREA)
  • User Interface Of Digital Computer (AREA)

Abstract

本申请提供一种显示设备中人机交互方法、装置、计算机设备和存储介质，其中，方法包括：显示设备获取用户当前图像数据，当检测到用户处于正视状态时，在不同显示区域显示多个不同待选数据，用户浏览待选数据，显示设备识别出关注的目标显示区域，读取与所述目标显示区域对应的待选数据，执行与读取的待选数据对应的操作。整个过程中，无需用户过多的操作，即可感知用户需求，便捷实现人机交互，给用户带来便捷操作。

Description

显示设备中人机交互方法、装置、计算机设备和存储介质 技术领域
本申请涉及人机交互技术领域，特别是涉及一种显示设备中人机交互方法、装置、计算机设备和存储介质。
背景技术
人机交互是指人与设备之间使用某种对话语言，以一定的交互方式，为完成确定任务的人与设备之间的信息交换过程。
随着科学技术的发展，人机交互技术的应用领域越来越宽广，小如收音机的播放按键，大至飞机上的仪表板、或是发电厂的控制室，用户都可以通过人机交互界面与系统交流，并进行操作。目前在人机交互技术中，人机交互方式单一，需要用户发出特定的语音指令，显示设备才会响应执行进一步操作。
传统人机交互方式，虽然在一定程度上可以实现人机交互功能，但是由于交互方式单一，需要预先设定一定特定手势动作，交互过程并不十分自然，在一定程度上给用户操作带来不便。
发明内容
基于此，有必要针对上述技术问题，提供一种便捷的显示设备中人机交互方法、装置、计算机设备和存储介质。
一种显示设备中人机交互方法，所述方法包括：
获取用户当前图像数据；
当检测到用户处于正视状态时，在不同显示区域显示多个不同待选数据；
识别用户关注的目标显示区域；
读取与所述目标显示区域对应的待选数据，执行与读取的待选数据对应的操作。
在其中一个实施例中，所述读取与所述目标显示区域对应的待选数据，执 行与读取的待选数据对应的操作之后，还包括：
当检测到用户未处于正视状态时，中断与读取的待选数据对应的操作。
在其中一个实施例中，所述当检测到用户处于正视状态时，在不同显示区域显示多个不同待选数据包括：
当检测到用户处于正视状态时，计时用户处于正视状态时间；
当用户处于正视状态时间超过预设时间阈值时，在不同显示区域显示多个不同待选数据。
在其中一个实施例中，所述当检测到用户处于正视状态时，在不同显示区域显示多个不同待选数据包括：
根据用户当前图像数据，采用正视识别技术，初步检测用户是否处于正视状态；
当初步检测结果为用户处于正视状态时，采集用户人脸图像；
根据用户人脸图像，进行识别处理，根据识别处理结果，在不同显示区域显示多个不同待选数据，所述识别处理包括人脸识别、年龄识别、表情识别以及性别识别中的至少一种。
在其中一个实施例中，所述当初步检测结果为用户处于正视状态时，采集用户人脸图像包括：
当初步检测结果为用户处于正视状态时，对用户进行人脸识别；
当识别用户为已知合法用户时，采集用户人脸图像。
在其中一个实施例中，上述显示设备中人机交互方法还包括：
当人脸识别出用户身份时，在不同显示区域显示多个与用户身份关联的不同待选数据。
在其中一个实施例中，采集用户人脸图像，之后还包括：
当用户人脸图像大小大于预设脸部图像大小阈值时，在不同显示区域显示多个不同待选数据。
在其中一个实施例中，所述识别用户关注的目标显示区域包括：
采用头部姿态识别技术，确认用户头部朝向及角度；
根据所述用户头部朝向及角度，识别用户浏览的显示区域，将用户浏览的 显示区域作为待选显示区域；
分别累计计时用户在不同待选显示区域中停留时间；
选取最先累计所述停留时间超过预设关注时间阈值对应的待选显示区域为用户关注的目标显示区域。
一种显示设备中人机交互装置，所述装置包括：
图像数据获取模块，用于获取用户当前图像数据；
显示模块，用于当检测到用户处于正视状态时，在不同显示区域显示多个不同待选数据；
识别模块，用于识别用户关注的目标显示区域；
执行模块，用于读取与所述目标显示区域对应的待选数据，执行与读取的待选数据对应的操作。
一种计算机设备，包括存储器和处理器，所述存储器存储有计算机程序，所述处理器执行所述计算机程序时实现如上述方法的步骤。
一种计算机可读存储介质，其上存储有计算机程序，所述计算机程序被处理器执行时实现如上述的方法的步骤。
上述显示设备中人机交互方法、装置、计算机设备和存储介质，显示设备获取用户当前图像数据，当检测到用户处于正视状态时，在不同显示区域显示多个不同待选数据，用户浏览待选数据，显示设备识别出关注的目标显示区域，读取与所述目标显示区域对应的待选数据，执行与读取的待选数据对应的操作。整个过程中，无需用户过多的操作，即可感知用户需求，便捷实现人机交互，给用户带来便捷操作。
附图说明
图1为一个实施例中显示设备中人机交互方法的应用环境图；
图2为一个实施例中显示设备中人机交互方法的流程示意图；
图3为另一个实施例中显示设备中人机交互方法的应用环境图；
图4为另一个实施例中显示设备中人机交互方法的流程示意图；
图5为另一个实施例中显示设备中人机交互方法的应用环境图；
图6为另一个实施例中显示设备中人机交互方法的应用环境图；
图7为另一个实施例中显示设备中人机交互方法的应用环境图；
图8为另一个实施例中显示设备中人机交互方法的应用环境图；
图9为一个实施例中显示设备中人机交互装置的结构框图；
图10为一个实施例中计算机设备的内部结构图。
具体实施方式
为了使本申请的目的、技术方案及优点更加清楚明白，以下结合附图及实施例，对本申请进行进一步详细说明。应当理解，此处描述的具体实施例仅仅用以解释本申请，并不用于限定本申请。
本申请提供的显示设备中人机交互方法，可以应用于如图1所示的应用环境中。其中，用户出现显示设备的可视角度范围内，显示设备(电视机)获取用户当前图像数据，当检测到用户处于正视状态时，在不同显示区域显示多个不同待选数据，用户浏览显示设备显示的待选数据，显示设备识别出用户关注的目标显示区域，显示设备读取与目标显示区域对应的待选数据，执行与读取的待选数据对应的操作。显示设备具体可以为电视机、平板电脑、拼接显示设备或车载显示设备等。
在一个实施例中，如图2所示，提供了一种显示设备中人机交互方法，以该方法应用于显示设备为例进行说明，包括以下步骤：
S200：获取用户当前图像数据。
显示设备获取用户当前图像数据，其获取的方式可以是通过内置或外置于显示设备的摄像头获取。具体来说，可以在显示设备上设置摄像头，摄像头360度转动拍摄用户图像，获取用户当前图像数据。更进一步来说，可以在显示设备正前方设置摄像头，摄像头采集进入显示设备可视角度范围内的用户当前图像数据。
S400：当检测到用户处于正视状态时，在不同显示区域显示多个不同待选数据。
针对步骤S200获取的用户当前图像数据进行正视识别，当检测到用户处于正视状态时，在不同显示区域显示多个不同待选数据。简单来说，在这里“正 视识别”可以理解一个“开关”动作，显示设备在检测到用户处于正视状态时，立即在不同显示区域显示多个不同待选数据。更进一步来说，可以通过比较用户当前图像数据与历史获取的用户处于正视状态下的图像数据来检测。在实际应用中，先获取图像采集设备采集的显示设备处于正视状态下的正视图像数据，显示设备处于正视状态下的正视图像数据可以是历史记录中采集好的数据，也可以是当场采集的数据。图像采集设备可以是摄像头等设备，在这里，显示设备处于正视状态下的正视图像数据是通过图像采集设备采集的，图像采集设备可以设置于显示设备上，还可以设置于显示设备的辅助设备或外围设备上，例如当设备为电视机时，图像采集设备可以设置于电视，也可以设置于与电视配套的机顶盒上。更具体来说，摄像头拍摄的显示设备处于正视状态下的正视图像数据，进行图像处理和图像目标坐标换算之后即可确定设备和用户人脸相对位置，即可以获取显示设备处于正视状态下用户的人脸图像数据。判定显示设备处于正视状态可以选择采用头部姿态估计(head pose estimation)或者视线跟踪(gaze tracking)等技术来实现。
S600：识别用户关注的目标显示区域。
采用头部姿态估计和视线跟踪等技术可以准确识别出用户关注的目标显示区域。具体来说，在显示设备上不同显示区域会显示不同的数据，用户在浏览到这些数据直接根据自身需求关注(注视)目标显示区域。在实际应用中可以基于视线跟踪技术识别用户关注的位置，累计计时用户目光在各显示区域停留时间，选取最先累计停留时间超过预设关注时间阈值对应的待选显示区域为用户关注的目标显示区域。例如，如图3所示，当前在电视机左中右三个显示区域中分别显示有拨打电话给孙女、大儿子、小儿子的内容，用户关注中间显示区域最久，即识别出用户关注的目标显示区域为中间显示区域。
S800：读取与目标显示区域对应的待选数据，执行与读取的待选数据对应的操作。
显示设备中存储有各显示区域于对应数据的关系，在步骤S600确定目标显示区域之后，显示设备直接读取存储的与目标显示区域对应的待选数据，运行该待选数据，执行相应操作。基于以图3所示实施例为例，在步骤S600中确定 中间显示区域为目标显示区域之后，显示设备读取“拨打电话给大儿子”的数据，执行“拨打电话给大儿子”的操作。非必要的，当显示设备不具备通信功能时，也可以是控制外围设备(通信设备)执行与读取的待选数据对应的操作。
上述显示设备中人机交互方法，显示设备获取用户当前图像数据，当检测到用户处于正视状态时，在不同显示区域显示多个不同待选数据，用户浏览待选数据，显示设备识别出关注的目标显示区域，读取与目标显示区域对应的待选数据，执行与读取的待选数据对应的操作。整个过程中，无需用户过多的操作，即可感知用户需求，便捷实现人机交互，给用户带来便捷操作。
如图4所示，在其中一个实施例中，步骤S800之后，还包括：
S900：当检测到用户未处于正视状态时，中断与读取的待选数据对应的操作。
当显示设备检测到用户未处于正视状态时，中断与读取的待选数据对应的操作。继续以图3所示实施例为例，电视机在执行拨打电话给大儿子的操作过程中，当检测到的用户没有处于正视状态时，中断该操作。非必要的，可以在有限时间范围内执行步骤S900。即在有限时间范围内(例如10秒)检测到用户未处于正视状态时，中断与读取的待选数据对应的操作。避免需要用户长期保持正视状态，给用户带来便捷。当用户想结束当前操作时，可以与显示设备处于“非正视”状态，电视机检测到用户未处于正视状态时，中断当前操作。例如用户当前想结束与大儿子通话，用户转过头去，电视机识别到用户未处于正视状态，即中断与大儿子通话。
如图4所示，在其中一个实施例中，S400包括：
S420：当检测到用户处于正视状态时，计时用户处于正视状态时间；
S440：当用户处于正视状态时间超过预设时间阈值时，在不同显示区域显示多个不同待选数据。
在这里，针对正视状态进一步加入预设时间阈值的要求，只有当用户处于正视状态时间超过预设时间阈值时，才会在不同显示区域显示多个不同待选数据。用户注视电视机(处于正视状态)一定时间(3秒)，电视机在不同显示区域显示多个不同待选数据。采用上述方式，可以避免用户在日常使用过程中误 操作，给用户带来便捷。
在其中一个实施例中，当检测到用户处于正视状态时，在不同显示区域显示多个不同待选数据包括：根据用户当前图像数据，采用正视识别技术，初步检测用户是否处于正视状态；当初步检测结果为用户处于正视状态时，采集用户人脸图像；根据用户人脸图像，进行识别处理，根据识别处理结果，在不同显示区域显示多个不同待选数据，所述识别处理包括人脸识别、年龄识别、表情识别以及性别识别中的至少一种。
在本实施例中，采用正视识别+人脸识别、或年龄识别、或表情识别或性别识别相结合的方式，更加准确显示用户的交互数据。具体来说，先根据用户当前图像数据，采用正视识别技术，初步检测判断用户是否处于正视状态，当为正视状态时，采集用户人脸图像，针对采集的正脸图像进行脸识别、年龄识别、表情识别以及性别识别中的至少一种处理，根据处理结果，在不同显示区域显示多个不同待选数据，例如当根据正视识别+人脸识别时，识别出用户身份，可以不同显示区域显示与用户身份关联的待选数据，如与用户的通讯录、用户亲属的头像等；当根据正视识别+人脸识别+表情识别时，识别出用户身份，并且初步识别出用户当前为开心的表情，可以在不同显示区域显示与身份关联且用户感兴趣的待选数据，例如用户喜爱的电视节目、用户喜爱的图片、照片等。
在其中一个实施例中，上述显示设备中人机交互方法还包括：当用户人脸图像大小大于预设脸部图像大小阈值时，在不同显示区域显示多个不同待选数据。
用户与显示设备距离不同，显示设备采集到的用户图像中用户人脸图像大小不同。而用户只有在与显示设备处于一定距离内的人机交互才为有效的人机交互，例如用户与电视机在进行人机交互过程，若用户与电视机距离几十米远，用户与电视机之间的人机交互显然为无效的人机交互动作。在本实施例中，基于上述原理和实际应用需求，只有当用户人脸图像大小大于预设脸部图像大小阈值时，才在不同显示区域显示多个不同待选数据。预设脸部图像大小阈值可以是在预设用户与显示设备距离时采集的脸部图像，例如预先采集的用户与电视设备距离3米时采集的脸部图像，获取对应的正脸图像大小作为预设脸部图 像大小阈值。
在其中一个实施例中，当初步检测结果为用户处于正视状态时，采集用户人脸图像包括：当初步检测结果为用户处于正视状态时，对用户进行人脸识别；当识别用户为已知合法用户时，采集用户人脸图像。
当检测到用户处于正视状态时，对用户进行人脸识别，判断用户是否为已知合法用户，当为已知合法用户时，采集其正脸图像。在本实施例中，采用正视识别+用户身份识别两重机制，识别是否需要在不同显示区域显示多个不同待选数据，避免出现误操作情况，给用户带来便捷。
在其中一个实施例中，上述显示设备中人机交互方法还包括：
当人脸识别出用户身份时，在不同显示区域显示多个与用户身份关联的不同待选数据。
待选数据是与用户身份关联的，例如当识别出用户身份时，可以在不同显示区域显示与用户关联的多个联系方式，例如可以在左中右三个显示区域分别显示女儿、儿子、母亲的联系方式。
在其中一个实施例中，识别用户关注的目标显示区域包括：采用头部姿态识别技术，确认用户头部朝向及角度；根据用户头部朝向及角度，识别用户浏览的显示区域，将用户浏览的显示区域作为待选显示区域；分别计时用户在不同待选显示区域中停留时间；选取停留时间最长的待选显示区域为用户关注的目标显示区域。
应该理解的是，虽然图2与图4的流程图中的各个步骤按照箭头的指示依次显示，但是这些步骤并不是必然按照箭头指示的顺序依次执行。除非本文中有明确的说明，这些步骤的执行并没有严格的顺序限制，这些步骤可以以其它的顺序执行。而且，图2与图4中的至少一部分步骤可以包括多个子步骤或者多个阶段，这些子步骤或者阶段并不必然是在同一时刻执行完成，而是可以在不同的时刻执行，这些子步骤或者阶段的执行顺序也不必然是依次进行，而是可以与其它步骤或者其它步骤的子步骤或者阶段的至少一部分轮流或者交替地执行
为更进一步详细解释本申请显示设备中人机交互方法的技术方案及其效 果，下面将采用应用于电视机为具体应用实例，并结合附图3、图5、图6、图7以及图8进行详细说明。在应用于电视机中时，本申请显示设备中人机交互方法包括步骤：
1、如图5所示，用户面向电视机屏幕，电视机采集用户图像数据。
2、电视机根据采集的用户图像数据判断用户是否处于正视状态，具体如图5和图3所示，用户正视电视机中央，电视机检测到用户处于正视状态，在电视机上不同显示区域显示多个不同待选数据。具体在电视机左中右分别显示孙女、大儿子以及小儿子三个联系人，其他空白区域显示旁白或解释性数据(也可以不现实这部分内容)。
3、如图6所示，用户左右微摆动头部，通过注视选定其中一个联系人，停顿3秒即可拨打其视频通话。电视机采用正视识别+用户身份识别两重识别机制，识别出当前用户为合法用户，且用户关注的目标显示区域为左侧区域。
4、电视机读取与左边区域对应的待选数据，并且执行对应操作。如图7所示，电视机读取拨打孙女视频通话的相关数据，向孙女发起视频通话。
5、如图8所示，当视频通话被接通时，在电视机上显示视频通话的图像。当用户需要中断、结束视频通话时，偏转头部，侧视电视机，电视机检测到当前用户处于非正视状态，结束本次视频通话。
如图9所示，一种显示设备中人机交互装置，装置包括：
图像数据获取模块200，用于获取用户当前图像数据；
显示模块400，用于当检测到用户处于正视状态时，在不同显示区域显示多个不同待选数据；
识别模块600，用于识别用户关注的目标显示区域；
执行模块800，用于读取与目标显示区域对应的待选数据，执行与读取的待选数据对应的操作。
上述显示设备中人机交互装置，图像数据获取模块200获取用户当前图像数据，当检测到用户处于正视状态时，显示模块400在不同显示区域显示多个不同待选数据，用户浏览待选数据，识别模块600识别出关注的目标显示区域，执行模块800读取与目标显示区域对应的待选数据，执行与读取的待选数据对 应的操作。整个过程中，无需用户过多的操作，即可感知用户需求，便捷实现人机交互，给用户带来便捷操作。
在其中一个实施例中上述显示设备中人机交互装置还包括：
中断模块，用于当检测到用户未处于正视状态时，中断与读取的待选数据对应的操作。
在其中一个实施例中，显示模块400还用于当检测到用户处于正视状态时，计时用户处于正视状态时间；当用户处于正视状态时间超过预设时间阈值时，在不同显示区域显示多个不同待选数据。
在其中一个实施例中，显示模块400还用于根据用户当前图像数据，采用正视识别技术，初步检测用户是否处于正视状态；当初步检测结果为用户处于正视状态时，采集用户人脸图像；根据用户人脸图像，进行识别处理，根据识别处理结果，在不同显示区域显示多个不同待选数据，所述识别处理包括人脸识别、年龄识别、表情识别以及性别识别中的至少一种。
在其中一个实施例中，显示模块400还用于当初步检测结果为用户处于正视状态时，对用户进行人脸识别；当识别用户为已知合法用户时，采集用户人脸图像。
在其中一个实施例中，显示模块400还用于当人脸识别出用户身份时，在不同显示区域显示多个与用户身份关联的不同待选数据。
在其中一个实施例中，识别模块600采用头部姿态识别技术，确认用户头部朝向及角度；根据用户头部朝向及角度，识别用户浏览的显示区域，将用户浏览的显示区域作为待选显示区域；分别累计计时用户在不同待选显示区域中停留时间；选取最先累计所述停留时间超过预设关注时间阈值对应的待选显示区域为用户关注的目标显示区域。
关于显示设备中人机交互装置的具体限定可以参见上文中对于显示设备中人机交互方法的限定，在此不再赘述。上述显示设备中人机交互装置中的各个模块可全部或部分通过软件、硬件及其组合来实现。上述各模块可以硬件形式内嵌于或独立于计算机设备中的处理器中，也可以以软件形式存储于计算机设备中的存储器中，以便于处理器调用执行以上各个模块对应的操作。
在一个实施例中，提供了一种计算机设备，该计算机设备可以是服务器，其内部结构图可以如图10所示。该计算机设备包括通过系统总线连接的处理器、存储器、网络接口和数据库。其中，该计算机设备的处理器用于提供计算和控制能力。该计算机设备的存储器包括非易失性存储介质、内存储器。该非易失性存储介质存储有操作系统、计算机程序和数据库。该内存储器为非易失性存储介质中的操作系统和计算机程序的运行提供环境。该计算机设备的数据库用于存储用户处于正视状态的图像数据。该计算机设备的网络接口用于与外部的终端通过网络连接通信。该计算机程序被处理器执行时以实现一种显示设备中人机交互方法。
本领域技术人员可以理解，图10中示出的结构，仅仅是与本申请方案相关的部分结构的框图，并不构成对本申请方案所应用于其上的计算机设备的限定，具体的计算机设备可以包括比图中所示更多或更少的部件，或者组合某些部件，或者具有不同的部件布置。
在一个实施例中，提供了一种计算机设备，包括存储器、处理器及存储在存储器上并可在处理器上运行的计算机程序，处理器执行计算机程序时实现以下步骤：
获取用户当前图像数据；
当检测到用户处于正视状态时，在不同显示区域显示多个不同待选数据；
识别用户关注的目标显示区域；
读取与目标显示区域对应的待选数据，执行与读取的待选数据对应的操作。
在一个实施例中，处理器执行计算机程序时还实现以下步骤：
当检测到用户未处于正视状态时，中断与读取的待选数据对应的操作。
在一个实施例中，处理器执行计算机程序时还实现以下步骤：
当检测到用户处于正视状态时，计时用户处于正视状态时间；当用户处于正视状态时间超过预设时间阈值时，在不同显示区域显示多个不同待选数据。
在一个实施例中，处理器执行计算机程序时还实现以下步骤：
根据用户当前图像数据，采用正视识别技术，初步检测用户是否处于正视状态；当初步检测结果为用户处于正视状态时，采集用户人脸图像；根据用户 人脸图像，进行识别处理，根据识别处理结果，在不同显示区域显示多个不同待选数据，所述识别处理包括人脸识别、年龄识别、表情识别以及性别识别中的至少一种。
在一个实施例中，处理器执行计算机程序时还实现以下步骤：
当人脸识别出用户身份时，在不同显示区域显示多个与用户身份关联的不同待选数据。
在一个实施例中，处理器执行计算机程序时还实现以下步骤：
当初步检测结果为用户处于正视状态时，对用户进行人脸识别；当识别用户为已知合法用户时，采集用户人脸图像。
在一个实施例中，处理器执行计算机程序时还实现以下步骤：
采用头部姿态识别技术，确认用户头部朝向及角度；根据用户头部朝向及角度，识别用户浏览的显示区域，将用户浏览的显示区域作为待选显示区域；分别累计计时用户在不同待选显示区域中停留时间；选取最先累计所述停留时间超过预设关注时间阈值对应的待选显示区域为用户关注的目标显示区域。
在一个实施例中，提供了一种计算机可读存储介质，其上存储有计算机程序，计算机程序被处理器执行时实现以下步骤：
获取用户当前图像数据；
当检测到用户处于正视状态时，在不同显示区域显示多个不同待选数据；
识别用户关注的目标显示区域；
读取与目标显示区域对应的待选数据，执行与读取的待选数据对应的操作。
在一个实施例中，计算机程序被处理器执行时还实现以下步骤：
当检测到用户未处于正视状态时，中断与读取的待选数据对应的操作。
在一个实施例中，计算机程序被处理器执行时还实现以下步骤：
当检测到用户处于正视状态时，计时用户处于正视状态时间；当用户处于正视状态时间超过预设时间阈值时，在不同显示区域显示多个不同待选数据。
在一个实施例中，计算机程序被处理器执行时还实现以下步骤：
根据用户当前图像数据，采用正视识别技术，初步检测用户是否处于正视状态；当初步检测结果为用户处于正视状态时，采集用户人脸图像；根据用户 人脸图像，进行识别处理，根据识别处理结果，在不同显示区域显示多个不同待选数据，所述识别处理包括人脸识别、年龄识别、表情识别以及性别识别中的至少一种。
在一个实施例中，计算机程序被处理器执行时还实现以下步骤：
当人脸识别出用户身份时，在不同显示区域显示多个与用户身份关联的不同待选数据。
在一个实施例中，计算机程序被处理器执行时还实现以下步骤：
当初步检测结果为用户处于正视状态时，对用户进行人脸识别；当识别用户为已知合法用户时，采集用户人脸图像。
在一个实施例中，处理器执行计算机程序时还实现以下步骤：
采用头部姿态识别技术，确认用户头部朝向及角度；根据用户头部朝向及角度，识别用户浏览的显示区域，将用户浏览的显示区域作为待选显示区域；分别累计计时用户在不同待选显示区域中停留时间；选取最先累计所述停留时间超过预设关注时间阈值对应的待选显示区域为用户关注的目标显示区域。
本领域普通技术人员可以理解实现上述实施例方法中的全部或部分流程，是可以通过计算机程序来指令相关的硬件来完成，的计算机程序可存储于一非易失性计算机可读取存储介质中，该计算机程序在执行时，可包括如上述各方法的实施例的流程。其中，本申请所提供的各实施例中所使用的对存储器、存储、数据库或其它介质的任何引用，均可包括非易失性和/或易失性存储器。非易失性存储器可包括只读存储器(ROM)、可编程ROM(PROM)、电可编程ROM(EPROM)、电可擦除可编程ROM(EEPROM)或闪存。易失性存储器可包括随机存取存储器(RAM)或者外部高速缓冲存储器。作为说明而非局限，RAM以多种形式可得，诸如静态RAM(SRAM)、动态RAM(DRAM)、同步DRAM(SDRAM)、双数据率SDRAM(DDRSDRAM)、增强型SDRAM(ESDRAM)、同步链路(Synchlink)DRAM(SLDRAM)、存储器总线(Rambus)直接RAM(RDRAM)、直接存储器总线动态RAM(DRDRAM)、以及存储器总线动态RAM(RDRAM)等。
以上实施例的各技术特征可以进行任意的组合，为使描述简洁，未对上述 实施例中的各个技术特征所有可能的组合都进行描述，然而，只要这些技术特征的组合不存在矛盾，都应当认为是本说明书记载的范围。
以上实施例仅表达了本申请的几种实施方式，其描述较为具体和详细，但并不能因此而理解为对发明专利范围的限制。应当指出的是，对于本领域的普通技术人员来说，在不脱离本申请构思的前提下，还可以做出若干变形和改进，这些都属于本申请的保护范围。因此，本申请专利的保护范围应以所附权利要求为准。

Claims (11)

  1. 一种显示设备中人机交互方法，所述方法包括：
    获取用户当前图像数据；
    当检测到用户处于正视状态时，在不同显示区域显示多个不同待选数据；
    识别用户关注的目标显示区域；
    读取与所述目标显示区域对应的待选数据，执行与读取的待选数据对应的操作。
  2. 根据权利要求1所述的方法，其特征在于，所述读取与所述目标显示区域对应的待选数据，执行与读取的待选数据对应的操作之后，还包括：
    当检测到用户未处于正视状态时，中断与读取的待选数据对应的操作。
  3. 根据权利要求1所述的方法，其特征在于，所述当检测到用户处于正视状态时，在不同显示区域显示多个不同待选数据包括：
    当检测到用户处于正视状态时，计时用户处于正视状态时间；
    当用户处于正视状态时间超过预设时间阈值时，在不同显示区域显示多个不同待选数据。
  4. 根据权利要求1所述的方法，其特征在于，所述当检测到用户处于正视状态时，在不同显示区域显示多个不同待选数据包括：
    根据用户当前图像数据，采用正视识别技术，初步检测用户是否处于正视状态；
    当初步检测结果为用户处于正视状态时，采集用户人脸图像；
    根据用户人脸图像，进行识别处理，根据识别处理结果，在不同显示区域显示多个不同待选数据，所述识别处理包括人脸识别、年龄识别、表情识别以及性别识别中的至少一种。
  5. 根据权利要求4所述的方法，其特征在于，所述当初步检测结果为用户处于正视状态时，采集用户人脸图像包括：
    当初步检测结果为用户处于正视状态时，对用户进行人脸识别；
    当识别用户为已知合法用户时，采集用户人脸图像。
  6. 根据权利要求5所述的方法，其特征在于，还包括：
    当人脸识别出用户身份时，在不同显示区域显示多个与用户身份关联的不同待选数据。
  7. 根据权利要求5所述的方法，其特征在于，采集用户人脸图像，之后还包括：
    当用户人脸图像大小大于预设脸部图像大小阈值时，在不同显示区域显示多个不同待选数据。
  8. 根据权利要求1所述的方法，其特征在于，所述识别用户关注的目标显示区域包括：
    采用头部姿态识别技术，确认用户头部朝向及角度；
    根据所述用户头部朝向及角度，识别用户浏览的显示区域，将用户浏览的显示区域作为待选显示区域；
    分别累计计时用户在不同待选显示区域中停留时间；
    选取最先累计所述停留时间超过预设关注时间阈值对应的待选显示区域为用户关注的目标显示区域。
  9. 一种显示设备中人机交互装置，其特征在于，所述装置包括：
    图像数据获取模块，用于获取用户当前图像数据；
    显示模块，用于当检测到用户处于正视状态时，在不同显示区域显示多个不同待选数据；
    识别模块，用于识别用户关注的目标显示区域；
    执行模块，用于读取与所述目标显示区域对应的待选数据，执行与读取的待选数据对应的操作。
  10. 一种计算机设备，包括存储器和处理器，所述存储器存储有计算机程序，其特征在于，所述处理器执行所述计算机程序时实现权利要求1至8中任一项所述方法的步骤。
  11. 一种计算机可读存储介质，其上存储有计算机程序，其特征在于，所述计算机程序被处理器执行时实现权利要求1至8中任一项所述的方法的步骤。
PCT/CN2019/087901 2018-05-31 2019-05-22 显示设备中人机交互方法、装置、计算机设备和存储介质 WO2019228236A1 (zh)

Priority Applications (2)

Application Number Priority Date Filing Date Title
US17/059,091 US11416068B2 (en) 2018-05-31 2019-05-22 Method and apparatus for human-computer interaction in display device, and computer device and storage medium
KR1020207037972A KR20210013631A (ko) 2018-05-31 2019-05-22 표시 장치에서의 인간-컴퓨터 상호 작용 방법, 장치, 컴퓨터 설비 및 저장 매체

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201810550530.1A CN108897589B (zh) 2018-05-31 2018-05-31 显示设备中人机交互方法、装置、计算机设备和存储介质
CN201810550530.1 2018-05-31

Publications (1)

Publication Number Publication Date
WO2019228236A1 true WO2019228236A1 (zh) 2019-12-05

Family

ID=64343726

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2019/087901 WO2019228236A1 (zh) 2018-05-31 2019-05-22 显示设备中人机交互方法、装置、计算机设备和存储介质

Country Status (4)

Country Link
US (1) US11416068B2 (zh)
KR (1) KR20210013631A (zh)
CN (1) CN108897589B (zh)
WO (1) WO2019228236A1 (zh)

Families Citing this family (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108897589B (zh) 2018-05-31 2020-10-27 刘国华 显示设备中人机交互方法、装置、计算机设备和存储介质
CN110555737A (zh) * 2019-09-06 2019-12-10 北京海益同展信息科技有限公司 内容呈现方法、装置和系统
CN111178294A (zh) * 2019-12-31 2020-05-19 北京市商汤科技开发有限公司 状态识别方法、装置、设备及存储介质
CN118192800A (zh) * 2020-03-31 2024-06-14 托比股份公司 用于对可视化数据进行预处理的方法、计算机程序产品和处理电路系统
CN113467614A (zh) * 2021-06-30 2021-10-01 北京市商汤科技开发有限公司 显示界面选择方法及装置

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6152563A (en) * 1998-02-20 2000-11-28 Hutchinson; Thomas E. Eye gaze direction tracker
CN101866215A (zh) * 2010-04-20 2010-10-20 复旦大学 在视频监控中采用视线跟踪的人机交互装置和方法
CN107239139A (zh) * 2017-05-18 2017-10-10 刘国华 基于正视的人机交互方法与系统
CN108897589A (zh) * 2018-05-31 2018-11-27 刘国华 显示设备中人机交互方法、装置、计算机设备和存储介质

Family Cites Families (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7762665B2 (en) * 2003-03-21 2010-07-27 Queen's University At Kingston Method and apparatus for communication between humans and devices
CN102957743A (zh) * 2012-10-18 2013-03-06 北京天宇朗通通信设备股份有限公司 数据推送方法及装置
KR102095765B1 (ko) * 2012-10-19 2020-04-01 삼성전자주식회사 디스플레이 장치 및 그 제어 방법

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6152563A (en) * 1998-02-20 2000-11-28 Hutchinson; Thomas E. Eye gaze direction tracker
CN101866215A (zh) * 2010-04-20 2010-10-20 复旦大学 在视频监控中采用视线跟踪的人机交互装置和方法
CN107239139A (zh) * 2017-05-18 2017-10-10 刘国华 基于正视的人机交互方法与系统
CN108897589A (zh) * 2018-05-31 2018-11-27 刘国华 显示设备中人机交互方法、装置、计算机设备和存储介质

Also Published As

Publication number Publication date
US20210223857A1 (en) 2021-07-22
US11416068B2 (en) 2022-08-16
KR20210013631A (ko) 2021-02-04
CN108897589A (zh) 2018-11-27
CN108897589B (zh) 2020-10-27

Similar Documents

Publication Publication Date Title
WO2019228236A1 (zh) 显示设备中人机交互方法、装置、计算机设备和存储介质
US10817705B2 (en) Method, apparatus, and system for resource transfer
KR102092931B1 (ko) 시선 추적 방법 및 이를 수행하기 위한 사용자 단말
US20110305394A1 (en) Object Detection Metadata
JP5662670B2 (ja) 画像処理装置、画像処理方法、及びプログラム
WO2020055657A1 (en) Liveness detection method, apparatus and computer-readable storage medium
US20150261997A1 (en) Apparatus and method for recognizing image
KR101978299B1 (ko) 콘텐츠 서비스 시스템에서의 콘텐츠 서비스를 위한 장치
CN111726689B (zh) 一种视频播放控制方法及装置
US20110141257A1 (en) Apparatus and method for registering plurality of facial images for face recognition
WO2019214442A1 (zh) 一种设备控制方法、装置、控制设备及存储介质
WO2019206243A1 (zh) 一种素材展示方法、终端和计算机存储介质
US9602872B2 (en) Display apparatus and control method thereof
WO2017101391A1 (zh) 一种放大视频图像的方法及装置
CN110557564A (zh) 相机的快速启动方法及装置
CN113780212A (zh) 用户身份核验方法、装置、设备及存储介质
WO2014126735A1 (en) Real time object scanning using a mobile phone and cloud-based visual search engine
US20210127071A1 (en) Method, system and computer program product for object-initiated redaction of surveillance video
CN110705356A (zh) 功能控制方法及相关设备
CN110969120B (zh) 图像处理方法及装置、电子设备、可读存储介质
CN111818385A (zh) 视频处理方法、视频处理装置及终端设备
US20220256094A1 (en) Method and System for Assisting a User Who is Looking at a Screen of a User Device
CN109547678B (zh) 一种处理方法、装置、设备及可读存储介质
CN105912253B (zh) 一种虚拟拍照按键的触发方法、装置及移动终端
WO2015100070A1 (en) Presenting information based on a video

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 19811861

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

ENP Entry into the national phase

Ref document number: 20207037972

Country of ref document: KR

Kind code of ref document: A

32PN Ep: public notification in the ep bulletin as address of the adressee cannot be established

Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205A DATED 13.04.2021)

122 Ep: pct application non-entry in european phase

Ref document number: 19811861

Country of ref document: EP

Kind code of ref document: A1