Classifications

• • G—PHYSICS
  • G05—CONTROLLING; REGULATING
  • G05B—CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS
  • G05B15/00—Systems controlled by a computer
  • G05B15/02—Systems controlled by a computer electric
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
  • H04L12/00—Data switching networks
  • H04L12/28—Data switching networks characterised by path configuration, e.g. LAN [Local Area Networks] or WAN [Wide Area Networks]
  • H04L12/2803—Home automation networks
  • H04L12/2807—Exchanging configuration information on appliance services in a home automation network
  • H04L12/2814—Exchanging control software or macros for controlling appliance services in a home automation network
• • G—PHYSICS
  • G05—CONTROLLING; REGULATING
  • G05B—CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS
  • G05B19/00—Programme-control systems
  • G05B19/02—Programme-control systems electric
  • G05B19/04—Programme control other than numerical control, i.e. in sequence controllers or logic controllers
  • G05B19/042—Programme control other than numerical control, i.e. in sequence controllers or logic controllers using digital processors
  • G05B19/0423—Input/output
• • G—PHYSICS
  • G05—CONTROLLING; REGULATING
  • G05B—CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS
  • G05B19/00—Programme-control systems
  • G05B19/02—Programme-control systems electric
  • G05B19/04—Programme control other than numerical control, i.e. in sequence controllers or logic controllers
  • G05B19/045—Programme control other than numerical control, i.e. in sequence controllers or logic controllers using logic state machines, consisting only of a memory or a programmable logic device containing the logic for the controlled machine and in which the state of its outputs is dependent on the state of its inputs or part of its own output states, e.g. binary decision controllers, finite state controllers
• • G—PHYSICS
  • G05—CONTROLLING; REGULATING
  • G05B—CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS
  • G05B19/00—Programme-control systems
  • G05B19/02—Programme-control systems electric
  • G05B19/418—Total factory control, i.e. centrally controlling a plurality of machines, e.g. direct or distributed numerical control [DNC], flexible manufacturing systems [FMS], integrated manufacturing systems [IMS] or computer integrated manufacturing [CIM]
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
  • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
  • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
  • G06F3/012—Head tracking input arrangements
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
  • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
  • G06F3/017—Gesture based interaction, e.g. based on a set of recognized hand gestures
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
  • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
  • G06F3/03—Arrangements for converting the position or the displacement of a member into a coded form
  • G06F3/0304—Detection arrangements using opto-electronic means
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
  • H04L12/00—Data switching networks
  • H04L12/28—Data switching networks characterised by path configuration, e.g. LAN [Local Area Networks] or WAN [Wide Area Networks]
  • H04L12/2803—Home automation networks
  • H04L12/2816—Controlling appliance services of a home automation network by calling their functionalities
  • H04L12/282—Controlling appliance services of a home automation network by calling their functionalities based on user interaction within the home
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
  • H04L12/00—Data switching networks
  • H04L12/28—Data switching networks characterised by path configuration, e.g. LAN [Local Area Networks] or WAN [Wide Area Networks]
  • H04L12/2803—Home automation networks
  • H04L12/2823—Reporting information sensed by appliance or service execution status of appliance services in a home automation network
  • H04L12/2827—Reporting to a device within the home network; wherein the reception of the information reported automatically triggers the execution of a home appliance functionality
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04W—WIRELESS COMMUNICATION NETWORKS
  • H04W4/00—Services specially adapted for wireless communication networks; Facilities therefor
  • H04W4/02—Services making use of location information
  • H04W4/029—Location-based management or tracking services
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04W—WIRELESS COMMUNICATION NETWORKS
  • H04W4/00—Services specially adapted for wireless communication networks; Facilities therefor
  • H04W4/18—Information format or content conversion, e.g. adaptation by the network of the transmitted or received information for the purpose of wireless delivery to users or terminals
• • G—PHYSICS
  • G05—CONTROLLING; REGULATING
  • G05B—CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS
  • G05B2219/00—Program-control systems
  • G05B2219/20—Pc systems
  • G05B2219/26—Pc applications
  • G05B2219/2642—Domotique, domestic, home control, automation, smart house
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
  • H04L12/00—Data switching networks
  • H04L12/28—Data switching networks characterised by path configuration, e.g. LAN [Local Area Networks] or WAN [Wide Area Networks]
  • H04L12/2803—Home automation networks
  • H04L2012/284—Home automation networks characterised by the type of medium used
  • H04L2012/2841—Wireless
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
  • H04L12/00—Data switching networks
  • H04L12/28—Data switching networks characterised by path configuration, e.g. LAN [Local Area Networks] or WAN [Wide Area Networks]
  • H04L12/2803—Home automation networks
  • H04L2012/2847—Home automation networks characterised by the type of home appliance used
  • H04L2012/2849—Audio/video appliances
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
  • H04L12/00—Data switching networks
  • H04L12/28—Data switching networks characterised by path configuration, e.g. LAN [Local Area Networks] or WAN [Wide Area Networks]
  • H04L12/2803—Home automation networks
  • H04L2012/2847—Home automation networks characterised by the type of home appliance used
  • H04L2012/285—Generic home appliances, e.g. refrigerators

Definitions

• the present application relates to the field of control, and in particular, to a control system and a control processing method and apparatus.
• Smart homes are an organic combination of various systems related to home life such as security, light control, curtain control, gas valve control, information household appliances, scene linkage, floor heating, health care, hygiene and epidemic prevention, security guard using advanced computer technologies, network
• various smart home devices are generally controlled through mobile phone APPs corresponding to the smart home devices, and the smart home devices are controlled using a method of virtualizing the mobile phone APPs as remote controls.
• a certain response waiting time exists during the control of the home devices.
• an effective solution has not yet been proposed.
• Embodiments of the present application provide a control system and a control processing method and apparatus to solve the technical problem of complex operation and low control efficiency in controlling home devices.
• a control system includes a collection unit to collect information in a predetermined space that includes a plurality of devices.
• the control system also includes a processing unit to determine, according to the collected information, pointing information of a user.
• the processing unit selects a target device to be controlled by the user from the plurality of devices according to the pointing information.
• the present application further provides a control processing method that includes collecting information in a predetermined space that includes a plurality of devices. The method also includes determining, according to the collected information, pointing information of a user. Further, the method includes selecting a target device to be controlled by the user from the plurality of devices according to the pointing information.
• the present application further provides a control processing apparatus that includes a first collection unit to collect information in a predetermined space that includes a plurality of devices.
• the control processing apparatus also includes a first determining unit to determine, according to the collected information, pointing information of a user.
• the control processing apparatus further includes a second determining unit to select a target device to be controlled by the user from the plurality of devices according to the pointing information.
• a processing unit determines pointing information of a user's face appearing in a predetermined space according to information collected by a collection unit, determines a to-be-controlled device according to the indication of the pointing information, and then controls the determined device.
• a device to be controlled by a user can be determined based on pointing information of the user's face in a predetermined space so as to control the device.
• This process requires only collecting multimedia information to achieve the goal of controlling the device.
• the user does not need to switch among various operation interfaces of applications for controlling a device.
• the technical problem of complex operation and low control efficiency in controlling home devices is therefore solved, thereby achieving the goal of directly controlling a device according to the collected information with a simple operation.
• FIG. 1 is a schematic diagram illustrating a control system 100 according to an embodiment of the present application.
• FIG. 2 is a structural block diagram illustrating a computer terminal 200 according to an embodiment of the present application.
• FIG. 3(a) is a flow diagram illustrating a control processing method 300 according to an embodiment of the present application.
• FIG. 3(b) is a flow diagram illustrating an alternative control processing method 350 according to an embodiment of the present application.
• FIG. 4 is a schematic structural diagram illustrating an alternative human-computer interaction system according to an embodiment of the present application
• FIG. 5 is a flow diagram of a method 500 illustrating an alternative human-computer interaction system according to an embodiment of the present application.
• FIG. 6 is a schematic diagram illustrating a control processing apparatus according to an embodiment of the present application.
• FIG. 1 is a schematic diagram of a control system 100 according to an embodiment of the present application.
• control system 100 includes a collection unit 101 and a processing unit 103.
• Collection unit 101 is configured to collect information in a predetermined space that includes a plurality of devices.
• the predetermined space may be one or more preset spaces, and areas included in the space may have fixed sizes or variable sizes.
• the predetermined space is determined based on a collection range of the collection unit. For example, the predetermined space may be the same as the collection range of the collection unit, or the predetermined space may be within the collection range of the collection unit.
• rooms of the user include an area A, an area B, an area C, an area D, and an area E.
• the area A is a space that changes, for example, a balcony. Any one or more of the area A, the area B, the area C, the area D, and the area E may be set as the predetermined space according to the collection capacity of the collection unit.
• the collected information may include multimedia information, an infrared signal, and so on.
• Multimedia information is a combination of computer and video technologies, and the multimedia information mainly includes sounds and images.
• the infrared signal can represent a feature of a detected object through a thermal state of the detected object.
• collection unit 101 may collect the information in the predetermined space through one or more sensors.
• the sensors include, but are not limited to, an image sensor, a sound sensor, and an infrared sensor.
• Collection unit 101 may collect environmental information and/or biological information in the predetermined space through the one or more sensors.
• the biological information may include image information, a sound signal, and/or biological sign information.
• collection unit 101 may also be implemented through one or more signal collectors (or signal collection apparatuses).
• collection unit 101 may include an image collection system that is configured to collect an image in the predetermined space such that the collected information includes the image.
• the image collection system may be a DSP (Digital Signal Processor, namely, digital signal processing) image collection system, which can convert collected analog signals in the predetermined space into digital signals of 0 or 1.
• the DSP image collection system can also modify, delete, and enhance the digital signals, and then interpret digital data back into analog data or an actual environment format in a system chip.
• the DSP image collection system collects an image in the predetermined space, converts the collected image into digital signals, modifies, deletes, and enhances the digital signals to correct erroneous digital signals, converts the corrected digital signals into analog signals to realize correction of analog signals, and determines the corrected analog signals as the final image.
• the image collection system may also be a digital image collection system, a multispectral image collection system, or a pixel image collection system.
• collection unit 101 includes a sound collection system which can collect a sound signal in the predetermined space using a sound receiver, a sound collector, a sound card, or the like such that the collected information includes the sound signal.
• Processing unit 103 is configured to determine, according to the collected information, pointing information of the user, and then select a target device to be controlled by the user from the plurality of devices according to the pointing information.
• the processing unit may determine, according to the collected information, pointing information of a user's face appearing in the
• facial information of the user is extracted from the collected information.
• Pose and spatial position information or the like of the user's face are determined based on the facial information, and pointing information is then generated. After the pointing information of the user's face has been determined, a user device pointed to by the pointing information is determined according to the pointing
• the pointing information of the user's face may be determined through pointing information of a facial feature point of the user.
• the information in the predetermined space when the information in the predetermined space contains human body information, information of one or more human facial feature points is extracted from the information.
• the pointing information of the user is determined based on the extracted information of the facial feature points, wherein the pointing information points to a device to be controlled by the user.
• information of a nose (the information contains a pointing direction of a certain local position of the nose, for example, a pointing direction of a nose tip) is extracted from the information, and the pointing information is determined based on the pointing direction of the nose.
• information of a crystalline lens of an eye is extracted from the information, wherein the information may contain a pointing direction of a reference position of the crystalline lens, the pointing information is determined based on the pointing direction of the reference position of the crystalline lens of the eye.
• the pointing information may be determined according to the information of the eye and the nose.
• one piece of pointing information of the user's face may be determined through the orientation and angle of the crystalline lens of the eye, while the other piece of pointing information of the user's face may also be determined through the orientation and angle of the nose.
• the pointing information of the user's face determined through the crystalline lens of the eye is consistent with the other piece of pointing information of the user's face determined through the nose, the pointing information of the user's face is determined as the pointing information of the user's face in the predetermined space.
• a device in the direction pointed to by the determined pointing information of the user's face is determined according to the pointing information, and the device in the pointed-to direction is determined as the to-be-controlled device.
• pointing information of a user's face in a predetermined space can be determined based on collected information in the predetermined space, and a device controlled by the user can be determined according to the pointing information of the user's face.
• processing unit 103 is configured to determine that a user appears in the predetermined space when a human body appears in the image, and determine pointing information of the user's face.
• processing unit 103 detects whether the user appears in the predetermined space, and when the user appears in the predetermined space, determines pointing information of the user's face based on the collected information in the
• the detecting whether the user appears in the predetermined space may be implemented through the following steps: detecting whether a human body feature appears in the image and, when a human body feature is detected in the image, determining that a user appears in the image in the predetermined space.
• image features of a human body may be pre-stored. After collection unit 101 collects an image, the image is identified using the pre-stored image features (namely, human body features) of the human body. If it is recognized that an image feature exists in the image, it is determined that the human body appears in the image.
• pre-stored image features namely, human body features
• processing unit 103 is configured to determine pointing information of the user's face according to the sound signal.
• processing unit 103 detects whether the user appears in the predetermined space according to the sound signal and, when the user appears in the predetermined space, determines pointing information of the user's face based on the collected information in the predetermined space. [045] The detecting whether the user appears in the predetermined space according to the sound signal may be implemented through the following steps:
• sound features for example, a human voice feature
• collection unit 101 collects a sound signal
• the sound signal is recognized using the pre-stored sound features of the human body. If it is recognized that a sound feature exists in the sound signal, it is determined that the sound signal comes from the human body.
• a collection unit collects information, and a processing unit performs human recognition according to the collected information.
• processing unit 103 determines pointing information of the user's face so that whether a human body exists in the predetermined space can be accurately detected.
• processing unit 103 determines pointing information of the human face, thereby improving the efficiency of determining the pointing information of the human face.
• a device to be controlled by a user can be determined based on pointing information of the user's face in a predetermined space so as to control the device.
• This process requires only collecting multimedia information to achieve the goal of controlling the device.
• the user does not need to switch among various operation interfaces of applications for controlling a device.
• the technical problem of complex operation and low control efficiency in controlling home devices in the prior art is therefore solved, thereby achieving the goal of directly controlling a device according to the collected information with a simple operation.
• FIG. 2 is a structural block diagram of a computer terminal 200 according to an embodiment of the present application.
• computer terminal 200 may include one or more (only one in the figure) processing units 202 (the processing units 202 may include, but are not limited to, a processing apparatus such as a microprocessing unit (MCU) or a programmable logic device (FPGA)), a memory configured to store data, a collection unit 204 configured to collect information, and a transmission module 206 configured to implement a communication function.
• processing units 202 may include, but are not limited to, a processing apparatus such as a microprocessing unit (MCU) or a programmable logic device (FPGA)
• MCU microprocessing unit
• FPGA programmable logic device
• memory configured to store data
• collection unit 204 configured to collect information
• transmission module 206 configured to implement a communication function.
• computer terminal 200 may further include more or fewer components than those shown in FIG. 2, or have a different configuration from that shown in FIG. 2.
• Transmission module 206 is configured to receive or send data via a network. Specifically, transmission module 206 may be configured to send a command generated by processing unit 202 to various controlled devices 210 (including the device to be controlled by the user in the aforementioned embodiment).
• a specific example of the aforementioned network may include a wireless network provided by a communication supplier of computer terminal 200.
• transmission module 206 includes a network adapter (network interface controller, NIC), which may be connected to other network devices through a base station so as to communicate via the Internet.
• transmission module 206 may be a radio frequency (RF) module, which is configured to communicate with controlled device 210 in a wireless manner.
• NIC network interface controller
• RF radio frequency
• Examples of the aforementioned network include, but are not limited to, an internet, an intranet, a local area network, a mobile communication network, and a
• FIG. 3(a) shows a flow diagram that illustrates a control processing method 300 according to an embodiment of the present application.
• method 300 begins at step S302 by collecting information in a predetermined space that includes a plurality of devices.
• Method 300 next moves to step S304 to determine, according to the collected information, pointing information of a user. Following this, method 300 moves to step S306 to select a target device to be controlled by the user from the plurality of devices according to the pointing information.
• a processing unit determines pointing information of a user's face appearing in the predetermined space according to
• a collection unit determines a to-be-controlled device according to the indication of the pointing information, and then controls the determined device.
• a device to be controlled by a user can be determined based on pointing information of the user's face in a
• Step S302 may be implemented by collection unit 101.
• predetermined space may be one or more preset spaces, and areas included in the space may have fixed sizes or variable sizes.
• the predetermined space is determined based on a collection range of the collection unit.
• the predetermined space may be the same as the collection range of the collection unit, or the predetermined space may be within the collection range of the collection unit.
• rooms of the user include an area A, an area B, an area C, an area D, and an area E.
• the area A is a space that changes, for example, a balcony. Any one or more of the area A, the area B, the area C, the area D, and the area E may be set as the predetermined space according to the collection capacity of the collection unit.
• the information may include multimedia information, an infrared signal, and so on.
• the multimedia information is a combination of computer and video technologies, and the multimedia information mainly includes sounds and images.
• the infrared signal can represent a feature of a detected object through a thermal state of the detected object.
• FIG. 3(b) shows a flow diagram that illustrates an alternative control processing method 350 according to an embodiment of the present application.
• method 350 begins at step S352 to collect information in a predetermined space, and then moves to step S354 to determine, according to the collected information, pointing information of a user's face appearing in the predetermined space. Following this, method 350 moves to step S356 to determine a device to be controlled by the user according to the pointing information.
• a device to be controlled by a user can be determined based on the pointing information of the user's face in a predetermined space so as to control the device.
• This process requires only collecting multimedia information to achieve the goal of controlling the device.
• the user does not need to switch among various operation interfaces of applications for controlling a device.
• the technical problem of complex operation and low control efficiency in controlling home devices in the prior art is therefore solved, thereby achieving the goal of directly controlling a device according to the collected information with a simple operation.
• facial information of the user is extracted from the collected information.
• Pose and spatial position information or the like of the user's face is
• pointing information is then generated. After the pointing information of the user's face is determined, a user device pointed to by the pointing information is determined according to the pointing information, and the user device is determined as the target device to be controlled by the user.
• the pointing information of the user's face may be determined through pointing information of a facial feature point of the user. Specifically, after the information in the predetermined space is collected, when the collected information in the predetermined space contains human body information, information of one or more human facial feature points is extracted from the information. The pointing information of the user is determined based on the extracted information of the facial feature points, wherein the pointing information points to a device to be controlled by the user.
• information of a nose (the information contains a pointing direction of a certain local position of the nose, for example, a pointing direction of a nose tip) is extracted from the information, and the pointing information is determined based on the pointing direction of the nose.
• information of a crystalline lens of an eye is extracted from the information, wherein the information may contain a pointing direction of a reference position of the crystalline lens, the pointing information is determined based on the pointing direction of the reference position of the crystalline lens of the eye.
• the pointing information may be determined according to the information of the eye and the nose.
• one piece of pointing information of the user's face may be determined through the orientation and angle of the crystalline lens of the eye.
• the other piece of pointing information of the user's face may also be determined through the orientation and angle of the nose. If the piece of pointing information of the user's face determined through the crystalline lens of the eye is consistent with the other piece of pointing information of the user's face determined through the nose, the pointing information of the user's face is determined as the pointing information of the user's face in the predetermined space.
• a device in the direction pointed to by the determined pointing information of the user's face is determined according to the pointing information, and the device in the pointed-to direction is determined as the to-be-controlled device.
• pointing information of a user's face in a predetermined space can be determined based on collected information in the predetermined space.
• a device controlled by the user can be determined according to the pointing information of the user's face so that by determining the controlled device using the pointing information of the user's face, the interaction between the human and the device is simplified, and the interaction experience is improved, thereby achieving the goal of controlling different devices in the predetermined space.
• the information includes an image.
• determining pointing information of a user according to the image includes determining that the image contains a human body feature, wherein the human body feature includes a head feature, acquiring a spatial position and a pose of the head feature from the image, and determining the pointing information according to the spatial position and the pose of the head feature so as to determine the target device in the plurality of devices.
• the determining pointing information according to the image includes judging whether a human body appears in the image and, when judging that the human body appears, acquiring a spatial position and a pose of a head of the human body.
• coordinate system includes an x axis, a y axis, and a z axis) is established for the predetermined space, it is judged whether a human body exists in the collected image according to the image, and when the human body appears, a position of a head feature of the human body is acquired, wherein f indicates the human head, j S spatial position coordinates of the human head, is an x-axis coordinate of the human head in the three-dimensional space coordinate system, is a y-axis coordinate of the human head in the three-dimensional space coordinate system, and is a z-axis coordinate of the human head in the three-dimensional space coordinate system.
• a pose of a human head is acquired, wherein is used to indicate an Euler angle of the human head, is used to indicate an angle of precession, is used to indicate an angle of nutation, and is used to indicate an angle of rotation, and then the pointing information is determined according to the determined position of the head feature and the determined pose of the head feature of the human body.
• a pointing ray is determined using the spatial position of the head feature of the human body as a starting point and the pose of the head feature as a direction.
• the pointing ray is used as the pointing information, and the device
• the target device (namely, the target device) to be controlled by the user is determined based on the pointing information.
• device coordinates of the plurality of devices corresponding to the predetermined space are determined.
• a device range of each device is determined based on a preset error range and the device coordinates of each device.
• a device corresponding to a device range pointed to by the pointing ray is determined as the target device, wherein if the pointing ray passes through the device range, it is determined that the pointing ray points to the device range.
• the device coordinates may be three-dimensional coordinates.
• three-dimensional coordinates of various devices in the predetermined space are determined, and a device range of each device is determined based on a preset error range and the three- dimensional coordinates of each device, and after the pointing ray is acquired. If the ray passes through a device range, a device corresponding to the device range is the device (namely, the target device) to be controlled by the user.
• pointing information of the human face is determined, thereby improving the efficiency of determining the pointing information of the human face.
• the method when judging that a human body appears, the method further includes determining a posture feature and/or a gesture feature in a human body feature in the image, and controlling the target device according to a command corresponding to the posture feature and/or the gesture feature.
• commands corresponding to posture features and/or gesture features may be preset, the set correspondence is stored in a data table, and after a posture feature and/or a gesture feature is identified, a command matching the posture feature and/or the gesture feature is read from the data table.
• this table records the correspondence between postures, gestures, and commands.
• a pose feature is used to indicate a pose of the human body (or user)
• a gesture feature is used to indicate a gesture of the human body (or user).
• the collected information includes a sound signal
• the determining pointing information of a user according to the sound signal includes: determining that the sound signal contains a human voice feature
• the sound signal is a sound produced by a human body.
• position information of the source of the sound signal in the predetermined space and a propagation direction of the sound signal are determined, and the pointing information is determined according to the position information and the propagation direction so as to determine the device (namely, the target device) to be controlled by the user.
• a sound signal in the predetermined space may be collected. After the sound signal is collected, it is determined according to the collected sound signal whether the sound signal is a sound signal produced by a human body. After the sound signal is determined as a sound signal produced by the human body, a source position and a propagation direction of the sound signal are further acquired, and the pointing information is determined according to the determined position information and propagation direction.
• a pointing ray is determined using the position information of the source of the sound signal in the predetermined space as a starting point and the propagation direction as a direction.
• the pointing ray is used as the pointing information.
• device coordinates of the plurality of devices corresponding to the predetermined space are determined.
• a device range of each device is determined based on a preset error range and the device coordinates of each device.
• a device corresponding to a device range pointed to by the pointing ray is determined as the target device. If the pointing ray passes through the device range, it is determined that the pointing ray points to the device range.
• the device coordinates may be three-dimensional coordinates.
• three-dimensional coordinates of various devices in the predetermined space are determined, and a device range of each device is determined based on a preset error range and the three- dimensional coordinates of each device, and after the pointing ray is acquired. If the ray passes through a device range, a device corresponding to the device range is the device (namely, the target device) to be controlled by the user.
• the user stands in the bedroom facing the balcony and produces a sound "Open” to the curtains on the balcony.
• a sound signal "Open” is collected, it is judged whether the sound signal "Open” is produced by a human body. After it is determined that the sound signal is produced by the human body, a source position and a propagation direction of the sound signal, namely, a position at which the human body produces the sound and a propagation direction of the sound, are acquired. Pointing information of the sound signal is then determined.
• pointing information can be determined not only through a human face but also through a human sound so that flexibility of human-computer interaction is further increased. Different approaches are also provided for determining the pointing information.
• the target device is controlled to execute the command, wherein the device is the device determined to be controlled by the user according to the pointing information.
• a speech signal may be converted through speech recognition into a speech command corresponding to different services recognizable by various devices.
• a device pointed to by the sound signal is then controlled through the instruction to perform a corresponding operation so that the devices can be controlled more conveniently, rapidly, and
• a microphone array is used to measure the speech propagation direction and sound production position, which can achieve a similar effect to that of recognizing the head pose and position in the image.
• a unified interaction platform may be installed to multiple devices in a scattered manner.
• image and speech collection systems are installed on all the multiple devices to separately perform human face recognition and pose judgment rather than performing unified judgment.
• the device is controlled to execute the command, wherein the device is the device determined to be controlled by the user according to the pointing information.
• the pointing information and the command may be determined through different information, thereby increasing flexibility of processing. For example, after lights are determined as devices to be controlled by the user, the lights are turned on after the user issues a light-up command. At this time, another piece of information in the predetermined space is further collected. For example, the user issues a Bright command, and then an operation of adjusting the brightness is further performed.
• the device may be further controlled by collecting another piece of information in the predetermined space so that various devices can be controlled continuously.
• the another piece of information may include at least one of the following: a sound signal, an image, and an infrared signal. That is, the device already controlled by the user may be further controlled through an image, a sound signal, or an infrared signal to perform a corresponding operation, thereby further improving the
• nondirectional speech and gesture commands are reused using directional information of a human face so that the same command can be used for multiple devices.
• pointing information and a command of the user may be determined through an infrared signal.
• pointing information of a face of a human body carried in the infrared signal is recognized.
• a posture or a gesture of the human body may be extracted from the infrared information for recognition so as to determine a control instruction (namely, the aforementioned command) of the user.
• a sound signal in the predetermined space may be collected.
• the sound signal is recognized to obtain a command corresponding to the sound signal, and the controlled device is controlled to execute the command.
• an infrared signal in the predetermined space may be collected.
• the infrared signal is recognized to obtain a command corresponding to the infrared signal, and the controlled device is controlled to execute the command.
• image recognition and speech recognition in the aforementioned embodiment of the present application may choose to use an open source software library.
• the image recognition may choose to use a relevant open source project, for example, openCV (Open Source Computer Vision Library, namely, cross-platform computer vision library), dlib (an open source, cross-platform, general-purpose library written using modern C++ techniques), or the like.
• the speech recognition may use a relevant open source speech project, for example, openAL (Open Audio Library, namely, cross-platform audio API) or HKT (Hidden Markov Model Toolkit).
• the computer software product is stored in a storage medium (for example, a ROM/RAM, a magnetic disk, or an optical disk) and includes several instructions for instructing a terminal device (which may be a mobile phone, a computer, a server, a network device, or the like) to perform the methods described in the embodiments of the present application.
• a storage medium for example, a ROM/RAM, a magnetic disk, or an optical disk
• a terminal device which may be a mobile phone, a computer, a server, a network device, or the like
• a control system 400 for example, a human-computer interaction system shown in FIG. 4 includes: a camera 401 or other image collection system, a microphone 402 or other audio signal collection system, an information processing system 403, a wireless command interaction system 404, and controlled devices (the controlled devices include the aforementioned device to be controlled by the user), wherein the controlled devices include: lights 4051, televisions 4053, and curtains 4055.
• the camera 401 and the microphone 402 in this embodiment are included in collection unit 101 in the embodiment shown in FIG. 1.
• Information processing system 403 and wireless command interaction system 404 are included in processing unit 103 in the embodiment shown in FIG. 1.
• the camera 401 and the microphone 402 are respectively configured to collect image information and audio information in the activity space of the user and transfer the collected information to information processing system 403 for processing.
• Information processing system 403 extracts pointing information of the user's face and a user instruction.
• Information processing system 403 includes a processing program and hardware platform, which may be implemented in a form including, but not limited to, a local architecture and a cloud architecture.
• wireless command interaction system 404 sends, using radio waves or in an infrared manner, the user instruction to the controlled devices 4051, 4053, 4055 specified by the pointing information of the user's face.
• the device in the embodiment of the present application may be an intelligent device, and the intelligent device may communicate with processing unit 103 in the embodiment of the present application.
• the intelligent device may also include a processing unit and a transmission or communication module.
• the intelligent device may be a smart home appliance, for example, a television, or the like.
• FIG. 5 shows a flow diagram of a method 500 illustrating an alternative human-computer interaction system according to an embodiment of the present application.
• the control system shown in FIG. 4 may control the device according to the steps shown in FIG. 5.
• method 500 begins at step S501 by starting the system. After the control system (for example, the human-computer interaction system) shown in FIG. 4 has been started, method 500 separately performs step S502 and step S503 to collect an image and a sound signal in a predetermined space.
• control system for example, the human-computer interaction system
• step S502 method 500 collects an image.
• An image in the predetermined space may be collected using an image collection system.
• method 500 moves to step S504 to recognize whether a human is present.
• human recognition is performed on the collected image to determine whether a human body exists in the predetermined space.
• method 500 separately performs step S505, step S506, and step S507.
• step S505 method 500 recognizes a gesture.
• a human gesture is recognized on the collected image in the predetermined space so as to acquire an operation to be performed by the user through a recognized gesture.
• method 500 moves to step 506 to match gesture commands.
• the human-computer interaction system matches the recognized human gesture with a gesture command stored in the system so as to control, through the gesture command, the controlled device to perform a corresponding operation.
• step S507 method 500 estimates a head pose.
• a human head pose is estimated on the collected image in the predetermined space so as to determine a device to be controlled by the user through a recognized head pose.
• step S508 method 500 estimates a head position.
• a human head position estimation is performed on the collected image in the predetermined space so as to determine a device to be controlled by the user through a recognized head position.
• step 507 and step 508 method 500 matches device orientations in step S509. In a three-dimensional space coordinate system established in the
• the human-computer interaction system determines coordinates of the to-be-controlled device indicated by the pointing information according to a pose Euler angle of the human head and spatial position coordinates of the head, wherein are respectively a horizontal coordinate, a longitudinal coordinate, and a vertical coordinate of the controlled device.
• the three-dimensional space coordinate system is established in the predetermined space, and the pose Euler angle of the human head and the spatial position coordinates of the head are obtained using the human-computer interaction system.
• a certain pointing error (or error range) ⁇ is allowed.
• a ray may be drawn using r f as the starting point and as the direction, and if the ray (namely, the aforementioned pointing ray) passes through a sphere (namely, the device range in the aforementioned embodiment) using as the center and as the radius, it is determined that the human face points to the target controlled device (namely, the device to be controlled by the user in the aforementioned embodiment).
• step S506 to step S508 are performed without precedence.
• method 500 also collects sound in step S503.
• a sound signal in the predetermined space may be collected using an audio collection system. After this, method 500 moves to step S510 to perform speech
• the collected sound signal is recognized to judge whether the sound signal is a sound produced by the human body.
• step S511 to perform speech command matching.
• the human-computer interaction system matches the recognized speech information with a speech command stored in the system so as to control, through the speech
• step S506 After step S506, step S509, and step S511 have been performed, method 500 performs command synthesis in step S512.
• the matched gesture command and speech command are synthesized with the controlled device to generate a synthetic command so as to instruct the controlled device to perform a synthetic operation.
• step S513 to perform command broadcast.
• the synthetic command is broadcast (namely, sent and propagated) to control each to-be-controlled device to perform a corresponding operation.
• the command may be sent in a manner including, but not limited to, radio communication and infrared remote control.
• step S5114 returns method 500 back to the start.
• the aforementioned human-computer interaction system includes an image processing part and a sound processing part.
• the image processing part is further divided into a human recognition unit and a gesture recognition unit.
• the image processing part first collects an image in the activity space (namely, the predetermined space) of the user, and then recognizes whether a human body image exists in the image.
• the flow separately enters into a head recognition unit and the gesture recognition unit.
• head recognition unit head pose estimation and head position estimation are performed, and then face orientation is solved by synthesizing the head pose and position.
• face orientation is solved by synthesizing the head pose and position.
• gesture recognition unit a gesture of the user in the image is recognized and matched with a gesture command, and if the matching is successful, the command is output.
• a sound signal is first collected, then speech recognition is performed on the sound signal to extract a speech command. If the extraction is successful, the command is output.
• the commands output at the head recognition unit and the speech processing part are synthesized with a target device address obtained according to the face orientation to obtain a final command. Therefore, directional information is provided to the human-computer interaction system through the pose of the human face to accurately point to a specific device.
• Using and reusing of multiple specific devices is made possible via a speech command and a gesture command.
• a speech command and a gesture command For example, when the user issues a speech command "Open/Turn on” facing different devices, the faced devices can be opened/turned on.
• a gesture command “Palm to fist” facing different devices, the faced devices can be closed or turned off, and the like.
• the delay and costs of human-computer interaction in the aforementioned embodiment may be reduced in the following manners.
• a specific image recognition chip ASIC Application Specific Integrated Circuit, namely, integrated circuit
• an FPGA Field-Programmable Gate Array
• an architecture such as x86 (a microprocessor) or arm (Advanced RISC Machines, namely, embedded RISC processor) may further be used to have low costs.
• a GPU Graphic Processing Unit, namely, a graphics processor
• all or some of processing programs are run on the cloud.
• FIG. 6 shows a schematic diagram illustrating a control processing apparatus 600 according to an embodiment of the present application.
• apparatus 600 includes a first collection unit 601 configured to collect information in a predetermined space that includes a plurality of devices.
• Apparatus 600 also includes a first determining unit 603 configured to determine, according to the collected information, pointing information of a user, and a second determining unit 605 configured to select a target device to be controlled by the user from the plurality of devices according to the pointing information.
• a processing unit determines pointing information of a face of a user appearing in a predetermined space according to information collected by a collection unit, and determines a to-be-controlled device according to indication of the pointing information, and then controls the
• a device to be controlled by a user can be determined based on pointing information of the user's face in a predetermined space so as to control the device.
• This process requires only collecting multimedia information to realize control on the device, without requiring the user to switch various operation interfaces of applications to realize control on the device.
• the technical problem of complex operation and low control efficiency in controlling home devices in the prior art is solved.
• the purpose of directly controlling a device according to collected information is achieved. Further, the operation is simple.
• the aforementioned predetermined space may be one or more preset spaces, and areas included in the space may have fixed sizes or variable sizes.
• the predetermined space is determined based on a collection range of the collection unit.
• the predetermined space may be the same as the collection range of the collection unit, or the predetermined space may be within the collection range of the collection unit.
• rooms of the user include an area A, an area B, an area C, an area D, and an area E.
• the area A is a space that changes, for example, a balcony. Any one or more of the area A, the area B, the area C, the area D, and the area E may be set as the predetermined space according to the collection capacity of the collection unit.
• the aforementioned information may include multimedia information, an infrared signal, and so on.
• the multimedia information is a combination of computer and video technologies, and mainly includes sounds and images.
• the infrared signal can represent a feature of a detected object through a thermal state of the detected object.
• facial information of the user is extracted from the information, pose and spatial position information, or the like of the user's face is determined based on the facial information, and pointing information is generated.
• a user device pointed to by the pointing information is determined according to the pointing information, and the user device is determined as the device to be controlled by the user.
• the pointing information of the user's face may be determined through pointing information of a facial feature point of the user. Specifically, after the information in the predetermined space is collected, when the information in the predetermined space contains human body information, information of one or more human facial feature points is extracted from the information. The pointing information of the user is determined based on the extracted information of the facial feature points, wherein the pointing information points to a device to be controlled by the user.
• information of a nose (the information contains a pointing direction of a certain local position of the nose, for example, a pointing direction of a nose tip) is extracted from the information, and the pointing information is determined based on the pointing direction of the nose.
• information of a crystalline lens of an eye is extracted from the information, wherein the information may contain a pointing direction of a reference position of the crystalline lens, the pointing information is determined based on the pointing direction of the reference position of the crystalline lens of the eye.
• the pointing information may be determined according to the information of the eye and the nose.
• one piece of pointing information of the user's face may be determined through the orientation and angle of the crystalline lens of the eye, while the other piece of pointing information of the user's face may also be determined through the orientation and angle of the nose.
• the pointing information of the user's face determined through the crystalline lens of the eye is consistent with the other piece of pointing information of the user's face determined through the nose, the pointing information of the user's face is determined as the pointing information of the user's face in the predetermined space.
• a device in the direction pointed to by the determined pointing information of the user's face is determined according to the pointing information, and the device in the pointed-to direction is determined as the to-be-controlled device.
• pointing information of a user's face in a predetermined space can be determined based on collected information in the predetermined space, and a device controlled by the user is determined according to the pointing information of the user's face.
• a device controlled by the user is determined according to the pointing information of the user's face.
• the first determining unit may include: a first feature determining module configured to determine that the image contains a human body feature, wherein the human body feature includes a head feature; a first acquisition module configured to acquire a spatial position and a pose of the head feature from the image; and a first information determining module configured to determine the pointing information according to the spatial position and the pose of the head feature so as to determine the target device in the plurality of devices.
• the first information determining module is specifically configured to determine a pointing ray using the spatial position of the head feature as a starting point and the pose of the head feature as a direction.
• the pointing ray is used as the pointing information.
• the apparatus further includes: a first recognition module configured to, when determining that the image contains the human body feature, acquire a posture feature and/or a gesture feature from the image comprising the human body feature; and a first control module configured to control the target device according to a command corresponding to the posture feature and/or the gesture feature.
• a posture and/or a gesture of the human body may further be recognized, and a device pointed to by the facial information is controlled through a preset control instruction corresponding to the posture and/or the gesture of the human body to perform a corresponding operation.
• An operation that a device is controlled to perform can be determined when the controlled device is determined so that the waiting time in human-computer interaction is reduced to a certain extent.
• the first determining unit when the information includes a sound signal, and the pointing information is determined according to the sound signal, the first determining unit further includes: a second feature determining module configured to determine that the sound signal contains a human voice feature; a second acquisition module configured to determine position information of a source of the sound signal in the predetermined space and a
• a second information determining module configured to determine the pointing information according to the position information of the source of the sound signal in the
• predetermined space and the propagation direction so as to determine the target device in the plurality of devices.
• the second information determining module is specifically configured to: determine a pointing ray using the position information of the source of the sound signal in the predetermined space as a starting point and the propagation direction as a direction; and use the pointing ray as the pointing information.
• pointing information can be determined not only through a human face but also through a human sound so that flexibility of human-computer interaction is further increased. Different approaches are also provided for determining the pointing information.
• the apparatus further includes: a second recognition module configured to, when determining that the sound signal contains the human voice feature, perform speech recognition on the sound signal to acquire a command corresponding to the sound signal; and a second control module configured to control the target device to execute the command.
• a speech signal may be converted through speech recognition into a speech command corresponding to different services that is recognizable by various devices.
• a device pointed to by the sound signal is then controlled through the instruction to perform a corresponding operation so that the devices can be controlled more conveniently, rapidly, and accurately.
• the apparatus further includes a second collection unit configured to collect another piece of information in the predetermined space.
• a recognition unit is configured to recognize the another piece of information to obtain a command corresponding to the another piece of information.
• a control unit is configured to control the device to execute the command, wherein the device is the device determined to be controlled by the user according to the pointing information.
• the pointing information of the user is determined by collecting image information in the predetermined space
• another piece of information in the predetermined space may be collected.
• the another piece of information is identified to obtain a command corresponding to the another piece of information.
• the device is controlled to execute the command, wherein the device is the device determined to be controlled by the user according to the pointing information. That is, in this embodiment, the pointing information and the command may be determined through different
• the another piece of information includes at least one of the following: a sound signal, an image, and an infrared signal. That is, the device already controlled by the user may be further controlled through an image, a sound signal, or an infrared signal to perform a corresponding operation, thereby further improving the experiencial effect of the human- computer interaction.
• nondirectional speech and gesture commands are reused using directional information of a human face so that the same command can be used for multiple devices.
• An embodiment of the present application further provides a storage medium.
• the storage medium may be used for storing program code executed by the control processing method provided in the
• the storage medium may be located in any computer terminal in a computer terminal group in a computer network, or located in any mobile terminal in a mobile terminal group.
• the storage medium is configured to store program code for executing the following steps: collecting information in a
• predetermined space determining, according to the information, pointing information of a face of a user appearing in the predetermined space; and determining a device to be controlled by the user according to the pointing information.
• a processing unit determines pointing information of a user's face appearing in a predetermined space according to information collected by a collection unit, determines a to-be-controlled device according to the indication of the pointing information, and then controls the determined device.
• a device to be controlled by a user can be determined based on pointing information of the user's face in a predetermined space so as to control the device.
• This process requires only collecting multimedia information to achieve the goal of controlling the device.
• the user does not need to switch among various operation interfaces of applications for controlling a device.
• the technical problem of complex operation and low control efficiency in controlling home devices in the prior art is therefore solved, thereby achieving the goal of directly controlling a device according to the collected information with a simple operation.
• communication connections may be implemented through some interfaces, and the indirect couplings or communication connections between units or modules may be implemented in electrical or other forms.
• the units described as separate parts may be or may not be physically separate, and the parts shown as units may be or may not be physical units, and not only can be located in one place, but also can be distributed onto a plurality of network units. Part or all of the units can be chosen to implement the purpose of the solutions of this embodiment according to actual requirements.
• respective functional units in respective embodiments of the present application may be integrated into one processing unit, or respective units may physically exist alone, or two or more units may be integrated into one unit.
• the integrated unit can be implemented in the form of hardware or in the form of a software functional unit.
• the integrated unit may be stored in a computer readable storage medium.
• the computer software product is stored in a storage medium and includes several instructions for instructing a computer device (which may be a personal computer, a server, a network device, or the like) to perform all or part of the steps in the methods described in the embodiments of the present application.
• the foregoing storage medium includes various media capable of storing program code, such as a USB flash drive, a readonly memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk, or an optical disk.

Landscapes

• Engineering & Computer Science (AREA)
• General Engineering & Computer Science (AREA)
• Automation & Control Theory (AREA)
• Physics & Mathematics (AREA)
• General Physics & Mathematics (AREA)
• Theoretical Computer Science (AREA)
• Computer Networks & Wireless Communication (AREA)
• Signal Processing (AREA)
• Human Computer Interaction (AREA)
• Manufacturing & Machinery (AREA)
• Quality & Reliability (AREA)
• User Interface Of Digital Computer (AREA)
• Selective Calling Equipment (AREA)

Priority Applications (2)

Applications Claiming Priority (2)

Publications (1)

Family

ID=61159612

Family Applications (1)

Country Status (6)

Cited By (2)

Families Citing this family (20)

Citations (7)

Family Cites Families (18)

• 2016
  • 2016-08-11 CN CN201610658833.6A patent/CN107728482A/zh active Pending
• 2017
  • 2017-05-10 TW TW106115504A patent/TW201805744A/zh unknown
  • 2017-08-10 US US15/674,147 patent/US20180048482A1/en not_active Abandoned
  • 2017-08-10 JP JP2019507757A patent/JP6968154B2/ja active Active
  • 2017-08-10 WO PCT/US2017/046276 patent/WO2018031758A1/en not_active Ceased
  • 2017-08-10 EP EP17840270.7A patent/EP3497467A4/en not_active Withdrawn

Patent Citations (7)

Non-Patent Citations (1)

Also Published As

Similar Documents

Legal Events