CN115393954A

CN115393954A - AR-based child behavior analysis method, terminal device and server

Info

Publication number: CN115393954A
Application number: CN202210988674.1A
Authority: CN
Inventors: 费雯悦
Original assignee: Beijing Hetu United Innovation Technology Co ltd
Current assignee: Beijing Hetu United Innovation Technology Co ltd
Priority date: 2022-08-17
Filing date: 2022-08-17
Publication date: 2022-11-25

Abstract

The application discloses a child behavior analysis method and device based on augmented reality AR, a server and a terminal device. The method comprises the following steps: acquiring monitoring video data of an activity area where a child is located; according to different children appearing in the monitoring video data, carrying out image segmentation processing on the monitoring video data to obtain a plurality of video image data sets respectively corresponding to the different children; identifying one or more child behaviors occurring in a first video image dataset corresponding to a first child to determine a child behavior analysis result, wherein the child behavior analysis result comprises the number of times and/or time of occurrence of the one or more child behaviors; converting a child behavior analysis result corresponding to the first video image data set into a first AR image; and sending the first AR image to a first terminal device. Utilize this application embodiment can improve children's control data utilization and rate, the head of a family of being convenient for knows children's behavioral performance fast.

Description

AR-based child behavior analysis method, terminal device and server

Technical Field

The application relates to the technical field of enhanced display, in particular to a child behavior analysis method based on AR, a terminal device, a server, a computer readable storage medium and a computer program product.

Background

At present, more and more kindergartens are provided with monitoring equipment in public areas, so that on one hand, monitoring pictures of kindergartens activities can be provided for parents of children, and the parents can be helped to comprehensively know the behaviors of the children, for example, the parents generally concern the sleeping quality of the children in the kindergartens, the interaction condition with children, the drinking frequency, the eating condition and the like, and the daily behavior performance of the children can be visually seen through monitoring videos; on the other hand, if special conditions occur, the monitoring video can help to be traceable and documented.

However, in most cases, because the free time of parents is limited, many parents cannot pay attention to and check the monitoring videos of children in time, even if the parents check the monitoring videos, the parents only randomly select some time periods to check the monitoring videos, and the parents cannot check the monitoring videos in the whole process, so that the parents cannot know the specific behavior state of the children comprehensively and accurately; of course, parents can also choose to communicate with teachers in a kindergarten, but the teachers can only feed back the general behavior habits of children and cannot clearly and accurately record all behaviors of the children. In addition, due to the considerations of data security, personal privacy and the like, the monitoring data of many education institutions is not actively opened to parents, and parents are difficult to know the behavior information of children.

Disclosure of Invention

In view of the above, embodiments of the present application provide an AR-based child behavior analysis method, a terminal device, a server, a computer-readable storage medium, and a computer program product, which are used to solve at least one of the above problems and can be popularized to middle and primary schools or various institutional sites mainly involving children.

In a first aspect, an embodiment of the present application provides a child behavior analysis method based on augmented reality AR, applied to a server, including: the method comprises the steps that a server obtains monitoring video data of an activity area where a child is located; the server carries out image segmentation processing on the monitoring video data according to different children appearing in the monitoring video data to obtain a plurality of video image data sets respectively corresponding to the different children; the server identifies one or more child behaviors appearing in a first video image data set corresponding to a first child to determine a child behavior analysis result, wherein the child behavior analysis result comprises the number of times and/or time of the one or more child behaviors; the server converts a child behavior analysis result corresponding to the first video image data set into a first AR image, wherein the first AR image comprises information of the occurrence times and/or time of the one or more child behaviors; and the server sends the first AR image to a first terminal device so that the first terminal device displays the first AR image.

In a second aspect, an embodiment of the present application provides a child behavior analysis method based on augmented reality AR, applied to a terminal device, including: the method comprises the steps that a terminal device sends request information to a server, the request information is used for requesting a child behavior analysis result corresponding to a first child, and the request information comprises user identity information; the terminal equipment receives and displays a first AR image sent by the server, wherein the first AR image comprises the information of the occurrence times and/or time of the one or more child behaviors; the first AR image is generated by the server based on a child behavior analysis result corresponding to a first video image data set, the first video image data set is from monitoring video data of an activity area where the first child is located, and the child behavior analysis result is determined by identifying one or more child behaviors of the first child in the first video image data set through the server.

In a third aspect, an embodiment of the present application provides an augmented reality AR-based child behavior analysis apparatus, including:

the acquisition module is used for acquiring monitoring video data of an activity area where the child is located;

the segmentation module is used for carrying out image segmentation processing on the monitoring video data according to different children appearing in the monitoring video data to obtain a plurality of video image data sets respectively corresponding to the different children;

the identification module is used for identifying one or more child behaviors appearing in a first video image data set corresponding to a first child to determine a child behavior analysis result, wherein the child behavior analysis result comprises the number of times and/or time of the one or more child behaviors;

a conversion module, configured to convert a child behavior analysis result corresponding to the first video image data set into a first AR image, where the first AR image includes information of the number of times and/or time of occurrence of the one or more child behaviors;

and the sending module is used for sending the first AR image to a first terminal device so that the first terminal device displays the first AR image.

In a fourth aspect, an embodiment of the present application provides an augmented reality AR-based child behavior analysis apparatus, including:

the sending module is used for sending request information to a server, the request information is used for requesting a child behavior analysis result corresponding to a first child, and the request information comprises user identity information;

the display module is used for receiving and displaying a first AR image sent by the server, wherein the first AR image comprises the information of the occurrence times and/or time of the one or more child behaviors; the first AR image is generated by the server based on a child behavior analysis result corresponding to a first video image data set, the first video image data set is from monitoring video data of an activity area where the first child is located, and the child behavior analysis result is determined by identifying one or more child behaviors of the first child in the first video image data set by the server.

In a fifth aspect, an embodiment of the present application provides a server device, including: a processor and a memory storing computer program instructions; the processor, when executing the computer program instructions, performs the steps of the method as described above.

In a sixth aspect, an embodiment of the present application provides a terminal device, including: a processor and a memory storing computer program instructions; the processor, when executing the computer program instructions, performs the steps of the method as described above.

In a seventh aspect, the present application provides a computer-readable storage medium, on which computer program instructions are stored, and when the computer program instructions are executed by a processor, the steps of the method are implemented as described above.

In an eighth aspect, the present application provides a computer program product, which includes computer program instructions, and when the computer program instructions are executed by a processor, the steps of the method are implemented as described above.

Utilize the scheme that this application embodiment provided, can discern and whole analysis to children's surveillance video, obtain children's behavioral characteristics in certain period of time, and play analysis result and corresponding picture with the mode of AR image, the head of a family does not go to specifically look over video content, also can accomplish comparatively accurate grasp to children's behavioural expression, the utilization ratio of children's monitoring data has been improved to a certain extent, be favorable to the head of a family in time to know children's action, the energy and the time that the head of a family paid for knowing children's action can be compressed by a wide margin to its convenient degree, certain social meaning and value have.

Drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings in the embodiments of the present application are briefly described below.

Fig. 1 is a schematic diagram of an AR system architecture based on a server and a terminal device according to an embodiment of the present application.

Fig. 2 is a schematic diagram of a virtual-real fusion image for AR navigation by using a mobile phone APP.

Fig. 3 is a block flow diagram of a server-side AR-based child behavior analysis method according to an embodiment of the present application.

Fig. 4 is a flowchart of an AR-based child behavior analysis method on the terminal device side according to an embodiment of the present application.

FIG. 5 is a flow chart illustrating a process for providing video of a kindergarten child according to an embodiment of the present application.

Fig. 6 is a block diagram showing a configuration of a child behavior analysis device based on AR on the service side according to an embodiment of the present invention.

Fig. 7 is a block diagram of a configuration of an AR-based child behavior analysis device on a terminal device side according to an embodiment of the present invention.

Fig. 8 is a schematic diagram of an electronic device for implementing the AR-based child behavior analysis method according to the embodiment of the present application.

Fig. 9 is a schematic diagram of a software structure of a terminal device according to an embodiment of the present application.

Detailed Description

The principles and spirit of the present application will be described below with reference to a number of exemplary embodiments. It is to be understood that these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the concept and spirit of the disclosure to those skilled in the art. The exemplary embodiments provided herein are only a part of the embodiments of the present application, and not all of the embodiments. All other embodiments obtained by a person of ordinary skill in the art based on the embodiments herein without inventive effort are within the scope of the present application.

As will be appreciated by one skilled in the art, embodiments of the present application may be embodied as a system, apparatus, device, method, computer-readable storage medium, or computer program product. Accordingly, the present application may be embodied in at least one of the following forms: entirely hardware, entirely software (including firmware, resident software, micro-code, etc.), or a combination of hardware and software.

According to the embodiment of the application, the application requests to protect an Augmented Reality (AR) testing method, an AR testing device, a terminal device, a server and a computer-readable storage medium.

In this document, terms such as first, second, third, and the like are used solely to distinguish one entity (or operation) from another entity (or operation), without necessarily requiring or implying any order or relationship between such entities (or operations).

The embodiment of the application can be applied to a server and a terminal device. Referring to fig. 1, a schematic diagram of an AR system architecture based on a server and a terminal device is schematically shown. The AR system architecture includes a server 10 and several terminal devices 20. In some examples, terminal device 20 is an AR device, which may be a dedicated AR device, such as a Head-mounted AR device (HMD), a smart glove, a piece of apparel, or other wearable electronic device. In some examples, the terminal device 20 may be a general-purpose AR device, such as a cell phone, a portable computer, a laptop, a tablet computer, a Virtual Reality (VR) device, or a vehicle-mounted device, which is installed with AR function software, and so on.

Taking an AR helmet or AR glasses as an example, a head-mounted display, a machine vision system, a mobile computer, and the like can be integrated and arranged in a device which can be worn in a binding manner, the device is provided with a display similar to glasses in shape and worn on the head of a user during work, and the device can transmit augmented reality information to the display or project the augmented reality information to eyeballs of the user, so that visual immersion of the user is enhanced. In some examples, the AR device also has a camera, which may be a wide-angle camera, a tele-camera, and also a structured light camera (also referred to as a point cloud depth camera, a 3D structured light camera, or a depth camera). The structured light camera is based on a 3D vision technology, and can acquire plane and depth information of an object. The structure light camera accessible near infrared laser instrument will have certain structural feature light to be projected by the shooting object on, gather the reverberation by infrared camera again, be handled by the treater chip, its theory of calculation is that the change according to the light signal that the object leads to calculates object position and depth information, presents the 3D image. The general terminal equipment such as a mobile phone presents a two-dimensional image, the depth of different positions on the image cannot be displayed, and 3D image information data can be shot and acquired by using a structured light camera, so that not only can information such as colors of different positions in the image be acquired, but also depth information of different positions can be acquired, and the method can be used for AR ranging. Of course, the common terminal device may also acquire the depth information of the 2D image based on the optical camera and by combining with a deep learning algorithm and the like, and may finally present the 3D image.

In some examples, terminal device 20 has AR-enabled software or application APP installed therein. The server 10 may be a management server or an application server of the software or APP. The server 10 may be a single server, a server cluster formed by multiple servers, a cloud server or a cloud server, and the like. A module having a networking function, such as a Wireless-Fidelity (Wifi) module, a bluetooth module, a 2G/3G/4G/5G communication module, etc., is integrated in the terminal device 20 so as to be connected to the server 10 through a network.

Taking APP with AR navigation function as an example, APP may have high-precision map navigation capability, environment understanding capability, virtual-real fusion rendering capability, and the like, APP may report current geographic location information to server 10 through terminal device 20, and server 10 provides AR navigation service for the user based on real-time geographic location information. Illustratively, taking the terminal device 20 as a mobile phone as an example, in response to the operation of starting the APP by the user, the mobile phone may start a camera to acquire an image of the real environment, then perform AR enhancement on the image of the real environment acquired by the camera through the system, blend or superimpose a rendered AR effect (e.g., a navigation route identifier, a road name, business information, an advertisement display, etc.) in the image of the real environment, and display the image of the virtual-real blend on the screen of the mobile phone.

For example, a first user may log in to a first user account through an APP installed in a cell phone, and a second user may log in to a second user account through software installed in AR glasses.

Fig. 2 schematically shows a virtual-real fusion image for AR navigation by using a mobile phone APP, wherein an indication arrow of AR navigation is superimposed on a real road surface and in a space in the image, and an electronic resource promoted by a merchant is displayed in a space in a building in a form that a parachute carries a small gift box. When a user walks with the mobile phone, the AR navigation route can be checked through the mobile phone screen, and digital information related to the environment can be obtained.

Embodiments of the present application relate to a terminal device and/or a server. The principles and spirit of the present application will be explained in detail below by way of a number of exemplary embodiments or representative implementations.

Fig. 3 schematically shows a virtual-real fused image for AR navigation by using a mobile phone terminal APP, wherein an indication arrow of AR navigation is superimposed on a real road surface and in a space in the figure, and electronic resources promoted by merchants are displayed in a space in a building in a manner that a parachute carries a small gift box. When a user walks with the mobile phone, the AR navigation route can be checked through the mobile phone screen, and the digital information related to the environment can be easily obtained.

Embodiments of the present application relate to a terminal device and/or a server. The principles and spirit of the present application will be explained in detail below by way of several exemplary embodiments or representative implementations.

The embodiment of the application provides an AR-based child behavior analysis method, which is applied to a server and refers to fig. 3, and the method includes the following steps:

s101, a server acquires monitoring video data of an activity area where a child is located;

s102, the server carries out image segmentation processing on the monitoring video data according to different children appearing in the monitoring video data to obtain a plurality of video image data sets respectively corresponding to the different children;

s103, the server identifies one or more child behaviors appearing in a first video image data set corresponding to a first child to determine a child behavior analysis result, wherein the child behavior analysis result comprises the number of times and/or time of the one or more child behaviors;

s104, the server converts the child behavior analysis result corresponding to the first video image data set into a first AR image, wherein the first AR image comprises the information of the occurrence times and/or time of the one or more child behaviors;

and S105, the server sends the first AR image to a first terminal device, so that the first terminal device displays the first AR image.

According to the method of the embodiment of the application, the server can process the content of the monitoring videos of the children (such as the daily monitoring videos of kindergarten, school, extraclass education institution, various children activity venues and the like), not only can the video content be divided according to different children, but also the video content of different children can be further identified and integrally analyzed, the behavior characteristics of the children in a certain time period are obtained, such as the times and time of occurrence of certain behaviors (the hands are held in class to answer questions, the afternoon nap time is less than 30 minutes) and the like, the analysis results and corresponding pictures are played in an AR image mode, and equivalently, the AR analysis results of behavior habits of different children in the monitoring videos are obtained, and parents can watch the results through equipment such as mobile phones and AR glasses. Therefore, the parents can accurately master the behavior of the children even if the parents do not specifically check the video content, the utilization rate of the monitoring data of the children is improved to a certain degree, the parents can timely know the behavior of the children, the education means is timely adjusted, the children are timely informed (or criticized), and good behavior habits are developed.

Corresponding to the processing on the server side, the embodiment of the present application further provides an AR-based child behavior analysis method, applied to a terminal device, and referring to fig. 4, including the following steps:

s201, a terminal device sends request information to a server, wherein the request information is used for requesting a child behavior analysis result corresponding to a first child, and the request information comprises user identity information;

s202, the terminal equipment receives and displays a first AR image sent by the server, wherein the first AR image comprises the information of the occurrence times and/or time of the one or more child behaviors; the first AR image is generated by the server based on a child behavior analysis result corresponding to a first video image data set, the first video image data set is from monitoring video data of an activity area where the first child is located, and the child behavior analysis result is determined by identifying one or more child behaviors of the first child in the first video image data set by the server.

According to the method of the embodiment of the application, at the terminal side, when a user such as a parent wants to check the AR analysis result of the monitoring video of the child, a request can be sent out, the server sends the AR analysis result of the corresponding child to the user terminal according to the request information, at the moment, the user can obtain the statistical result (such as the occurrence frequency and/or time of the child behavior) after the child behavior is identified and analyzed, and can see that the statistical result is played in an AR image mode, so that the method is clear, visual and easy to obtain, the convenience degree of the method can greatly compress the energy and time of the parent for knowing the child behavior, and has certain social significance and value.

According to an embodiment of the present application, optionally, the following processing is further performed:

(1) matching the child face image appearing in the monitoring video data with the child face image prestored in the server;

(2) after a plurality of video image data sets respectively corresponding to different children are obtained, the identity ID information of the different children is bound with the corresponding video image data sets.

For example, for a young child in a kindergarten or a middle or large child in a school of primary and secondary schools, if facial images of the child are stored in the server, the monitoring data of the child are identified and matched, then binding is performed, and when a user request is received, behavior analysis can be performed and an AR analysis result can be sent. Optionally, the video data of the child face image which is not pre-stored in the server is not processed, so that the waste of computing resources is avoided.

(1) before the first AR image is sent to a first terminal device, the server receives request information from the first terminal device, wherein the request information is used for requesting a child behavior analysis result corresponding to the first child, and the request information comprises user identity information;

(2) and the server verifies the user identity information, and if the user identity information passes the verification, the first AR image is allowed to be sent to the first terminal equipment.

That is to say, when request information of checking the AR analysis result by the user is received, the identity of the user needs to be verified, the AR analysis result is normally sent if the verification is passed, and no processing is performed if the verification is not passed, thereby ensuring the security of the data information.

According to the embodiment of the present application, optionally, the first AR image further includes image and/or video information of the one or more child behaviors.

For example, the AR analysis result may be made in a form in which pictures, videos, and characters correspond to each other, so that parents can quickly browse key information.

According to an embodiment of the application, optionally, the child behavior comprises at least one of: work and rest conditions, dining conditions, game interaction states, drinking frequency, toilet frequency, sleeping conditions, dining conditions, dressing conditions, class listening states, question answering conditions, and task making states.

The method covers the more common behavior types of the children in places such as kindergartens, schools, extraclass education institutions, various children activity venues and the like, can adopt a pre-trained neural network model for identification, and counts the times and time of the various behaviors of the children within one day, so as to achieve the purpose of integrally analyzing video contents.

In order to more clearly illustrate the advantages that can be obtained by the embodiments of the present application, the following describes the processing procedure of the embodiments of the present application in detail based on specific examples.

As an example, the following description will be made by taking an AR behavior analysis process of a kindergarten child Zhang III in a kindergarten as an example, and fig. 5 schematically shows a process of processing a kindergarten-related monitoring video, which is roughly as follows:

1) Acquiring a monitoring video of a kindergarten or a school, and segmenting and warehousing related video images of corresponding characters of a baby (or a student) by using an image matching Artificial Intelligence (AI) algorithm;

2) Correspondingly analyzing the video time and the behavior expression of each child and generating data;

3) Parents carry out face verification to infants or students, obtain relevant videos and analysis data of corresponding characters through verification, and show the relevant videos and analysis data to the parents in an AR mode.

Specifically, the server obtains video data of a middle class where zhang san of the kindergarten is located, then performs image processing on the video data, mainly performs face matching on children by using an AI algorithm, and performs segmentation processing on the video data after matching. Similarly, the video image data sets corresponding to each child in the class, such as the video image data sets of the children 1 to n in fig. 5, may be stored in the database 1.

Then, performing analysis calculation on the video image data set in the database 1, and using the trained neural network model to identify the child behavior category in the video image, for example, analyzing and identifying the child behavior may be: work and rest time, dining conditions, game interactions, class attendance states, answer questions, water drinking frequency, toilet frequency, sleep quality, dining conditions, job status, self-standing ability, and the like. Time, occurrence times and the like can be counted, and an AR analysis result is formed by combining the corresponding images and can be stored in the database 2 as analysis result data of the children 1 to n.

After the request information is received, the identity of the requester is verified, the face of the target child who wants to check the analysis result can be verified, after the verification is passed, the analysis result of the target child is searched in the database 2 and displayed in an AR virtual image mode, and therefore parents can conveniently and visually obtain the content of the analysis result. The user device is not limited to a cell phone, AR glasses, etc.

For example, referring to fig. 5, the results of AR analysis for children zhang san are shown as follows:

breakfast time: only the staple food is eaten and the vegetables are not eaten.

The game time: collide with children.

Afternoon nap time: difficulty in falling asleep, and less than 30 minutes of sleep time.

Class hours: raise hands 3 times, answer 3 times, vague 2 times, steal whisper 2 times.

To summarize: today, three children in the kindergarten have active school performance, but have behaviors of vague nerve and joint; the afternoon nap is difficult, and the afternoon nap is also recommended at regular time on weekends, so that good habits are developed.

The display of the AR analysis results shown in fig. 5 also includes a display of the monitoring image. In addition, parents of middle and large children who are school students in primary schools are more concerned about how children listen and speak and answer questions in the class of schools, and the parents can also be used as main identification and analysis and AR display contents of the system.

Based on the series of processing, the behaviors of the children generally concerned by parents can be identified, analyzed and AR displayed, the content is detailed and appropriate, the primary and secondary are clear, and the method is clear at a glance, and a very convenient implementation mode is provided for the parents to quickly know the requirements of behavior expression habits of the children.

In accordance with the method provided by the present application, there is also provided an AR-based child behavior analysis device 100, as shown in fig. 6, the device 100 comprising:

the acquisition module 110 is configured to acquire monitoring video data of an activity area where an infant is located;

a segmentation module 120, configured to perform image segmentation processing on the monitored video data according to different infants appearing in the monitored video data, so as to obtain a plurality of video image data sets corresponding to the different infants, respectively;

a recognition module 130, configured to recognize one or more child behaviors occurring in a first video image dataset corresponding to a first child to determine a child behavior analysis result, where the child behavior analysis result includes a number of times and/or a time of occurrence of the one or more child behaviors;

a conversion module 140, configured to convert an infant behavior analysis result corresponding to the first video image data set into a first AR image, where the first AR image includes information of the number of times and/or time of occurrence of the one or more infant behaviors;

a sending module 150, configured to send the first AR image to a first terminal device, so that the first terminal device displays the first AR image.

In accordance with the method provided by the present application, there is also provided an AR-based child behavior analysis device 200, as shown in fig. 7, the device 200 comprising:

a sending module 210, configured to send request information to a server, where the request information is used to request a child behavior analysis result corresponding to a first child, and the request information includes user identity information;

a display module 220, configured to receive and display a first AR image sent by the server, where the first AR image includes information of times and/or time of occurrence of the one or more child behaviors; the first AR image is generated by the server based on a child behavior analysis result corresponding to a first video image dataset, the first video image dataset is from monitoring video data of an activity area where a first child is located, and the child behavior analysis result is determined by the server through recognition of one or more child behaviors of the first child in the first video image dataset.

It should be noted that, as will be clear to those skilled in the art, for convenience and brevity of description, the specific working processes of the above-described systems, modules and units may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again.

Those skilled in the art will appreciate that the embodiments described herein are illustrative of preferred embodiments and that the acts, steps, modules, or units described in connection with the embodiments disclosed herein are not required to be utilized in any particular manner. In the foregoing embodiments, the description of each embodiment in the embodiments of the present application has an emphasis, and reference may be made to relevant descriptions of other embodiments for parts that are not described in detail in a certain embodiment.

Fig. 8 is a schematic structural diagram of an electronic device 10 according to an embodiment of the present application, where the electronic device 10 includes a processor 11, a memory 12, and a communication bus for connecting the processor 11 and the memory 12, where a computer program that can be executed on the processor 11 is stored in the memory 12, and the processor 11 executes the computer program to perform or implement steps in the methods according to the embodiments of the present application. The electronic device 10 may be a server in the embodiment of the present application, and the electronic device 10 may also be a cloud server. The electronic device 10 may also be a terminal device or an AR device in the embodiment of the present application. The electronic device 10 may also be a cloud server. The electronic device 10 also includes a communication interface for receiving and transmitting data.

In some embodiments, the processor 11 may be a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), an Application Processor (AP), a modem processor, an Image Signal Processor (ISP), a controller, a video codec, a Digital Signal Processor (DSP), a baseband processor, a neural Network Processor (NPU), etc.; the processor 11 may also be other general purpose processors, application Specific Integrated Circuits (ASICs), field-Programmable Gate arrays (FPGAs) or other Programmable logic devices, discrete Gate or transistor logic devices, discrete hardware components, etc. The general purpose processor may be a microprocessor, any conventional processor, etc. The NPU can rapidly process input information by referring to a biological neural network structure and can continuously perform self-learning. Applications such as intelligent recognition, such as image recognition, face recognition, semantic recognition, voice recognition, text understanding, etc., may be implemented by the NPU electronic device 10.

In some embodiments, the memory 12 may be an internal storage unit of the electronic device 10, such as a hard disk or a memory of the electronic device 10; the memory 12 may also be an external storage device of the electronic device 10, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), etc. provided on the electronic device 10. The memory 12 may also include both internal and external memory units of the electronic device 10. The memory 12 may be used for storing an operating system, application programs, a BootLoader (BootLoader), data, and other programs, such as program code of a computer program. The memory 12 includes, but is not limited to, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), or a portable read-only memory (CD-ROM). The memory 12 is used for storing program codes executed by the electronic device 10 and transmitted data. The memory 12 may also be used to temporarily store data that has been output or is to be output.

Those skilled in the art will appreciate that fig. 8 is merely an example of the electronic device 10 and does not constitute a limitation of the electronic device 10, and that the electronic device 10 may include more or fewer components than those shown, or some of the components may be combined, or different components may be included, such as input output devices, network access devices, etc.

Fig. 9 is a schematic diagram of a software structure of a terminal device according to an embodiment of the present application. Taking the mobile phone operating system as an Android system as an example, in some embodiments, the Android system is divided into four layers, which are: the system comprises an application program layer, an application program Framework (FWK), a system layer and a hardware abstraction layer, wherein the layers communicate with each other through a software interface.

First, the application layer may include a plurality of application packages, which may be various application apps such as call, camera, video, navigation, weather, instant messenger, education, and may also be an application app based on AR technology.

Second, the application framework layer FWK provides an Application Programming Interface (API) and a programming framework for the application program of the application layer. The application framework layer may include some predefined functions, such as functions for receiving events sent by the application framework layer.

The application framework layer may include a window manager, a resource manager, and a notification manager, among others.

Wherein, the window manager is used for managing the window program. The window manager can obtain the size of the display screen, judge whether a status bar exists, lock the screen, intercept the screen and the like. Content providers are used to store and retrieve data and make it accessible to applications. The data may include video, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.

Among other things, the resource manager provides various resources, such as localized strings, icons, pictures, layout files, video files, and the like, to the application.

The notification manager enables the application program to display notification information in the status bar, can be used for conveying notification type messages, can automatically disappear after being stopped for a short time, and does not need user interaction. Such as a notification manager used to inform download completion, message alerts, etc. The notification manager may also be a notification that appears in the form of a chart or scrollbar text in a status bar at the top of the system, such as a notification of a running application in the background, or a notification that appears on the screen in the form of a dialog window. For example, prompting text information in the status bar, sounding a prompt tone, vibrating the electronic device, flashing an indicator light, etc.

In addition, the application framework layer may also include a view system that includes visual controls, such as controls to display text, controls to display pictures, and the like. The view system may be used to build applications. The display interface can be composed of one or more views, for example, a view for displaying text and a view for displaying pictures can be included on the display interface of the short message notification icon.

Third, the system layer may include a plurality of functional modules, such as a sensor service module, a physical state recognition module, a three-dimensional graphics processing library (e.g., openGLES), and so on.

The sensor service module is used for monitoring sensor data uploaded by various sensors in a hardware layer and determining the physical state of the mobile phone; the physical state recognition module is used for analyzing and recognizing user gestures, human faces and the like; the three-dimensional graphic processing library is used for realizing three-dimensional graphic drawing, image rendering, synthesis, layer processing and the like.

In addition, the system layer may also include a surface manager and a media library. The surface manager is used to manage the display subsystem and provide a fusion of the 2D and 3D layers for multiple applications. The media library supports a variety of commonly used audio, video format playback and recording, and still image files, among others.

Finally, the hardware abstraction layer is a layer between hardware and software. The hardware abstraction layer may include a display driver, a camera driver, a sensor driver, and the like, and is used for driving related hardware of the hardware layer, such as a display screen, a camera, a sensor, and the like.

Embodiments of the present application also provide a computer-readable storage medium, which stores a computer program or instructions, and the computer program or instructions implement the steps of the method designed in the above embodiments when executed.

Embodiments of the present application also provide a computer program product, which includes a computer program or instructions, and the computer program or instructions implement the steps of the method designed in the above embodiments when executed. Illustratively, the computer program product may be a software installation package.

It should be clear to a person skilled in the art that the methods, steps or functions of related modules/units described in the embodiments of the present application can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented in software, it may be implemented in whole or in part in the form of a computer program product or in the form of computer program instructions executed by a processor. Wherein the computer program product comprises at least one computer program instruction, which may consist of corresponding software modules, which may be stored in RAM, flash memory, ROM, EPROM, EEPROM, registers, hard disk, a removable hard disk, a compact disc read only memory (CD-ROM), or any other form of storage medium known in the art. The computer program instructions may be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another computer readable storage medium. For example, the computer program instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center by wired or wireless means. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device, such as a server, a data center, etc., that includes one or more of the available media. The usable medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium, or a semiconductor medium (e.g., SSD), among others.

With regard to each apparatus/product described in the above embodiments, the modules/units included in the apparatus/product may be software modules/units, or hardware modules/units, or may be part of the software modules/units and part of the hardware modules/units. For example, for an application or a device/product integrated on a chip, each of the modules/units included in the device/product may be implemented by hardware such as a circuit, or at least a part of the modules/units may be implemented by a software program running on a processor integrated inside the chip, and the remaining part of the modules/units may be implemented by hardware such as a circuit. For another example, for an application or a device/product integrated in a terminal, each module/unit included in the device/product may be implemented by using hardware such as a circuit, or at least a part of the module/unit may be implemented by using a software program and run on a processor integrated in the terminal, and the remaining part of the module/unit may be implemented by using hardware such as a circuit.

It should be understood that the scope of the present application is not limited thereto, and any person skilled in the art can easily conceive various equivalent modifications or substitutions within the technical scope of the present application, and these modifications or substitutions should be covered within the scope of the present application.

Claims

1. A children behavior analysis method based on Augmented Reality (AR) is applied to a server, and comprises the following steps:

the method comprises the steps that a server obtains monitoring video data of an activity area where a child is located;

the server carries out image segmentation processing on the monitoring video data according to different children appearing in the monitoring video data to obtain a plurality of video image data sets respectively corresponding to the different children;

the server identifies one or more child behaviors appearing in a first video image data set corresponding to a first child to determine child behavior analysis results, wherein the child behavior analysis results comprise the number of times and/or time of the one or more child behaviors appearing;

the server converts a child behavior analysis result corresponding to the first video image data set into a first AR image, wherein the first AR image comprises information of the occurrence times and/or time of the one or more child behaviors;

and the server sends the first AR image to a first terminal device so that the first terminal device displays the first AR image.

2. The method of claim 1, further comprising:

matching the child face image appearing in the monitoring video data with the child face image prestored in the server;

after a plurality of video image data sets respectively corresponding to different children are obtained, the identity ID information of the different children is bound with the corresponding video image data sets.

3. The method of claim 1, further comprising:

before the first AR image is sent to a first terminal device, the server receives request information from the first terminal device, the request information is used for requesting a child behavior analysis result corresponding to the first child, and the request information comprises user identity information;

and the server verifies the user identity information, and if the user identity information passes the verification, the first AR image is allowed to be sent to the first terminal equipment.

4. The method of claim 1, wherein the first AR picture further comprises image and/or video information of the one or more child behaviors.

5. The method of claim 1, wherein the child behavior comprises at least one of: work and rest condition, dining condition, game interaction state, drinking frequency, toilet frequency, sleep condition, dining condition, dressing condition, class listening state, question answering condition, and task doing state.

6. A child behavior analysis method based on Augmented Reality (AR) is applied to a terminal device, and comprises the following steps:

the method comprises the steps that terminal equipment sends request information to a server, wherein the request information is used for requesting a child behavior analysis result corresponding to a first child, and the request information comprises user identity information;

the terminal equipment receives and displays a first AR image sent by the server, wherein the first AR image comprises the information of the occurrence times and/or time of one or more child behaviors of the first child; the first AR image is generated by the server based on a child behavior analysis result corresponding to a first video image data set, the first video image data set is from monitoring video data of an activity area where the first child is located, and the child behavior analysis result is determined by identifying one or more child behaviors of the first child in the first video image data set by the server.

7. The method of claim 6, wherein the first AR image further comprises image and/or video information of the one or more child behaviors.

8. The method of claim 6, wherein the child behavior comprises at least one of: work and rest condition, dining condition, game interaction state, drinking frequency, toilet frequency, sleep condition, dining condition, dressing condition, class listening state, question answering condition, and task doing state.

9. An Augmented Reality (AR) -based child behavior analysis device applied to a server, the device comprising:

and the sending module is used for sending the first AR image to first terminal equipment so as to enable the first terminal equipment to display the first AR image.

10. The utility model provides a children's action analytical equipment based on augmented reality AR which characterized in that is applied to terminal equipment, the device includes:

11. A server-side device, comprising: a processor and a memory storing computer program instructions; the processor, when executing the computer program instructions, implements the method of any of claims 1-5.

12. A terminal device, comprising: a processor and a memory storing computer program instructions; the processor, when executing the computer program instructions, implements the method of any of claims 6-8.

13. A computer-readable storage medium, having stored thereon computer program instructions, which when executed by a processor, implement the method of any one of claims 1-8.

14. A computer program product comprising computer program instructions which, when executed by a processor, implement the method according to any one of claims 1-8.