KR20240081641A - Method and system for generating growth diary using character recognition - Google Patents

Abstract

A method and system for generating a growth diary using person recognition are disclosed. A growth diary generation method performed by a diary generation system according to an embodiment includes the steps of recognizing person information and pose information by collecting image information of a person captured through a camera included in an electronic device; Classifying an image frame including the person information and pose information from the collected image information of the person; Generating a growth diary of the person through mapping change information to an image frame including the classified person information and pose information using a deep learning-based synthetic model; and visualizing changes in the person based on the created growth diary of the person.

Description

Method and system for generating growth diary using character recognition {METHOD AND SYSTEM FOR GENERATING GROWTH DIARY USING CHARACTER RECOGNITION}

The explanation below is about techniques for creating a growth diary.

As the use of portable devices increases, services and content are evolving to suit the smart environment, and device users' service and content usage patterns are also changing to match this trend.

Industries and services related to photography are also changing to suit the smart environment, and in particular, SNS (Social Network Service) services based on sharing 'photos' containing personal status information are rapidly spreading. I'm going. For example, by uploading photos taken with a smart phone to the Internet, children's photos or videos can be created as a growth album, stored online, and opened whenever they want. Furthermore, recreating photos with past friends or childhood photos with parents as an adult is still a hot topic. In this way, if a growth diary service that contains the child's growth process from childhood is provided, it is expected that the entire family will be able to recall happy memories.

For reference, in Republic of Korea Patent No. 10-0900322 (2009.06.02 notice, ‘Growth album service method using avatar’), 3D data measuring the subject's body size using a 3D scanner is periodically provided from the user. A technology has been disclosed that creates a 3D avatar identical to the subject, stores it, and provides the user with a picture of the subject changing as he or she grows up in the form of a growth album.

A method and system for storing image frames containing person information and pose information recognized from the person's image information captured through a camera of an electronic device for each person, and generating a growth diary of the person's moments within the image frames stored for each person. can be provided.

A method and system for visualizing a growth diary can be provided by synthesizing the growth process or aging process using image information and converting it into a video format or continuous image format.

A growth diary generation method performed by a diary generation system includes the steps of recognizing person information and pose information by collecting image information of a person captured through a camera included in an electronic device; Classifying an image frame including the recognized person information and pose information from the collected image information of the person; Generating a growth diary of the person through mapping change information to an image frame including the classified person information and pose information using a deep learning-based synthetic model; and visualizing changes in the person based on the created growth diary of the person.

The step of recognizing the person information and pose information may include recognizing the person information and pose information using image information of the person captured from each camera of different electronic devices through a cloud service.

The step of recognizing the person information and pose information includes recognizing person information including face information of the person from the collected image information of the user using a face recognition model, and using feature points of the recognized person information. It may include detecting facial expression and pose information of the recognized person.

The person information is provided in real time by face recognition in the Raspberry Pi as the camera and the Raspberry Pi on which WebOS is installed are linked through the WebOS API, and the pose information is provided as person information through face recognition in the Raspberry Pi. As is recognized, the pose of the person is recognized using OpenPose, and as the recognized pose of the person is judged to be the same pose as the pose recognized before the preset standard, the image information including the pose information is divided into frames. and can be saved as an image.

In the classifying step, an image frame including the recognized person information and pose information is classified from the collected image information of the person using preset person information and pose information to be collected, and the classified person information and A step of automatically classifying image frames containing pose information by person may be included.

The classifying step may include extracting physical characteristics, including height and weight, of the recognized person from an image frame including the recognized person information and pose information.

The classifying step may include selecting an image frame most similar to the recognized person information or pose information through comparison of the image frame with previous image frames of the image frame.

The step of generating the growth diary of the person includes the growth of the person based on video or continuous images by synthesizing change information including the growth process or aging process for the image frame containing the face information and pose information of the classified person. It may include the step of creating a diary.

The visualizing step includes transmitting changes in the visualized person based on the generated growth diary of the person to the electronic device or another electronic device other than the electronic device, and the electronic device or the electronic device. Electronic devices other than the above can be provided with changes in the visualized person based on the created growth diary of the person as they are connected to the diary service through input of each user's ID and password.

The diary creation system includes: a recognition unit that recognizes person information and pose information by collecting image information of a person captured through a camera included in an electronic device; a classification unit that classifies image frames including the person information and pose information from the collected image information of the person; A generation unit that generates a growth diary of the person through mapping change information to an image frame containing the classified person information and pose information using a deep learning-based synthesis model; and a visualization unit that visualizes changes in the person based on the created growth diary of the person.

Even if you are far away, you can share the past and present of your loved ones at any time through the diary service in real time, and record your daily life in various places in your home.

Figure 1 is a diagram for explaining the general operation of a diary creation system, according to one embodiment.
Figure 2 is a diagram for explaining the specific operation of a diary creation system, according to one embodiment.
Figure 3 is a block diagram for explaining the configuration of a diary creation system, according to one embodiment.
Figure 4 is a flowchart for explaining a method for generating a growth diary, according to one embodiment.
Figure 5 is an example for explaining an operation of recognizing pose information of a person, according to an embodiment.
Figure 6 is an example for explaining an operation of visualizing changes in a person based on the person's growth diary, according to an embodiment.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings.

Figure 1 is a diagram for explaining the general operation of a diary creation system, according to one embodiment.

The diary creation system 100 recognizes person information and pose information in the image information of the person captured through the camera of the electronic device 110, and creates an image frame containing the person information and pose information recognized from the image information of the person. You can save and visualize the person's moments within the saved image frame by time to provide the person's growth or aging process through a diary service. At this time, the diary creation system 100 sets the pose information for the person, so that when the person takes a pose determined from the recognized person information in the image information of the person, several seconds of video can be divided into image frames and stored. Additionally, the diary creation system 100 may provide person information by analyzing the person's physical characteristics, such as height and weight, from image information.

For example, the diary creation system 100 learns image information of a person captured through a camera of an electronic device, synthesizes the growth process or aging process, and converts it into a video or continuous image form to provide a growth diary. . Alternatively, a growth diary can be created using a deep learning model installed directly on electronic devices such as smart phones and smart TVs. Alternatively, the diary creation system 100 can create a growth diary through a cloud AI engine for image information uploaded as the cloud service is accessed from different electronic devices such as smart phones and smart TVs, and the produced growth diary It can be transmitted to each electronic device. For example, different electronic devices can access the electronic device manufacturer's cloud or other third party's cloud based on login information by entering an ID and password. Image information of a person captured from each camera of different electronic devices may be transmitted to a cloud service. The diary creation system 100 recognizes person information and pose information using image information of a person captured from each camera of different electronic devices through a cloud service, automatically classifies the image information for each person, and automatically classifies the image information for each person for a period selected by the user or automatically. You can create a growth diary by combining character information during the recommended period.

Referring to Figure 2, it is a diagram to explain the specific operation of the diary creation system. The diary creation system 100 may collect image information of a person photographed 210 through an electronic device including a camera. For example, electronic devices may include smart phones, mobile phones, navigation devices, computers, laptops, digital broadcasting terminals, PDAs (Personal Digital Assistants), PMPs (Portable Multimedia Players), tablet PCs, smart TVs, etc. there is. These electronic devices may have a built-in camera or may be configured with a camera attached through a separate installation. In the embodiment, collection of image information of a person captured through a smart TV, smart phone, or camera will be described as an example. For example, when a child's video is captured, the child's image and video can be manually captured using the camera of the parent's smartphone, and the child's image and video can be captured manually using the camera of the smart TV in the living room. Capture can be taken automatically. Alternatively, an image of the child may be captured as the motion recognition service is activated within a time range set by the user.

The diary creation system 100 can collect image information about a person by linking a camera of an electronic device and RaspberryPi 4. The diary creation system 100 can recognize a person's face information in real time from collected image information. At this time, if the person information is already set, the diary creation system 100 may start recognizing the person's pose information. If the person assumes preset pose information, the diary creation system 100 can store the image frame by dividing the image information of the person into frames for a certain period of time along with the phrase that the person has been recognized through the display. At this time, since video information must be collected in real time and person information and pose information within the video information must be recognized, video information can be collected from a Raspberry Pi directly connected to the webcam rather than a server computer. Afterwards, the recognized image information can be transmitted to a server computer (diary creation system) to allow image analysis and processing.

The diary creation system 100 may recognize person information and pose information from the collected image information of the person (220). The diary creation system 100 can recognize a person's face information using a deep learning-based face recognition model for person recognition. First, the diary creation system 100 can set the person information and pose information to be collected. The diary creation system 100 provides a user interface for setting character information and pose information, and can receive character information and pose information set from the user through the provided user interface. The diary creation system 100 may store an image frame including the person information and pose information set as the person information and pose information are captured from the collected image information of the person.

For example, the diary creation system 100 may recognize a partial or entire area of a person from image information of the person. For example, when the diary creation system 100 wants to generate a growth diary about the growth process or aging process of a person's facial area, it can perform face recognition by extracting only the facial area from image information. Alternatively, when the diary creation system 100 wants to generate a growth diary about the growth process or aging process of the entire area of the person, the diary generation system 100 may recognize facial information and pose information from image information including both the face and body. . In addition, because the pixel information or size information of image information captured using each electronic device may be different, the diary creation system 100 changes the image information captured using each electronic device to a preset standard. Afterwards, face recognition and recognition of pose information can be performed. Additionally, the diary creation system 100 may distinguish background information and object information included in image information and then perform face recognition and pose information recognition using an image frame containing only object information.

The diary creation system 100 may receive face information and pose information recognized through a camera of an electronic device. After linking the Raspberry Pi 4 with WebOS installed and the camera using the WebOS API, face recognition can be performed in real time on the Raspberry Pi. For example, face recognition can be performed using a deep learning network in electronic devices such as smart TVs and smart phones. First, after learning a deep learning-based model for the face of the person to be recognized, face recognition is performed in real time, and when the face of the previously recognized person is recognized, pose recognition is performed. Additionally, pose recognition can also be performed in real time on Raspberry Pi. Once person information is recognized through face recognition, the person's pose information is recognized using OpenPose. If you assume the same pose as the previously recognized pose information, the 3-second video is cut into frames and saved as image frames at 2 fps (2 images per second, a total of 6 images). The diary creation system 100 can analyze video information received from the Raspberry Pi frame by frame and classify albums.

For example, the diary creation system 100 may perform an album classification 230 task that stores image frames for each person. The diary creation system 100 may classify image frames containing character information and pose information of a person recognized from the collected image information of the person. The diary creation system 100 can measure physical characteristics such as height and weight of the person in the stored image frame. The diary creation system 100 measures the physical characteristics of a person using a model for measuring physical characteristics using stored image frames, and includes the physical characteristics of the person, the frame in which the person's physical characteristic is measured, and the frame in which the frame was photographed. Time information can be stored. For example, the diary creation system 100 may measure the height of a person using a model for measuring the height of the person in an image frame, and store the frame in which the height was measured and time information of the frame. The diary creation system 100 may classify and store image frames that are similar to the set pose information among the stored image frames. The diary creation system 100 can process classified image frames so that they can be visualized through a user interface. The diary creation system 100 may select (240) an image frame in which a person is clearly photographed and is similar to the set pose information among the classified image frames for synthesis of the growth process or aging process.

More specifically, the diary creation system 100 can extract character information, pose information, and key information of a person through data processing 250 using OpenCV and OpenPose. OpenCV stands for Open Source Computer Vision, and is a machine learning-based open source library that can be used for image processing. It supports various languages such as C++, JAVA, and Phthon, and OpenCV can be used using Python. OpenPose is also an open source library that extracts important parts of a person's (user's) body as feature points and can detect facial expression information or pose information using the extracted feature points (Figure 5). For example, feature points may be extracted for a person's joints, and pose information may be detected using the extracted feature points.

At this time, if OpenCV and OpenPose take a long time to recognize faces and poses, it may be difficult to properly save image frames. In order to overcome this, the diary generation system 100 learns as much as possible to increase recognition time, or continues to temporarily store the face after face recognition is completed. When the pose is recognized, the remaining stored portion is deleted except for the subsequent images. If pose recognition does not occur for a certain period of time, you can delete all temporarily stored frames and wait.

The diary creation system 100 can establish an environment for creating a person's growth diary, connect various functions within the app, and store various data. The diary creation system 100 may implement a user interface that allows selection of functions that visualize analysis results of stored image frames and changes over time. The diary creation system 100 can visualize a person's growth diary. The diary creation system 100 may provide a user interface related to the diary service. The diary creation system 100 may perform the role of reprocessing and storing the user's body data and image data in the diary service database.

The diary creation system 100 can estimate and store physical characteristics such as height and weight of a person through image analysis, and recognize and store parts that are significantly different compared to the previous image. At this time, guessing the person's physical characteristics can also be processed using OpenCV. The diary creation system 100 selects which image frame is most similar to the preset pose among the image frames using the PCK metric (a metric using the distance between the pose feature point and the actual feature point), and uses features such as the guessed key. It can be stored in the database along with . The diary creation system 100 stores unstructured data analyzed and processed from the person's image information in a NoSQL-type database, and stores structured data such as the person's physical characteristics analyzed through deep learning in a relational database. You can use a relational database. At this time, excessive time may be consumed in the process of referencing an external database and decompressing the image data during the operation of loading the image frame. In this case, the database response time can be reduced by using the local DB supported by WebOS itself or by changing the database storage space from S3 to EBS (Elastic Block Store) and making it internal storage.

The diary creation system 100 may generate a person's growth diary through mapping change information to an image frame including face information and pose information of a classified person. The diary creation system 100 can selectively synthesize change information for image frames including face information and pose information of classified people (240). For example, the diary creation system 100 combines various information, such as collected change information related to the physical characteristics of the person and public data on physical development of an age group similar to the person, to create a person's growth diary about the person's growth information or aging information. can be created. Alternatively, the diary creation system 100 may generate a person's growth diary by mapping change information to an image frame containing face information and pose information of a classified person using a deep learning-based synthetic model (260) ). At this time, the deep learning-based synthesis model may be constructed to generate the person's growth diary by synthesizing the person's growth information or aging information, and GAN may be used. The diary creation system 100 uses flipbook animation and slideshow format with a certain number of frames in the chronological order of a series of image frames to effectively visualize changes over time in the person with information accumulated for the continuous use of the person. The same effect can be produced. The diary creation system 100 can produce a growth diary (270) by converting it into a video format or a continuous image format. At this time, when the diary creation system 100 needs to compress the person's growth diary, it can preferentially perform non-lossy compression. If lossy compression is unavoidably necessary, the diary generation system 100 may perform lossy compression by preferentially losing the color of the image rather than losing pixels.

The diary creation system 100 may visualize changes in the character based on the created growth diary of the character. For example, changes in a person based on a growth diary can be visualized through WebOS of an electronic device. The application (application for diary service) to be installed on webOS devices can provide users with a clear and effective UI/UX by utilizing libraries such as sandstone library and billboard.js for graphs based on the Enact framework. At this time, the camera and scene recognition function is registered as an external service so that the function that starts video recording when a specific person is positioned in front of the camera with a set pose can operate even when the application is not running in the foreground, and the service is connected to the heartbeat of webOS. You can subscribe to the service. Additionally, the diary creation system 100 can use PostgreSQL, which supports various variable types so that various physical data other than height can be added.

According to one embodiment, it is possible to provide a universal and convenient diary service that can store memories of precious people that can be used on any electronic device with a camera because it is equipped with WebOS. In addition, the nature of the home life screen can be maximized in that daily life can be recorded in various places in the home. Through this, you can capture memories of your loved ones in front of electronic devices with WebOS anytime, anywhere.

It is expected that integrated management will be possible by building a network between devices that can utilize the developed services just by installing webOS and cameras in electronic products. Through this, you can share the past and present of your loved ones at any time through the diary service in real time, even if you are far away.

FIG. 3 is a block diagram for explaining the configuration of a diary generation system in one embodiment, and FIG. 4 is a flowchart for explaining a growth diary generation method in one embodiment.

The processor of the diary creation system 100 may include a recognition unit 310, a classification unit 320, a creation unit 330, and a visualization unit 340. These processor components may be expressions of different functions performed by the processor according to control instructions provided by program codes stored in the diary generation system. The processor and its components may control the diary generation system to perform steps 410 to 440 included in the growth diary generation method of FIG. 4. At this time, the processor and its components may be implemented to execute instructions according to the code of an operating system included in the memory and the code of at least one program.

The processor may load the program code stored in the file of the program for the growth diary creation method into memory. For example, when a program is executed in a diary creation system, the processor can control the diary creation system to load program code from the program file into memory under the control of the operating system. At this time, each of the recognition unit 210, classification unit 220, generation unit 230, and visualization unit 240 executes the command of the corresponding portion of the program code loaded into the memory to perform subsequent steps (310 to 340). These may be different functional expressions of the processor for executing.

In step 310, the recognition unit 210 may recognize person information and pose information by collecting image information of the person captured through a camera included in the electronic device. The recognition unit 210 may recognize person information and pose information using image information of a person captured from each camera of different electronic devices through a cloud service. For example, the recognition unit 210 sends first image information of a person captured through a first electronic device including a camera and second image information of a person captured through a second electronic device including a camera to a cloud service. It can be collected through . The recognition unit 210 recognizes person information including the face information of the person from the user's image information collected using a face recognition model, and uses the characteristic points of the recognized person information to determine the facial expression and pose information of the recognized person. It can be detected.

In step 320, the classification unit 220 may classify an image frame including person information and pose information of a person recognized from the collected image information of the person. The classification unit 220 classifies image frames containing person information and pose information recognized from the image information of the person collected using preset person information and pose information to be collected, and classifies the classified person information and pose information. Containing image frames can be automatically classified by person. The classifier 220 may extract physical characteristics, including height and weight, of the recognized person from an image frame including recognized person information and pose information. The classification unit 220 may select an image frame that is most similar to the recognized person information or pose information through comparison of the image frame with previous image frames.

In step 330, the generation unit 230 may generate a growth diary of a person through mapping change information on an image frame containing face information and pose information of a classified person using a deep learning-based synthesis model. . The generator 230 may generate a video or continuous image-based growth diary of a person by synthesizing change information including a growth process or an aging process with an image frame including face information and pose information of a classified person. .

In step 340, the visualization unit 240 may visualize changes in the character based on the created growth diary of the character. The visualization unit 240 may transmit changes in the visualized person based on the generated growth diary of the person to an electronic device or another electronic device other than the electronic device. Alternatively, the visualization unit 240 may provide changes to the visualized person through logging in to the diary service from a desired electronic device. Additionally, the visualization unit 240 can share a person's growth process and aging process anytime, anywhere on each electronic device through a diary service.

Figure 6 is an example for explaining an operation of visualizing changes in a person based on the person's growth diary, according to an embodiment.

The diary creation system 100 may visualize changes in the person based on the person's growth diary, including the growth process or aging process, through the diary service screen 600. The diary creation system 100 may visualize changes in the person according to the growth diary according to the time direction (eg, forward time, reverse time) set by the user through the diary service screen 600. At this time, changes from the person's childhood to old age may be played in the form of a slide or video on the diary service screen 600. Alternatively, the changes from old age to childhood may be played in the form of slides or videos.

The device described above may be implemented with hardware components, software components, and/or a combination of hardware components and software components. For example, devices and components described in embodiments may include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), etc. , may be implemented using one or more general-purpose or special-purpose computers, such as a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. A processing device may execute an operating system (OS) and one or more software applications that run on the operating system. Additionally, a processing device may access, store, manipulate, process, and generate data in response to the execution of software. For ease of understanding, a single processing device may be described as being used; however, those skilled in the art will understand that a processing device includes multiple processing elements and/or multiple types of processing elements. It can be seen that it may include. For example, a processing device may include a plurality of processors or one processor and one controller. Additionally, other processing configurations, such as parallel processors, are possible.

Software may include a computer program, code, instructions, or a combination of one or more of these, which may configure a processing unit to operate as desired, or may be processed independently or collectively. You can command the device. Software and/or data may be used on any type of machine, component, physical device, virtual equipment, computer storage medium or device to be interpreted by or to provide instructions or data to a processing device. It can be embodied in . Software may be distributed over networked computer systems and stored or executed in a distributed manner. Software and data may be stored on one or more computer-readable recording media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc., singly or in combination. Program instructions recorded on the medium may be specially designed and configured for the embodiment or may be known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. -Includes optical media (magneto-optical media) and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, flash memory, etc. Examples of program instructions include machine language code, such as that produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc.

As described above, although the embodiments have been described with limited examples and drawings, various modifications and variations can be made by those skilled in the art from the above description. For example, the described techniques are performed in a different order than the described method, and/or components of the described system, structure, device, circuit, etc. are combined or combined in a different form than the described method, or other components are used. Alternatively, appropriate results may be achieved even if substituted or substituted by an equivalent.

Therefore, other implementations, other embodiments, and equivalents of the claims also fall within the scope of the claims described below.

Claims (10)

In the growth diary generation method performed by the diary generation system,
Recognizing person information and pose information by collecting image information of a person captured through a camera included in an electronic device;
Classifying an image frame including the recognized person information and pose information from the collected image information of the person;
Generating a growth diary of the person through mapping change information to an image frame including the classified person information and pose information using a deep learning-based synthetic model; and
Step of visualizing changes in the character based on the created character’s growth diary
How to create a growth diary including. According to paragraph 1,
The step of recognizing the person information and pose information is,
Recognizing person information and pose information using image information of a person captured from each camera of different electronic devices through a cloud service
How to create a growth diary including. According to paragraph 2,
The step of recognizing the person information and pose information is,
Recognizing person information including face information of the person from the collected image information of the user using a face recognition model, and detecting expression and pose information of the recognized person using feature points of the recognized person information.
How to create a growth diary including. According to paragraph 3,
As the personal information is linked to the Raspberry Pi with WebOS installed and the camera through the WebOS API, facial recognition is performed in real time on the Raspberry Pi.
As the pose information is recognized through face recognition in the Raspberry Pi, the pose of the person is recognized using OpenPose, and the pose of the recognized person is the same as the pose recognized before the preset standard. As determined, the image information including the pose information is divided into frames and stored as images.
A method for creating a growth diary, characterized in that: According to paragraph 1,
The classification step is,
Classify image frames containing the recognized person information and pose information from the collected image information of the person using preset person information and pose information to be collected, and image frames containing the classified person information and pose information. Steps to automatically classify frames by person
How to create a growth diary including. According to clause 5,
The classification step is,
Extracting physical characteristics including height and weight of the recognized person from an image frame including the recognized person information and pose information.
How to create a growth diary including. According to clause 6,
The classification step is,
Selecting an image frame most similar to the recognized person information or pose information by comparing the image frame with previous image frames of the image frame.
How to create a growth diary including. According to paragraph 1,
The step of creating a growth diary of the person is,
Creating a growth diary of the person based on video or continuous images by synthesizing change information including the growth process or aging process with the image frame containing the face information and pose information of the classified person.
How to create a growth diary including. According to paragraph 1,
The visualization step is,
Transmitting changes in the person visualized based on the created growth diary of the person to the electronic device or another electronic device other than the electronic device
Including,
The electronic device or other electronic device other than the electronic device is connected to the diary service through input of a user ID and password, and is provided with changes in the person visualized based on the created growth diary of the person.
A method for creating a growth diary, characterized in that: In the diary creation system,
A recognition unit that recognizes person information and pose information by collecting image information of the person captured through a camera included in the electronic device;
a classification unit that classifies image frames including the person information and pose information from the collected image information of the person;
A generation unit that generates a growth diary of the person through mapping change information to an image frame containing the classified person information and pose information using a deep learning-based synthesis model; and
Visualization unit that visualizes changes in the character based on the character's growth diary created above
A growth diary creation system that includes.

Images