KR20110125431A - Method and apparatus for generating life log in portable termianl - Google Patents

Abstract

PURPOSE: A method and device for generating a life log in portable terminal are provided to generate a life log through the situation of a user. CONSTITUTION: A sound environment recognizing unit(212) recognizes a sound environment corresponding to a sound signal inputted from a microphone among the preset sound environment. An image environment recognizing unit(214) recognizes an image environment corresponding to an image signal inputted from a camera among the preset image environments. A situation determining unit(220) determines a user situation corresponding to a sound environment recognition result and an image environment recognition result among the preset situation models. The situation determining unit records the determined user situation as a life log.

Description

METHOD AND APPARATUS FOR GENERATING LIFE LOG IN PORTABLE TERMIANL}

The present invention relates to a method and apparatus for generating a lifelog in a portable terminal, and more particularly, to a method and apparatus for determining a situation of a user using a microphone and a camera and thereby generating a lifelog.

Recently, with the development of portable terminals and increased utilization, life log services have been provided to record information that can be obtained in a user's daily life and to search as needed. That is, the life log service acquires and records information in a user's daily life from various sensors such as a camera, a global positioning system (GPS), an illuminance sensor, a geomagnetic sensor, and a thermometer, and then uses the recorded information later. To help.

The life log service provided in the related art collects various user's surrounding situation data such as time, location, brightness and user's gaze by mounting cameras and wearable devices that match the user's eyes on the user's body, and then transmits the data to the web server. By saving, the user can check his or her life log through a web service.

As described above, the conventional lifelog service requires mounting a large number of sensors on the user's body in order to obtain various information of the user. However, the mounting of a large number of sensors on the user's body as well as causing unnaturalness in the user's daily life, as well as the user's rejection has a disadvantage that it is difficult to be commercially practical. In addition, in the conventional lifelog technique, the collected information, for example, time, location, brightness, and user's gaze image is transmitted and stored in a simple form on a web server, and thus a comprehensive situation such as a user's behavior or an environment in which the user is placed. There is a disadvantage that does not represent.

The present invention has been made to solve the above problems, and an object of the present invention is to provide a method and apparatus for generating a lifelog in a portable terminal.

Another object of the present invention is to provide a method and apparatus for determining a situation of a user by using an image and a sound input from a microphone and a camera in a portable terminal, and generating a life log through the same.

It is still another object of the present invention to provide a scheduling method and apparatus for processing and recognizing an image and sound input from a microphone and a camera in a portable terminal.

According to a first aspect of the present invention for achieving the above objects, the method for generating a lifelog in a portable terminal includes the steps of recognizing a sound environment corresponding to a sound signal input from a microphone among preset sound environments; A process of recognizing an image environment corresponding to an image signal input from a camera among image environments, a process of determining a user situation corresponding to the acoustic environment recognition result and the image environment recognition result among preset context models; And recording a user situation in a life log.

According to a second aspect of the present invention for achieving the above object, the life log generation apparatus of a portable terminal includes a microphone for receiving an audio signal, a camera for receiving an image signal, and the microphone from among the preset acoustic environments. An acoustic environment recognition unit for recognizing an acoustic environment corresponding to an input audio signal, an image environment recognition unit for recognizing an image environment corresponding to an image signal input from the camera among preset image environments, and preset situation models And a situation determination unit for determining a user situation corresponding to the acoustic environment recognition result and the image environment recognition result, and recording the determined user situation in a life log.

The present invention determines the user's situation using the image and sound input from the microphone and the camera in the portable terminal, and generates a life log through this, so that the user does not purchase or possess a separate device, the life log in daily life It can be created, the life log only has the effect of knowing the user's life patterns. In addition, by processing and recognizing the image and sound in accordance with the scheduling method proposed in the present invention, it is possible to reduce the time spent processing and recognizing the image and sound.

1 is a diagram illustrating an example of determining a user situation according to a surrounding environment of a user according to an embodiment of the present invention;
2 is a block diagram of a portable terminal according to an embodiment of the present invention;
3 is a diagram illustrating a video signal analysis procedure in a portable terminal according to an embodiment of the present invention;
4 is a diagram illustrating an example of limiting an analysis area according to an image signal analysis procedure in a portable terminal according to an embodiment of the present invention;
5 is a diagram illustrating a scheduling technique for processing video and audio in a portable terminal according to an embodiment of the present invention;
6 is a diagram illustrating a situation statistical model of a portable terminal according to an embodiment of the present invention;
7 is a diagram illustrating an example of determining a user situation through a video, audio, and situation statistics model in a portable terminal according to an embodiment of the present invention;
8 is a diagram illustrating a lifelog storage method in a portable terminal according to an embodiment of the present invention;
9 is a diagram illustrating a procedure of generating and storing a lifelog in a portable terminal according to an embodiment of the present invention;

Hereinafter, exemplary embodiments of the present invention will be described with reference to the accompanying drawings. In describing the present invention, when it is determined that a detailed description of a related known function or configuration may unnecessarily obscure the subject matter of the present invention, the detailed description thereof will be omitted.

Hereinafter, a description will be given of a technology of determining a user's situation using a video and a sound input from a microphone and a camera in a portable terminal, and generating a lifelog.

The portable terminal according to the present invention recognizes the surrounding environment through the sound input from the microphone and the image input from the camera, and determines the user's situation based on the recognized result. For example, as shown in FIG. 1, the portable terminal analyzes the sound input from the microphone to recognize the car sound 100, the music sound 102, the roaring sound 104, and from the camera. After analyzing the input image to recognize that the person is walking image 106, the car sound 100, the music sound 102, the roaring sound 104 and the walking image 106 by combining the user's situation is It may be determined that the situation is during the street movement 110.

2 is a block diagram of a portable terminal according to an exemplary embodiment of the present invention. As shown in FIG. 2, the portable terminal includes an input unit 200, a recognition result accumulator 210, a situation determination unit 220, a situation statistical model 230, and a storage unit 240. The input unit 200 includes a microphone 202 and a camera 204, and the recognition result accumulator 210 includes an acoustic environment recognizer 212 and an image environment recognizer 214.

First, the input unit 200 receives an audio signal and an image signal necessary for determining a user situation and provides the received result to the accumulator 210. That is, the input unit 200 receives various sound signals around the user through the microphone 202, and the camera 204 receives an image signal around the user through a camera sensor. The input unit 200 is activated every predetermined period, receives the sound signal and the image signal for a predetermined time and provides the recognition result accumulator 210, and when the predetermined time expires, deactivates until the next cycle. Return to

As a result of the recognition, the accumulator 210 analyzes the sound signal and the image signal input during the accumulation time from the input unit 200 for each preset short period (eg, within 3 seconds) to recognize the sound environment and the image environment. The cumulative result accumulated during the cumulative time is accumulated and provided to the situation determiner 220. In particular, the recognition result accumulator 210 recognizes an acoustic environment by analyzing the sound signal through the acoustic environment recognizer 212, and analyzes an image signal through the image environment recognizer 214. Recognize.

The acoustic environment recognizer 212 searches for and recognizes an acoustic environment most similar to the input acoustic signal with reference to Gaussian Mixture Models for the previously stored acoustic environment. Here, the Gaussian mixed models for the acoustic environment may represent energy characteristics for each acoustic environment, and the acoustic environment may include, for example, a loud noise, a car sound, a music sound, a noise in a closed space such as a bag, It can be classified into office noise, human voice, subway noise, quiet public place noise, water flowing sound and loud sound (or loud noise). That is, the acoustic environment recognition unit 212 extracts a MFCC (Mel Frequency Cepstral Coefficient), which is an energy characteristic of an acoustic signal input from the microphone 202, and then extracts the extracted energy characteristic from among acoustic environments represented by the Gaussian mixture model. Search for an acoustic environment with a maximum likelihood value for.

In this case, when the sound signal includes a sound greater than or equal to a threshold value, it is difficult to distinguish other voices, noises, or sounds, and thus, the acoustic environment recognition unit 212 determines whether the sound signal includes a loud sound. First, when the loud sound is included, the sound signal may be divided into a loud sound environment, and when the loud sound is not included, an energy feature may be extracted from the sound signal to search for the sound environment. will be.

The image environment recognizer 214 analyzes the input image signal according to a preset criterion to recognize the image environment. That is, the image environment recognition unit 214 recognizes the image environment by checking the low light, the movement, the presence of the face, the indoor and outdoor for the frame of the input image signal. The image environment recognizing unit 214 determines the low light by measuring the intensity (or brightness) of the light for the corresponding image frame, determines whether it is moving by using a motion vector, detects the skin color The presence of a face is determined, and whether the face is indoors or outdoors is determined by using color and texture features. For example, the image environment recognizer 214 may measure a gray scale representing brightness of an input image frame and compare the average gray scale with a threshold to determine whether the image frame is low light. Can be. In addition, the image environment recognizer 214 divides an input image frame into a plurality of subblocks, and then measures an average gray scale for each subblock, and according to the number of subblocks having a gray scale less than or equal to a threshold. It may be determined whether the image frame is low light. In addition, the image environment recognizer 214 may determine whether the image frame is moved by extracting a motion vector from the plurality of image frames using a 'Lucas-Kanade algorithm' or 'Pyramid algorithm'. have. In addition, the image environment recognition unit 214 detects an area having a skin color in the image frame, and then the detected area is a preset maximum and minimum size condition, a horizontal-to-vertical ratio condition, and a ratio of the detection area to other areas. It may be determined whether the face is included in the image frame by checking whether the condition is satisfied. In addition, the image environment recognizer 214 divides the image frame into a plurality of subblocks, extracts the color and texture features of each subblock, and compares the image frame with a preset threshold value. It may be determined whether the image corresponds to the outdoor image.

In particular, the image environment recognizing unit 214 may analyze the image frame in the order as shown in FIG. 3 to process the input image signal in real time. First, the image environment recognizing unit 214 determines whether the image frame is low light 301, and if the image frame is low light, the procedure for determining whether the movement, the presence of the face, indoors and outdoors are omitted. This is because, when the image frame is low light, information on image recognition is insufficient. On the other hand, when the image frame is not low light, the image environment recognizer 214 determines whether there is a face 303, and if there is a face, determines whether it is indoor or outdoor 305. Here, the image environment recognition unit 214 may omit the procedure of determining whether to move if a face exists in the image frame. This is because, when a face is detected in the image frame, information for detecting movement may be insufficient. On the other hand, if a face does not exist in the image frame, the image environment recognizer 214 determines whether to move (307) and whether it is indoor or outdoor (305). When analyzing the image frame as described above, it is possible to process the input image in real time by improving the image processing speed by eliminating unnecessary processes.

In addition, the image environment recognition unit 214 may analyze the corresponding image frame by limiting an analysis region as shown in FIG. 4 to process the input image signal in real time. That is, the image environment recognition unit 214 determines whether the image frame is low light for the entire region of the image frame, and checks whether there is a face. When the face exists in the image, the remaining area except for the area 410 where the face exists may be limited to the analysis area to determine whether the indoor / outdoor and the movement. As such, when the analysis region of the image frame is limited, the amount of computation required for image processing may be reduced to process the input image in real time.

Here, as illustrated in FIG. 5, the acoustic environment recognizer 212 and the image environment recognizer 214 may process the sound signal and the image signal at different times. In this case, the sound environment recognition unit 212 and the image environment recognition unit 214 outputs the sound signal and the image signal simultaneously at a point in time after the input of the sound signal and the image signal starts, the sound The operation is performed in consideration of the time and data necessary for determining the environment, and in consideration of the time and data necessary for determining the video environment. In FIG. 5, the sound environment recognition unit 212 and the image environment recognition unit 214 output the sound environment determination result and the image determination result at the same time three seconds after the sound signal and the image signal are input. ) Shows the timing of operation. As shown in FIG. 5, the acoustic environment recognizer 212 recognizes an acoustic environment by processing an acoustic signal at an acoustic time point that is periodically repeated for three seconds, and the image environment recognizer 214 recognizes the acoustic environment. The video signal is processed at a video point that is periodically repeated for a second. Here, the sound viewpoint and the image viewpoint do not overlap. The image environment recognizing unit 214 may first determine whether low light is in order to determine the image environment for the three seconds, check whether there is a movement, and then determine whether there is a face and whether it is indoor or outdoor. Here, when the image frame is low light, it is not necessary to perform an operation for determining whether to move the rest, whether there is a face, and whether it is indoor or outdoor. Therefore, it is important to first determine whether the low light is low.

The situation statistical model 230 represents a statistical model for sound and image signals for a plurality of predetermined situations. For example, the situation statistical model is divided into offices, restaurants / cafes, games, shopping malls, cars / buses, classrooms, streets, large marts, subways, outdoor situations, and homes based on living / moving spaces in daily life. Can be. The statistical models for each situation collect sound and image signals for a long time (several hours) for each situation, and classify the collected sound and image signals for each short period (for example, 3 seconds) to recognize the acoustic environment and the image environment. After cognition is performed, cognitive results are accumulated and acquired. In this case, as shown in FIG. 6, the statistical model for each situation may be represented as a two-dimensional (acoustic / image environment) histogram using the accumulated cognitive result. As shown in FIG. 6, the statistical model for each situation represents the characteristics of each situation. For example, the statistical model of a large mart shows high indoor / loudness and moving / loudness, and the statistical model of subway shows high indoor / subway noise and indoor / loud noise.

When the cumulative result is provided from the cognitive result accumulator 210, the situation determiner 220 compares the probabilistic distance between the situational models stored in the situation statistical model 230 and the cumulative result, thereby providing the probability distance. Determines the closest situation statistical model as the user situation. For example, as illustrated in FIG. 7, the situation determination unit 220 may include a histogram representing a sound environment recognition result and an image environment recognition result accumulated for 5 minutes, and a large mart stored in the situation statistical model 230. A situation statistical model having the shortest probability distance calculated by calculating a stochastic distance between histograms representing situational models such as subways and streets may be determined as the user's situation.

When the user situation is determined, the situation determination unit 220 maps a time index to the determined user situation information as shown in FIG. 8 and stores the time index in the storage unit 240. Here, the time index is a time point when the user situation is determined, and may include date information and time information. The mapping of the determined user context with the time index is performed so that the user can easily search for the past situation as needed. That is, to allow the user to search for past situations through date and time unit search.

The storage unit 240 stores various programs and data necessary for generating a lifelog in the portable terminal. In particular, the storage unit 240 stores user context information mapped with a time index under the control of the context determination unit 220. Here, the user context information to which the time index is mapped may be referred to as a user's life log.

9 is a flowchart illustrating a procedure of generating and storing a lifelog in a portable terminal according to an exemplary embodiment of the present invention.

Referring to FIG. 9, in step 901, the terminal receives various sound signals and video signals around a user through the microphone 202 and the camera 204.

Thereafter, the terminal distinguishes the sound signal from the video signal in order to process the sound signal and the video signal input in step 903, and proceeds to step 905 for the audio signal and to step 911 for the video signal.

In step 905, the terminal determines whether a loud sound equal to or greater than a threshold value is included in the sound signal input during the preset short section. If the sound signal includes a loud sound above the threshold, the terminal proceeds to step 909 and recognizes that the sound environment corresponding to the sound signal is a loud sound environment.

On the other hand, when the sound signal does not include a loud sound above the threshold, the terminal proceeds to step 907 to extract the energy characteristic of the sound signal, and proceeds to step 909 to create an acoustic environment according to the extracted energy feature Be aware. That is, the terminal searches for and recognizes an acoustic environment having a maximum likelihood value for the extracted energy feature among the Gaussian mixture models for the previously stored acoustic environment. Here, the Gaussian mixed model for the acoustic environment may represent an energy characteristic for each acoustic environment, and the acoustic environment may include, for example, a loud noise, a car sound, a music sound, a noise in a closed space such as a bag, It can be divided into office noise, human voice, subway noise, quiet public noise, running water and loud noise.

Here, after checking whether the loud sound is included, when the loud sound is not included, a feature of the sound signal is extracted to recognize an acoustic environment, but the procedure of checking whether the loud sound is included is omitted, and You can also extract features.

On the other hand, the terminal measures the intensity of the video frame corresponding to the video signal input during the preset short section in step 911 to check whether or not the low light, and if the low light proceeds to step 915 and the environment of the video signal is low Be aware of the lighting environment. On the other hand, if the corresponding video frame is not low light, the terminal detects a motion vector, skin color, color and texture of the frame corresponding to the video signal in step 913 to determine whether the corresponding frame is moving or includes a face. In step 915, the image environment of the corresponding frame is recognized according to the determination result. In this case, the terminal may analyze the video frame in the order as shown in FIG. 3 or limit the analysis area as shown in FIG. 4 to analyze the video frame in real time to process the input video signal in real time. have. In particular, as shown in FIG. 5, the terminal may process the video signal at a time when the sound signal is not processed.

When the sound environment of the sound signal is recognized and the video environment of the video signal is recognized, the terminal accumulates the sound environment result and the video environment result for the preset short section in step 917. In this case, the terminal may accumulate the recognition result and may represent the two-dimensional histogram as shown in FIG. 6.

Thereafter, the terminal proceeds to step 919 to check whether the preset cumulative time expires. If the cumulative time has not expired, the terminal returns to step 901 to perform the following steps again.

On the other hand, when the cumulative time expires, the terminal determines the user's situation using the cumulative result in step 921. That is, the terminal compares the probabilistic distance between the plurality of pre-stored situation statistical models and the cumulative result and determines the situation statistical model having the shortest probabilistic distance as the user situation. Here, the plurality of situation statistical models may be represented as two-dimensional histograms as statistical models for sound and image signals for a plurality of predetermined situations. For example, the situation statistical models may characterize the acoustic and video signals of offices, restaurants / cafés, games, shopping malls, cars / buses, classrooms, streets, hypermarkets, subways, outdoor situations, and homes. have.

In step 923, the terminal maps a time index to the determined user context information as shown in FIG. 8 and stores the time index as a lifelog. Here, the time index is a time point when the user situation is determined, and may include date information and time information. Here, the storage of the determined user situation by mapping with the time index is to allow the user to easily search for the past situation by searching the date and time unit as needed.

Thereafter, the terminal terminates the algorithm according to the present invention.

Meanwhile, in the detailed description of the present invention, specific embodiments have been described, but various modifications may be made without departing from the scope of the present invention. Therefore, the scope of the present invention should not be limited to the described embodiments, but should be determined not only by the scope of the following claims, but also by the equivalents of the claims.

200: input unit 202: microphone
204: Camera 210: Cognitive Result Accumulator
212: acoustic environment recognition unit 214: image environment recognition unit
220: situation determination unit 230: situation statistics model
240: storage unit

Claims (18)

Recognizing a sound environment corresponding to a sound signal input from a microphone among preset sound environments;
Recognizing a video environment corresponding to a video signal input from a camera among preset video environments;
Determining a user context corresponding to the acoustic environment recognition result and the image environment recognition result among preset context models;
And recording the determined user situation into a life log.
The method of claim 1,
The acoustic environments represent energy characteristics for each of a plurality of preset acoustic environments,
The acoustic environments may include at least one of a loud noise, a car sound, a music sound, a noise in an enclosed space such as a bag, an office noise, a human voice, a subway noise, a quiet public place noise, a water flowing sound, and a loud sound. Method comprising a.
The method of claim 2,
The process of recognizing the acoustic environment,
Extracting energy characteristics of an acoustic signal input during a predetermined period from the microphone;
And recognizing an acoustic environment having a maximum likelihood value with respect to the extracted energy feature among the preset acoustic environments as an acoustic environment corresponding to the acoustic signal.
The method of claim 1,
The video environments are distinguished according to at least one of low light, mobility, presence of face, and indoors and outdoors.
The method of claim 4, wherein
Recognizing the video environment,
Checking whether an image signal input during a preset period from the camera satisfies the at least one condition;
And recognizing a video environment corresponding to the video signal according to whether the at least one condition is satisfied.
The method of claim 1,
The audio signal and the video signal are processed at different points in time within a preset section.
The method of claim 1,
The situation models represent recognition result statistics of an audio signal and an image signal for each of a plurality of predetermined situations.
The situation models may include models for at least one of an office, a restaurant / café, an event, a shopping mall, a car / bus, a classroom, a street, a large mart, a subway, an outdoor situation, and a home.
The method of claim 7, wherein
The process of determining a situation corresponding to the acoustic environment recognition result and the image environment recognition result among preset context models may include:
Accumulating the acoustic environment recognition result and the image environment recognition result for a predetermined time interval;
And determining the situation model having the shortest probabilistic distance from the cumulative result among the situation models as a user situation.
The method of claim 1,
The process of recording the determined user situation in the life log,
Obtaining time information at the time of determining the user situation;
And mapping the time information to the determined user context and recording the time information.
A microphone receiving an audio signal,
A camera receiving an image signal,
An acoustic environment recognizing unit recognizing an acoustic environment corresponding to a sound signal input from the microphone among preset acoustic environments;
An image environment recognizing unit recognizing an image environment corresponding to an image signal input from the camera among preset image environments;
A life log generation of a portable terminal, comprising: a situation determination unit for determining a user situation corresponding to the acoustic environment recognition result and the image environment recognition result among preset situation models and recording the determined user situation as a life log Device.
The method of claim 10,
The acoustic environments represent energy characteristics for each of a plurality of preset acoustic environments,
The acoustic environments may include at least one of a loud noise, a car sound, a music sound, a noise in an enclosed space such as a bag, an office noise, a human voice, a subway noise, a quiet public place noise, a water flowing sound, and a loud sound. Apparatus comprising a.
The method of claim 10,
The acoustic environment recognizing unit extracts an energy feature of an acoustic signal input during a preset period, and includes an acoustic environment having a maximum likelihood value with respect to the extracted energy feature among the preset acoustic environments. Recognizing as a device.
The method of claim 10,
The imaging environment is characterized in that the classification according to at least one of the low light, whether the movement, the presence of the face, indoors and outdoors.
The method of claim 13,
And the image environment recognizer recognizes an image environment corresponding to the image signal by checking whether an image signal input during a preset period satisfies the at least one condition.
The method of claim 10,
And the audio signal and the video signal are processed at different points in time within a preset section.
The method of claim 10,
The situation models represent recognition result statistics of an audio signal and an image signal for each of a plurality of predetermined situations.
The situation models may include a model for at least one of an office, a restaurant / café, a game watch, a shopping mall, a car / bus, a classroom, a street, a large mart, a subway, an outdoor situation, and a home.
17. The method of claim 16,
The situation determination unit may be configured to determine, as a user situation, a situation model having a shortest probabilistic distance from an acoustic environment recognition result and an image environment recognition result accumulated during a predetermined time period among the situation models.
The method of claim 10,
And the situation determination unit obtains time information at the time point of determining the user situation and maps and records the time information to the determined user situation.

Images