JP6363561B2 - Information processing system - Google Patents

Description

  The present invention relates to an information processing system that processes information such as images and sounds recorded at a life event such as a wedding so as to reflect the mental state of a user who is the main subject of information and the surrounding people. It is.

  Conventionally, in life events such as weddings, the images and videos of the target users are recorded and edited as appropriate to present information during the event as a presentation, or information can be collected on an album or DVD after the end of the life event. There is a service to present (see Patent Document 1).

JP 2001-150834 A

  However, the technology disclosed in Patent Document 1 merely provides information such as photos and videos taken on the day of a life event, and provides information reflecting the main role of the life event and the mental state of the event participants. There was a problem that it was not possible to provide sufficient information to recall the impressions and thoughts on the day of the life event.

  The present invention has been made in view of the above problems, and an object of the present invention is to provide an information processing system capable of processing information so as to reflect the mental states of users and event participants who are main players in life events, for example. It is to provide.

An information processing system according to the present invention includes a biometric information of a user who is a main subject and peripheral information representing the situation of the user and its surroundings as input, and recording means for storing the information, and the biometric information of the user And event determination means for determining whether an event has occurred based on at least one of the peripheral information, and information processing means for processing the peripheral information based on the event information extracted by the event determination means The event determination means extracts an event based on the biological information of the user and determines an event weight, and further extracts an event based on the peripheral information or an operation related to acquisition of the peripheral information and determines an event weight. And the information processing means includes the event extracted based on the biological information of the user and the peripheral information or the peripheral information. Process that integrates the events that can be regarded as the same among the events extracted based on the operation related to acquisition, and respond to the event according to the event weight for each event that is integrated with the event that does not have the event that can be regarded as the same The peripheral information is processed .

Further, the information processing system of the present invention includes a biometric information of a user who is a main subject and peripheral information representing the user and the surrounding situation, and a recording unit for storing these information, and the user's Event determining means for determining whether or not an event has occurred based on at least one of biological information and peripheral information; and information processing means for processing the peripheral information based on event information extracted by the event determining means; It comprises a peripheral information of the user includes at least the still image taken the user or around, the event determination means, the time at which the number of shots of the still picture per unit time exceeds a predetermined threshold event thereby determining the time of occurrence, as the number of shots of the still picture per unit time is large, heavy as the weight of the event is greater Give have rows, the information processing unit, in response to the weight of the event, and is characterized in that processing the peripheral information corresponding to the event.

Further, the information processing system of the present invention includes a biometric information of a user who is a main subject and peripheral information representing the user and the surrounding situation, and a recording unit for storing these information, and the user's Event determining means for determining whether or not an event has occurred based on at least one of biological information and peripheral information; and information processing means for processing the peripheral information based on event information extracted by the event determining means; the provided peripheral information of the user includes at least a moving image that the user or around, the event determination unit is configured to determine when the zoom-in operation of the moving image photographing is started and the time the event occurred , the longer the zoom duration from the zoom-in operation start until the zoom-out operation start, the weight of the event increases There row weighting, the information processing unit, in response to the weight of the event, and is characterized in that processing the peripheral information corresponding to the event.

  According to the present invention, the biometric information of the user who is the main target person and the peripheral information representing the user and the surrounding situation are stored, and an event has occurred based on at least one of the biometric information of the user and the peripheral information By determining whether or not and processing the peripheral information based on the extracted event information, the peripheral information can be processed to reflect the mental state of the user and the surrounding people. As a result, in the present invention, it is possible to realize provision of information sufficient to evoke the impressions and thoughts on the day of the life event as compared with the prior art.

  In the present invention, the event determination means extracts an event based on the user's biological information and determines the event weight, and further extracts the event based on the peripheral information or an operation related to acquisition of the peripheral information, and the event weight. The information processing means processes the peripheral information corresponding to this event according to the weight of the event, for example, by converting each peripheral information corresponding to a plurality of characteristic events in a series of life events into the event Can be processed according to the degree of importance. In the present invention, the peripheral information is processed based on the event information extracted based on the user's biological information, and the peripheral information is determined based on the user's peripheral information or the event information extracted based on the operation related to the acquisition of the peripheral information. By processing the peripheral information can be processed separately when the user's biological information shows a specific change and when the peripheral information or the operation related to the acquisition of the peripheral information shows a specific change. Providing information on whether biometric information shows a specific change and whether peripheral information or an operation related to acquisition of peripheral information shows a specific change is the same or not It becomes possible.

  In the present invention, the event determination means extracts an event based on the user's biological information and determines the event weight, and further extracts the event based on the peripheral information or an operation related to acquisition of the peripheral information, and the event weight. The information processing means integrates the event extracted based on the user's biological information and the event that can be regarded as the same among the events extracted based on the operation related to the peripheral information or the peripheral information acquisition, For each event integrated with an event for which there is no event that can be regarded as the same, by processing the peripheral information corresponding to the event according to the event weight, the event extracted from the user's biological information and the peripheral information or Correlate events extracted based on operations related to peripheral information acquisition Rukoto can, can be associated with each other and a peripheral information related to specific events for people in the vicinity of the other than the peripheral information and the user related to the specific events for the user himself. Thereby, in this invention, surrounding information can be processed so that the mental state of a user and its surrounding people may be reflected.

  In the present invention, as the weight of the event increases, the peripheral information corresponding to the event is emphasized, for example, by processing each peripheral information corresponding to a plurality of characteristic events in a series of life events, It can be processed to reveal the importance of the event.

  In the present invention, the user's biological information includes at least one of heart rate, body temperature, blood pressure, and blood glucose level, and the event determination unit determines the occurrence time of the event based on the change of the user's biological information. By doing so, for example, it is possible to estimate a user's mental state change in a series of life events. Further, in the present invention, by weighting so that the weight of the event increases as the change in the user's biometric information increases, for example, the event having a large change in the user's biometric information is weighted through a series of life event periods. The peripheral information corresponding to a characteristic event in a series of life events can be processed to reflect the mental state of the user.

  Further, in the present invention, the surrounding information of the user includes at least one of a still image obtained by photographing the user or the periphery thereof, a moving image obtained by photographing the user or the periphery thereof, and sound of the user or the surroundings thereof, for example, It is possible to estimate changes in the surrounding evaluation in a series of life events.

  Further, in the present invention, the surrounding information of the user includes at least a still image obtained by photographing the user or the periphery thereof, and the event determination unit performs an event when the number of still images taken per unit time exceeds a predetermined threshold. For example, it is possible to estimate a change in the evaluation of surroundings in a series of life events, and to determine a characteristic event that a person other than the user feels worth leaving a still image. Can be extracted. Further, in the present invention, by increasing the number of still images taken per unit time, the weight of the event is increased so that, for example, a person other than the user can view the still image throughout a series of life event periods. Events that you deserve to leave can be weighted, and peripheral information corresponding to characteristic events in a series of life events can be processed to reflect the mental state of people around you other than the user it can.

  In the present invention, it is assumed that the peripheral information of the user includes at least a moving image obtained by shooting the user or the periphery thereof, and the event determination unit determines the time when the zoom-in operation of moving image shooting is started as the event occurrence time. Thus, for example, it is possible to estimate a change in the surrounding evaluation in a series of life events, and it is possible to extract a characteristic event that is felt worthy to be noticed by people other than the user. Also, in the present invention, the longer the zoom-in duration from the start of the zoom-in operation to the start of the zoom-out operation, the higher the weight of the event, so that the weight of the event is increased. It is possible to weight events that have long felt worthy of attention, and process peripheral information corresponding to characteristic events in a series of life events to reflect the mental state of people around the user can do.

  In the present invention, the user's surrounding information includes at least one of a still image obtained by photographing the user or the periphery thereof, a moving image obtained by photographing the user or the periphery thereof, a sound of the user or the surroundings thereof, and a person other than the user. It is assumed that biometric information is included, and this biometric information includes at least one of heart rate, body temperature, blood pressure, and blood glucose level, and the event determination unit generates an event occurrence time based on a change in biometric information of a person other than the user For example, it is possible to estimate a mental state change of surrounding people other than the user in a series of life events. Further, in the present invention, an event having a large change in the biological information of a person other than the user, for example, through a series of life event periods is performed by weighting so that the weight of the event increases as the change in the biological information increases. The peripheral information corresponding to a characteristic event in a series of life events can be processed to reflect the mental state of people around the user other than the user.

It is a system configuration figure for explaining one example of a presentation system of living body information concerning an embodiment of the invention. It is a flowchart explaining operation | movement of the presentation system of the biometric information which concerns on embodiment of this invention. It is a flowchart explaining the method to determine the generation | occurrence | production time and weight of an event from a user's biometric information in embodiment of this invention. It is a figure which shows the example of the time-sequential change of the heart rate which is a user's biometric information in a life event. It is the figure which plotted the fluctuation | variation of the heart rate of FIG. It is a flowchart explaining the method to determine the generation | occurrence | production time and weight of an event from a user's periphery information in embodiment of this invention. It is a figure which shows the example which made the histogram the number of images by which the still image was image | photographed during the life event for every minute. It is a flowchart explaining the other method of determining the generation | occurrence | production time and weight of an event from a user's periphery information in embodiment of this invention. It is a figure which shows the example which made the histogram the time when zoom-in operation continued during imaging | photography of the moving image in a life event. It is a figure explaining an example of the processing method of a user's periphery information in embodiment of this invention. It is a flowchart explaining operation | movement of the information processing part in embodiment of this invention. It is a figure which shows an example of the result of the process of the information processing part in embodiment of this invention. It is a flowchart explaining other operation | movement of the information processing part in embodiment of this invention.

  Hereinafter, the present invention will be described with reference to the drawings with regard to preferred embodiments, but the present invention is not limited to such embodiments.

  FIG. 1 is a system configuration diagram for explaining an example of an information processing system 100 according to an embodiment of the present invention. The information processing system 100 is characterized by a biological information acquisition unit 120 that acquires biological information of a main target person (hereinafter referred to as a user) 110 in a life event, a peripheral information acquisition unit 140 that acquires peripheral information of the user 110, and biological information. Whether or not an event has occurred based on at least one of the recording unit 150 that stores the information acquired by the acquisition unit 120 and the peripheral information acquisition unit 140 and storing the information and the biological information and peripheral information of the user 110 An event determination unit 160 that determines whether the information is processed, an information processing unit 170 that processes peripheral information based on the event information extracted by the event determination unit 160, and an information output unit 180 that outputs the peripheral information processed by the information processing unit 170 It is to provide.

  If the life event is a wedding as in the example of FIG. 1, the user 110 who is the main target is, for example, a groom or a bride, and the event participants 130 other than the user 110 are wedding attendees.

  The biometric information acquisition unit 120 may be any sensor that can be worn by the user 110 and participate in a life event, such as a wearable exercise meter, a GPS (Global Positioning System) transmitter, an altimeter, a heart rate / electrocardiograph. A thermometer, a sphygmomanometer, a blood glucose meter, etc. can be suitably used.

  In particular, the heart rate of the user 110 acquired by the heart rate / electrocardiograph, the body temperature of the user 110 acquired by the thermometer, the blood pressure of the user 110 acquired by the sphygmomanometer, and the blood glucose level of the user 110 acquired by the blood glucose meter are autonomic nerves. It is known that it fluctuates depending on the function of the system and endocrine system, is effective for estimating the mental state of the user 110 such as tension / relaxation, and can be suitably used in view of the object of the present invention.

  The peripheral information acquisition unit 140 may be an electronic device that can be worn by an event participant 130 other than the user 110 or can be carried by the event participant 130 and participate in a life event. As the peripheral information acquisition unit 140, a general camera, a video camera, a microphone, a wearable sensor worn by the event participant 130, or the like can be used.

  Here, the surrounding information of the user 110 is information representing the situation of the user 110 and its surroundings, a still image obtained by photographing the user 110 or its surroundings, a moving image obtained by photographing the user 110 or its surroundings, the user 110 or its surroundings. It includes at least one of the surrounding sounds. Furthermore, as will be described later, it may include biological information of the event participant 130 that is measured by a wearable sensor worn by the event participant 130 separately from these pieces of information.

  The recording unit 150, the event determination unit 160, the information processing unit 170, and the information output unit 180 are realized by a computer having a CPU (Central Processing Unit), a storage device, and an interface, and a program that controls these hardware resources. be able to. The CPU executes processing described below according to a program stored in the storage device, and functions as a recording unit 150, an event determination unit 160, an information processing unit 170, and an information output unit 180.

  FIG. 2 is a flowchart for explaining the operation of the information processing system 100 of the present embodiment. In the information processing system 100, the biological information of the user 110 is acquired by the biological information acquisition unit 120 in the life event, and the event participant 130 other than the user 110 acquires the peripheral information of the user 110 using the peripheral information acquisition unit 140. An information acquisition step (step S1 in FIG. 2), an information recording step (step S2 in FIG. 2) for storing the biometric information and peripheral information of the user 110 acquired in the information acquisition step in the recording unit 150, and a recording unit in the information recording step 150, an event extraction step (step S3 in FIG. 2) in which the event determination unit 160 determines the occurrence time and event weight of a specific event based on the biological information and peripheral information recorded in 150, and the event extracted in the event extraction step. An information processing step for processing the peripheral information of the user 110 based on the information (see FIG. And-up S4), the information output step (FIG. 2, step S5 for outputting processed information in the information processing step) and to run. Hereinafter, each step will be described sequentially.

  In the information acquisition step, it is only necessary that the user 110 wears the biometric information acquisition unit 120 and at least one event participant 130 wears or carries the peripheral information acquisition unit 140. You can experience life events naturally without being conscious of acquiring information.

  That is, the biometric information acquisition unit 120 automatically always acquires the biometric information of the user 110 during the life event, and the peripheral information acquisition unit 140 always automatically acquires the peripheral information of the user 110 during the life event ( FIG. 2 step S1). At this time, it is desirable to match the time measured by the internal clock of the biological information acquisition unit 120 with the time measured by the internal clock of the peripheral information acquisition unit 140.

  In the information recording step, the recording unit 150 records the information acquired in the information acquisition step (step S2 in FIG. 2). The biometric information acquisition unit 120 transmits the acquired biometric information with measurement time information added thereto, or records the biometric information with the measurement time information added to a recording medium such as a memory card. The peripheral information acquisition unit 140 transmits the acquired peripheral information with measurement time information added thereto, or records the peripheral information with the measurement time information added to a recording medium.

  When it is necessary to process and output the peripheral information of the user 110 during the life event (for example, video playback during the life event), the biological information and the peripheral information are obtained from the biological information acquisition unit 120 and the peripheral information acquisition unit 140. It is necessary to perform wireless transmission or wired transmission. In this case, the recording unit 150 may be a wireless communication unit (not shown) that receives biological information and peripheral information wirelessly transmitted from the biological information acquisition unit 120 and the peripheral information acquisition unit 140, or the biological information acquisition unit 120 and the peripheral information acquisition. A wired communication unit (not shown) that receives information wired from the unit 140 is provided.

  Further, when it is necessary to process and output the peripheral information after the life event ends (for example, printing an album or writing to a DVD), the recording unit 150 includes information recorded on the recording medium of the biological information acquisition unit 120, Information recorded on the recording medium of the peripheral information acquisition unit 140 may be read and stored. In this case, the recording unit 150 includes an information reading unit (not shown) that reads information from the recording media of the biological information acquisition unit 120 and the peripheral information acquisition unit 140.

  In the event extraction step, the event determination unit 160 determines a specific event occurrence time and an event weight from the biological information and peripheral information stored in the recording unit 150 (step S3 in FIG. 2). Here, a method for determining the occurrence time and weight of a specific event from the biometric information of the user 110 and a method of determining from the peripheral information of the user 110 will be described in order.

  FIG. 3 is a flowchart for explaining a method of determining the event occurrence time and weight based on the biometric information of the user 110 acquired by the biometric information acquisition unit 120 and recorded in the recording unit 150. Here, a case where the biological information of the user 110 is a heart rate will be described as an example.

  FIG. 4 shows an example of time-series data in which the heart rate of the groom (user 110) in a life event (here, a wedding) is recorded. The heart rate of the user 110 increases or decreases in a series of life events to reflect the mental state of the user 110. In particular, the time when the heart rate continues to increase or decrease can be considered as the time when a unique event has occurred for the user 110.

  Therefore, the event determination unit 160 takes the extreme value (maximum value or minimum value) of the heart rate fluctuation (differentiation) of the user 110 (step S10 in FIG. 3), and the absolute value of the extreme value of the heart rate fluctuation is predetermined. Is exceeded (step S11 in FIG. 3), it is determined that an event peculiar to the user 110 has occurred, and the time when the absolute value of the extreme value of heart rate fluctuation exceeds the threshold is set as the event occurrence time ( FIG. 3 step S12). The time that should be adopted as the event occurrence time may be acquired from the measurement time information added to the biological information of the user 110 by the biological information acquisition unit 120.

  FIG. 5 shows a plot of a variation (differentiation) of a part of the heart rate shown in FIG. In the graph shown in FIG. 5, if points where the fluctuation of the heart rate takes an extreme value and the absolute value of the extreme value exceeds a predetermined threshold (0.1 in the present embodiment) are extracted, 301 to 305 are obtained. Five points are obtained. Therefore, the times of these five points 301 to 305 can be determined as event occurrence times, respectively. In the example of FIG. 5, the maximum value of the point 302 is the largest. There was actually a unique event for the groom that the groom entered the ceremony at the time of this point 302.

  The event determination unit 160 repeats the processes of steps S10 to S12 every predetermined cycle (for example, in milliseconds), and when an instruction to end measurement of biometric information is given from the outside, or the user 110 measures biometric information. When the biometric information acquisition unit 120 is turned off and the power is turned off (YES in step S13 in FIG. 3), the absolute value of the extreme value determined as the event occurrence in the processing of steps S10 to S12 is determined as the weight of the event ( FIG. 3 step S14). The event determination unit 160 performs such weight determination for each event.

  In the example of FIG. 5, the absolute value A1 of the local minimum value at point 301, the absolute value A2 of the local maximum value at point 302, the absolute value A3 of the local minimum value at point 303, the absolute value A4 of the local maximum value at point 304, and the local minimum value at point 305 The absolute value A5 is determined as the weight of each of the five events. That is, the event determination unit 160 determines that the event weight increases as the absolute value of the extreme value of the heart rate fluctuation increases.

  In the description with reference to FIG. 3, the weight of the event is determined after an instruction to end measurement of biometric information is given from the outside or after the user 110 stops measuring biometric information. As an example of an instruction to end biological information measurement from the outside, for example, an instruction input by an operator of the information processing system 100 (for example, a staff member of an event management company) at the end of a life event, or peripheral information in a life event There are instructions input by an operator of the information processing system 100 for processing / output.

  However, the event weight determination process is not limited to the example described in FIG. The event determination unit 160 determines that an event has occurred because an extreme value appears in the fluctuation of the heart rate while the measurement of the heart rate continues (while the processing of steps S10 to S12 is continued). Sometimes, it is possible to immediately determine the absolute value of the extreme value of the heart rate fluctuation as the event weight.

  Further, if a normalization reference value (for example, the maximum value that can be expected of the absolute value of the extreme value) is set in advance, the event determination unit 160 normalizes the absolute value of the extreme value of the heart rate fluctuation. Can also be determined as the event weight.

As described above, the event determination unit 160 can determine the event occurrence time and weight from the biological information of the user 110.
In measuring the heart rate and heart rate fluctuation, low-pass filter processing may be applied to the measured heart rate and heart rate fluctuation in order to exclude the influence of high-frequency noise and fluctuation.

  In the above description, the case where the biological information of the user 110 is a heart rate is described as an example. However, the biological information of the user 110 may be information other than the heart rate, such as body temperature, blood pressure, and blood sugar level. It is possible to determine the event occurrence time and weight by the same method as in the case of the heart rate.

  In this case, only one of the heart rate, body temperature, blood pressure, and blood glucose level is included as the biological information of the user 110, and the event occurrence time and weight may be determined by the above method for this one, The biometric information of the user 110 may include a plurality of heart rate, body temperature, blood pressure, and blood glucose level, and the event occurrence time and weight may be determined individually for each of the plurality of biometric information by the above-described method.

  Next, FIG. 6 is a flowchart for explaining a method for determining the event occurrence time and weight from the peripheral information of the user 110 acquired by the peripheral information acquisition unit 140 and recorded in the recording unit 150. Here, a case where the peripheral information of the user 110 is a still image will be described as an example.

  FIG. 7 shows an example of time-series data in which the number of still images (photographs) taken during a life event (here, a wedding) is histogrammed every minute. The time when the number of shots per unit time by the peripheral information acquisition unit 140 (camera) is large indicates that the photographer (event participant 130) has decided to leave many photographs, and there is a unique event. It is considered the time of occurrence.

  Accordingly, when the number of still images taken per unit time exceeds a predetermined threshold (step S20 in FIG. 6), the event determination unit 160 determines that a unique event has occurred for the event participant 130, and per unit time. The time when the number of still images taken exceeds the threshold is set as the event occurrence time (step S21 in FIG. 6). The time that should be adopted as the event occurrence time may be acquired from the measurement time information added to the peripheral information of the user 110 by the peripheral information acquisition unit 140.

  In the example shown in FIG. 7, when points where the number of still images taken per unit time (here, one minute) exceeds a predetermined threshold (four in this embodiment) are extracted, seven points 401 to 407 are extracted. Points are earned. Therefore, the times of these seven points 401 to 407 can be determined as the occurrence times of the seven events.

  In the example of FIG. 7, an event of bride entry occurs at the time of point 401, an event of handing over the bride from the bride to the groom occurs at time of point 402, and the bride and groom at the time of points 403 and 404. A ring exchange event occurs, the bride opens a veil at the time of point 405, a wedding kiss of the groom and the bride occurs, and an event of signing a marriage certificate occurs at the time of point 406. An event of leaving the bride and groom occurred at noon.

  The event determination unit 160 performs the processing of steps S20 and S21 every unit time. For example, when there is an instruction to end the shooting of a still image from the outside, or the event participant 130 stops shooting and the surrounding information acquisition unit 140 ( When the power of the camera is turned off (YES in step S22 in FIG. 6), the number of still images to be photographed determined to have occurred in the processing in steps S20 and S21 is determined as the weight of the event (step S23 in FIG. 6). . The event determination unit 160 performs such weight determination for each event.

  In the example of FIG. 7, the number of shots at point 401 is 6, the number of shots at point 402 is 5, the number of shots at point 403 is 7, the number of shots at point 404 is 5, the number of shots at point 405 is 9, and the point 406 is shot. Five shots and six shots of point 407 are determined as the weights of seven events, respectively. That is, the event determination unit 160 determines that the event weight increases as the number of still images taken per unit time increases.

  In the description using FIG. 6, the weight of the event is determined after an instruction to end the shooting of the still image is given from the outside or the event participant 130 stops the shooting. As an instruction to end shooting from the outside, for example, an instruction input by an operator of the information processing system 100 (for example, a staff member of an event operating company) at the end of a life event, or processing / output of peripheral information during a life event Therefore, there are instructions input by the operator of the information processing system 100.

  However, as in the case of the biometric information of the user 110, the event weight determination process is not limited to the example of FIG. The event determination unit 160 determines that an event is detected when the number of still images taken per unit time exceeds the threshold while still images are being taken (while the processing in steps S20 and S21 is continued). When it is determined that it has occurred, the number of shots can be immediately determined as the event weight.

  In addition, if a normalization reference value (for example, the maximum value that can be predicted for the number of shots) is set in advance, the event determination unit 160 uses a value obtained by normalizing the number of shots of still images per unit time as the event weight. It is also possible to determine as

  In the description using FIG. 6, the case where the peripheral information of the user 110 is a still image is described as an example. However, even when the peripheral information of the user 110 is a moving image, an event is performed in the same manner as in the case of a still image. Occurrence time and weight can be determined. However, in the case of a moving image, an event is extracted based on an operation related to image shooting, not the image itself.

  FIG. 8 is a flowchart for explaining a method of determining an event occurrence time and weight from a moving image acquired by the peripheral information acquisition unit 140 and recorded in the recording unit 150, and FIG. 9 is a life event (wedding here) It is a figure which shows the example which made the histogram the time when zoom-in operation which took a to-be-photographed object large during imaging | photography of this moving image was continued. The time when the zoom-in operation of the peripheral information acquisition unit 140 (video camera) is performed indicates that the photographer (event participant 130) has determined that it is better to pay attention, and an event unique to the event participant 130 It is considered the time when this occurred.

  Therefore, when the zoom-in operation for the peripheral information acquisition unit 140 is started (step S30 in FIG. 8), the event determination unit 160 determines that a unique event has occurred for the event participant 130, and the zoom-in operation is started. The time is set as the event occurrence time (step S31 in FIG. 8). Similarly to the above, the time to be adopted as the event occurrence time may be acquired from the measurement time information added to the peripheral information of the user 110 by the peripheral information acquisition unit 140.

  However, in the method for determining the event occurrence time and weight based on the zoom operation for moving image shooting, the peripheral information acquisition unit 140 (video camera) transmits the peripheral information acquisition unit to the computer constituting the event determination unit 160. It is necessary to transmit 140 pieces of operation information wirelessly or by wire. By transmitting such operation information, the event determination unit 160 can detect a zoom-in operation or a zoom-out operation on the peripheral information acquisition unit 140.

  In FIG. 9, when the point where the zoom-in operation for the peripheral information acquisition unit 140 is started is extracted, three points 501 to 503 are obtained. Therefore, the times of these three points 501 to 503 can be determined as the occurrence times of three events, respectively. In the example of FIG. 9, an event of entering the bride occurred at the time of point 501, an event of ring exchange occurred at the time of point 502, and an event of signing the marriage certificate occurred at the time of point 503. .

  The event determination unit 160 performs the processing of steps S30 and S31 every unit time, and when an instruction to end the shooting of the moving image is given from the outside, or the event participant 130 stops shooting and the surrounding information acquisition unit 140 (video) When the camera is turned off (YES in step S32 in FIG. 8), the zoom-in duration from the zoom-in operation start to the zoom-out operation start determined as an event occurrence in steps S30 and S31 is used as the weight of the event. Determine (step S33 in FIG. 8). The event determination unit 160 performs such weight determination for each event.

  In the example of FIG. 9, the zoom-in duration from the start of the zoom-in operation at the point 501 to the start of the zoom-out operation is 2 minutes, the zoom-in duration from the start of the zoom-in operation at the point 502 to the start of the zoom-out operation is 2 minutes, and the zoom-in operation at the point 503 is started. The zoom-in duration of 3 minutes from the start of the zoom-out operation to 3 minutes is determined as the weight of each of the three events. That is, the event determination unit 160 determines that the weight of the event increases as the zoom-in duration continues.

  Note that the event weight determination process is not limited to the example of FIG. 8, as in the case of the biometric information and still image of the user 110. When the event determination unit 160 determines that an event has occurred when a zoom-in operation is started while shooting of a moving image continues (while the processes of steps S30 and S31 are continued), It is also possible to immediately determine the zoom-in duration from the zoom-in operation start to the zoom-out operation start as an event weight.

  In addition, if a normalization reference value (for example, a maximum value that can be predicted for the zoom-in duration) is set in advance, the event determination unit 160 may determine a value obtained by normalizing the zoom-in duration as the event weight. Is possible.

  In the above description, the case where the peripheral information of the user 110 is a still image or a moving image is described as an example. However, even when the peripheral information of the user 110 is audio, the event occurrence time and weight can be determined. Is possible.

  When the surrounding information of the user 110 is voice, the event determination unit 160 generates a unique event for the event participant 130 when the intensity of the voice collected by the surrounding information acquisition unit 140 (microphone) exceeds a predetermined threshold. The time when the sound intensity exceeds the threshold may be set as the event occurrence time. Further, the event determination unit 160 may determine the intensity of the sound determined as the occurrence of the event as the weight of the event. In addition, as with other information, if a normalization reference value (for example, a maximum value that can be predicted for voice intensity) is set in advance, the event determination unit 160 uses the value obtained by normalizing the voice intensity as an event. It is also possible to determine it as the weight.

  Note that the peripheral information of the user 110 includes only one of a still image, a moving image, and audio, and the event occurrence time and weight may be determined for the one by the above-described method. Peripheral information includes a plurality of still images, moving images, and audio, and the event occurrence time and weight may be determined individually for each of the plurality of peripheral information by the above-described method.

  Further, in the case where the peripheral information of the user 110 includes the biological information (at least one of heart rate, body temperature, blood pressure, blood glucose level) of the event participant 130 other than the user 110, it has been described with reference to FIGS. By using the same method, the event occurrence time and weight specific to the event participant 130 can be determined.

  However, in the present embodiment, the peripheral information of the user 110 is processed using the processing result of the event determination unit 160 as described later, and therefore, when the biological information of the event participant 130 is acquired as the peripheral information of the user 110. Needs to acquire at least one of a still image, a moving image, and sound as peripheral information to be processed separately from the biological information of the event participant 130.

  In this case, only the biometric information of the event participant 130 included in the peripheral information of the user 110 may be used to determine the event occurrence time and weight by the same method as described in FIGS. The event occurrence time and weight may be determined individually for each of at least one of the moving image and the sound and the biological information of the event participant 130 by the above-described method.

  Next, in the information processing step, the information processing unit 170 processes the peripheral information of the user 110 based on the event information extracted by the event determination unit 160 (step S4 in FIG. 2). More specifically, the information processing unit 170 processes the peripheral information corresponding to the event to be emphasized as the event weight increases.

  FIG. 10 is a diagram for explaining an example of a processing method of peripheral information of the user 110. According to the example of FIG. 10, the events 601 to 606 extracted when the absolute value of the extreme value of the fluctuation of the groom's heart rate at the wedding exceeds the threshold value 0.15, It can be seen that the events 607 to 612 extracted when the number of still images taken exceeds the threshold value 4 have occurred.

  In the present embodiment, an event extracted based on the biological information of the user 110 and an event that can be regarded as the same among the events extracted based on the peripheral information of the user 110 are integrated, and a final weight of each event is calculated. To do. FIG. 11 is a flowchart for explaining the operation of the information processing unit 170 of the present embodiment.

  First, the information processing unit 170 normalizes the weight of each event extracted by the event determination unit 160 based on the biometric information of the user 110, and each of the event determination unit 160 extracted based on the peripheral information of the user 110. The event weight is normalized (step S40 in FIG. 11).

  When the biological information of the user 110 is a heart rate, the normalization of the event weights extracted based on the heart rate fluctuation is the extreme of the heart rate fluctuation obtained from the start of the heart rate measurement to the end of the measurement. It is performed so that the maximum value of the absolute values is 1. Even when the biological information of the user 110 is information other than the heart rate, such as body temperature, blood pressure, and blood glucose level, the event weights can be normalized by the same method as in the case of the heart rate.

  Similarly, the surrounding information of the user 110 includes biological information (at least one of heart rate, body temperature, blood pressure, blood glucose level) of the event participant 130, and an event extracted based on the fluctuation of the biological information is included. In some cases, the normalization of the event weight is performed so that the maximum value of the absolute values of the extreme values of the fluctuation of the biological information obtained from the start to the end of the measurement of the biological information is 1.

  In addition, when a still image is included as the peripheral information of the user 110 and there is an event extracted based on the number of still images taken per unit time, normalization of the weight of the event is performed from the start of still image shooting. The maximum value of the number of still images taken per unit time counted until the end is set to 1.

  In addition, when a moving image is included as the peripheral information of the user 110 and an event is extracted based on the start point of the zoom-in operation, the normalization of the weight of the event is measured from the start of moving image shooting to the end of shooting. It is performed so that the maximum value of the zoomed-in durations thus obtained becomes 1.

  When there is a sound included as the peripheral information of the user 110 and there is an event extracted based on the sound intensity, the normalization of the weight of the event is performed based on the sound intensity measured from the sound recording start to the recording end. The maximum value is set to 1.

  Note that normalization of event weights by the information processing unit 170 is not an essential process of the present invention. As described above, when the event determination unit 160 normalizes the event weight, the event weight may not be normalized in the process of step S40 of the information processing unit 170.

  Next, the information processing unit 170 can regard each event extracted by the event determination unit 160 based on the biometric information of the user 110 and each event extracted by the event determination unit 160 based on the peripheral information of the user 110 as the same. It is determined whether an event exists (step S41 in FIG. 11).

  When the occurrence time of the event extracted based on the biometric information of the user 110 is the same as the occurrence time of the event extracted based on the peripheral information of the user 110, the information processing unit 170 Are considered as the same event and are subject to integration. In addition, the information processing unit 170 determines that the event occurrence time extracted based on the peripheral information of the user 110 is within a predetermined time range centered on the event occurrence time extracted based on the biometric information of the user 110. If included, these two events whose occurrence times are close to each other are regarded as the same event.

  If there is an event that can be regarded as the same among the event extracted based on the biological information of the user 110 and the event extracted based on the peripheral information of the user 110 (YES in step S41), the information processing unit 170 is the same. A plurality of events that can be considered as a single event are integrated (step S42 in FIG. 11).

  At this time, the information processing unit 170 is extracted based on the occurrence time of the event extracted based on the biometric information of the user 110 included in the event to be integrated, and the peripheral information of the user 110 included in the event to be integrated. The event occurrence time, the intermediate value of these event occurrence times, and the like are set as the event occurrence time after the integration process (step S43 in FIG. 11). The method for determining the event occurrence time after the integration process can be selected in advance and set in the information processing unit 170.

  The information processing unit 170 also includes an event weight extracted based on the biometric information of the user 110 included in the event to be integrated and an event extracted based on the peripheral information of the user 110 included in the event to be integrated. And the added result is used as the event weight after the integration process (step S44 in FIG. 11).

  The information processing unit 170 performs the processing of steps S41 to S44 for all of the events extracted based on the biological information of the user 110 and the events extracted based on the peripheral information of the user 110 (step S45 in FIG. 11).

  FIG. 12 is a diagram showing an example of the results of the processes in steps S40 to S45 described above. Here, the result processed by the information processing unit 170 for the events 601 to 612 described in FIG. 10 is shown. As described in FIG. 10, the events 601 to 606 are events extracted based on the biological information of the user 110 (groom's heart rate at the wedding), and the events 607 to 612 are the peripheral information (still images) of the user 110. It is an event extracted based on.

  In the example of FIG. 10, an event of entering the groom occurs at the time 601, an event of entering the bride at the times 602 and 607, and handing the bride from the bride to the groom at the times 603 and 608. An event occurs, a ring exchange event occurs at a time 609, a wedding kiss event occurs at a time 604, 610, an event of signing a marriage certificate occurs at a time 605, 611, and 606 , An event of leaving the bride and groom occurred at the time of 612.

  “602 + 607” in the column 701 in FIG. 12 indicates an event 602 extracted based on the change in the heart rate of the groom at the wedding and an event 607 extracted based on the number of still images taken per minute during the wedding. Are integrated into one. “603 + 608” indicates that the event 603 extracted based on the groom's heart rate fluctuation and the event 608 extracted based on the number of still images taken are integrated into one. Also, “604 + 610” indicates that the event 604 extracted based on the groom's heart rate fluctuation and the event 610 extracted based on the number of still images taken are integrated into one. The same applies to other events.

  A column 702 in FIG. 12 represents the absolute value of the extreme value of the change in the heart rate of the groom at the wedding, that is, the weight of the events 601 to 606 determined by the event determination unit 160. The event 609 is an event extracted based on the number of still images taken per minute during the wedding, and is not an event extracted based on the variation of the heart rate of the groom, so is described in the column 702. The weight is 0.

  A column 703 in FIG. 12 represents the weights of the events 601 to 606 normalized by the information processing unit 170. In the weights of the events 601 to 606 shown in the column 702, the maximum value serving as a reference value for weight normalization is the weight 0.5 of the event 601. Therefore, the information processing unit 170 normalizes the weights of the events 601 to 606 so that the weight of the event 601 is 1.

  A column 704 in FIG. 12 represents the number of still images taken per minute during the wedding, that is, the weights of the events 607 to 612 determined by the event determination unit 160. The event 601 is an event extracted based on the change in the heart rate of the groom during the wedding and is not an event extracted based on the number of still images taken. Therefore, the weight described in the column 704 is 0. It has become.

  A column 705 in FIG. 12 represents the weights of the events 607 to 612 normalized by the information processing unit 170. In the weights of the events 607 to 612 shown in the column 704, the maximum value serving as a reference value for weight normalization is the weight 9 of the event 610. Therefore, the information processing unit 170 normalizes the weights of the events 607 to 612 so that the weight of the event 610 is 1.

  The column 706 in FIG. 12 represents the event weight after the processing in step S44. That is, since the information processing unit 170 integrates the events 602 and 607, a value 1.01 obtained by adding the normalized weight 0.34 of the event 602 and the normalized weight 0.67 of the event 607 is the integrated value. The weight of the event. Also, since the information processing unit 170 integrates the events 603 and 608, a value 0.9 obtained by adding the normalized weight 0.34 of the event 603 and the normalized weight 0.56 of the event 608 is the post-integration processing. The weight of the event. Since the information processing unit 170 integrates the events 604 and 610, a value 1.34 obtained by adding the normalized weight 0.34 of the event 604 and the normalized weight 1 of the event 610 is the event after the integration processing. It becomes the weight of. The same applies to the integration of 605 and 611 and the integration of 606 and 612.

  For the event 601 extracted based on the change in the heart rate of the groom during the wedding, there is no event that can be regarded as the same among the events extracted based on the number of still images taken, so the integration is not performed. The normalized weight 1 becomes the final weight of the event 601 as it is.

  Similarly, with respect to the event 609 extracted based on the number of still images to be photographed, there is no event that can be regarded as the same among the events extracted based on the variation of the groom's heart rate, and therefore, integration is not performed and normalization is performed. The weight 0.78 thus used becomes the final weight of the event 609 as it is.

  In the present embodiment, the event weight A extracted based on the biometric information of the user 110 included in the event to be integrated and the peripheral information of the user 110 included in the event to be integrated are extracted. The event weight B is the same weight, and the result of adding A and B (A + B) is the event weight after integration processing. However, the present invention is not limited to this, and the weight occupies the event weight after integration processing. The ratio of A and B may be changed. As an example, for example, the weight of the event after the integration process may be (2A + B). As a result, it is possible to realize event integration focusing on specific changes in the mental state of the user 110 (specific changes in the biological information of the user 110).

  Next, the information processing unit 170 completes the processes of steps S41 to S44 for all the events extracted based on the biological information of the user 110 and the events extracted based on the peripheral information of the user 110 (FIG. 11). In step S45, YES), peripheral information of the user 110 corresponding to each event is processed based on the weight of the event (step S46 in FIG. 11).

  When the peripheral information to be processed is a still image, the information processing unit 170 processes the size of each still image so that the ratio of the areas of the still images is equal to the ratio of the event weights. A still image corresponding to an event refers to a still image whose shooting time is equal to the event occurrence time. The photographing time of the still image may be acquired from the measurement time information added to the still image by the peripheral information acquisition unit 140.

  The information processing unit 170 may process the size of a still image having no corresponding event as having a predetermined weight smaller than the maximum normalized weight (1 in the present embodiment). . Thus, as the weight of the corresponding event increases, the size of the still image increases.

  Similarly, when the peripheral information to be processed is a moving image, the information processing unit 170 adjusts the size of each frame of the moving image so that the ratio of the area of each frame of the moving image is equal to the ratio of the weight of the event. Process. A moving image frame corresponding to an event refers to a frame whose shooting time is equal to the event occurrence time. As in the case of a still image, the shooting time may be acquired from the measurement time information added to each frame of the moving image by the peripheral information acquisition unit 140.

  The information processing unit 170 may process the size of an image frame having no corresponding event as having a predetermined weight smaller than the normalized maximum weight. Thus, as the weight of the corresponding event increases, the size of the moving image frame increases.

  When the peripheral information to be processed is a still image or a moving image, the size of the processed image is automatically adjusted so as to be within a predetermined maximum size. Specifically, the adjustment may be performed so that one image fits in the maximum size, or the peripheral area of the image may be cut so as to fit in the maximum size.

  In addition, when the peripheral information to be processed is sound, the information processing unit 170 adjusts the volume of each section of the sound so that the sound intensity ratio becomes equal to the event weight ratio. An audio section corresponding to an event is a section whose recording time is equal to the event occurrence time. As in the case of a still image, the recording time may be acquired from the measurement time information added to the audio data by the peripheral information acquisition unit 140.

The information processing unit 170 may adjust the sound volume for a voice section having no corresponding event as having a predetermined weight smaller than the maximum normalized weight. Thus, the greater the weight of the corresponding event, the louder the sound. When the peripheral information to be processed is audio, the volume is automatically limited so that the processed volume falls within a predetermined maximum volume.
Above, the process of the information processing part 170 is complete | finished.

  As for the peripheral information of the user 110, the peripheral information on which the event is extracted and the peripheral information to be processed may be the same or different. For example, when an event is extracted based on the biological information of the event participant 130 other than the user 110 and at least one of a still image, a moving image, and a sound is processed, peripheral information and a processing target as a basis for the event extraction The surrounding information will be different. Further, when an event is extracted based on the number of still images to be photographed and the still image is processed, the peripheral information on which the event is extracted and the peripheral information to be processed are the same.

Next, the information output unit 180 outputs the peripheral information processed by the information processing unit 170 (step S5 in FIG. 2).
When the peripheral information processed by the information processing unit 170 is a still image, the peripheral information output includes display display, printing of an album, recording on a recording medium such as a DVD, and the like.

When the peripheral information processed by the information processing unit 170 is a moving image, examples of the peripheral information output include display on a display and recording on a recording medium such as a DVD.
When the peripheral information processed by the information processing unit 170 is audio, the peripheral information output includes a speaker output and recording on a recording medium such as a DVD.

  As described above, in the present embodiment, an event unique to the user 110 is extracted based on the biological information of the user 110, and an event unique to the event participant 130 is extracted based on the peripheral information of the user 110. By integrating the event extracted based on the biological information of the user 110 and the event extracted based on the peripheral information of the user 110, the peripheral information of the user 110 is processed based on the weight of the event. The surrounding information when the mental state of the event shows a specific change is emphasized according to the weight of the event, and the peripheral information when the mental state of the event participant 130 shows the specific change is set as the event weight. The surrounding information can be added to reflect the mental state of the user 110 and the event participant 130. It can be to output. As a result, in the present embodiment, as compared with the conventional technique disclosed in Patent Document 1, it is possible to realize information provision sufficient to evoke the impressions and thoughts on the day of the life event.

  In this embodiment, the event extracted based on the biological information of the user 110 and the event that can be regarded as the same among the events extracted based on the peripheral information of the user 110 are integrated. When the user's 110 biological information shows a specific change, and when the event participant's 130 mental state shows a specific change (the surrounding information shows a specific change). If they match, the weight of the event to be integrated is added and increased, so that the peripheral information can be emphasized as an important event.

  Note that when the peripheral information to be processed is a still image or a moving image, the information processing unit 170 is processed to reflect the mental information of the user 110 (an event extracted based on the biological information of the user 110). Whether the surrounding information is processed by reflecting the mental state of the event participant 130 (the event extracted based on the surrounding information of the user 110), or the surrounding information is both the user 110 and the event participant 130 The character information explaining the reason for the processing may be added to the still image or the moving image to be processed so that it can be understood that the image has been processed reflecting the mental state.

  Thereby, the person who has seen the peripheral information output from the information processing unit 170 has a specific change in the mental state of the user 110 (when the biological information of the user 110 has a specific change). It is possible to recognize whether or not the event participant 130 matches or does not match when the mental state of the event participant 130 shows a specific change (when the peripheral information shows a specific change).

  For example, in the example of FIGS. 10 and 12, the event 602 is an event extracted based on the variation of the groom's heart rate, and the event 607 is an event extracted based on the number of still images taken. And 607 are merged, the still image or the moving image corresponding to the merged event is added with character information explaining that it has been processed to reflect the mental state of the user 110 and the event participant 130. .

  Therefore, a person who has seen the peripheral information output from the information processing unit 170 shows a specific change in the mental state of the user 110 and the mental state of the event participant 130 when the bride enters, It can be recognized that the feelings of the participants 130 match.

  In the example of FIGS. 10 and 12, the event 609 is an event extracted based on the number of still images taken, and is not an event extracted based on the change in the groom's heart rate, and therefore corresponds to the event 609. Character information explaining that the still image or the moving image is processed to reflect the mental state of the event participant 130 is added.

  Therefore, the person who has seen the peripheral information output from the information processing unit 170 does not change greatly in the mental state of the user 110 when the ring is exchanged, and the mental state of the event participant 130 is not changed. A specific change is shown, and it can be recognized that the feeling of the user 110 and the event participant 130 is different.

  Note that event integration is not an essential process of the present invention. That is, the information processing unit 170 does not have to perform the processes of steps S41 to S45 in FIG. A flowchart of the operation of the information processing unit 170 in this case is shown in FIG. After performing the process of step S40, the information processing unit 170 performs the process of step S46 separately for the event extracted by the event determination unit 160 based on the biological information of the user 110 and the event extracted based on the peripheral information of the user 110. Can be done. Even when events are not integrated, character information explaining the reason for processing can be added to a still image or moving image to be processed, as described above.

  Moreover, in this Embodiment, although the event is extracted about each of the biometric information and peripheral information of the user 110, you may make it extract an event only about either one information.

  The present invention can be applied to a technique for processing information so as to reflect the mental state of a user who is a main target of information such as images and sounds, and the surrounding people.

  DESCRIPTION OF SYMBOLS 100 ... Information processing system, 110 ... User, 120 ... Biometric information acquisition part, 130 ... Event participant, 140 ... Peripheral information acquisition part, 150 ... Recording part, 160 ... Event determination part, 170 ... Information processing part, 180 ... Information Output part.

Claims (3)

Recording means for storing biometric information of a user who is a main target person and peripheral information representing the situation of the user and its surroundings, and storing these information,
Event determining means for determining whether an event has occurred based on at least one of the user's biological information and peripheral information;
Information processing means for processing the peripheral information based on the event information extracted by the event determination means ,
The event determination means extracts an event based on the user's biological information and determines an event weight, and further extracts an event based on the peripheral information or an operation related to acquisition of the peripheral information and determines an event weight. ,
The information processing means performs a process of integrating an event extracted based on the user's biological information and an event that can be regarded as the same among the events extracted based on the peripheral information or an operation related to peripheral information acquisition, and the same An information processing system for processing the peripheral information corresponding to an event according to the weight of the event for each event integrated with an event for which no event can be considered . Recording means for storing biometric information of a user who is a main target person and peripheral information representing the situation of the user and its surroundings, and storing these information,
Event determining means for determining whether an event has occurred based on at least one of the user's biological information and peripheral information;
Information processing means for processing the peripheral information based on the event information extracted by the event determination means,
The peripheral information of the user includes at least a still image obtained by photographing the user or the periphery thereof,
The event determination means determines the time when the number of still images taken per unit time exceeds a predetermined threshold as the occurrence time of the event, and as the number of still images taken per unit time increases, the event Weight so that the weight of
The information processing system, wherein the information processing means processes the peripheral information corresponding to the event according to the weight of the event . Recording means for storing biometric information of a user who is a main target person and peripheral information representing the situation of the user and its surroundings, and storing these information,
Event determining means for determining whether an event has occurred based on at least one of the user's biological information and peripheral information;
Information processing means for processing the peripheral information based on the event information extracted by the event determination means,
The peripheral information of the user includes at least a moving image that captures the user or the periphery thereof,
The event determining means determines the time when the zoom-in operation for moving image shooting is started as the event occurrence time, and the longer the zoom-in duration from the start of the zoom-in operation to the start of the zoom-out operation, the greater the weight of the event Weight so that
The information processing system, wherein the information processing means processes the peripheral information corresponding to the event according to the weight of the event .