JP2018033365A - Information processing device, information processing method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To detect an estrous behavior of a cow from an image of the cow taken with a camera installed in a cowhouse.SOLUTION: An information processing device includes image acquisition means for acquiring moving image data taken with an imaging device, region specification means for extracting one frame image from the moving image data acquired by the image acquisition means, and specifying a region where an object is shown from the extracted frame image, appearance determination means for determining whether or not the object shown in the region specified by the region specification means is an object making a movement to be detected, and movement determination means for determining whether or not the object is making a movement to be detected using the moving image information in the region when it is determined by the determination means that the object is the object making the movement to be detected.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an information processing apparatus, an information processing method, and a program.

  In livestock production (dairy farming and beef cattle breeding), the breeding performance of cattle greatly affects management. In order to improve the reproductive performance, it is important to insemination by detecting the timing when the cow is in estrus. In general, the estrus behavior of cattle includes riding behavior, an increase in the amount of behavior, etc., and the breeder determines the estrus time by observing the behavior of the cattle.

  However, since estrus behavior is often performed during night hours, the breeder may miss it.

  It is desired to improve the breeding performance and strengthen the production base of the livestock industry by constructing an alternative system that assists the breeder's observation.

  Therefore, Patent Document 1 describes a mechanism in which an infrared sensor is attached to the back of the target cow and the presence or absence of riding is determined based on the sensor response and its length.

  Patent Document 2 describes a mechanism for determining estrus by measuring cow roaring and cow behavior by using a microphone and a sensor installed in the barn.

  Conventionally, as a method for detecting the estrus of a cow using a sensor, there are (1) a method for monitoring the number of steps and (2) a method for monitoring an amount of action, which are commercialized.

JP 2016-42843 A Japanese Patent Laying-Open No. 2015-89342

  However, the method of patent document 1 has a problem that it is necessary to attach a sensor to all the cows to be managed, and that the introduction cost and work are troublesome, and that the device attached to the body causes the cow's stress. .

  Moreover, the method of patent document 2 has the problem that a dispersion | variation generate | occur | produces according to the distance of the microphone attached to the cowshed and the object cow. In addition, there is a problem that the amount of action cannot be measured and the estrus cannot be determined unless the estrus target cow is restrained in a specific area of the barn where the sensor is attached.

  In addition, in the method of monitoring the number of steps and the method of monitoring the amount of behavior, the amount of behavior of the cow increases in summer and there is an individual difference in the amount of behavior, so the change in these states is a change due to estrus There is a problem that estrus cannot be found with high accuracy.

  In view of the above problems, an object of the present invention is to provide a mechanism for detecting estrus behavior of a cow from captured moving image data.

  An information processing apparatus according to the present invention extracts an image acquisition unit that acquires moving image data captured by an imaging device, and extracts one frame image from the moving image data acquired by the image acquisition unit, and an object is extracted from the extracted frame image. An area specifying means for specifying a reflected area; an appearance determining means for determining whether an object shown in the area specified by the area specifying means is an object performing a detection target; and Motion determining means for determining whether the object is performing the detection target motion using the moving image information in the area when the means determines that the object is performing the detection target motion. It is characterized by.

  The information processing method of the present invention is an information processing method in an information processing apparatus, wherein an image acquisition unit of the information processing apparatus acquires moving image data captured by an imaging device, and the information processing An area specifying unit of the apparatus extracts one frame image from the moving image data acquired by the image acquisition process, specifies an area in which an object is shown from the extracted frame image, and the information processing apparatus An appearance determination unit that determines whether an object shown in the region specified by the region specifying step is an object performing a detection target operation; and a motion determination unit of the information processing apparatus. If it is determined in the determination step that the object is performing an operation as a detection target, the object performs an operation as a detection target using moving image information in the region. Characterized in that it comprises a determining motion determination step whether the a.

  The program of the present invention is a program that can be executed in an information processing apparatus, and the information processing apparatus is acquired by an image acquisition unit that acquires moving image data captured by an imaging device, and the image acquisition unit. One frame image is extracted from the moving image data, the area specifying means for specifying the area in which the object is shown from the extracted frame image, and the object shown in the area specified by the area specifying means is the operation to be detected If it is determined by the determination means that the object is being detected, the object is detected using the moving image information in the area. A program for functioning as motion determination means for determining whether or not a target motion is being performed.

  According to the present invention, it is possible to efficiently detect the estrus behavior of a cow from captured moving image data.

The figure which shows an example of the system configuration | structure of the information processing system in a present Example Block diagram showing an example of the hardware configuration of the network camera terminal 101, the server apparatus 102, and the client terminal 103 Flow chart explaining the overall image of the cow estrus determination process Flowchart showing the estrus determination process in step S105 Flow chart showing details of estrus determination process based on appearance information The flowchart which shows the detail of the estrus judgment processing by movement information The figure which shows an example of the teacher data of appearance information The schematic diagram which shows the result of having extracted the candidate rectangle about 1 frame (t) in a moving image, and calculating the similarity degree of the image in the said candidate rectangle, and the learned image Schematic diagram showing frame image (t) and frame images before and after the frame image (t) The figure which shows the candidate rectangle (target object area | region) extracted from each frame image

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

  FIG. 1 is a diagram illustrating an example of a system configuration of an information processing system according to the present embodiment.

  The system according to the present invention is a system in which a network camera terminal (imaging device) 101, a server device (information processing device) 102, and a client terminal 103 are communicably connected.

  The cow A in the cowshed is photographed by the network camera terminal 101 installed in the cowshed, and the photographed video data is transmitted to the server apparatus 102. The server apparatus 102 analyzes the video data and determines the presence or absence of the riding action. If it is determined by the analysis process that there is a riding action, the client terminal 103 is notified.

  FIG. 2 is a block diagram illustrating an example of a hardware configuration of the network camera terminal 101, the server apparatus 102, and the client terminal 103 according to the embodiment of the present invention.

  As shown in FIG. 2, in the network camera terminal 101, the server apparatus 102, and the client terminal 103, a CPU (Central Processing Unit) 201, a ROM (Read Only Memory) 202, and a RAM (Random Access Memory) 203 via the system bus 200. , A storage device 204, an input controller 205, an audio input controller 206, a video controller 207, a memory controller 208, and a communication I / F controller 209 are connected.

  The CPU 201 comprehensively controls each device and controller connected to the system bus 200.

  The ROM 202 or the storage device 204 is a control program executed by the CPU 201 such as a basic input / output system (BIOS) or an operating system (OS), a computer-readable program for realizing this information processing method, and various necessary programs. Holds data (including data table).

  The RAM 203 functions as a main memory, work area, and the like for the CPU 201. The CPU 201 implements various operations by loading a program or the like necessary for executing the processing from the ROM 202 or the storage device 204 to the RAM 203 and executing the loaded program.

  The input controller 205 controls input from an input device such as a keyboard / touch panel 210. The input device is not limited to this, and may be a touch panel capable of detecting a position touched by a plurality of fingers, such as a mouse or a multi-touch screen.

  Various instructions can be performed when the user presses (touches with a finger or the like) in accordance with an icon, a cursor, or a button displayed on the touch panel.

  This input device is used to input a destination for a communication destination that can be used by various communication devices.

  The voice input controller 206 controls input from the microphone 211. Voice input from the microphone 211 can be recognized.

  The video controller 207 controls display on an external output device such as the display 212. The display includes a display of a notebook computer integrated with the main body. The external output device is not limited to a display, and may be a projector, for example. Further, for devices that can be received by the touch operation described above, it is also possible to receive input from the keyboard / touch panel 210.

  Note that the video controller 207 can control a video memory (VRAM) for display control, and a part of the RAM 203 can be used as a video memory area, or a dedicated video memory can be provided separately. Is possible.

  In the present invention, the first video memory area used for display when the user normally uses the information processing apparatus and the display overlaid on the display content of the first video memory area when a predetermined screen is displayed. A second video memory area used in The number of video memory areas is not limited to two, and a plurality of video memory areas can be provided as long as the resources of the information processing apparatus permit.

  The memory controller 208 controls access to the external memory 213. The external memory is connected via an adapter to an external storage device (hard disk), flexible disk (FD), or PCMCIA card slot that stores boot programs, various applications, font data, user files, editing files, and various data. A compact flash (registered trademark) memory or the like can be used.

  The communication I / F controller 209 is connected to and communicates with an external device via the network 214, and executes communication control processing in the network. For example, communication using TCP / IP, telephone lines such as ISDN, and communication using 3G lines of mobile phones are possible.

  The storage device 204 is a medium for permanently storing information, and the form is not limited to a storage device such as a hard disk. For example, a medium such as SSD (Solid State Drive) may be used.

  It can also be used as a temporary memory area during various processes performed by the communication terminal in the present embodiment.

  Next, the overall image of the cow estrus determination process in this embodiment will be described with reference to the flowchart shown in FIG.

  In step S101, it is determined whether or not the CPU 201 of the network camera terminal 101 installed in the barn is ready for photographing.

  If photographing is possible (step S101: YES), the process proceeds to step S102.

  If the photographing is impossible (step S101: NO), the process of this flowchart is terminated.

  In step S102, the CPU 201 of the network camera terminal 101 acquires one frame image of the captured video.

  In step S <b> 103, the CPU 201 of the network camera terminal 101 transfers the one frame image acquired in step S <b> 102 to the server apparatus 102.

  In step S104, the CPU 201 of the server apparatus 102 receives the one-frame image transmitted from the network camera terminal 101 in step S103.

  The moving image data captured by the network camera terminal 101 may be transmitted to the server device, and one frame image may be acquired from the moving image data received by the server device.

  In step S105, the CPU 201 of the server apparatus 102 performs an estrus determination process using the image received in step S104. Details of the estrus determination process will be described with reference to the flowcharts of FIGS.

  In step S106, the CPU 201 of the server apparatus 102 determines whether or not the estrus determination flag is True by the estrus determination process in step S105.

  If it is True (step S106: YES), the process proceeds to step S107.

  If it is False (step S106: NO), the process returns to step S101.

  In step S107, the CPU 201 of the server apparatus 102 notifies the client terminal 103 that there is an estrus cow.

  Next, the estrus determination process in step S105 will be described using the flowcharts of FIGS.

  In step S201, the CPU 201 of the server apparatus 102 executes estrus determination processing based on appearance information (still image).

  Details of the estrus determination process based on appearance information will be described with reference to the flowchart of FIG.

  FIG. 5 is a flowchart showing estrus determination processing based on appearance information in step S201.

  In step S <b> 301, the CPU 201 of the server apparatus 102 reads learned feature amount weight data.

  For example, a plurality of images of cows having an estrus behavior (for example, riding behavior) as shown in FIG. 7 (that is, images showing actions and actions of detection targets) are learned, and feature amounts are obtained from these images. Calculate it.

  In step S <b> 302, the CPU 201 of the server apparatus 102 extracts a region where an object is shown as a candidate rectangle from the processing target image (the image received in step S <b> 104). The extraction of the candidate rectangle can be realized by using a known technique such as “Faster R-CNN”.

  In step S303, the CPU 201 of the server apparatus 102 determines whether there is a candidate rectangle extracted by the process in step S302.

  If it exists (step S303: YES), the process proceeds to step S304.

  If it does not exist (step S303: NO), the processing shown in this flowchart is terminated.

  In the case after step S310 is executed, it is a process for determining whether or not there is a candidate rectangle for which the processes from steps S303 to S310 have not been executed. That is, in the present embodiment, the processes in steps S303 to S310 are repeatedly executed for each candidate rectangle extracted in step S302.

  In step S304, the CPU 201 of the server apparatus 102 calculates an image feature amount in the candidate rectangle to be processed.

  In step S305, the CPU 201 of the server apparatus 102 compares the feature amount calculated in step S304 with the feature amount of the learned image read in step S301.

  FIG. 8 is a schematic diagram showing a result of extracting a candidate rectangle for one frame (t) in a moving image and calculating a similarity between an image in the candidate rectangle and a learned image.

  In the example of FIG. 8, the coordinates of the candidate rectangle in the frame (t) are (x1, y1, x2, y2), and the similarity is 0.9.

  In step S306, the CPU 201 of the server apparatus 102 determines whether the similarity of the feature amount is equal to or greater than a threshold value as a result of the comparison in step S305. That is, it is determined whether the object in the candidate rectangle is a cow performing estrus behavior (whether it is an object performing the detection target operation).

  If it is equal to or greater than the threshold (step S306: YES), the process proceeds to step S307.

  If the threshold value is not reached (step S306: NO), the process proceeds to step S309.

  In step S307, the CPU 201 of the server apparatus 102 sets the appearance determination flag to True.

  In step S308, the CPU 201 of the server apparatus 102 holds the region information of the candidate rectangle to be processed. The area information is information such as coordinate information (x1, y1, x2, y2), for example.

  In step S309, the CPU 201 of the server apparatus 102 sets the appearance determination flag to False.

  In step S310, CPU201 of server apparatus 102 acquires a candidate rectangle which has not performed processing of S303-S310 among candidate rectangles extracted at step S302.

  When the processes of S303 to S310 are executed for all candidate rectangles extracted in step S302 (step S303: NO), the process of this flowchart is terminated, and the process proceeds to step S202.

  In step S202, the CPU 201 of the server apparatus 102 determines whether True is set in the appearance determination flag.

  If True is set (step S202: YES), the process proceeds to step S203.

  If False is set (step S202: NO), the process proceeds to step S207.

  In step S203, the CPU 201 of the server apparatus 102 acquires the area information held in step S308.

  In step S204, the CPU 201 of the server apparatus 102 executes an estrus determination process using motion information (moving image).

  Details of the estrus determination process based on motion information will be described with reference to the flowchart of FIG.

  FIG. 6 is a flowchart showing the estrus determination process based on the motion information in step S204.

  In step S401, the CPU 201 of the server apparatus 102 reads information on a target object area (an area specified by the coordinate information held in step S308).

  In step S402, the CPU 201 of the server apparatus 102 determines the presence / absence of area information. If there is no area information, the process proceeds to step S411. If there is area information, the process proceeds to step S403.

  In step S403, the CPU 201 of the server apparatus 102 determines whether there is a frame image before the processing target frame image.

  If it exists (step S403: YES), the process proceeds to step S404.

  If it does not exist (step S403: NO), the process proceeds to step S411.

  In step S404, the CPU 201 of the server apparatus 102 acquires an image of the target object area from the previous frame. That is, an area specified by the same coordinate information as the coordinate information held in step S308 is acquired from the image of the previous frame.

  In step S405, the CPU 201 of the server apparatus 102 acquires an image of the target object region from the processing target frame image.

  In step S406, the CPU 201 of the server apparatus 102 calculates an optical flow feature amount from the two images acquired in steps S404 and S405.

  As a method for calculating the OpticalFlow feature value, there are some known techniques, but any technique may be used. In this embodiment, the OpticalFlow feature value is used for calculating the feature value of the motion information. However, the method for calculating the feature value of the motion information is not limited to the method using the OpticalFlow, and any method may be used. .

  In step S407, the CPU 201 of the server apparatus 102 calculates the feature amount of the motion information by dividing the OpticalFlow feature amount calculated in step S406 by the size of the target object region.

  In step S408, the CPU 201 of the server apparatus 102 determines whether the feature amount of the motion information calculated in step S407 is greater than or equal to a threshold value. That is, it is determined based on the movement of the cow whether or not it is performing estrus behavior (whether or not the movement is a movement to be detected based on the feature amount calculated from the movement).

  If it is equal to or greater than the threshold (step S408: YES), the process proceeds to step S409.

  If the threshold is not satisfied (step S408: NO), the process proceeds to step S410.

  In step S409, the CPU 201 of the server apparatus 102 sets True to the motion determination flag.

  In step S410, the CPU 201 of the server apparatus 102 acquires the next target object area, and proceeds to processing for the target object area. Then, the process shown in this flowchart is terminated, and the process returns to step S205.

  In step S411, the CPU 201 of the server apparatus 102 sets False to the motion determination flag. Then, the process shown in this flowchart is terminated, and the process returns to step S205.

  In step S205, the CPU 201 of the server apparatus 102 determines whether or not True is set in the motion determination flag as a result of the processing in step S204. That is, it is determined whether or not an estrus action is detected by an estrus action determination process using motion information.

  If the motion determination flag is True (step S205: YES), the process proceeds to step S206.

  If the motion determination flag is False (step S205: NO), the process proceeds to step S207.

  In step S206, the CPU 201 of the server apparatus 102 sets the estrus determination flag to True since it is determined that there is an estrus cow by the processing of steps S201 and S204.

  In step S207, the CPU 201 of the server apparatus 102 sets the estrus determination flag to False because the cow in estrus could not be detected by the processing in steps S201 and S204.

  In step S208, the CPU 201 of the server apparatus 102 holds the processing target frame image. The frame image held in this step is used in the process for calculating the motion feature amount in the process for the next frame image.

  FIG. 9 is a schematic diagram showing a frame image (t) and frame images before and after the frame image (t). FIG. 10 is a diagram showing candidate rectangles (target object regions) extracted from each frame image in FIG.

  As described above, the present invention first detects whether or not an estrus act is performed based on appearance information (still image), identifies an area where a cow performing the act is shown, and identifies the identified area. It is characterized by determining whether the estrus act is performed by motion information (moving image).

  In this way, by using not only the appearance information but also the movement information, it is difficult to determine whether or not it is an estrus act only by the appearance information (for example, when only a riding behavior or two cows are Even when standing on top of each other), it becomes possible to make a more accurate determination.

  In addition, in the case of only moving image information, since the processing target data size is large, it takes a long time for the determination process, but the area where the cow performing estrus behavior is shown by the still image is specified, and the specified area is On the other hand, by performing determination processing using motion information, processing for a small data size can be performed, so that processing time can be shortened.

  Moreover, since the presence or absence of the cow's riding behavior is discovered without attaching a special sensor to the cow's body, it is possible to reduce the cost, the labor of attachment, and the stress on the cow. In addition, since the estrus behavior can be detected wherever a cow can be photographed with a camera, there is no restriction on the whereabouts of the cow. In addition, since it is determined whether the estrus is caused by a riding action or the like, it is possible to determine the estrus state with higher accuracy than the determination based on the number of steps or the amount of action.

  Although the present embodiment has been described on the assumption that it is applied to detection of estrus behavior of cattle, the application target of the present invention is not limited to detection of estrus behavior of cattle, and a mechanism for detecting specific movement from video data As such, it can be applied to various objects.

  The program in the present invention is a program that causes a computer to execute the processes of FIGS. The program in the present invention may be a program for each process in FIGS.

  As described above, a recording medium that records a program that implements the functions of the above-described embodiments is supplied to a system or apparatus, and a computer (or CPU or MPU) of the system or apparatus stores the program stored in the recording medium. It goes without saying that the object of the present invention can also be achieved by reading and executing.

  In this case, the program itself read from the recording medium realizes the novel function of the present invention, and the recording medium recording the program constitutes the present invention.

  As a recording medium for supplying the program, for example, a flexible disk, hard disk, optical disk, magneto-optical disk, CD-ROM, CD-R, DVD-ROM, magnetic tape, nonvolatile memory card, ROM, EEPROM, silicon A disk or the like can be used.

  Further, by executing the program read by the computer, not only the functions of the above-described embodiments are realized, but also an OS (operating system) operating on the computer based on an instruction of the program is actually It goes without saying that a case where the function of the above-described embodiment is realized by performing part or all of the processing and the processing is included.

  Furthermore, after the program read from the recording medium is written to the memory provided in the function expansion board inserted into the computer or the function expansion unit connected to the computer, the function expansion board is based on the instructions of the program code. It goes without saying that the case where the CPU or the like provided in the function expansion unit performs part or all of the actual processing and the functions of the above-described embodiments are realized by the processing.

  The present invention may be applied to a system constituted by a plurality of devices or an apparatus constituted by a single device. Needless to say, the present invention can be applied to a case where the present invention is achieved by supplying a program to a system or apparatus. In this case, by reading a recording medium storing a program for achieving the present invention into the system or apparatus, the system or apparatus can enjoy the effects of the present invention.

  Furthermore, by downloading and reading a program for achieving the present invention from a server, database, etc. on a network using a communication program, the system or apparatus can enjoy the effects of the present invention. In addition, all the structures which combined each embodiment mentioned above and its modification are also included in this invention.

101 network camera terminal 102 server apparatus 103 client terminal

Claims (5)

Image acquisition means for acquiring moving image data shot by the imaging device;
A region specifying unit for extracting one frame image from the moving image data acquired by the image acquiring unit, and specifying a region in which an object is shown from the extracted frame image;
Appearance determining means for determining whether an object shown in the area specified by the area specifying means is an object performing a detection target operation;
When it is determined by the determination means that the object is performing the detection target motion, using the moving image information in the region, the motion determination means for determining whether the object is performing the detection target motion;
An information processing apparatus comprising: An extraction means for extracting an area corresponding to the specified area in a frame before the extracted frame;
The movement determination unit determines whether the object is performing a detection target operation by using the region extracted by the extraction unit and the image in the region specified by the region specification unit. The information processing apparatus according to claim 1. A notification means for notifying that the motion of the detection target is being performed when the motion determination means determines that the object is performing the motion of the detection target;
The information processing apparatus according to claim 1, further comprising: An information processing method in an information processing apparatus,
An image acquisition step in which the image acquisition means of the information processing apparatus acquires moving image data captured by the imaging apparatus;
A region specifying step in which the region specifying unit of the information processing apparatus extracts one frame image from the moving image data acquired by the image acquiring step, and specifies a region in which an object is reflected from the extracted frame image;
An appearance determination step in which the appearance determination unit of the information processing apparatus determines whether the object shown in the region specified by the region specification step is an object performing a detection target operation;
When the movement determination unit of the information processing apparatus determines that the object to be detected is moving in the determination step, the object performs the detection target movement using moving image information in the region. A motion determination step of determining whether or not
An information processing method comprising: A program executable in the information processing apparatus,
The information processing apparatus;
Image acquisition means for acquiring moving image data shot by the imaging device;
A region specifying unit for extracting one frame image from the moving image data acquired by the image acquiring unit, and specifying a region in which an object is shown from the extracted frame image;
Appearance determining means for determining whether an object shown in the area specified by the area specifying means is an object performing a detection target operation;
When it is determined by the determination means that the object is performing the motion of the detection target, the moving image information in the area is used to function as a motion determination means for determining whether the object is performing the motion of the detection target. Program for.

Images