The costs and benefits of automated behavior classification

Clara Bird, PhD Student, OSU Department of Fisheries, Wildlife, and Conservation Sciences, Geospatial Ecology of Marine Megafauna Lab

“Why don’t you just automate it?” This is a question I am frequently asked when I tell someone about my work. My thesis involves watching many hours of drone footage of gray whales and meticulously coding behaviors, and there are plenty of days when I have asked myself that very same question. Streamlining my process is certainly appealing and given how wide-spread and effective machine learning methods have become, it is a tempting option to pursue. That said, machine learning is only appropriate for certain research questions and scales, and it’s important to consider these before investing in using a new tool.

The application of machine learning methods to behavioral ecology is called computational ethology (Anderson & Perona, 2014). To identify behaviors from videos, the model tracks individuals across video frames and identifies patterns of movement that form a behavior. This concept is similar to the way we identify a whale as traveling if it’s moving in a straight line and as foraging if it’s swimming in circles within a small area (Mayo & Marx, 1990, check out this blog to learn more). The level of behavioral detail that the model is able to track  depends on the chosen method (Figure 1, Pereira et al., 2020). These methods range from tracking each animal as a simple single point (called a centroid) to tracking the animal’s body positioning in 3D (this method is called pose estimation), which range from providing less detailed to more detailed behavior definitions. For example, tracking an individual as a centroid could be used to classify traveling and foraging behaviors, while pose estimation could identify specific foraging tactics. 

Figure 1. Figure from Pereira et al. (2020) illustrating the different methods of animal behavior tracking that are possible using machine learning.

Pose estimation involves training the machine learning algorithm to track individual anatomical features of an individual (e.g., the head, legs, and tail of a rat), meaning that it can define behaviors in great detail. A behavior state could be defined as a combination of the angle between the tail and the head, and the stride length. 

For example, Mearns et al. (2020) used pose estimation to study how zebrafish larvae in a lab captured their prey. They tracked the tail movements of individual larvae when presented with prey and classified these movements into separate behaviors that allowed them to associate specific behaviors with prey capture (Figure 2). The authors found that these behaviors occurred in a specific sequence, that the behaviors kept the prey within the larvae’s line of sight, and that the sequence was triggered by visual cues.  In fact, when they removed the visual cue of the prey, the larvae terminated the behavior sequence, meaning that the larvae are continually choosing to do each behavior in the sequence, rather than the sequence being one long behavior event that is triggered only by the initial visual cue. This study is a good example of the applicability of machine learning models for questions aimed at kinematics and fine-scale movements. Pose estimation has also been used to study the role of facial expression and body language in rat social communication (Ebbesen & Froemke, 2021). 

Figure 2. Excerpt from figure 1 of Mearns et al. (2020) illustrating (A) the camera set up for their experiment, (B) how the model tracked the eye angles and tail of the larvae fish, (C) the kinematics extracted from the footage. In panel (C) the top plot shows how the eyes converged on the same object (the prey) during prey capture event, the middle plot shows when the tail was curved to the left or the right, and the bottom plot shows the angle of the tail tip relative to the body.

While previous machine learning methods to track animal movements required individuals to be physically marked, the current methods can perform markerless tracking (Pereira et al., 2020). This improvement has broadened the kinds of studies that are possible. For example, Bozek et al., (2021) developed a model that tracked individuals throughout an entire honeybee colony and showed that certain individual behaviors were spatially distributed within the colony (Figure 3). Machine learning enabled the researchers to track over 1000 individual bees over several months, a task that would be infeasible for someone to do by hand. 

Figure 3. Excerpt from figure 1 of Bozek et al., (2021) showing how individual bees and their trajectories were tracked.

These studies highlight that the potential benefits of using machine learning when studying fine scale behaviors (like kinematics) or when tracking large groups of individuals. Furthermore, once it’s trained, the model can process large quantities of data in a standardized way to free up time for the scientists to focus on other tasks.

While machine learning is an exciting and enticing tool, automating behavior detection via machine learning could be its own PhD dissertation. Like most things in life, there are costs and benefits to using this technique. It is a technically difficult tool, and while applications exist to make it more accessible, knowledge of the computer science behind it is necessary to apply it effectively and correctly. Secondly, it can be tedious and time consuming to create a training dataset for the model to “learn” what each behavior looks like, as this step involves manually labeling examples for the model to use. 

As I’ve mentioned in a previous blog, I came quite close to trying to study the kinematics of gray whale foraging behaviors but ultimately decided that counting fluke beats wasn’t necessary to answer my behavioral research questions. It was important to consider the scale of my questions (as described in Allison’s blog) and I think that diving into more fine-scale kinematics questions could be a fascinating follow-up to the questions I’m asking in my PhD. 

For instance, it would be interesting to quantify how gray whales use their flukes for different behavior tactics. Do gray whales in better body condition beat their flukes more frequently while headstanding? Does the size of the fluke affect how efficiently they can perform certain tactics? While these analyses would help quantify the energetic costs of different behaviors in better detail, they aren’t necessary for my broad scale questions. Consequently, taking the time to develop and train a pose estimation machine learning model is not the best use of my time.

That being said, I am interested in applying machine learning methods to a specific subset of my dataset. In social behavior, it is not only useful to quantify the behaviors exhibited by each individual but also the distance between them. For example, the distance between a mom and her calf can be indicative of the calves’ dependence on its mom (Nielsen et al., 2019). However, continuously measuring the distance between two individuals throughout a video is tedious and time intensive, so training a machine learning model could be an effective use of time. I plan to work with an intern this summer to develop a machine learning model to track the distance between pairs of gray whales in our drone footage and then relate this distance data with the manually coded behaviors to examine patterns in social behavior (Figure 4).  Stay tuned to learn more about our progress!

Figure 4. A mom and calf pair surfacing together. Image collected under NOAA/NMFS permit #21678

Did you enjoy this blog? Want to learn more about marine life, research, and conservation? Subscribe to our blog and get a weekly alert when we make a new post! Just add your name into the subscribe box on the left panel.  


Anderson, D. J., & Perona, P. (2014). Toward a Science of Computational Ethology. Neuron84(1), 18–31.

Bozek, K., Hebert, L., Portugal, Y., Mikheyev, A. S., & Stephens, G. J. (2021). Markerless tracking of an entire honey bee colony. Nature Communications12(1), 1733.

Ebbesen, C. L., & Froemke, R. C. (2021). Body language signals for rodent social communication. Current Opinion in Neurobiology68, 91–106.

Mayo, C. A., & Marx, M. K. (1990). Surface foraging behaviour of the North Atlantic right whale, Eubalaena glacialis , and associated zooplankton characteristics. Canadian Journal of Zoology68(10), 2214–2220.

Mearns, D. S., Donovan, J. C., Fernandes, A. M., Semmelhack, J. L., & Baier, H. (2020). Deconstructing Hunting Behavior Reveals a Tightly Coupled Stimulus-Response Loop. Current Biology30(1), 54-69.e9.

Nielsen, M., Sprogis, K., Bejder, L., Madsen, P., & Christiansen, F. (2019). Behavioural development in southern right whale calves. Marine Ecology Progress Series629, 219–234.

Pereira, T. D., Shaevitz, J. W., & Murthy, M. (2020). Quantifying behavior to understand the brain. Nature Neuroscience23(12), 1537–1549.

It Takes a Village to Raise a PhD Student

By Rachel Kaplan, PhD student, Oregon State University College of Earth, Ocean, and Atmospheric Sciences and Department of Fisheries, Wildlife, and Conservation Sciences, Geospatial Ecology of Marine Megafauna Lab

This year in late February is the 2022 Ocean Sciences Meeting, an interdisciplinary bonanza of ocean scientists from all over the world. The conference will be held online this year as a precaution against Covid-19, and a week of virtual talks and poster sessions will cover new research in diverse topics from microbial ecology to ocean technology to whale vocalizations.

The meeting will also include my first poster presentation at a major conference, and so I have the typical grad student jitters that accompany each new thing I do (read more about the common experience of “imposter syndrome” here). This poster is the first time since starting graduate school and joining Project OPAL that I’m trying to craft a full science story that connects whales, their prey, and oceanographic conditions.

Learning how to do the analyses to assess and quantify these connections has involved plenty of head-scratching and periodic frustration on my part, but it has also offered a surprisingly joyful and even moving experience. In my efforts to troubleshoot a problem with my prey analysis, I’ve reached out to nearly everyone who works with krill acoustic data on the West Coast. Every single person has been incredibly welcoming and ready to help me, and excited to learn about my work in return. This experience has made me realize how many people I have on my team, and that even strangers are willing to support me on the whacky journey that is a PhD.

Through these collaborations, I am learning to analyze the acoustic signal of krill, small animals that are important food for whales foraging off the coast of Oregon and beyond. As part of Project OPAL, we plan to compare krill swarms with whale survey data to learn about the types of aggregations that whales are drawn to. From the perspective of a hungry whale, not all krill are created equal.

Analysis of a layer of krill in the upper ocean. The blue color in the top panel indicates scattering of acoustic signal by the krill, and the outline in the bottom panel shows the results of an algorithm programmed to detect krill aggregations.

In addition to developing great remote relationships through this work, the ability to meet in person as we continue adapting to life during the pandemic has absolutely not lost its thrill. After over a year of meetings and collaborating on Zoom, I was delighted to meet GEMM Lab postdoc Solène Derville this January, after she journeyed from her home in New Caledonia to Oregon. It was so exciting to see her in real life (we’re more similar in height than I knew!) and a few minutes into our first lunch together she was already helping me refine my analysis plans and think of new approaches.

Our interaction also made me think about how impressive the GEMM Lab is. The first two people Solène saw upon her arrival in Oregon were me and fellow GEMM Lab student Allison Dawn, two newer members who joined the lab after her last trip to Oregon. Without a moment of hesitation, Allison stepped up to give Solène a ride to Newport from Corvallis to finish her long journey. The connection our lab has developed and maintained during a pandemic, across borders and time zones, is special.

Hiking on gorgeous days is just one of the many benefits of being in the same place! This adventure included spotting a whale blow off the coast and a lot of GEMM excitement.

As I look out at the next few weeks until the Ocean Sciences meeting, and out towards the rest of my PhD, I inevitably feel worried about all I need to accomplish. But, I know that the dynamics in our lab and the other collaborative relationships I’m forming are what will carry me through. Every meeting and new connection reminds me that I’m not doing this alone. I’m grateful that there’s a team of people who are ready and willing to help me muddle my way through my first Principal Components Analysis, puzzle over algorithm errors, and celebrate with me as we make progress.

The benefits of play: A review of cetacean behavior.

Miranda Mayhall, Graduate Student, OSU Department of Fisheries, Wildlife, & Conservation Sciences, Geospatial Ecology of Marine Megafauna Lab.

Beluga whale. Photo credit:

Since coming back from winter holiday, things have picked back up to my normal pace of GO! and I’ve taken little to no “down time” in my awaken hours. As a grad student who is also a mother to an active 11-year-old daughter and two dogs, my days are packed. Although I do enjoy a life of steady movement and accomplishment, I also need to do “nothing” sometimes, like a recluse who needs to see the sun on occasion. So, this evening I decided that I would have a night of fun and I took my daughter to see a movie. We haven’t been to the movies much since the pandemic started, but it is one of our most beloved things to do. I heard the theatres were like ghost towns since the recent omicron surge anyway, so we showed up and were one of two families there. We picked a comedy and ordered a bucket of popcorn, nachos (no jalapeños, just the cheese), slurpies, soft pretzels, and sour patch kids (I told the cashier to have two wheelchairs ready to haul us out of there post feast). We laughed and sang and by the near end of the movie, I had a moment of self-realization: I felt really relaxed. This epiphany was synaptically followed by thinking about how cetaceans engage in play.

Humans often recognize play through sports or games, and mostly through smiling and the vocalization of laughter. If we’re laughing it usually means that we are not aggressing. From what we currently understand, play in cetaceans has evolved as an ontogenetic behavior in many species for the purposes of developing survival skills (Paulos et al., 2010). This “purpose of play” makes a lot of sense, and I see it in my dogs when they are growling, snapping, tugging rope, and chasing each other in the yard. They are having the time of their lives and certainly not really fighting one another, yet they are also clearly practicing important skills if they were to come across predators or prey in the wild.

Two dolphins play-fighting.

Most cetaceans vocalize often, whether in the form of pulsed calls, whistles, screams, songs, clicks or combination calls. The element of play associated with a utilized sound or other behavior opens the door for cetaceans to develop important social relationships among conspecifics, as well as developing crucial survival skills (Paulos et al., 2010). To quantify the vocal signals produced by cetacean species, researchers examine their complex repertoires to understand more about the function of certain sounds made specifically during play (Boisseau, 2004). Bottlenose dolphins provide each other with a distinct signal, pulse whistles that start around 13 kHz and end at around 10 kHz (Fig 1), to tell one another that the behavior they are exhibiting is play rather than aggression (Blomqvist et al., 2005).

Figure 1. Spectrogram of bottlenose dolphin pulse whistles during play. Blomqvist et al., 2005.

Cetacean play is defined as behavior that is spontaneous, intentional, pleasurable, and rewarding (Hill et al., 2017). Although cetacean play is conducted in a relaxed setting when there is no immediate need for survival, it has a role in growth and sociability (Hill et al., 2017). For example, cetaceans participate in interspecies play, where they actively engage with one another for no apparent ecological benefit (excluding periods of symbiotic behavior, such as working together to herd prey). Yet, these periods of interspecies play may suggest that these animals are comfortable practicing for real world situations with one another. Large baleen whales have few predators and thus have opportunities to engage in play with pods of dolphins. In some cases, large baleen whales such as humpback and gray whales will lift smaller mammals out of the water, possibly to practice for maternal care (Hill et al., 2017).

Gray whales swim/interact with white-sided dolphins, playing with one another. Image credit:

Cetaceans engage in play not only with one another, but as solitary individuals as well. This play (which can occur parallel to conspecifics simultaneously) includes surfing, aerial breaches and leaps, slapping the surface of the water with a fin or tail fluke, and erratic swimming (Paulos et al., 2010). Some cetaceans play with objects they find in the wild. One example being bowhead whales, which are known to balance, sink, and lift logs (Paulos et al., 2010).

Another interesting cetacean play behavior is bubble blowing. Though humpback whales blow bubbles as a means of trapping prey while foraging (Moreno & Macgregor, 2019), beluga whales, particularly females, blow mouth ring bubbles and perform blowhole bursts when engaging in solitary play (Hill et al., 2011). Just for the fun of it. It appears that cetaceans also need to be actively involved in “nothing” sometimes, as there is some good use for it. For me, engaging in play is a way to reset and relax, which is necessary even for those us who gain a lot of pleasure from our accomplishments. As I sit in the desolate theatre connecting with my daughter and nurturing my own needs, I feel completely justified in my relaxing night off. Pass the nachos, please.

Beluga mouth ring bubble. Photo credit:

Literature Cited

Blomqvist, C., Mello, I., Amundin, M. 2005. An acoustic play-fight signal in bottlenose dolphins (Tursiops truncatus) in human care. Aquatic Mammals, 31 (2), 187-194. 

Boisseau, O. 2004. Quantifying the acoustic repertoire of a population: The vocalizations of free-ranging bottlenose dolphins in Fiordland, New Zealand. The Journal of the Acoustical Society of America, 117, 2318-2329.

Hill, H., Dietrich, S., Cappiello, B. 2017. Learning to play: A review and theoretical investigation of the development mechanisms and functions of cetacean play. Learning & Behavior, 45, 335-354.

Hill, H., Kahn, M., Brilliott, L., Roberts, B., Gutierrez, C. 2011. Beluga (Delphinaptera leucas) bubble bursts: surprise, protection, or play? International Journal of Comparative Psychology, 24, 235-243.

Moreno, K. & Macgregor, R. 2019. Bubble trails, bursts, rings, and more: A review of multiple bubble types produced by cetaceans. Animal Behavior and Cognition, 6 (2), 105-126.

Paulos, R., Trone, M., Kuczaj II, S. 2010. Play in wild and captive cetaceans. International Journal of Comparative Psychology, 23, 701-722.

Provine, R. 2016. Laughter as an approach to vocal evolution: The bipedal theory. Psychonomic Bulletin & Review, 24, ­238-244.

New year’s hindsight: will it ever be the same?

By Solène Derville, Postdoc, OSU Department of Fisheries, Wildlife, and Conservation Science, Geospatial Ecology of Marine Megafauna Lab

As I sit down at my desk during the first week of 2022 to write the first blog of this new year, more than ever before I feel like I am at a pivotal time. Standing in front of an invisible frontier, contemplating the past, and anxiously looking ahead.

Globally, 2021 was yet another challenging year. The COVID pandemic is persisting in endless waves of contamination and new variants. Climate change is all the more on our minds as the COP26 failed to live up to the expectations of many.

For me personally, 2021 was a very strange year too. I recovered from an accident I had in November 2020 that shook me to the bones and pushed me into living life to its fullest. On the other hand, the pandemic prevented me from moving to Oregon and I have been remotely working on the OPAL project for a year. I feel very lucky to participate in this work and I have enjoyed every bit of time I have spent on my computer processing data and teasing out the ecological drivers of whale distribution in Oregon. Yet, despite the numerous zoom meeting and email exchanges, I have been frustrated by the long-distance relationship I had with my dear GEMM lab colleagues and friends. Like so many others, I have felt the tow of the virtual life the pandemic has imposed on us.

As I reflect on the mixed feelings I am experiencing in this first week of 2022, I realize that the global context we live in and my individual questionings are intertwined. The pandemic and environmental issues triggered the same ethical and philosophical questions about individual responsibility, freedom, and equity. For instance, why should I make sacrifices that will cost me a lot personally but only have a very minor effect on the broader scale? The year 2021 has confronted us with a harsh reality: however strongly you believe your answer to the above question is the right one, other people might think otherwise.

The term eco-anxiety has emerged in recent years to describe people suffering from ‘persistent worries about the future of Earth and the life it shelters’. These symptoms of chronic fear are rising worldwide, which sadly but frankly, is only normal given that the degradation of our climate and biosphere deserves our full attention. More disturbingly, I found out that eco-anxiety is mostly affecting children and young people around the globe. Despite acting for the environment on an everyday basis and working as a conservation biologist, I can relate to this feeling of overwhelming helplessness.

In the first week of this new year, I would like to turn this distress into motivation to act and do better. To that extent, ‘adaptation’ is the word that keeps coming up to my mind. In biology, adaptation is the process of change by which an organism or species becomes better suited to its environment. Contrary to ‘acclimation’ that refers to a temporary change occurring on the short term, adaptation is a more profound evolution occurring at the scale of multiple generations. Somewhat, we need to combine the best of both worlds, adapt profoundly but adapt fast.

As I stayed at my family house in Toulouse (France) during the last couple weeks, I went through my old stuff in the room I occupied as a teenager and found a note book written by a 13 year-old Solène. I smiled at my words “One day, I will become a Biologist so that maybe I can save our beautiful planet, […] it’s the only thing that matters”. I was both impressed by the strength of the conviction I was holding to back then and stunned that I have now reached a place, as an independent adult and early career marine ecologist, where I could actually put these words in action.

So here is my 2022 New Year’s resolution: despite the waves of anxiety that sometimes hit us, let’s keep fighting our battles and trust that we can make this world a better place!

“Sometimes you have the feeling that nothing makes sense anymore, and sometimes it just feels right.”
A picture of myself taken during a research cruise in New Caledonia this summer. We were searching for humpback whales in the Chesterfield archipelago (South Pacific), one of the most remote and pristine reef in the world (Photo credit: Marine Reveilhac, mission MARACAS/IRD/Opération Cétacés/WWF/GouvNC/Parc naturel de la mer de Corail).