It’s hard to believe that it’s already been four and half months since we started the field season (check out Lisa’s blog for a recap of where we began), but as of this weekend the GRANITE project’s 8th field season has officially ended! As the gray whales wrap up their foraging season and start heading south for the winter, it’s time for us to put our gear into storage, settle into a new academic year, and start processing the data we spent so much time collecting.
The field season can be quite an intense time (40 days equaling over 255 hours on the water!), so we often don’t take a moment to reflect until the end. But this season has been nothing short of remarkable. As you may remember from past blogs, the past couple years (2020-21) have been a bit concerning, with lower whale numbers than previously observed. Since many of us started working on the project during this time, most of us were expecting another similar season. But we were wrong in the best way. From the very first day, we saw more whales than in previous years and we identified whales from our catalog that we hadn’t seen in several years.
We identified friends – old and new!
This season we had 224 sightings of 63 individual whales. Of those 63, 51 were whales from our catalog (meaning we have seen them in a previous season). Of these 51 known whales, we only saw 20 of them last year! This observation brings up interesting questions such as, where did most of these whales forage last year? Why did they return to this area this year? And, the classic end of season question, what’s going to happen next year?
We also identified 12 whales that were not in our catalog, making them new to the GEMM lab. Two of our new whales are extra exciting because they are not just new to us but new to the population; we saw two calves this year! We were fortunate enough to observe two mom-calf pairs in July. One pair was of a “new” mom in our catalog and her calf. We nicknamed this calf “Roly-poly” because when we found this mom-calf pair, we recorded some incredible drone footage of “roly-poly” continuously performing body rolls while their mom was feeding nearby (video 1).
The other pair includes a known GEMM lab whale, Luna, and her calf (currently nicknamed “Lunita”). We recently found “Lunita” feeding on their own in early October (Image 2), meaning that they are now independent from its mom (for more on mom-calf behavior check out Celest’s recent blog). We’ll definitely be on the lookout for Roly-Poly and Lunita next year!
We flew, we scooped, we collected heaps of data!
From our previous blogs you probably know that in addition to photo-ID images, our other two most important forms of data collection are drone flights (for body condition and behavior data) and fecal samples (for hormone analysis). And this season was a success for both!
We conducted 124 flights over 49 individual whales. The star of these flights was a local favorite Scarlett who we flew over 18 different times. These repeat samples are crucial data for us because we use them to gain insight into how an individual’s body condition changes throughout the season. We also recorded loads of behavior data, collecting footage of different foraging tactics like headstanding, side-swimming, and surfacing feeding on porcelain crab larvae (video 2)!
We also collected 61 fecal samples from 26 individual whales (Image 3). The stars of that dataset were Soléand Peak who tied with 7 samples each. These hard-earned samples provide invaluable insight into the physiology and stress levels of these individuals and are a crucial dataset for the project.
On top of all that amazing data collection we also collected acoustic data with our hydrophones, prey data from net tows, and biologging data from our tagging efforts. Our hydrophones were in the water all summer recording the sounds that the whales are exposed to, and they were successfully recovered just a few weeks ago (Image 4)! We also conducted 69 net tows to sample the prey near where the whales were feeding and identify which prey the whales might be eating (Image 5). Lastly, we had two very successful tagging weeks during which we deployed (and recovered!) a total of 9 tags, which collected over 30 hours of data (Image 6; check out Kate’s blog for more on that).
All in all, it’s been an incredible season. We’ve seen the return of old friends, collected lots of awesome data, and had some record-breaking days (28 whales in one day!). As we look toward the analysis phase of the year, we’re excited to dig into our eight-year dataset and work to understand what might explain the increase in whales this year.
To end on a personal note, looking through photos to put in this blog was the loveliest trip down memory lane (even though it only ended a few days ago) – I am so honored and proud to be a part of this team. The work we do is hard; we spend long hours on a small boat together and it can be a bit grueling at times. But, when I think back on this season, my first thoughts are not of the times I felt exhausted or grumpy, but of all the joy we felt together. From the incredible whale encounters to the revitalizing snacks to the off-key sing alongs, there is no other team I would rather do this work with, and I so look forward to seeing what next season brings. Stay tuned for more updates from team GRANITE!
Did you enjoy this blog? Want to learn more about marine life, research, and conservation? Subscribe to our blog and get a weekly alert when we make a new post! Just add your name into the subscribe box below!
Hello! My name is Kate Colson and I am a master’s student at the University of British Columbia, co-supervised by Dr. Andrew Trites of the Marine Mammal Research Unit and Dr. Leigh Torres of the GEMM Lab. As part of my thesis work, I have had the opportunity to spend the summer field season with Leigh and the GEMM Lab team.
For my master’s I am studying the foraging energetics of Pacific Coast Feeding Group (PCFG) gray whales as part of the much larger Gray whale Response to Ambient Noise Informed by Technology and Ecology (GRANITE) project. Quantifying the energy expenditure of PCFG gray whales during foraging can help establish a baseline for how disturbance impacts the ability of this unique population to meet their energy needs. Additionally, determining how many calories are burned during different PCFG foraging behaviors might help explain why some gray whales are in better body condition than others.
To understand how much energy different PCFG foraging behaviors cost, I am using data from suction cup tags we have temporarily applied on PCFG gray whales (Figure 1). You can read more about the why the GEMM Lab started using these tags in an earlier blog here. What I want to talk about in this blog is how exactly we can use this tag data to estimate energy expenditure of PCFG gray whales.
The suction cups tags used in this project have many data sensors that are useful for describing the movement of the tagged whale including accelerometers, magnetometers, gyroscopes, and pressure sensors, and all are sampling at high frequencies. For example, the accelerometer is taking 400 measurements per second! The accelerometer, magnetometer, and gyroscope take measurements in 3 dimensions along the X, Y, and Z-axes. The whale’s movement around the X-axis indicates roll (if the whale is swimming on its side), while movement around the Y-axis indicates pitch (if the whales head is oriented towards the surface or the sea floor). Changes in the whale’s movement around the Z-axis indicates if the whale is changing its swimming direction. Together, all of these sensors can describe the dive profile, body orientation, fluking behavior, and fine-scale body movements of the animal down to the second (Figure 2). This allows for the behavior of the tagged whale to be specifically described for the entirety of the tag deployment.
Although these suction cup tags are a great advancement in collecting fine-scale data, they do not have a sensor that actually measures the whale’s metabolism, or rate of calories burned by the whale. Thus, to use this fine-scale tag data as an estimate for energy expenditure, a summary metric must be calculated from the data and used as a proxy. The most common metric found in the literature is Overall Dynamic Body Acceleration (ODBA) and many papers have been published discussing the pros and cons of using ODBA as a proxy for energy expenditure (Brown et al., 2013; Gleiss et al., 2011; Halsey, 2017; Halsey et al., 2011; Wilson et al., 2020). The theory behind ODBA is that because an animal’s metabolic rate is primarily comprised of movement costs, then measuring the acceleration of the body is an effective way of determining energy expenditure. This theory might seem very abstract, but if you have ever worn a Fitbit or similar fitness tracking device to estimate how many calories you’ve burned during a workout, the same principle applies. Those fitness devices use accelerometers and other sensors, to measure the movement of your limbs and produce estimates of energy used.
So now that we’ve established that the goal of my research is to essentially use these suction cup tags as Fitbits for PCFG gray whales, let’s look at how accelerometry data has been used to detect foraging behavior in large whales so far. Many accelerometry tagging studies have used rorquals as a focal species (see Shadwick et al. (2019) for a review). Well-known rorqual species include humpback, fin, and blue whales. These species forage by using lunges to bulk feed on dense prey patches in the water column. Foraging lunges are indicated by isolated periods of high acceleration that are easily detectable in the tag data (Figure 3; Cade et al., 2016; Izadi et al., 2022).
However, gray whales feed very differently from rorquals. Gray whales primarily suction feed on the benthos, using their head to dig into the sediment and filter prey out of the mud using their baleen. Yet, PCFG gray whales often perform many other foraging behaviors such as headstanding and side-swimming (Torres et al., 2018). Additionally, PCFG gray whales tend to feed in water depths that are often shallower than their body length. This shallow depth makes it difficult to isolate signals of foraging in the accelerometry data from random variation in the data and separate the tag data into periods of foraging behaviors (Figure 4).
But there is still hope! Thanks to the GEMM Lab’s previous work describing the foraging behavior of the PCFG sub-group using drone footage, and the video footage available from the suction cup tags deployed on PCFG gray whales, the body orientation calculated from the tag data can be a useful indication of foraging. Specifically, high body roll is apparent in many foraging behaviors known to be used by the PCFG, and when the tag data indicates that the PCFG gray whale is rolled onto its sides, lots of sediment (and sometimes even swarms of mysid prey) is seen in the tag video footage. Therefore, I am busy isolating these high roll events in the collected tag data to identify specific foraging events.
My next steps after isolating all the roll events will be to use other variables such as duration of the roll event and body pitch (i.e., if the whales head is angled down), to define different foraging behaviors present in the tag data. Then, I will use the accelerometry data to quantify the energetic cost of performing these behaviors, perhaps using ODBA. Hopefully when I visit the GEMM Lab again next summer, I will be ready to share which foraging behavior leads to PCFG gray whales burning the most calories!
Brown, D. D., Kays, R., Wikelski, M., Wilson, R., & Klimley, A. P. (2013). Observing the unwatchable through acceleration logging of animal behavior. Animal Biotelemetry, 1(1), 1–16. https://doi.org/10.1186/2050-3385-1-20
Cade, D. E., Friedlaender, A. S., Calambokidis, J., & Goldbogen, J. A. (2016). Kinematic diversity in rorqual whale feeding mechanisms. Current Biology, 26(19), 2617–2624. https://doi.org/10.1016/j.cub.2016.07.037
Gleiss, A. C., Wilson, R. P., & Shepard, E. L. C. (2011). Making overall dynamic body acceleration work: On the theory of acceleration as a proxy for energy expenditure. Methods in Ecology and Evolution, 2(1), 23–33. https://doi.org/10.1111/j.2041-210X.2010.00057.x
Halsey, L. G. (2017). Relationships grow with time: A note of caution about energy expenditure-proxy correlations, focussing on accelerometry as an example. Functional Ecology, 31(6), 1176–1183. https://doi.org/10.1111/1365-2435.12822
Halsey, L. G., Shepard, E. L. C., & Wilson, R. P. (2011). Assessing the development and application of the accelerometry technique for estimating energy expenditure. Comparative Biochemistry and Physiology – A Molecular and Integrative Physiology, 158(3), 305–314. https://doi.org/10.1016/j.cbpa.2010.09.002
Izadi, S., Aguilar de Soto, N., Constantine, R., & Johnson, M. (2022). Feeding tactics of resident Bryde’s whales in New Zealand. Marine Mammal Science, 1–14. https://doi.org/10.1111/mms.12918
Shadwick, R. E., Potvin, J., & Goldbogen, J. A. (2019). Lunge feeding in rorqual whales. Physiology, 34, 409–418. https://doi.org/10.1152/physiol.00010.2019
Torres, L. G., Nieukirk, S. L., Lemos, L., & Chandler, T. E. (2018). Drone up! Quantifying whale behavior from a new perspective improves observational capacity. Frontiers in Marine Science, 5, 1–14. https://doi.org/10.3389/fmars.2018.00319
Wilson, R. P., Börger, L., Holton, M. D., Scantlebury, D. M., Gómez-Laich, A., Quintana, F., Rosell, F., Graf, P. M., Williams, H., Gunner, R., Hopkins, L., Marks, N., Geraldi, N. R., Duarte, C. M., Scott, R., Strano, M. S., Robotka, H., Eizaguirre, C., Fahlman, A., & Shepard, E. L. C. (2020). Estimates for energy expenditure in free-living animals using acceleration proxies: A reappraisal. Journal of Animal Ecology, 89(1), 161–172. https://doi.org/10.1111/1365-2656.13040
Celest Sorrentino, University of California, Santa Barbara, Department of Ecological, Evolution, and Marine Biology, GEMM Lab NSF REU intern
Are you thinking “Did anyone proofread this blog beforehand? Don’t they know how to spell SLEEP?” I completely understand this concern, but not to fear: the spelling of SLEAP is intentional! We’ll address that clickbait in just a moment.
My name is Celest Sorrentino, a first-generation Latina undergrad who leaped at the opportunity to depart from the beaches of Santa Barbara, California to misty Newport, Oregon to learn and grow as a scientist under the influential guidance of Clara Bird, Dr. Leigh Torres and the powerhouse otherwise known as the GEMM lab. As a recent NSF REU (Research Experience for Undergraduates) intern in the GEMM Lab at Oregon State University, I am thrilled to have the chance to finally let you in on the project Clara, Leigh and I have been working on all summer. Ready for this?
Our project uses a deep-learning platform called SLEAP A.I. ( https://sleap.ai/) that can predict and track multiple animals in video to track gray whale mother calf pairs in drone footage. We also took this project a step further and explored how the distance between a gray whale mother and her calf, a proxy for calf independence, varied throughout the season and by different calf characteristics.
In this story, we’ve got a little bit for everyone: the dynamic duo of computer vision and machine learning for my data scientist friends, and ecological inquest for my cetacean researcher friends.
About the Author
Before we begin, I’d like to share that I am not a data scientist. I’ve only ever taken one coding class. I also do not have years of gray whale expertise under my belt (not yet at least!). I’m entering my 5th year at University of California, Santa Barbara as a double major in Ecology and Evolution (BS) as well as Italian Studies (BA). I am sharing this information to convey the feasibility of learning how to use machine-learning as a solution to streamline the laborious task of video analysis, which would permit more time towards answering your own ecological question, as we did here.
Hundreds of Hours of Drone footage
Since 2016, the GEMM Lab has been collecting drone footage of gray whales off the Oregon Coast to observe gray whale behavior in more detail (Torres et al. 2018). Drones have been shown to increase observational time of gray whales by a three-fold (Torres et al. 2018), including the opportunity to revisit the video with fresh eyes at any time one pleases. The GEMM Lab has flow over 500 flights in the past 6 years, including limited footage of gray whale mother-calf pairs. Little is known about gray whale mother-calf dynamics and even less about factors that influence calf development. As we cannot interview hundreds of gray-whale mother-calf pairs to develop a baseline for this information, we explore potential proxies for calf development instead (similar to how developmental benchmarks are used for human growth).
Distance and Development
During our own life journey, each of us became less and less dependent on our parents to survive on our own. Formulating our first words so that we can talk for ourselves, cracking an egg for our parents so that we can one day cook for ourselves, or even letting go of their hand when crossing the street. For humans, we spend many years with our kin preparing for these moments, but gray whale mother-calf pairs only have a few months after birth until they separate. Gray whale calves are born on their wintering grounds in Baja California, Mexico (around February), migrate north with their mothers to the foraging grounds, and are then weaned during the foraging season (we think around August). This short time with their mother means that they have to become independent pretty quickly (about 6 months!).
Distance between mother and calf can be considered a measure of independence because we would expect increased distance between the pair as calf independence increases. In a study by Nielson et al (2019), distance between Southern Right Whale mother-calf pairs was found to increase as the calf grew, indicating that it can serve as a good proxy for independence. The moment a mother-calf pair separates has not been documented, but the GEMM lab has footage of calves during the foraging season pre-weaning that can be used to investigate this process. However, video analysis is no easy feat: video analysis can range from post-processing, diligent evaluation, and video documentation (Torres et al. 2018). Although the use of UAS has become a popular method for many researchers, the extensive time required for video analysis is a limitation. As mentioned in Clara’s blog, the choice to pursue different avenues to streamline this process, such as automation through machine learning, is highly dependent on the purpose and the kind of questions a project intends to answer.
In a world where modern technology is constantly evolving to cater towards making everyday tasks easier, machine learning leads the charge with its ability for a machine to perform human tasks. Deep learning is a subset of machine learning in which the model learns and adapts the ability to perform a task given a dataset. SLEAP (Social LEAP Estimation of Animal Poses) A.I. is an open-source deep-learning framework created to be able to track multiple subjects, specifically animals, throughout a variety of environmental conditions and social dynamics. In previous cases, SLEAP has tracked animals with distinct morphologies and conditions such as mice interactions, fruit flies engaging in courtship, and bee behavior in a petri dish (Pereira 2020). While these studies show that SLEAP could help make video analysis more efficient, these experiments were all conducted on small animals and in controlled environments. However, large megafauna, such as gray whales, cannot be cultivated and observed in a controlled Petri dish. Could SLEAP learn and adapt to predict and track gray whales in an uncontrolled environment, where conditions are never the same (ocean visibility, sunlight, obstructions)?
In order to establish a model within SLEAP, we split our mother-calf drone video dataset into training (n=9) and unseen/testing (n=3) videos. Training involves teaching the model to recognize gray whales, and necessitated me to label every four frames using the following labels (anatomical features): rostrum, blowhole, dorsal, dorsal-knuckle, and tail (Fig. 1). Once SLEAP was trained and able to successfully detect gray whales, we ran the model on unseen video. The purpose of using unseen video was to evaluate whether the model could adapt and perform on video it had never seen before, eliminating the need for a labeler to retrain it.
We then extracted the pixel coordinates for the mom and calf, calculated the distance between their respective dorsal knuckles, and converted the distance to meters using photogrammetry (see KC’s blog for a great explanation of these methods). The distance between each pair was then summarized on a daily scale as the average distance and the standard deviation. Standard deviation was explored to understand how variable the distance between mother-calf pair was throughout the day. We then looked at how distance and the standard deviation of distance varied by day of year, calf Total Length (TL), and calf Body Area Index (BAI; a measure of body condition). We hypothesized that these three metrics may be drivers of calf independence (i.e., as the calf gets longer or fatter it becomes more independent from its mother).
SLEAP A.I. was able to successfully detect and track gray whale mother-calf pairs across all videos (that’s a total of 1318 frames!). When evaluating how the average distance changed across Day of Year, calf Total length, and calf BAI, the plots did not demonstrate the positive relationship we anticipated (Fig 2A). However, when evaluating the standard deviation of distance across Day of Year, calf Total Length, and calf BAI, we did notice that there does appear to be an increase in variability of distance with an increase in Day of Year and calf Total length (Fig 2B)
These results are monumental! We demonstrated the feasibility to use AI to create a model that can track gray whale pairs in drone footage, which is a fantastic tool that can be applied to updated datasets in the future. As more footage of gray whale mother-calf pairs are collected, this video can be quickly uploaded to SLEAP for model evaluation, predictions can be exported, and results subsequently included in the distance analysis to update our plots and increase our understanding. Our data currently provide a preliminary understanding of how the distance between mother-calf pairs changes with Day of Year, Total length, and BAI, but we are now able to continue updating our dataset as we collect more drone footage.
I suppose you can say I did mislead you a bit with my title, as I have lost some SLEEP recently. But, not over video analysis per say but rather in the form of inspiration. Inspiration toward expanding my understanding of machine learning so that it can be applied toward answering pressing ecological questions. This project has only propelled me to dig my heels in and investigate further the potential of machine learning to analyze dense datasets for huge knowledge gains.
This project was made possible in partnership by the continuous support by Clara Bird, Dr. Leigh Torres, KC Bierlich, and the entire GEMM Lab!
Nielsen, M., Sprogis, K., Bejder, L., Madsen, P., & Christiansen, F. (2019). Behavioural development in southern right whale calves. Marine Ecology Progress Series, 629, 219–234. https://doi.org/10.3354/meps13125
Pereira, Talmo D., Nathaniel Tabris, Junyu Li, Shruthi Ravindranath, Eleni S. Papadoyannis, Z. Yan Wang, David M. Turner, et al. “SLEAP: Multi-Animal Pose Tracking.” Preprint. Animal Behavior and Cognition, September 2, 2020. https://doi.org/10.1101/2020.08.31.276246.
Torres, Nieukirk, S. L., Lemos, L., & Chandler, T. E. (2018). Drone Up! Quantifying Whale Behavior From a New Perspective Improves Observational Capacity. Frontiers in Marine Science, 5. https://doi.org/10.3389/fmars.2018.00319
“What if I’m wrong? What if I make a mistake?” When I began my career after completing my undergraduate degree, these questions echoed constantly in my head as the stakes were raised and my work was taken more seriously. Of course, this anxiety was not new. As a student, my worst fear had been poor performance in class. Post-undergrad, I was facing the possibility of making a mistake that could impact larger research projects and publications.
Gaining greater responsibility and consequences is a fact of life and an intrinsic part of growing up. As I wrap up my third year of graduate school, I’ve been reflecting on how learning to take on this responsibility as a scientist has been a crucial part of my journey thus far.
A scientist’s job is to ask, and try to answer, questions that no one knows the answer to – which is both terrifying and exciting. It feels a bit like realizing that grown-ups don’t have all the answers as a kid. Becoming comfortable with the fact that my work often involves making decisions that no one definitively can say are wrong or right has been one of my biggest challenges of grad school. The important thing to remember, I’ve learned, is that I’m not making wild guesses – I’m being trained to make the best, most informed decisions possible. And, hopefully, with more experience will come greater confidence.
Through grad school I have learned to take on this responsibility both in the field and the lab, although each brings different experiences. In the field, the stakes can feel higher because the decisions we make affect not just the quality of the data, but the safety of the team (which is always the top priority). I felt this most acutely throughout my first summer as a drone pilot. As a pilot, I am responsible for the safety of the team, the drone, and the quality of the data. As a new pilot, I intensely felt this pressure and would come home feeling more exhausted than usual. Now, in my second field season in this role, I’ve become more comfortable and am slowly building confidence in my abilities as I gain more and more experience.
I have also had a similar experience in the lab. Once it’s time to work on the analysis of a project, I choose how to clean, analyze, and interpret the data. As a young scientist, every step of the process involves learning new skills and making decisions that I don’t feel entirely qualified to make. When I started analysis for my first PhD chapter, I felt overwhelmed by deciding how to standardize my data, what kind of analysis to perform, and what indices to calculate. And, since it’s my first chapter, I felt further overwhelmed by the worry that any decision I made would become a later regret in a future part of my PhD.
Recently, the most daunting decision has been how to standardize my data. For my first chapter, I am investigating individual specialization of gray whale foraging behavior. The results of this question are not only important for conservation, but for my subsequent work (check out these previous blogs from January 2021and April 2022 for more on this research question). While there is a wealth of literature to draw analysis inspiration from, most of these studies use discrete prey capture data, while I am working with continuous behavior data. So, to make my data points comparable to one another, I need to standardize the behavior observation time of each drone flight to account for the potential bias introduced by recording one individual for more time than another. After experiencing an internal roller coaster of having an idea, thinking it through, deciding it was terrible and restarting the cycle, I was reminded that turning to lab mates and collaborators is the best way to work through a problem.
So, I had as many conversations as I could with my advisor, committee members, and peers. My thinking clarified with every conversation, and I gained confidence in the justification behind my decision. I cannot fully express the comfort that comes from hearing a trusted advisor say, “that makes ecological sense to me”. These conversations have also helped me remember that I am not alone in my worry and that I am not failing because I have these doubts. While I may never be 100% convinced that I’ve made the right decision, I feel much better knowing that I’ve talked it through with the brilliant group of scientists around me. And as I enter an analysis-intensive phase of my PhD, I am extremely grateful to have this community around to challenge, advise, and support me.
Did you enjoy this blog? Want to learn more about marine life, research, and conservation? Subscribe to our blog and get a weekly alert when we make a new post! Just add your name into the subscribe box below!
Dr. KC Bierlich, Postdoctoral Scholar, OSU Department of Fisheries, Wildlife, & Conservation Sciences, Geospatial Ecology of Marine Megafauna (GEMM) Lab
In a previous blog, I discussed the importance of incorporating measurement uncertainty in drone-based photogrammetry, as drones with different sensors, focal length lenses, and altimeters will have varying levels of measurement accuracy. In my last blog, I discussed how to incorporate photogrammetric uncertainty when combining multiple measurements to estimate body condition of baleen whales. In this blog, I will highlight our recent publication in Frontiers in Marine Science (https://doi.org/10.3389/fmars.2022.867258) led by GEMM Lab’s Dr. Leigh Torres, Clara Bird, and myself that used these methods in a collaborative study using imagery from four different drones to compare gray whale body condition on their breeding and feeding grounds (Torres et al., 2022).
Most Eastern North Pacific (ENP) gray whales migrate to their summer foraging grounds in Alaska and the Arctic, where they target benthic amphipods as prey. A subgroup of gray whales (~230 individuals) called the Pacific Coast Feeding Group (PCFG), instead truncates their migration and forages along the coastal habitats between Northern California and British Columbia, Canada (Fig. 1). Evidence from a recent study lead by GEMM Lab’s Lisa Hildebrand (see this blog) found that the caloric content of prey in the PCFG range is of equal or higher value than the main amphipod prey in the Arctic/sub-Arctic regions (Hildebrand et al., 2021). This implies that greater prey density and/or lower energetic costs of foraging in the Arctic/sub-Arctic may explain the greater number of whales foraging in that region compared to the PCFG range. Both groups of gray whales spend the winter months on their breeding and calving grounds in Baja California, Mexico.
In January 2019 an Unusual Mortality Event (UME) was declared for gray whales due to the elevated numbers of stranded gray whales between Mexico and the Arctic regions of Alaska. Most of the stranded whales were emaciated, indicating that reduced nutrition and starvation may have been the causal factor of death. It is estimated that the population dropped from ~27,000 individuals in 2016 to ~21,000 in 2020 (Stewart & Weller, 2021).
During this UME period, between 2017-2019, the GEMM Lab was using drones to monitor the body condition of PCFG gray whales on their Oregon coastal feeding grounds (Fig. 1), while Christiansen and colleagues (2020) was using drones to monitor gray whales on their breeding grounds in San Ignacio Lagoon (SIL) in Baja California, Mexico. We teamed up with Christiansen and colleagues to compare the body condition of gray whales in these two different areas leading up to the UME. Comparing the body condition between these two populations could help inform which population was most effected by the UME.
The combined datasets consisted of four different drones used, thus different levels of photogrammetric uncertainty to consider. The GEMM Lab collected data using a DJI Phantom 3 Pro, DJI Phantom 4, and DJI Phantom 4 Pro, while Christiansen et al., (2020) used a DJI Inspire 1 Pro. By using the methodological approach described in my previous blog (here, also see Bierlich et al., 2021a for more details), we quantified photogrammetric uncertainty specific to each drone, allowing cross-comparison between these datasets. We also used Body Area Index (BAI), which is a standardized relative measure of body condition developed by the GEMM Lab (Burnett et al., 2018) that has low uncertainty with high precision, making it easier to detect smaller changes between individuals (see blog here, Bierlich et al., 2021b).
While both PCFG and ENP gray whales visit San Ignacio Lagoon in the winter, we assume that the photogrammetry data collected in the lagoon is mostly of ENP whales based on their considerably higher population abundance. We also assume that gray whales incur low energetic cost during migration, as gray whale oxygen consumption rates and derived metabolic rates are much lower during migration than on foraging grounds (Sumich, 1983).
Interestingly, we found that gray whale body condition on their wintering grounds in San Ignacio Lagoon deteriorated across the study years leading up to the UME (2017-2019), while the body condition of PCFG whales on their foraging grounds in Oregon concurrently increased. These contrasting trajectories in body condition between ENP and PCFG whales implies that dynamic oceanographic processes may be contributing to temporal variability of prey available in the Arctic/sub-Arctic and PCFG range. In other words, environmental conditions that control prey availability for gray whales are different in the two areas. For the ENP population, this declining nutritive gain may be associated with environmental changes in the Arctic/sub-Arctic region that impacted the predictability and availability of prey. For the PCFG population, the increase in body condition across years may reflect recovery of the NE Pacific Ocean from the marine heatwave event in 2014-2016 (referred to as “The Blob”) that resulted with a period of low prey availability. These findings also indicate that the ENP population was primarily impacted in the die-off from the UME.
Surprisingly, the body condition of PCFG gray whales in Oregon was regularly and significantly lower than whales in San Ignacio Lagoon (Fig. 2). To further investigate this potential intrinsic difference in body condition between PCFG and ENP whales, we compared opportunistic photographs of gray whales feeding in the Northeastern Chukchi Sea (NCS) in the Arctic collected from airplane surveys. We found that the body condition of PCFG gray whales was significantly lower than whales in the NCS, further supporting our finding that PCFG whales overall have lower body condition than ENP whales that feed in the Arctic (Fig. 3).
This difference in body condition between PCFG and ENP gray whales raises some really interesting and prudent questions. Does the lower body condition of PCFG whales make them less resilient to changes in prey availability compared to ENP whales, and thus more vulnerable to climate change? If so, could this influence the reproductive capacity of PCFG whales? Or, are whales that recruit into the PCFG adapted to a smaller morphology, perhaps due to their specialized foraging tactics, which may be genetically inherited and enables them to survive with reduced energy stores?
These questions are on our minds here at the GEMM Lab as we prepare for our seventh consecutive field season using drones to collect data on PCFG gray whale body condition. As discussed in a previous blog by Dr. Alejandro Fernandez Ajo, we are combining our sightings history of individual whales, fecal hormone analyses, and photogrammetry-based body condition to better understand gray whales’ reproductive biology and help determine what the consequences are for these PCFG whales with lower body condition.
Did you enjoy this blog? Want to learn more about marine life, research, and conservation? Subscribe to our blog and get a weekly message when we post a new blog. Just add your name and email into the subscribe box below.
Bierlich, K. C., Hewitt, J., Bird, C. N., Schick, R. S., Friedlaender, A., Torres, L. G., … & Johnston, D. W. (2021). Comparing Uncertainty Associated With 1-, 2-, and 3D Aerial Photogrammetry-Based Body Condition Measurements of Baleen Whales. Frontiers in Marine Science, 1729.
Bierlich, K. C., Schick, R. S., Hewitt, J., Dale, J., Goldbogen, J. A., Friedlaender, A.S., et al. (2021b). Bayesian Approach for Predicting Photogrammetric Uncertainty in Morphometric Measurements Derived From Drones. Mar. Ecol. Prog. Ser. 673, 193–210. doi: 10.3354/meps13814
Burnett, J. D., Lemos, L., Barlow, D., Wing, M. G., Chandler, T., & Torres, L. G. (2018). Estimating morphometric attributes of baleen whales with photogrammetry from small UASs: A case study with blue and gray whales. Marine Mammal Science, 35(1), 108–139.
Christiansen, F., Rodrı́guez-González, F., Martı́nez-Aguilar, S., Urbán, J., Swartz, S., Warick, H., et al. (2021). Poor Body Condition Associated With an Unusual Mortality Event in Gray Whales. Mar. Ecol. Prog. Ser. 658, 237–252. doi:10.3354/meps13585
Hildebrand, L., Bernard, K. S., and Torres, L. G. (2021). Do Gray Whales Count Calories? Comparing Energetic Values of Gray Whale Prey Across Two Different Feeding Grounds in the Eastern North Pacific. Front. Mar. Sci. 8. doi: 10.3389/fmars.2021.683634
Stewart, J. D., and Weller, D. (2021). Abundance of Eastern North Pacific Gray Whales 2019/2020 (San Diego, CA: NOAA/NMFS)
Sumich, J. L. (1983). Swimming Velocities, Breathing Patterns, and Estimated Costs of Locomotion in Migrating Gray Whales, Eschrichtius Robustus. Can. J. Zoology. 61, 647–652. doi: 10.1139/z83-086
Torres, L.G., Bird, C., Rodrigues-Gonzáles, F., Christiansen F., Bejder, L., Lemos, L., Urbán Ramírez, J., Swartz, S., Willoughby, A., Hewitt., J., Bierlich, K.C. (2022). Range-wide comparison of gray whale body condition reveals contrasting sub-population health characteristics and vulnerability to environmental change. Frontiers in Marine Science. 9:867258. https://doi.org/10.3389/fmars.2022.867258
When I wrote my first blog on individual specialization well over a year ago, I just skimmed the surface of the literature on this topic and only started to recognize the importance of studying individual specialization. The question, “is there individual specialization in the PCFG of gray whales?” is the focus of my first thesis chapter and the results will affect all my subsequent work. Therefore, the literature and concepts of individual specialization are a focus of my literature review and studies.
In my previous blog I focused on common characteristics of individuals that are similarly specialized as an underlying driver of individual specialization. While these characteristics (often attributable to age, sex, or physical traits) are important to consider, I’ve learned that the list of drivers of individual specialization is long and that many variables are dynamic. Of all the drivers I’ve learned about, competition is among the most common.
Competition is a major driver of individual specialization, and a common driver of competition is resource availability. When resource availability decreases, whether caused by increasing population density or changing environmental conditions, competition for that resource increases. As competition increases, individuals have a choice. They can choose to engage in competition, either by racing, fighting, or sharing , which can be costly, or they can diffuse the competition by focusing on a different resource. This second approach would be considered niche partitioning in the prey dimension. Niche partitioning is a way for individuals to share ecological space by using different resources. Essentially, individuals can share habitat without having to engage in direct competition by pursuing different prey types .
This switch to different prey types can change the degree of individual specialization present in the population (Figure 1). But the direction of the change is not constant. If all individuals were pursuing the same prey type under low competition conditions but then switched to different alternate prey types under high competition, then individual specialization would increase (Figure 1a). This direction has been observed across a range of species including sharks , otters –, dolphins , , stickleback fish , , largemouth bass , banded mongoose , fur seals , and baleen whales .
However, if individuals were pursuing different prey types under low competition conditions (maybe because of underlying differences such as age or sex) but then switched to the same alternate prey types under high competition, diet overlap would increase, and individual specialization would decrease (Figure 1b). Furthermore, an individual might not switch to an entirely new prey type but instead add prey items to its diet . This diet expansion under competition would also decrease individual specialization. Fewer studies have reported this direction but it’s been found in the common bumblebee  and in several neotropical vertebrate species , .
Interestingly, its hypothesized that individual specialization driven by competition is one of the factors that facilitates the formation and existence of stable groups . For example, a study on resident female dolphins in Sarasota Bay, FL, USA found that females with high spatial overlap used distinct foraging specializations (Figure 2). This study illustrates how partitioning prey enabled spatial and social coexistence. A study on banded mongooses reached a similar conclusion . They found that specialization was highest in the biggest groups (with the most competition) and not explained by sex, age, or other inherent differences. They hypothesized that specialization increasing with competition reduced conflict and allowed the groups to remain stable. This study also highlighted the role of learning to determine an individual’s specialization.
Learning drives the distribution of knowledge throughout a population, which can lead to either specialization or generalization. ‘One-to-one’ learning, where one individual learns from one demonstrator, tends to promote individual specialization . This form of transmission drives specialization because the individuals who learn the specialization tend to then carry on using, and eventually teaching, that specialization . A common example of ‘one-to-one’ learning is vertical transmission from parent to offspring. It has been shown to transmit specializations in dolphins  and otters . ‘One-to-one’ learning can occur outside of parent-offspring pairs; non-parent-offspring ‘one-to-one’ learning has been shown to drive specialization in banded mongooses (Figure 3).
However, other forms of social learning can promote more generalized foraging strategies. ‘Many-to-one’ or ‘one-to-many’ learning can reduce the presence of specialization in species ,  as can the presence of conformity in a group , .
The multiple drivers of specialization and their dynamic quality means that it is important to contextualize specialization. For example, a study on four species of neotropical frogs found varying degrees of specialization across multiple populations of each species . The degree of specialization was dependent on a variety of drivers including predation and both intra- and inter-specific competition. Notably, the direction of the relationship between degree of specialization and each driver was species specific. This study highlights that one species may not always be more specialized than another, but that a populations’ specialization is context dependent.
Therefore, it is important to not only be aware of the degree of specialization present in a population, but to also understand its dynamics and drivers. These relationships can then be used to understand how, and why, a population may react to competition from other species, predators, and changes in resource availability . A population’s specialization can also affect the specialization of other populations and community dynamics , therefore, it’s important to consider and study individual specialization on both the population and community level. I am excited to start using our incredible six-year dataset to start investigating these questions for PCFG gray whales, so stay tuned for results!
Did you enjoy this blog? Want to learn more about marine life, research, and conservation? Subscribe to our blog and get a weekly alert when we make a new post! Just add your name into the subscribe box on the left panel.
 M. Taborsky, M. A. Cant, and J. Komdeur, The Evolution of Social Behaviour. Cambridge: Cambridge University Press, 2021. doi: 10.1017/9780511894794.
 E. R. Pianka, “Niche Overlap and Diffuse Competition,” vol. 71, no. 5, pp. 2141–2145, 1974.
 P. Matich et al., “Ecological niche partitioning within a large predator guild in a nutrient-limited estuary,” Limnol. Oceanogr., vol. 62, no. 3, pp. 934–953, 2017, doi: https://doi.org/10.1002/lno.10477.
 S. D. Newsome et al., “The interaction of intraspecific competition and habitat on individual diet specialization: a near range-wide examination of sea otters,” Oecologia, vol. 178, no. 1, pp. 45–59, May 2015, doi: 10.1007/s00442-015-3223-8.
 M. T. Tinker, G. Bentall, and J. A. Estes, “Food limitation leads to behavioral diversification and dietary specialization in sea otters,” Proc. Natl. Acad. Sci., vol. 105, no. 2, pp. 560–565, Jan. 2008, doi: 10.1073/pnas.0709263105.
 M. T. Tinker, M. Mangel, and J. A. Estes, “Learning to be different: acquired skills, social learning, frequency dependence, and environmental variation can cause behaviourally mediated foraging specializations,” Evol. Ecol. Res., vol. 11, pp. 841–869, 2009.
 M. T. Tinker et al., “Structure and mechanism of diet specialisation: testing models of individual variation in resource use with sea otters,” Ecol. Lett., vol. 15, no. 5, pp. 475–483, 2012, doi: 10.1111/j.1461-0248.2012.01760.x.
 S. Rossman et al., “Foraging habits in a generalist predator: Sex and age influence habitat selection and resource use among bottlenose dolphins (Tursiops truncatus),” Mar. Mammal Sci., vol. 31, no. 1, pp. 155–168, 2015, doi: https://doi.org/10.1111/mms.12143.
 L. G. Torres, “A kaleidoscope of mammal , bird and fish : habitat use patterns of top predators and their prey in Florida Bay,” vol. 375, pp. 289–304, 2009, doi: 10.3354/meps07743.
 M. S. Araújo et al., “Network Analysis Reveals Contrasting Effects of Intraspecific Competition on Individual Vs. Population Diets,” Ecology, vol. 89, no. 7, pp. 1981–1993, 2008, doi: 10.1890/07-0630.1.
 R. Svanbäck and D. I. Bolnick, “Intraspecific competition drives increased resource use diversity within a natural population,” Proc. R. Soc. B Biol. Sci., vol. 274, no. 1611, pp. 839–844, Mar. 2007, doi: 10.1098/rspb.2006.0198.
 D. E. Schindler, J. R. Hodgson, and J. F. Kitchell, “Density-dependent changes in individual foraging specialization of largemouth bass,” Oecologia, vol. 110, no. 4, pp. 592–600, May 1997, doi: 10.1007/s004420050200.
 C. E. Sheppard et al., “Intragroup competition predicts individual foraging specialisation in a group-living mammal,” Ecol. Lett., vol. 21, no. 5, pp. 665–673, 2018, doi: 10.1111/ele.12933.
 L. Kernaléguen, J. P. Y. Arnould, C. Guinet, and Y. Cherel, “Determinants of individual foraging specialization in large marine vertebrates, the Antarctic and subantarctic fur seals,” J. Anim. Ecol., vol. 84, no. 4, pp. 1081–1091, 2015, doi: 10.1111/1365-2656.12347.
 E. M. Keen and K. M. Qualls, “Respiratory behaviors in sympatric rorqual whales: the influence of prey depth and implications for temporal access to prey,” J. Mammal., vol. 99, no. 1, pp. 27–40, Feb. 2018, doi: 10.1093/jmammal/gyx170.
 R. H. MacArthur and E. R. Pianka, “On Optimal Use of a Patchy Environment,” Am. Nat., vol. 100, no. 916, pp. 603–609, 1966, doi: 10.1086/282454.
 C. Fontaine, C. L. Collin, and I. Dajoz, “Generalist foraging of pollinators: diet expansion at high density,” J. Ecol., vol. 96, no. 5, pp. 1002–1010, 2008, doi: 10.1111/j.1365-2745.2008.01405.x.
 R. Costa-Pereira, V. H. W. Rudolf, F. L. Souza, and M. S. Araújo, “Drivers of individual niche variation in coexisting species,” J. Anim. Ecol., vol. 87, no. 5, pp. 1452–1464, 2018, doi: 10.1111/1365-2656.12879.
 M. M. Pires, P. R. Guimarães Jr, M. S. Araújo, A. A. Giaretta, J. C. L. Costa, and S. F. dos Reis, “The nested assembly of individual-resource networks,” J. Anim. Ecol., vol. 80, no. 4, pp. 896–903, 2011, doi: 10.1111/j.1365-2656.2011.01818.x.
 M. S. Araújo, D. I. Bolnick, and C. A. Layman, “The ecological causes of individual specialisation,”Ecol. Lett., vol. 14, no. 9, pp. 948–958, 2011, doi: https://doi.org/10.1111/j.1461-0248.2011.01662.x.
 C. E. Sheppard, R. Heaphy, M. A. Cant, and H. H. Marshall, “Individual foraging specialization in group-living species,” Anim. Behav., vol. 182, pp. 285–294, Dec. 2021, doi: 10.1016/j.anbehav.2021.10.011.
 S. Wild, S. J. Allen, M. Krützen, S. L. King, L. Gerber, and W. J. E. Hoppitt, “Multi-network-based diffusion analysis reveals vertical cultural transmission of sponge tool use within dolphin matrilines,” Biol. Lett., vol. 15, no. 7, p. 20190227, Jul. 2019, doi: 10.1098/rsbl.2019.0227.
 L. M. Aplin, D. R. Farine, J. Morand-Ferron, A. Cockburn, A. Thornton, and B. C. Sheldon, “Experimentally induced innovations lead to persistent culture via conformity in wild birds,” Nature, vol. 518, no. 7540, pp. 538–541, Feb. 2015, doi: 10.1038/nature13998.
 E. Van de Waal, C. Borgeaud, and A. Whiten, “Potent Social Learning and Conformity Shape a Wild Primate’s Foraging Decisions,” Science, Apr. 2013, doi: 10.1126/science.1232769.
 D. I. Bolnick et al., “Why intraspecific trait variation matters in community ecology,” Trends Ecol. Evol., vol. 26, no. 4, pp. 183–192, Apr. 2011, doi: 10.1016/j.tree.2011.01.009.
Dr. KC Bierlich, Postdoctoral Scholar, OSU Department of Fisheries, Wildlife, & Conservation Sciences, Geospatial Ecology of Marine Megafauna (GEMM) Lab
The recent advancement in drones (or unoccupied aircraft systems, UAS) has greatly enhanced opportunities for scientists across a broad range of disciplines to collect high-resolution aerial imagery. Wildlife researchers in particular have utilized this technology to study large elusive animals, such as whales, to observe their behavior (see Clara Bird’s blog) and obtain morphological measurements via photogrammetry (see previous blog for a brief history on photogrammetry and drones). However, obtaining useful measurement data is not as easy as flying the drone and pressing record. For this blog, I will provide a brief overview on the basics of using photogrammetry to extract morphological measurements from images collected with drones, as well as the associated uncertainty from using different drone platforms.
During my PhD at Duke University, I co-developed an open-source photogrammetry software called MorphoMetriX to measure whales in images I collected using drones (Fig. 1) (Torres and Bierlich, 2020) (see this blog for some fieldwork memoirs!). The software is designed to be flexible, simple to use, and customizable without knowledge of scripting languages. Using MorphoMetriX, measurements are made in pixels and then multiplied by the ground sampling distance (GSD) to convert to standard units (e.g., meters) (Fig. 2A). GSD represents the distance on the ground each pixel represents (i.e., the linear size of the pixel) and therefore sets the scale of the image (i.e., cm per pixel). Figure 2A describes how GSD is dependent on the camera sensor, focal length lens, and altitude. Thus, drones equipped with different cameras and focal length lenses will have inherent differences in GSD as altitude increases (Fig. 2B). A larger GSD increases the length each pixel represents in a photo and results in a lower resolution image, potentially obscuring important features in the photo and introducing measurement error.
Obtaining accurate altitude information is a key component in obtaining accurate measurements. All drones are equipped with a barometer, which measures altitude from changes in pressure. In general, barometers usually yield low accuracy in the altitude recorded, particularly for low-cost sensors commonly found on small, off-the-shelf drones (Wei et al., 2016). Dawson et al. (2017) added a laser altimeter (i.e., LightWare SF11/C, https://www.mouser.com//datasheet//2//321//28054-SF11-Laser-Altimeter-Manual-Rev8-1371857.pdf) to a drone, which yields higher accuracy in the altitude recorded. Since then, several studies have adopted use of a laser altimeter to study different species of baleen whales (i.e., Gough et al., 2019; Christiansen et al., 2018).
The first chapter of my dissertation, which was published last year in Marine Ecology Progress Series, compared the accuracy of several drones equipped with different camera sensors, focal length lenses, and a barometer vs. laser altimeter (Bierlich et al., 2021). We flew each drone over a known sized object floating at the surface and collected images at various altitudes (between 10 – 120 m). We used the known size of the floating object to determine the percent error of each measurement at each altitude. We found that 1) there is a lot of variation in measurement error across the different drones when using a barometer to measure altitude and 2) using a laser altimeter dramatically reduces measurement error for each drone (Fig. 3).
These findings are important because if a study is analyzing measurements that are from more than one drone, the uncertainty associated with those measurements must be taken into account to know if measurements are reliable and comparable. For instance, let’s say we are comparing the body length of two different populations and found that population A is significantly longer than population B. From looking at Figure 3, that significant difference in length between population A and B could be unreliable as the difference may be due to the bias introduced by the type of drone, camera sensor, focal length lens, and whether a barometer or laser altimeter was used for recording altitude. In other words, without incorporating uncertainty associated with each measurement, how do you trust your measurement?
Hence, the National Institute of Standards and Technology (NIST) states that a measurement is complete only when accompanied by a quantitative statement of its uncertainty (Taylor & Kuyatt, 1994). In our Bierlich et al. (2021) study, we develop a Bayesian statistical model where we use the measurements of the known-sized object floating at the surface (what was used for Fig. 3) as training data to predict the lengths of unknown-sized whales. This Bayesian approach views data and the underlying parameters that generated the data (such as the mean or standard deviation) as random, and thus can be described by a statistical distribution. Using Bayes’ Theorem, a model of the observed data (called the likelihood function), is combined with prior knowledge pertaining to the underlying parameters (called the prior probability distribution) to form the posterior probability distribution, which serves as updated knowledge about the underlying parameter. For example, if someone told me they saw a 75 ft blue whale, I would not be phased. But if someone told me they saw a 150 ft blue whale, I would be skeptical – I’m using prior knowledge to determine the probability of this statement being true.
The posterior probability distribution produced by this Bayesian approach can also serve as new prior information for subsequent analyses. Following this framework, we used the known-sized objects to first estimate the posterior probability distribution for error for each drone. We then used that posterior probability distribution for error specific to each drone platform as prior information to form a posterior predictive distribution for length of unknown-sized whales. The length of an individual whale can then be described by the mean of this second posterior predictive distribution, and its uncertainty defined as the variance or an interval around the mean (Fig. 4).
For over half a decade, the GEMM Lab has been collecting drone images of Pacific Coast Feeding Group (PCFG) of gray whales off the coast of Oregon to measure their morphology and body condition (see GRANITE Project Blog). We have been using several different types of drones equipped with different cameras, focal length lenses, barometers, and laser altimeters. These measurements from different drones will inherently have different levels of error associated with them, so adapting these methods for incorporating uncertainty will be key to ensure our measurements are comparable and analyses are robust. To do this, we fly over a known-sized board (1 m) at the start of each flight to use as training data to generate a posterior predictive distribution for length of the an unknown-sized PCFG gray whale that we fly over (Fig. 5). Likewise, we are working closely with several other collaborators who are also using different drones. Incorporating measurement uncertainty from drones used across research labs and in different environments will help ensure robust analyses and provide great opportunity for some interesting comparisons – such as differences in gray whale body condition on their feeding grounds in Oregon vs. their breeding grounds in Baja, Mexico, and morphological comparisons with other baleen whale species, such as blue and humpback whales. We are currently wrapping up measurement from thousands of boards (Fig. 5) and whales (Fig. 1) from 2016 – 2021, so stay tuned for the results!
Bierlich, K.C., Schick, R.S., Hewitt, J., Dale, J., Goldbogen, J.A., Friedlaender, A.S., Johnston D.J. (2021). A Bayesian approach for predicting photogrammetric uncertainty in morphometric measurements derived from UAS. Marine Ecology Progress Series. DOI: https://doi.org/10.3354/meps13814
Christiansen F, Vivier F, Charlton C, Ward R, Amerson A, Burnell S, Bejder L (2018) Maternal body size and condition determine calf growth rates in southern right whales. Mar Ecol Prog Ser 592: 267−281
Dawson SM, Bowman MH, Leunissen E, Sirguey P (2017) Inexpensive aerial photogrammetry for studies of whales and large marine animals. Front Mar Sci 4: 366
Gough, W.T., Segre, P.S., Bierlich, K.C., Cade, D.E., Potvin, J., Fish, F. E., Dale, J., di Clemente, J., Friedlaender, A.S., Johnston, D.W., Kahane-Rapport, S.R., Kennedy, J., Long, J.H., Oudejans, M., Penry, G., Savoca, M.S., Simon, M., Videsen, S.K.A., Visser, F., Wiley, D.N., Goldbogen, J.A. (2019). Scaling of swimming performance in baleen whales. Journal of Experimental Biology, 222(20).https://doi.org/10.1242/jeb.204172
Taylor, B. N., and Kuyatt, C. E. (1994). Guidelines for Evaluating and Expressing the Uncertainty of NIST Measurement Results. Washington, DC: National Institute of Standards and Technology. 1–25.
Torres, W.I., & Bierlich, K.C. (2020). MorphoMetriX: a photogrammetric measurement GUI for morphometric analysis of megafauna. Journal of Open Source Software, 5(45), 1825. https://doi.org/10.21105/joss.01825
Wei S, Dan G, Chen H (2016) Altitude data fusion utilizing differential measurement and complementary filter. IET Sci Meas Technol (Singap) 10: 874−879
“Why don’t you just automate it?” This is a question I am frequently asked when I tell someone about my work. My thesis involves watching many hours of drone footage of gray whales and meticulously coding behaviors, and there are plenty of days when I have asked myself that very same question. Streamlining my process is certainly appealing and given how wide-spread and effective machine learning methods have become, it is a tempting option to pursue. That said, machine learning is only appropriate for certain research questions and scales, and it’s important to consider these before investing in using a new tool.
The application of machine learning methods to behavioral ecology is called computational ethology (Anderson & Perona, 2014). To identify behaviors from videos, the model tracks individuals across video frames and identifies patterns of movement that form a behavior. This concept is similar to the way we identify a whale as traveling if it’s moving in a straight line and as foraging if it’s swimming in circles within a small area (Mayo & Marx, 1990, check out this blog to learn more). The level of behavioral detail that the model is able to track depends on the chosen method (Figure 1, Pereira et al., 2020). These methods range from tracking each animal as a simple single point (called a centroid) to tracking the animal’s body positioning in 3D (this method is called pose estimation), which range from providing less detailed to more detailed behavior definitions. For example, tracking an individual as a centroid could be used to classify traveling and foraging behaviors, while pose estimation could identify specific foraging tactics.
Pose estimation involves training the machine learning algorithm to track individual anatomical features of an individual (e.g., the head, legs, and tail of a rat), meaning that it can define behaviors in great detail. A behavior state could be defined as a combination of the angle between the tail and the head, and the stride length.
For example, Mearns et al. (2020) used pose estimation to study how zebrafish larvae in a lab captured their prey. They tracked the tail movements of individual larvae when presented with prey and classified these movements into separate behaviors that allowed them to associate specific behaviors with prey capture (Figure 2). The authors found that these behaviors occurred in a specific sequence, that the behaviors kept the prey within the larvae’s line of sight, and that the sequence was triggered by visual cues. In fact, when they removed the visual cue of the prey, the larvae terminated the behavior sequence, meaning that the larvae are continually choosing to do each behavior in the sequence, rather than the sequence being one long behavior event that is triggered only by the initial visual cue. This study is a good example of the applicability of machine learning models for questions aimed at kinematics and fine-scale movements. Pose estimation has also been used to study the role of facial expression and body language in rat social communication (Ebbesen & Froemke, 2021).
While previous machine learning methods to track animal movements required individuals to be physically marked, the current methods can perform markerless tracking (Pereira et al., 2020). This improvement has broadened the kinds of studies that are possible. For example, Bozek et al., (2021) developed a model that tracked individuals throughout an entire honeybee colony and showed that certain individual behaviors were spatially distributed within the colony (Figure 3). Machine learning enabled the researchers to track over 1000 individual bees over several months, a task that would be infeasible for someone to do by hand.
These studies highlight that the potential benefits of using machine learning when studying fine scale behaviors (like kinematics) or when tracking large groups of individuals. Furthermore, once it’s trained, the model can process large quantities of data in a standardized way to free up time for the scientists to focus on other tasks.
While machine learning is an exciting and enticing tool, automating behavior detection via machine learning could be its own PhD dissertation. Like most things in life, there are costs and benefits to using this technique. It is a technically difficult tool, and while applications exist to make it more accessible, knowledge of the computer science behind it is necessary to apply it effectively and correctly. Secondly, it can be tedious and time consuming to create a training dataset for the model to “learn” what each behavior looks like, as this step involves manually labeling examples for the model to use.
As I’ve mentioned in a previous blog, I came quite close to trying to study the kinematics of gray whale foraging behaviors but ultimately decided that counting fluke beats wasn’t necessary to answer my behavioral research questions. It was important to consider the scale of my questions (as described in Allison’s blog) and I think that diving into more fine-scale kinematics questions could be a fascinating follow-up to the questions I’m asking in my PhD.
For instance, it would be interesting to quantify how gray whales use their flukes for different behavior tactics. Do gray whales in better body condition beat their flukes more frequently while headstanding? Does the size of the fluke affect how efficiently they can perform certain tactics? While these analyses would help quantify the energetic costs of different behaviors in better detail, they aren’t necessary for my broad scale questions. Consequently, taking the time to develop and train a pose estimation machine learning model is not the best use of my time.
That being said, I am interested in applying machine learning methods to a specific subset of my dataset. In social behavior, it is not only useful to quantify the behaviors exhibited by each individual but also the distance between them. For example, the distance between a mom and her calf can be indicative of the calves’ dependence on its mom (Nielsen et al., 2019). However, continuously measuring the distance between two individuals throughout a video is tedious and time intensive, so training a machine learning model could be an effective use of time. I plan to work with an intern this summer to develop a machine learning model to track the distance between pairs of gray whales in our drone footage and then relate this distance data with the manually coded behaviors to examine patterns in social behavior (Figure 4). Stay tuned to learn more about our progress!
Did you enjoy this blog? Want to learn more about marine life, research, and conservation? Subscribe to our blog and get a weekly alert when we make a new post! Just add your name into the subscribe box on the left panel.
Anderson, D. J., & Perona, P. (2014). Toward a Science of Computational Ethology. Neuron, 84(1), 18–31. https://doi.org/10.1016/j.neuron.2014.09.005
Bozek, K., Hebert, L., Portugal, Y., Mikheyev, A. S., & Stephens, G. J. (2021). Markerless tracking of an entire honey bee colony. Nature Communications, 12(1), 1733. https://doi.org/10.1038/s41467-021-21769-1
Ebbesen, C. L., & Froemke, R. C. (2021). Body language signals for rodent social communication. Current Opinion in Neurobiology, 68, 91–106. https://doi.org/10.1016/j.conb.2021.01.008
Mayo, C. A., & Marx, M. K. (1990). Surface foraging behaviour of the North Atlantic right whale, Eubalaena glacialis , and associated zooplankton characteristics. Canadian Journal of Zoology, 68(10), 2214–2220. https://doi.org/10.1139/z90-308
Mearns, D. S., Donovan, J. C., Fernandes, A. M., Semmelhack, J. L., & Baier, H. (2020). Deconstructing Hunting Behavior Reveals a Tightly Coupled Stimulus-Response Loop. Current Biology, 30(1), 54-69.e9. https://doi.org/10.1016/j.cub.2019.11.022
Nielsen, M., Sprogis, K., Bejder, L., Madsen, P., & Christiansen, F. (2019). Behavioural development in southern right whale calves. Marine Ecology Progress Series, 629, 219–234. https://doi.org/10.3354/meps13125
Pereira, T. D., Shaevitz, J. W., & Murthy, M. (2020). Quantifying behavior to understand the brain. Nature Neuroscience, 23(12), 1537–1549. https://doi.org/10.1038/s41593-020-00734-z
Over six field seasons the GEMM lab team has conducted nearly 500 drone flights over gray whales, equaling over 100 hours of footage. These hours of footage are the central dataset for my PhD dissertation, so it’s up to me to process them all. This process can be challenging, tedious, and daunting, but it is also quite fun and a privilege to be the one person who gets to watch all the footage. It’s fascinating to get to know the whales and their behaviors and pick up on patterns. It motivates me to get through this video processing step and start doing the data analysis. Recently, it’s been especially fun to notice patterns that I’ve seen mentioned in the literature. One example is adult social behavior.
There are two categories of social behavior that I’m interested in studying: maternal behavior, defined as interactions between a mom and its calf, and general social behaviors, defined as social interactions between non-mom/calf pairs. In this blog I’ll focus on general social behaviors, but if you’re interested in maternal behavior check out this blog. General social behavior, which I’ll refer to as social behavior moving forward, includes tactile interactions and promiscuous behaviors (Torres et al. 2018; Clip 1). While gray whales in the PCFG range are primarily foraging, researchers have observed increases in social behavior towards the end of the foraging season (Stelle et al., 2008; Torres et al., 2018). We think that this indicates that the whales are starting to focus less on feeding and more on breeding. This tradeoff of foraging vs. socializing time is interesting because it comes at an energetic cost.
Broadly, animals need to balance the energetic demands of survival with those of reproduction. They need to reproduce to pass on their genes, but reproduction is energetically demanding, and animals also need to survive and grow to be able to reproduce. The decision to reproduce is costly because reproduction requires energetic investment and time investment since animals do not forage (gaining energy) when they are socializing. Consequently, only animals with sufficient energy reserves (i.e., body condition) to invest in reproduction actually engage in reproduction. Given these costs associated with reproduction, we expect to see a relationship between social behavior and body condition (Green, 2001) with mainly animals in good body condition engaging in social behavior because these animals have sufficient reserves to sustain the cost. Furthermore, since body condition is an indicator of foraging success and prey availability, environmental conditions can also affect social behavior and reproduction through this pathway.
Rahman et al. (2014) used a lab experiment to study the relationship between nutritional stress and male guppy courtship behavior (Figure 1). In their experiment they tested for the effects of both decreased diet quantity and quality on the frequency of male courtship behaviors. Rahman et al (2014) found that individuals in the low-quantity group were significantly smaller than those in the high-quality group and that diet quantity had a significant effect on the frequency of courtship behaviors. Males fed a low-quantity diet performed fewer courtship behaviors. Interestingly, there was no significant effect of diet quality on courtships behavior, although there was some evidence of an interaction effect, which suggests that within the low-quantity group, males fed with high-quality food performed more courtship behaviors that those fed with low-quality food. This study is interesting because it shows how foraging success (diet quantity and quality) can affect courting behavior.
However, guppies are not the ideal species for comparison to gray whales because gray whales and guppies have quite different life history traits. A more fitting comparison would be with an example species with more in common with gray whales, such as viviparous capital breeders. Viviparous animals develop the embryo inside the body and give live birth. Capital breeders forage to build energy reserves and then rely on those energy reserves during reproduction. Surprisingly, I found asp vipers to be a good example species for comparison to gray whales.
Asp vipers (Figure 2) are viviparous snakes who are considered capital breeders because they forage prior to hibernation, and then begin reproduction immediately following hibernation without additional foraging. Naulleau & Bonnet (1996) conducted a field study on female asp vipers to determine if there was a difference in body condition at the start of the breeding season between females who reproduced or not during that season. To do this they marked individuals and measured their body condition at the start of the breeding season and then recaptured those individuals at the end of the breeding season and recorded whether the individual had reproduced. Interestingly, they found that there was a strongly significant difference in body condition between females that did and did not reproduce. In fact, they discovered that no female below a certain body condition value reproduced, meaning that they found a body condition threshold for reproduction.
Additionally, a study on water pythons found that their body condition threshold for reproduction shifted over time in response to prey availability (Madsen & Shine, 1999). These authors found that females lowered their threshold after several consecutive years of poor prey availability. These studies are really exciting to me because they address questions that the GRANITE project team is interested in tackling.
Understanding the relationship between body condition and reproduction in gray whales is an important puzzle piece for our work. The aim of the GRANITE project is to understand how the effects of stressors on individual whales scales up to population level impacts (read Lisa’s blog to learn more). Reproduction rates play a big role in population dynamics, so it is important to understand what factors affect reproduction. Since we’re studying these whales on their foraging grounds, assessing body condition provides an important link between foraging behavior and reproduction.
For example, if an individual’s response to a stressor is to forage less, that may lead to poorer body condition, meaning that they may be less likely to reproduce. While reduced reproduction in one individual may not have a big effect on the population, the same response from multiple individuals could impact the population’s dynamics (i.e., increasing or decreasing abundance). Understanding these different relationships between behavior, body condition, and reproduction rates is a big undertaking, but it’s exciting to be a member of the GRANITE team as this strong group of scientists works to bring together different data streams to work on this big picture question. We’re all deep into data processing right now so stay tuned over the next few years to learn more about gray whale social behavior and to find out if fat whales are more social than skinny whales.
Did you enjoy this blog? Want to learn more about marine life, research, and conservation? Subscribe to our blog and get weekly updates and more! Just add your name into the subscribe box on the left panel.
Green, A. J. (2001). Mass/Length Residuals: Measures of Body Condition or Generators of Spurious Results? Ecology, 82(5), 1473–1483. https://doi.org/10.1890/0012-9658(2001)082[1473:MLRMOB]2.0.CO;2
Madsen, T., & Shine, R. (1999). The adjustment of reproductive threshold to prey abundance in a capital breeder. Journal of Animal Ecology, 68(3), 571–580. https://doi.org/10.1046/j.1365-2656.1999.00306.x
Naulleau, G., & Bonnet, X. (1996). Body Condition Threshold for Breeding in a Viviparous Snake. Oecologia, 107(3), 301–306.
Rahman, M. M., Kelley, J. L., & Evans, J. P. (2013). Condition-dependent expression of pre- and postcopulatory sexual traits in guppies. Ecology and Evolution, 3(7), 2197–2213. https://doi.org/10.1002/ece3.632
Rahman, M. M., Turchini, G. M., Gasparini, C., Norambuena, F., & Evans, J. P. (2014). The Expression of Pre- and Postcopulatory Sexually Selected Traits Reflects Levels of Dietary Stress in Guppies. PLOS ONE, 9(8), e105856. https://doi.org/10.1371/journal.pone.0105856
Stelle, L. L., Megill, W. M., & Kinzel, M. R. (2008). Activity budget and diving behavior of gray whales (Eschrichtius robustus) in feeding grounds off coastal British Columbia. Marine Mammal Science, 24(3), 462–478. https://doi.org/10.1111/j.1748-7692.2008.00205.x
Torres, L. G., Nieukirk, S. L., Lemos, L., & Chandler, T. E. (2018). Drone up! Quantifying whale behavior from a new perspective improves observational capacity. Frontiers in Marine Science, 5(SEP). https://doi.org/10.3389/fmars.2018.00319
With the GRANITE field season officially over, we are now processing all of the data we collected this summer. For me, I am starting to go through all the drone videos to take snapshots of each whale to measure their body condition. As I go through these videos, I am reflecting on the different experiences I am fortunate enough to have with flying different drones, in different environments, over different species of baleen whales: blue, humpback, Antarctic minke, and now gray whales. Each of these species have a different morphological design and body shape (Woodward et al., 2006), which leads to different behaviors that are noticeable from the drone. Drones create immense opportunity to learn how whales thrive in their natural environments [see previous blog for a quick history], and below are some of my memories from above.
I first learned how drones could be used to study the morphology and behavior of large marine mammals during my master’s degree at Duke University, and was inspired by the early works of John Durban (Durban et al., 2015, 2016) Fredrick Christiansen (Christiansen et al., 2016) and Leigh Torres (Torres et al., 2018). I immediately recognized the value and utility of this technology as a new tool to better monitor the health of marine mammals. This revelation led me to pursue a PhD with the Duke University Marine Robotics and Remote Sensing (MaRRS) Lab led by Dr. Dave Johnston where I helped further develop tools and methods for collecting drone-based imagery on a range of species in different habitats.
When flying drones over whales, there are a lot of moving parts; you’re on a boat that is moving, flying something that is moving, following something that is moving. These moving elements are a lot to think about, so I trained hard, so I did not have to think about each step and flying felt intuitive and natural. I did not grow up playing video games, so reaching this level of comfort with the controls took a lot of practice. I practiced for hours over the course of months before my first field excursion and received some excellent mentorship and training from Julian Dale, the lead engineer in the MaRRS Lab. Working with Julian and the many hours of training helped me establish a solid foundation in my piloting skills and feel confident working in various environments on different species.
Blue whales offshore of Monterey, California.
In 2017 and 2018 I was involved in collaborative project with the MaRRS Lab and Goldbogen Lab at Stanford University, where we tagged and flew drones over blue whales offshore of Monterey, California. We traveled about an hour offshore and reliably found groups of blue whales actively feeding. Working offshore typically brought a large swell, which can often make landing the drone back into your field partner’s hands tricky as everything is bobbing up and down with the oscillations of the swell. Fortunately, we worked from a larger research vessel (~56 ft) and quickly learned that landing the drone in the stern helped dampen the effects of bobbing up and down. The blue whales we encountered often dove to a depth of around 200 m for about 20-minute intervals, then come to the surface for only a few minutes. This short surface period provided only a brief window to locate the whale once it surfaced and quickly fly over it to collect the imagery needed before it repeated its dive cycle. We learned to be patient and get a sense of the animal’s dive cycle before launch in order to time our flights so the drone would be in the air a couple of minutes before the whale surfaced.
Once over the whales, the streamlined body of the blue whales was noticeable, with their small, high aspect ratio flippers and fluke that make them so well adapted for fast swimming in the open ocean (Fig. 1) (Woodward et al., 2006). I also noticed that because these whales are so large (often 21 – 24 m), I often flew at higher altitudes to be able fit them within the field of view of the camera. It was also always shocking to see how small the tagging boat (~8 m) looked when next to Earth’s largest creatures.
Antarctic minke whales and humpback whales along the Western Antarctic PeninsulaA lot of the data included in my dissertation came from work along the Western Antarctic Peninsula (WAP), which had a huge range of weather conditions, from warm and sunny days to cold and snowy/foggy/rainy/windy/icy days. A big focus was often trying to keep my hands warm, as it was often easier to fly without gloves in order to better feel the controls. One of the coldest days I remember was late in the season in mid-June (almost winter!) in Wilhemina Bay where ice completely covered the bay in just a couple hours, pushing the whales out into the Gerlache Strait; I suspect this was the last ice-free day of the season. Surprisingly though, the WAP also brought some of the best conditions I have ever flown in. Humpback and Antarctic minke whales are often found deep within the bays along the peninsula, which provided protection from the wind. So, there were times where it would be blowing 40 mph in the Gerlache Strait, but calm and still in the bays, such as Andvord Bay, which allowed for some incredible conditions for flying. Working from small zodiacs (~7 m) allowed us more maneuverability for navigating around or through the ice deep in the bays (Fig. 2)
Flying over Antarctic minke whale was always rewarding, as they are very sneaky and can quickly disappear under ice flows or in the deep, dark water. Flying over them often felt like a high-speed chase, as their small streamlined bodies makes them incredibly quick and maneuverable, doing barrel rolls, quick banked turns, and swimming under and around ice flows (Fig. 3). There would often be a group between 3-7 individuals and it felt like they were playing tag with each other – or perhaps with me!
Humpbacks displayed a wide range of behaviors along the WAP. Early in the season they continuously fed throughout the entire day, often bubble net feeding in groups typically of 2-5 animals (Fig. 4). For as large as they are, it was truly amazing to see how they use their pectoral fins to perform quick accelerations and high-speed maneuvering for tight synchronized turns to form bubble nets, which corral and trap their krill, their main food source (Fig. 4) (Woodward et al., 2006). Later in the season, humpbacks switched to more resting behavior in the day and mostly fed at night, taking advantage of the diel vertical migration of krill. This behavior meant we often found humpbacks snoozing at the surface after a short dive, as if they were in a food coma. They also seemed to be more curious and playful with each other and with us later in the season (Fig. 5).
We also encountered a lot of mom and calf pairs along the WAP. Moms were noticeably skinny compared to their plump calf in the beginning of the season due to the high energetic cost of lactation (Fig. 6). It is important for moms to regain this lost energy throughout the feeding season and begin to wean their calves. I often saw moms refusing to give milk to their nudging calf and instead led teaching lessons for feeding on their own.
Gray whales off Newport, Oregon
All of these past experiences helped me quickly get up to speed and jump into action with the GRANITE field team when I officially joined the GEMM Lab this year in June. I had never flown a DJI Inspire quadcopter before (the drone used by the GEMM Lab), but with my foundation piloting different drones, some excellent guidance from Todd and Clara, and several hours of practice to get comfortable with the new setup, I was flying over my first gray whale by day three of the job.
The Oregon coast brings all sorts of weather, and some days I strangely found myself wearing a similar number of layers as I did in Antarctica. Fog, wind, and swell could all change within the hour, so I learned to make the most of weather breaks when they came. I was most surprised by how noticeably different gray whales behave compared to the blue, Antarctic minke, and humpback whales I had grown familiar with watching from above. For one, it is absolutely incredible to see how these huge whales use their low-aspect ratio flippers and flukes (Woodward et al., 2006) to perform low-speed, highly dynamic maneuvers to swim in very shallow water (5-10 m) so close to shore (<1m sometimes!) and through kelp forest or surf zones close to the beach. They have amazing proprioception, or the body’s ability to sense its movement, action, and position, as gray whales often use their pectoral fins and fluke to stay in a head standing position (see Clara Bird’s blog) to feed in the bottom sediment layer, all while staying in the same position and resisting the surge of waves that could smash them against the rocks (Video 1) . It is also remarkable how the GEMM Lab knows each individual whale based on natural skin marks, and I started to get a better sense of each whale’s behavior, including where certain individuals typically like to feed, or what their dive cycle might be depending on their feeding behavior.
I feel very fortunate to be a part of the GRANITE field team and to contribute to data collection efforts. I look forward to the data analysis phase to see what we learn about how the morphology and behavior of these gray whales to help them thrive in their environment.
Christiansen, F., Dujon, A. M., Sprogis, K. R., Arnould, J. P. Y., and Bejder, L. (2016).Noninvasive unmanned aerial vehicle provides estimates of the energetic cost of reproduction in humpback whales. Ecosphere 7, e01468–18.
Durban, J. W., Fearnbach, H., Barrett-Lennard, L. G., Perryman, W. L., & Leroi, D. J. (2015). Photogrammetry of killer whales using a small hexacopter launched at sea. Journal of Unmanned Vehicle Systems, 3(3), 131-135.
Durban, J. W., Moore, M. J., Chiang, G., Hickmott, L. S., Bocconcelli, A., Howes, G., et al.(2016). Photogrammetry of blue whales with an unmanned hexacopter. Mar. Mammal Sci. 32, 1510–1515.
Torres, L. G., Nieukirk, S. L., Lemos, L., & Chandler, T. E. (2018). Drone up! Quantifying whale behavior from a new perspective improves observational capacity. Frontiers in Marine Science, 5, 319.
Woodward, B. L., Winn, J. P., and Fish, F. E. (2006). Morphological specializations of baleen whales associated with hydrodynamic performance and ecological niche. J. Morphol. 267, 1284–1294.