computer software – Geospatial Ecology of Marine Megafauna Laboratory

Measure faster! New tools for automatically obtaining body length and body condition of whales from drone videos

Dr. KC Bierlich, Postdoctoral Scholar, OSU Department of Fisheries, Wildlife, & Conservation Sciences, Geospatial Ecology of Marine Megafauna Lab

Monitoring the body length and body condition of animals can help provide important information on the health status of individuals and their populations, and can even serve as early warning signs if a population is adapting to habitat changes or is at risk of collapse (Cerini et al., 2023). As discussed in previous blogs, drone-based photogrammetry provides a method for non-invasively collecting important size measurements of whales, such as for detecting differences in body condition and length between populations, and even diagnosing pregnancy. Thus, using drones to collect measurement data on the growth, body condition, and pregnancy rates of whales can help expedite population health assessments to elicit conservation and management actions.

However, it takes a long time to manually measure whales filmed in drone imagery. For every video collected, an analyst must carefully watch each video and manually select frames with whales in good positions for measuring (flat and straight at the surface). Once frames are selected, each image must then be ranked and filtered for quality before finally measuring using a photogrammetry software, such as MorphoMetriX. This entire manual processing pipeline ultimately delays results, which hinders the ability to rapidly assess population health. If only there was a way to automate this process of obtaining measurements…

Well now there is! Recently, a collaboration between researchers from the GEMM Lab, CODEX, and OSU’s Department of Engineering and Computer Science published a manuscript introducing automated methods for obtaining body length and body condition measurements (Bierlich et al., 2024). The manuscript describes two user-friendly models: 1) “DeteX”, which automatically detects whales in drone videos to output frames for measuring and 2) “XtraX”, which automatically extracts body length and body condition measurements from input frames (Figure 1). We found that using DeteX and XtraX produces measurements just as good as manual measurement (Coefficient of Variation < 5%), while substantially reducing the processing time by almost 90%. This increased efficiency not only saves hours (weeks!) of manual processing time, but enables more rapid assessments of populations’ health.

Future steps for DeteX and XtraX are to adapt the models so that measurements can be extracted from multiple whales in a single frame, which could be particularly useful for analyzing images containing mothers with their calf. We also look forward to adapting DeteX and XtraX to accommodate more species. While DeteX and XtraX was trained using only gray whale imagery, we were pleased to see that these models performed well when trialing on imagery of a blue whale (Figure 2). These results are encouraging because it shows that the models can be adapted to accommodate other species with different body shapes, such as belugas or beaked whales, with the inclusion of more training data.

We are excited to share these methods with the drone community and the rest of this blog walks through the features and steps for running DeteX and XtraX to make them even easier to use.

Figure 1. Overview of DeteX and XtraX for automatically obtaining body length and body condition measurements from drone-based videos.

Figure 2. Example comparing manual (MorphoMetriX) vs. automated (XtraX) measurements of a blue whale.

DeteX and XtraX walkthrough

Both DeteX and XtraX are web-based applications designed to be intuitive and user-friendly. Instructions to install and run DeteX and XtraX are available on the CODEX website. Once DeteX is launched, the default web-browser automatically opens the application where the user is asked to select 1) the folder containing the drone-based videos to analyze and 2) the folder to save output frames (Figure 3). Then, the user can select ‘start’ to begin. The default for DeteX is set to analyze the entire video from start to finish at one frame per second; if recording a video at 30 frames per second, the last (or 30^th) frame is processed for each second in the video. There is also a “finetune” version of DeteX that offers users much more control, where they can change these default settings (Figure 4). For example, users can change the defaults to increase the number of frames processed per second (i.e., 10 instead of 1), to target a specific region in the video rather than the entire video, and adjust the “detection model threshold” to change the threshold of confidence the model has for detecting a whale. These specific features for enhanced control may be particularly helpful when there is a specific surfacing sequence that a user wants to have more flexibility in selecting specific frames for measuring.

Figure 3. A screenshot of the DeteX web-based application interface.

Figure 4. The DeteX “finetune” version provides more control for users to change the default settings to target a specific region in the video (here between 3 min 00 sec and 3 min 05 sec), change the number of frames per second to process (now 10 per second), and the detection threshold, or level of confidence for identifying a whale in the video (now a higher threshold at 0.9 instead of the default at 0.8).

Once output frames are generated by DeteX, the user can select which frames to input into XtraX to measure. Once XtraX is launched, the default web-browser automatically opens the application where the user is asked to select 1) the folder containing the frames to measure and 2) the folder to save the output measurements. If the input frames were generated using DeteX, the barometric altitude is automatically extracted from the file name (note, that altitudes collected from a LiDAR altimeter can be joined in the XtraX output .csv file to then calculate measurements using this altitude). The image width (pixels) is automatically extracted from the input frame metadata. Users can then input specific camera parameters, such as sensor width (mm) and the focal length of the camera (mm), the launch height of the drone (i.e., if launching from hand when on a boat), and the region along the body to measure body condition (Figure 5). This region along the body is called the Head-Tail range and is identified as the area where most lipid storage takes place to estimate body condition. To run, the user selects “start”. XtraX then will output a .png file of each frame showing the keypoints (used for the body length measurement) and the shaded region (used for the body condition estimate) along the body to help visual results so users can filter for quality (Figure 6). XtraX also outputs a single .csv containing all the measurements (in meters and pixels) with their associated metadata.

Figure 5. User interface for XtraX. The user specifies a folder containing the images to measure and a folder to save the outputs measurements, and then can enter in camera specifications, the launch height of the drone (to be added to the barometer altitude) and the range of body widths to include in the body condition measurement (in the case, 0.2 and 0.7 correspond to body region between widths 20% and 70% of the total length, respectively).

Figure 6. Example output from XtraX showing (red) keypoints along the body to measure body length and the (green) shaded region used for body condition.

We hope this walkthrough is helpful for researchers interested in using and adapting these tools for their projects. There is also a video tutorial available online. Happy (faster) measuring!

References

Bierlich, K. C., Karki, S., Bird, C. N., Fern, A., & Torres, L. G. (2024). Automated body length and body condition measurements of whales from drone videos for rapid assessment of population health. Marine Mammal Science, e13137. https://doi.org/10.1111/mms.13137

Cerini, F., Childs, D. Z., & Clements, C. F. (2023). A predictive timeline of wildlife population collapse. Nature Ecology & Evolution, 7(3), 320–331. https://doi.org/10.1038/s41559-023-01985-2

An ‘X’travaganza! Introducing the Marine Mammal Institute’s Center of Drone Excellence (CODEX)

Dr. KC Bierlich, Postdoctoral Scholar, OSU Department of Fisheries, Wildlife, & Conservation Sciences, Geospatial Ecology of Marine Megafauna Lab

Drones are becoming more and more prevalent in marine mammal research, particularly for non-invasively obtaining morphological measurements of cetaceans via photogrammetry to identify important health metrics (see this and this previous blog). For example, the GEMM Lab uses drones for the GRANITE Project to study Pacific Coast Feeding Group (PCFG) gray whales and we have found that PCFG whales are skinnier and morphologically shorter with smaller skulls and flukes compared to the larger Eastern North Pacific (ENP) population. The GEMM Lab has also used drones to document variation in body condition across years and within a season, to diagnose pregnancy, and even measure blowholes.

While drone-based photogrammetry can provide major insight into cetacean ecology, several drone systems and protocols are used across the scientific community in these efforts, and no consistent method or centralized framework is established for quantifying and incorporating measurement uncertainty associated with these different drones. This lack of standardization restricts comparability across datasets, thus hindering our ability to effectively monitor populations and understand the drivers of variation (e.g., pollution, climate change, injury, noise).

We are excited to announce the Marine Mammal Institute’s (MMI) Center of Drone Excellence (CODEX), which focuses on developing analytical methods for using drones to non-invasively monitor marine mammal populations. CODEX is led by GEMM Lab member’s KC Bierlich, Leigh Torres, and Clara Bird and consists of other team members within and outside OSU. We draw from many years of trials, errors, headaches, and effort working with drones to study cetacean ecology in a variety of habitats and conditions on many different species.

Already CODEX has developed several open-source hardware and software tools. We developed, produced, and published LidarBoX (Bierlich et al., 2023), which is a 3D printed enclosure for a LiDAR altimeter system that can be easily attached and swapped between commercially available drones (i.e., DJI Inspire, DJI Mavic, and DJI Phantom) (Figure 1). Having a LidarBoX installed helps researchers obtain altitude readings with greater accuracy, yielding morphological measurements with less uncertainty. Since we developed LidarBoX, we have received over 35 orders to build this unit for other labs in national and international universities.

Figure 1. A ‘LidarBoX’ attached to a DJI Inspire 2. The LidarBoX is a 3D printed enclosure containing a LiDAR altimeter to help obtain more accurate altitude readings.

Additionally, CODEX recently released MorphoMetriX version 2 (v2), an easy-to-use photogrammetry software that provides users with the flexibility to obtain custom morphological measurements of megafauna in imagery with no knowledge of any scripting language (Torres and Bierlich, 2020). CollatriX is a user-friendly software for collating multiple MorphoMetriX outputs into a single dataframe and linking important metadata to photogrammetric measurements, such as altitude measured with a LidarBoX (Bird and Bierlich, 2020). CollatriX also automatically calculates several body condition metrics based on measurements from MorphoMetriX v2. CollatriX v2 is currently in beta-testing and scheduled to be released late Spring 2024.

Figure 2. An example of a Pygmy blue whale imported into MorphoMetriX v2, open-source photogrammetry software.

CODEX also recently developed two automated tools to help speed up the laborious manual processing of drone videos for obtaining morphological measurements (Bierlich & Karki et al., in revision). DeteX is a graphical user interface (GUI) that uses a deep learning model for automated detection of cetaceans in drone-based videos. Researchers can input their drone-based videos and DeteX will output frames containing whales at the surface. Users can then select which frames they want to use for measuring individual whales and then input these selected frames into XtraX, which is a GUI that uses a deep learning model to automatically extract body length and body condition measurements of cetaceans (Figure 4). We found automated measurements from XtraX to be similar (within 5%) of manual measurements. Importantly, using DeteX and XtraX takes about 10% of the time it would take to manually process the same videos, demonstrating how these tools greatly speed up obtaining key morphological data while maintaining accuracy, which is critical for effectively monitoring population health.

Figure 3. An example of an automated body length (top) and body condition (bottom) measurement of a gray whale using XtraX (Bierlich & Karki et al., in revision).

CODEX is also in the process of developing Xcertainty, an R package that uses a Bayesian statistical model to quantify and incorporate uncertainty associated with measurements from different drones (see this blog). Xcertainty is based on the Bayesian statistical model developed by Bierlich et al., (2021b; 2021a), which has been utilized by many studies with several different drones to compare body condition and body morphology across individuals and populations (Bierlich et al., 2022; Torres et al., 2022; Barlow et al., 2023). Rather than a single point-estimate of a length measurement for an individual, Xcertainty produces a distribution of length measurements for an individual so that the length of a whale can be described by the mean of this distribution, and its uncertainty as the the variance or an interval around the mean (Figure 4). These outputs ensure measurements are robust and comparable across different drones because they provide a measure of the uncertainty around each measurement. For instance, a measurement with more uncertainty will have a wider distribution. The uncertainty associated with each measurement can be incorporated into analyses, which is key when detecting important differences or changes in individuals or populations, such as changes in body condition (blog).

Figure 4. An example of a posterior predictive distribution for total length of an individual blue whale produced by the ‘Xcertainty’ R package. The black bars represent the uncertainty around the mean value (the black dot) – the longer black bars represent the 95% highest posterior density (HPD) interval, and the shorter black bars represent the 65% HPD interval.

CODEX has integrated all these lessons learned, open-source tools, and analytical approaches into a single framework of suggested best practices to help researchers enhance the quality, speed, and accuracy of obtaining important morphological measurements to manage vulnerable populations. These tools and frameworks are designed to be accommodating and accessible to researchers on various budgets and to facilitate cross-lab collaborations. CODEX plans to host workshops to educate and train researchers using drones on how to apply these tools within this framework within their own research practices. Potential future directions for CODEX include developing a system for using drones to drop suction-cup tags on whales and to collect thermal imagery of whales for health assessments. Stay up to date with all the CODEX ‘X’travaganza here: https://mmi.oregonstate.edu/centers-excellence/codex.

Huge shout out to Suzie Winquist for designing the artwork for CODEX!

References

Barlow, D.R., Bierlich, K.C., Oestreich, W.K., Chiang, G., Durban, J.W., Goldbogen, J.A., Johnston, D.W., Leslie, M.S., Moore, M.J., Ryan, J.P. and Torres, L.G., 2023. Shaped by Their Environment: Variation in Blue Whale Morphology across Three Productive Coastal Ecosystems. Integrative Organismal Biology, [online] 5(1). https://doi.org/10.1093/iob/obad039.

Bierlich, K., Karki, S., Bird, C.N., Fern, A. and Torres, L.G., n.d. Automated body length and condition measurements of whales from drone videos for rapid assessment of population health. Marine Mammal Science.

Bierlich, K.C., Hewitt, J., Bird, C.N., Schick, R.S., Friedlaender, A., Torres, L.G., Dale, J., Goldbogen, J., Read, A.J., Calambokidis, J. and Johnston, D.W., 2021a. Comparing Uncertainty Associated With 1-, 2-, and 3D Aerial Photogrammetry-Based Body Condition Measurements of Baleen Whales. Frontiers in Marine Science, 8. https://doi.org/10.3389/fmars.2021.749943.

Bierlich, K.C., Hewitt, J., Schick, R.S., Pallin, L., Dale, J., Friedlaender, A.S., Christiansen, F., Sprogis, K.R., Dawn, A.H., Bird, C.N., Larsen, G.D., Nichols, R., Shero, M.R., Goldbogen, J., Read, A.J. and Johnston, D.W., 2022. Seasonal gain in body condition of foraging humpback whales along the Western Antarctic Peninsula. Frontiers in Marine Science, 9(1036860), pp.1–16. https://doi.org/10.3389/fmars.2022.1036860.

Bierlich, K.C., Schick, R.S., Hewitt, J., Dale, J., Goldbogen, J.A., Friedlaender, A.S. and Johnston, D.W., 2021b. Bayesian approach for predicting photogrammetric uncertainty in morphometric measurements derived from drones. Marine Ecology Progress Series, 673, pp.193–210. https://doi.org/10.3354/meps13814.

Bird, C. and Bierlich, K.C., 2020. CollatriX: A GUI to collate MorphoMetriX outputs. Journal of Open Source Software, 5(51), pp.2323–2328. https://doi.org/10.21105/joss.02328.

Torres, L.G., Bird, C.N., Rodríguez-González, F., Christiansen, F., Bejder, L., Lemos, L., Urban R, J., Swartz, S., Willoughby, A., Hewitt, J. and Bierlich, K.C., 2022. Range-Wide Comparison of Gray Whale Body Condition Reveals Contrasting Sub-Population Health Characteristics and Vulnerability to Environmental Change. Frontiers in Marine Science, 9(April), pp.1–13. https://doi.org/10.3389/fmars.2022.867258.

Torres, W. and Bierlich, K.C., 2020. MorphoMetriX: a photogrammetric measurement GUI for morphometric analysis of megafauna. Journal of Open Source Software, 5(45), pp.1825–1826. https://doi.org/10.21105/joss.01825.

The Land of Maps and Charts: Geospatial Ecology

By Alexa Kownacki, Ph.D. Student, OSU Department of Fisheries and Wildlife, Geospatial Ecology of Marine Megafauna Lab

I love maps. I love charts. As a random bit of trivia, there is a difference between a map and a chart. A map is a visual representation of land that may include details like topology, whereas a chart refers to nautical information such as water depth, shoreline, tides, and obstructions.

Map of San Diego, CA, USA. (Source: San Diego Metropolitan Transit System)

Chart of San Diego, CA, USA. (Source: NOAA)

I have an intense affinity for visually displaying information. As a child, my dad traveled constantly, from Barrow, Alaska to Istanbul, Turkey. Immediately upon his return, I would grab our standing globe from the dining room and our stack of atlases from the coffee table. I would sit at the kitchen table, enthralled at the stories of his travels. Yet, a story was only great when I could picture it for myself. (I should remind you, this was the early 1990s, GoogleMaps wasn’t a thing.) Our kitchen table transformed into a scene from Master and Commander—except, instead of nautical charts and compasses, we had an atlas the size of an overgrown toddler and salt and pepper shakers to pinpoint locations. I now had the world at my fingertips. My dad would show me the paths he took from our home to his various destinations and tell me about the topography, the demographics, the population, the terrain type—all attribute features that could be included in common-day geographic information systems (GIS).

Uncle Brian showing Alexa where they were on a map of Maui, Hawaii, USA. (Photo: Susan K. circa 1995)

As I got older, the kitchen table slowly began to resemble what I imagine the set from Master and Commander actually looked like; nautical charts, tide tables, and wind predictions were piled high and the salt and pepper shakers were replaced with pencil marks indicating potential routes for us to travel via sailboat. The two of us were in our element. Surrounded by visual and graphical representations of geographic and spatial information: maps. To put my map-attraction this in even more context, this is a scientist who grew up playing “Take-Off”, a board game that was “designed to teach geography” and involved flying your fleet of planes across a Mercator projection-style mapboard. Now, it’s no wonder that I’m a graduate student in a lab that focuses on the geospatial aspects of ecology.

A precocious 3-year-old Alexa, sitting with the airplane pilot asking him a long list of travel-related questions (and taking his captain’s hat). Photo: Susan K.

So why and how did geospatial ecology became a field—and a predominant one at that? It wasn’t that one day a lightbulb went off and a statistician decided to draw out the results. It was a progression, built upon for thousands of years. There are maps dating back to 2300 B.C. on Babylonian clay tablets (The British Museum), and yet, some of the maps we make today require highly sophisticated technology. Geospatial analysis is dynamic. It’s evolving. Today I’m using ArcGIS software to interpolate mass amounts of publicly-available sea surface temperature satellite data from 1981-2015, which I will overlay with a layer of bottlenose dolphin sightings during the same time period for comparison. Tomorrow, there might be a new version of software that allows me to animate these data. Heck, it might already exist and I’m not aware of it. This growth is the beauty of this field. Geospatial ecology is made for us cartophiles (map-lovers) who study the interdependency of biological systems where location and distance between things matters.

Alexa’s grandmother showing Alexa (a very young cartographer) how to color in the lines. Source: Susan K. circa 1994

In a broader context, geospatial ecology communicates our science to all of you. If I posted a bunch of statistical outputs in text or even table form, your eyes might glaze over…and so might mine. But, if I displayed that same underlying data and results on a beautiful map with color-coded symbology, a legend, a compass rose, and a scale bar, you might have this great “ah-ha!” moment. That is my goal. That is what geospatial ecology is to me. It’s a way to SHOW my science, rather than TELL it.

Would you like to see this over and over again…?

A VERY small glimpse into the enormous amount of data that went into this map. This screenshot gave me one point of temperature data for a single location for a single day…Source: Alexa K.

Or see this once…?

Map made in ArcGIS of Coastal common bottlenose dolphin sightings between 1981-1989 with a layer of average sea surface temperatures interpolated across those same years. A picture really is worth a thousand words…or at least a thousand data points…Source: Alexa K.

For many, maps are visually easy to interpret, allowing quick message communication. Yet, there are many different learning styles. From my personal story, I think it’s relatively obvious that I’m, at least partially, a visual learner. When I was in primary school, I would read the directions thoroughly, but only truly absorb the material once the teacher showed me an example. Set up an experiment? Sure, I’ll read the lab report, but I’m going to refer to the diagrams of the set-up constantly. To this day, I always ask for an example. Teach me a new game? Let’s play the first round and then I’ll pick it up. It’s how I learned to sail. My dad described every part of the sailboat in detail and all I heard was words. Then, my dad showed me how to sail, and it came naturally. It’s only as an adult that I know what “that blue line thingy” is called. Geospatial ecology is how I SEE my research. It makes sense to me. And, hopefully, it makes sense to some of you!

Alexa’s dad teaching her how to sail. (Source: Susan K. circa 2000)

Alexa’s first solo sailboat race in Coronado, San Diego, CA. Notice: Alexa’s dad pushing the bow off the dock and the look on Alexa’s face. (Source: Susan K. circa 2000)

Alexa mapping data using ArcGIS in the Oregon State University Library. (Source: Alexa K circa a few minutes prior to posting).

I strongly believe a meaningful career allows you to highlight your passions and personal strengths. For me, that means photography, all things nautical, the great outdoors, wildlife conservation, and maps/charts. If I converted that into an equation, I think this is a likely result:

Photography + Nautical + Outdoors + Wildlife Conservation + Maps/Charts = Geospatial Ecology of Marine Megafauna

Or, better yet:

? + ⚓ + ? + ? + ? = GEMM Lab

This lab was my solution all along. As part of my research on common bottlenose dolphins, I work on a small inflatable boat off the coast of California (nautical ✅, outdoors ✅), photograph their dorsal fin (photography ✅), and communicate my data using informative maps that will hopefully bring positive change to the marine environment (maps/charts ✅, wildlife conservation✅). Geospatial ecology allows me to participate in research that I deeply enjoy and hopefully, will make the world a little bit of a better place. Oh, and make maps.

Share this:

Share this:

Share this: