Sept. 29, 2021

Oregon State Physics is leading a Department of Energy Office of Science funded project to design computing and software infrastructure for the DUNE experiment.   DUNE is a future neutrino experiment that will aim a neutrino beam from Fermilab, in Batavia Illinois, at a very large detector in the Homestake mine in Lead, South Dakota.  The experiment is currently under construction with a 5% prototype running at CERN in 2018 and 2022 and the full detector expected in 2029. These experiments generate data at rates of 1-2 GB/sec, or 30 PB/year which must be stored, processed and distributed to over 1,000 scientists worldwide.

The LBNF neutrino beam traveling (right to left) from Fermilab in Illinois to the Sanford Lab in South Dakota

The project “Essential Computing and Software Development for the DUNE experiment” is funded for 3M$ over 3 years, shared among 4 Universities (Oregon State, Colorado State, Minnesota and Wichita State) and three national laboratories (Argonne National Laboratory, Fermi National Laboratory and Brookhaven National Laboratory). The collaborators will work with colleagues worldwide on advanced data storage systems, high performance computing and databases in support of the DUNE physics mission.  See for more information on the experiment.

PI Heidi Schellman (Oregon State Physics) leads the DUNE computing and software consortium which is responsible for the international DUNE computing project. Physics graduate student Noah Vaughan helps oversee the global grid processing systems that DUNE uses for data reconstruction and simulation and recent graduate Amit Bashyal helped design the DUNE/LBNF beamline.  Graduate student Sean Gilligan is performing a statistical analysis of data transfer patterns to help optimize the design of the worldwide data network.  Postdoc Jake Calcutt recently joined us from Michigan State University and is designing improved methods for producing data analysis samples for the ProtoDUNE experiment at CERN.

One of the major thrusts of the Oregon State project is the design of robust data storage and delivery systems optimized for data integrity and reproducibility.  30 PB/year of data will be distributed worldwide and processed through a complex chain of algorithms. End users need to know the exact provenance of their data –  how was it produced, how was it processed, was any data lost – to ensure scientific reproducibility over the decades that the experiments will run.  Preliminary versions of the data systems have already led to results from the protoDUNE prototype experiments at CERN which are described in and

As an example of this work, three Oregon State Computer Science Majors (Lydia Brynmoor, Zach Lee and Luke Penner) worked with Fermilab scientist Steven Timm on a global monitor for the Rucio storage system shown below. This illustrates test data transfers between compute sites in the US, Brazil and Europe. The dots indicate compute sites in the DUNE compute grid while the lines illustrate test transfers. 

Data transfer diagram for DUNE data.

Other projects will be a Data Dispatcher which optimizes the delivery of data to CPU’s across the DUNE compute systems and monitoring of data streaming between sites.

Jennifer Hobbs – Data Scientist

Jen Hobbs worked in the Schellman group as an undergraduate and beginning grad student at Northwestern.  She and Howard Budd (Rochester) built the testing system for all 32,000 scintillating fibers for the MINERvA experiment using LabView.  I knew she was good when she got frustrated with the slow device drivers and rewrote them during her first summer at Fermilab. When MINERvA was complete she switched to neuroscience so she could continue doing hands-on stuff in the lab.

Her dissertation was on the relation between physical motion and neural signals from rat whiskers and won the Journal of Experimental Biology Outstanding Paper Award in 2015.

She then became a data scientist applying her expertise in quantitative studies of motion to – SPORTS!

She’s featured in the most recent issue of Symmetry Magazine

Kaseylin Yoke shows her poster on MINERvA data quality at the SURE Science Symposium.

Junior Physics major Kaseylin Yoke was awarded a $5,000 SURE Science Summer Fellowship by the College of Science.  The SURE Fellowship supports students to do research over the summer.  She spent the summer learning to analyze data from the MINERvA experimentand finished by spending two weeks in Illinois at Fermilab working with graduate students Amit Bashyal and Sean Gilligan and seeing the apparatus in the flesh.

She delivered a very popular poster on her work at the SURE Science Symposium in early September.

Chris Pratt analyzed Z boson decays on the D0 experiment while getting degrees in Integrated Science and Mathematics (with a certificate in Finance from Kellogg) at Northwestern.  He uses the data analysis skills he learned in the Schellman group as an Associate Analyst at NERA Economic Consulting in Chicago.

Chris and Miranda at the Great Wall.  A long way from the Quad Cities!
Chris and Miranda at the Great Wall. A long way from the Quad Cities!

Gabriel Nowak is a Junior in the Physics program at Oregon State.  He’s been in the Schellman group for 6 months and is writing a data comparator for the MINERvA and MicroBooNE experiments at Fermilab.


Gabriel Nowak working on the data comparator in the Fermilab neutrino control room.
Gabriel Nowak working on the data comparator in the Fermilab neutrino control room.


This link compares two versions of the MINERvA simulation code.  It combines  the CDF data validation code with Gabriel’s histogram generator.

Jason Stein, former physics student, now a neuroscientist.
Jason Stein, former physics student, now a neuroscientist.

Jason Stein wrote his undergraduate thesis with the Schellman group on “Theoretical Calculation of the Charge Asymmetry Uncertainties Using the CTEQ6 Parton Distribution Function Set.” as a student in the Integrated Science Program at Northwestern University. He also helped create the D0 experiment luminosity readout system.  He went on to graduate study and postdoctoral fellowships in neuroscience at UCLA and has just accepted a faculty position in genetics and neuroscience at the University of North Carolina, Chapel Hill.  See his lab page at .


The Luminosity group in 2005, Tim Andeen, Michelle Reschke, Sahal Yacoob, Terry Toole, Marco Verzocchi, Elizabeth Gallas and Jason Stein
The Luminosity group in 2005, Tim Andeen, Michelle Reschke, Sahal Yacoob, Terry Toole, Marco Verzocchi, Elizabeth Gallas and Jason Stein


Vasiliy Kuznetsov worked with the Schellman group on the MINERvA experiment starting in his freshman year at Northwestern.  He used Python and Postgres to make a client-server high voltage control system and a tracking database for calibration constants.  He ended up graduating Magna cum Laude with majors in Mathematics and Economics with a minor in Business Institutions.  He now uses the data skills he learned on MINERvA as a Software Engineer at facebook

Vasiliy enjoys the Bay Area

Brandon Walker graduated from Northwestern in 2010 with Bachelor’s degrees in Physics and Astronomy and in Mathematics.  He did his honors thesis in the Schellman group on `An Algorithm for Particle Tracking and Analysis of Muons in the Main Injector Experiment v-A (MINERvA).”


He is currently a doctoral student in Medical Physics at the University of Wisconsin at Madison.

Brandon Walker (center) helps assemble the MINERvA detector 300 feet below Fermilab
Brandon Walker (center) helps assemble the MINERvA detector 300 feet below Fermilab

For his PhD, he’s designing and building a modular multi-source electron beam scanner for high speed computed tomography and 3D printing applications. The system would enable ultra-fast CT scans for improved image quality in cardiac imaging and could be a game changer for 3D printing. The Wisconsin Alumni Research Foundation (WARF) has filed two patents for the project, one in 2014 and another in 2015 (patents pending).

He has also co-founded 2 startup companies. Formula Database, GelCombs and has his own consulting company that does quality assurance for radiation and diagnostic imaging products.