Oregon State Physics is leading a Department of Energy Office of Science funded project to design computing and software infrastructure for the DUNE experiment. DUNE is a future neutrino experiment that will aim a neutrino beam from Fermilab, in Batavia Illinois, at a very large detector in the Homestake mine in Lead, South Dakota. The experiment is currently under construction with a 5% prototype running at CERN in 2018 and 2022 and the full detector expected in 2029. These experiments generate data at rates of 1-2 GB/sec, or 30 PB/year which must be stored, processed and distributed to over 1,000 scientists worldwide.
The project “Essential Computing and Software Development for the DUNE experiment” is funded for 3M$ over 3 years, shared among 4 Universities (Oregon State, Colorado State, Minnesota and Wichita State) and three national laboratories (Argonne National Laboratory, Fermi National Laboratory and Brookhaven National Laboratory). The collaborators will work with colleagues worldwide on advanced data storage systems, high performance computing and databases in support of the DUNE physics mission. See https://www.dunescience.org/ for more information on the experiment.
PI Heidi Schellman (Oregon State Physics) leads the DUNE computing and software consortium which is responsible for the international DUNE computing project. Physics graduate student Noah Vaughan helps oversee the global grid processing systems that DUNE uses for data reconstruction and simulation and recent graduate Amit Bashyal helped design the DUNE/LBNF beamline. Graduate student Sean Gilligan is performing a statistical analysis of data transfer patterns to help optimize the design of the worldwide data network. Postdoc Jake Calcutt recently joined us from Michigan State University and is designing improved methods for producing data analysis samples for the ProtoDUNE experiment at CERN.
One of the major thrusts of the Oregon State project is the design of robust data storage and delivery systems optimized for data integrity and reproducibility. 30 PB/year of data will be distributed worldwide and processed through a complex chain of algorithms. End users need to know the exact provenance of their data – how was it produced, how was it processed, was any data lost – to ensure scientific reproducibility over the decades that the experiments will run. Preliminary versions of the data systems have already led to results from the protoDUNE prototype experiments at CERN which are described in https://doi.org/10.1088/1748-0221/15/12/P12004 and https://doi.org/10.1051/epjconf/202024511002.
As an example of this work, three Oregon State Computer Science Majors (Lydia Brynmoor, Zach Lee and Luke Penner) worked with Fermilab scientist Steven Timm on a global monitor for the Rucio storage system shown below. This illustrates test data transfers between compute sites in the US, Brazil and Europe. The dots indicate compute sites in the DUNE compute grid while the lines illustrate test transfers.
Other projects will be a Data Dispatcher which optimizes the delivery of data to CPU’s across the DUNE compute systems and monitoring of data streaming between sites.
Jen Hobbs worked in the Schellman group as an undergraduate and beginning grad student at Northwestern. She and Howard Budd (Rochester) built the testing system for all 32,000 scintillating fibers for the MINERvA experiment using LabView. I knew she was good when she got frustrated with the slow device drivers and rewrote them during her first summer at Fermilab. When MINERvA was complete she switched to neuroscience so she could continue doing hands-on stuff in the lab.
Her dissertation was on the relation between physical motion and neural signals from rat whiskers and won the Journal of Experimental Biology Outstanding Paper Award in 2015.
Junior Physics major Kaseylin Yoke was awarded a $5,000 SURE Science Summer Fellowship by the College of Science. The SURE Fellowship supports students to do research over the summer. She spent the summer learning to analyze data from the MINERvA experimentand finished by spending two weeks in Illinois at Fermilab working with graduate students Amit Bashyal and Sean Gilligan and seeing the apparatus in the flesh.
She delivered a very popular poster on her work at the SURE Science Symposium in early September.
Chris Pratt analyzed Z boson decays on the D0 experiment while getting degrees in Integrated Science and Mathematics (with a certificate in Finance from Kellogg) at Northwestern. He uses the data analysis skills he learned in the Schellman group as an Associate Analyst at NERA Economic Consulting in Chicago.
Jason Stein wrote his undergraduate thesis with the Schellman group on “Theoretical Calculation of the Charge Asymmetry Uncertainties Using the CTEQ6 Parton Distribution Function Set.” as a student in the Integrated Science Program at Northwestern University. He also helped create the D0 experiment luminosity readout system. He went on to graduate study and postdoctoral fellowships in neuroscience at UCLA and has just accepted a faculty position in genetics and neuroscience at the University of North Carolina, Chapel Hill. See his lab page at http://www.steinlab.org/ .
Vasiliy Kuznetsov worked with the Schellman group on the MINERvA experiment starting in his freshman year at Northwestern. He used Python and Postgres to make a client-server high voltage control system and a tracking database for calibration constants. He ended up graduating Magna cum Laude with majors in Mathematics and Economics with a minor in Business Institutions. He now uses the data skills he learned on MINERvA as a Software Engineer at facebook
Brandon Walker graduated from Northwestern in 2010 with Bachelor’s degrees in Physics and Astronomy and in Mathematics. He did his honors thesis in the Schellman group on `An Algorithm for Particle Tracking and Analysis of Muons in the Main Injector Experiment v-A (MINERvA).”
He is currently a doctoral student in Medical Physics at the University of Wisconsin at Madison.
For his PhD, he’s designing and building a modular multi-source electron beam scanner for high speed computed tomography and 3D printing applications. The system would enable ultra-fast CT scans for improved image quality in cardiac imaging and could be a game changer for 3D printing. The Wisconsin Alumni Research Foundation (WARF) has filed two patents for the project, one in 2014 and another in 2015 (patents pending).
He has also co-founded 2 startup companies. Formula Database, GelCombs and has his own consulting company that does quality assurance for radiation and diagnostic imaging products.