Oregon State Physics is leading a Department of Energy Office of Science funded project to design computing and software infrastructure for the DUNE experiment. DUNE is a future neutrino experiment that will aim a neutrino beam from Fermilab, in Batavia Illinois, at a very large detector in the Homestake mine in Lead, South Dakota. The experiment is currently under construction with a 5% prototype running at CERN in 2018 and 2022 and the full detector expected in 2029. These experiments generate data at rates of 1-2 GB/sec, or 30 PB/year which must be stored, processed and distributed to over 1,000 scientists worldwide.
The LBNF neutrino beam traveling (right to left) from Fermilab in Illinois to the Sanford Lab in South Dakota
The project “Essential Computing and Software Development for the DUNE experiment” is funded for 3M$ over 3 years, shared among 4 Universities (Oregon State, Colorado State, Minnesota and Wichita State) and three national laboratories (Argonne National Laboratory, Fermi National Laboratory and Brookhaven National Laboratory). The collaborators will work with colleagues worldwide on advanced data storage systems, high performance computing and databases in support of the DUNE physics mission. See https://www.dunescience.org/ for more information on the experiment.
PI Heidi Schellman (Oregon State Physics) leads the DUNE computing and software consortium which is responsible for the international DUNE computing project. Physics graduate student Noah Vaughan helps oversee the global grid processing systems that DUNE uses for data reconstruction and simulation and recent graduate Amit Bashyal helped design the DUNE/LBNF beamline. Graduate student Sean Gilligan is performing a statistical analysis of data transfer patterns to help optimize the design of the worldwide data network. Postdoc Jake Calcutt recently joined us from Michigan State University and is designing improved methods for producing data analysis samples for the ProtoDUNE experiment at CERN.
One of the major thrusts of the Oregon State project is the design of robust data storage and delivery systems optimized for data integrity and reproducibility. 30 PB/year of data will be distributed worldwide and processed through a complex chain of algorithms. End users need to know the exact provenance of their data – how was it produced, how was it processed, was any data lost – to ensure scientific reproducibility over the decades that the experiments will run. Preliminary versions of the data systems have already led to results from the protoDUNE prototype experiments at CERN which are described in https://doi.org/10.1088/1748-0221/15/12/P12004 and https://doi.org/10.1051/epjconf/202024511002.
As an example of this work, three Oregon State Computer Science Majors (Lydia Brynmoor, Zach Lee and Luke Penner) worked with Fermilab scientist Steven Timm on a global monitor for the Rucio storage system shown below. This illustrates test data transfers between compute sites in the US, Brazil and Europe. The dots indicate compute sites in the DUNE compute grid while the lines illustrate test transfers.
Data transfer diagram for DUNE data.
Other projects will be a Data Dispatcher which optimizes the delivery of data to CPU’s across the DUNE compute systems and monitoring of data streaming between sites.
His thesis combines his work on neutrino fluxes for the DUNE and MINERvA experiments, which has been submitted to JINST and a new high statistics measurement of anti-neutrino quasi-elastic scattering which we’re writing up.
He is moving to Argonne National Laboratory where he will be working on the HCP-CCE project and the DUNE collaboration.
The answer to one of the most fundamental questions in physics may lie in the universe’s most abundant matter particle: the neutrino. But interpreting and verifying data from neutrino experiments require reducing uncertainties in some very complicated particle interactions.
To advance this pivotal work, Fermilab scientist Laura Fields received a U.S. Department of Energy Early Career Research Award, an annual recognition of rising scientists in universities and national laboratories. The five-year $2.5 million grant will support Fields in identifying and reducing uncertainties in the composition of neutrino beams — improving measurements and boosting confidence in physics experiments with important implications.
According to the working theories, the Big Bang should have generated roughly equal amounts of matter and antimatter, which annihilate one another upon contact. However, from what we can tell, the universe seems to be composed almost entirely of matter (otherwise our universe would be empty or perhaps full of the mysterious dark matter).
To pin down the root of this mystery, scientists are looking to neutrinos — ghostly particles that sail through the universe and pass undetected through your body by the trillion every second — for clues. Aside from being frustratingly difficult to detect, neutrinos have another strange quality: They shape-shift. There are three different types, or “flavors,” of neutrinos (electron, muon and tau types), and the particles oscillate between these states as they travel.
In the search for the root of the matter-antimatter asymmetry, scientists are studying whether neutrinos and antineutrinos oscillate in the same way. To test this hypothesis, the international Deep Underground Neutrino Experiment, or DUNE, hosted by Fermilab, will use magnetic instruments called focusing horns to produce beams of neutrinos and antineutrinos and measure their oscillation rates with two different detectors — a near detector on the lab site and a far detector underground in South Dakota. A difference between the shape-shifting tendencies of matter and antimatter particles would violate the supposed symmetry and could help scientists understand why everything exists.
Precise measurements of neutrino oscillations require precise simulations. Toolkits such as Geant4, widely used in high-energy physics, simulate the passage of particles through matter, predicting what happens when particles interact with nuclei to form new particles. But the models are not completely accurate.
“The problem is that we don’t really know very well how many neutrinos are in our beam or what their flavor and energy composition is,” Fields said. “Because of all of these uncertainties, what Geant4 predicts will happen in the beamline may not be exactly what happens.”
Fields is working to reduce the uncertainty in the number, energy and flavor — known collectively as the “flux” — of the neutrinos below the current estimate of 10%.
Scientists make up for flux uncertainties by taking more data on similar particle interactions in different contexts. The extra data on neutrinos and the particles that produce them allow for more accurate simulations and more confident inferences.
“In addition to oscillations, DUNE will also make a lot of measurements at the near detector looking for beyond-the-Standard Model physics,” Fields said. “And the better we know our beam, the more certain we can be that any unexpected data is due to some exciting new physics rather than not understanding our equipment.”
A focusing horn focuses beams of charged particles, some of which eventually decay into neutrinos. Part of Fields’s work is to measure the charged particles after they’ve been focused. To date, nobody has precisely studied these particles downstream of a focusing horn. Photo: Reidar Hahn, Fermilab
Fermilab accelerators produce the DUNE neutrino beams by shooting protons at a solid target. The proton-target interaction gives off charged particles that are focused into beams and then decay into neutrinos. Because the parent particles are charged and focused, the chargeless neutrinos continue on in the same direction.
Fields’s proposal aims to cut the beam flux uncertainty in half by studying and modeling the interactions that produce neutrinos in two different ways.
First, other experiments, including the NA61 at CERN where Fields is a collaborator, already study target-based nuclear interactions. Fields plans to bring a DUNE target core to the NA61 experiment and analyze the data to learn more about the particles that decay into neutrinos.
Second, the proposal calls for measuring the neutrino-producing particles after they’re focused by a magnetic horn. To date, nobody has precisely studied these particles downstream of a focusing horn. To pull it off, Fields and her colleagues will use another Fermilab experiment, called EMPHATIC, and build a precision-motion table to move the detectors and scan the whole horn.
Aside from funding the construction of the precision-motion table and the transport and powering of a focusing horn, the Early Career Research Award grant will fund 50% of Fields’s time and cover two postdoctoral research associates who will help with data collection and analysis.
The data for uncertainty reduction will be made public and eventually used to tune Geant4 simulations and improve nuclear physics models. At the very least, Fields’s work will boost the accuracy and reliability of DUNE’s findings.
“Laura is doing the foundational work that is critical for an experiment like DUNE to succeed,” said James Amundson, Scientific Computing Division head at Fermilab. “Her effort should make the whole process more efficient so we can get more physics out of the experiment.”
Fermilab neutrino research is supported by the DOE Office of Science.
Fermilab is supported by the Office of Science of the U.S. Department of Energy. The Office of Science is the single largest supporter of basic research in the physical sciences in the United States and is working to address some of the most pressing challenges of our time. For more information, visit science.energy.gov.
Postdoc Mateus Carneiro has moved on to a position at Brookhaven National Laboratory but he left us with a really nice paper in Physics Review Letters.
M.Carneiro et al. [MINERvA], “High-Statistics Measurement of Neutrino Quasielasticlike Scattering at 6 GeV on a Hydrocarbon Target”, Phys. Rev. Lett. 124, no.12, 121801 (2020), doi:10.1103/PhysRevLett.124.121801
Graduate Student Amit Bashyal did many of the cross-checks for this complicated measurement and wrote a summary for FermiNews which is copied below. Amit’s thesis topic is the parallel measurement for anti-neutrinos.
Hard to believe you can play pool with neutrinos, but certain neutrino interaction events are closer to the game than you think.
In these charged-current quasielastic interactions — let’s call them CCQE interactions for short — a neutrino strikes a particle in an atom’s nucleus — a proton or a neutron. Two particles emerge from the collision. One is a muon, a heavier cousin of the electron. The other is either a proton (if the stationary particle is a neutron) or a neutron (if the stationary particle is a proton).
The neutrino interactions that result from these quasielastic reactions are like the collisions between balls in a game of pool: You can guess the energy of the incoming neutrino by measuring the direction and energy of only one of the outgoing particles, provided you know the types of all four particles that were in the interaction in the first place and the original direction of the neutrino.
CCQE interactions are an important interaction mode of neutrinos in current and future neutrino oscillation experiments, such as the international Deep Underground Neutrino Experiment, hosted by Fermilab.
They are similar to the elastic interactions every pool player knows except in one important way: The weak nuclear force allows the particles to change from one kind into another, hence the “quasielastic” name. In this subatomic pool game, the cue ball (neutrino) strikes a stationary red ball (proton), which emerges from the collision as an orange ball (neutron).
This display shows a CCQE-like event reconstructed in the MINERvA detector. Image: MINERvA
Since most modern neutrino experiments use targets made of heavy nuclei ranging from carbon to argon, nuclear effects and correlations between the neutrons and protons inside the nucleus can cause significant changes in the observed interaction rates and modifications to the estimated neutrino energy.
At MINERvA, scientists identify the CCQE interactions by a long muon track left in the particle detector and potentially one or more proton tracks. However, this experimental signature can sometimes be produced by non-CCQE interactions due to nuclear effects inside the target nucleus. Similarly, nuclear effects can also modify the final-state particles to make a CCQE event look like a non-CCQE event and vice versa.
Since nuclear effects can make it challenging to identify a true CCQE event, MINERvA reports measurements based on the properties of the final-state particles only and calls them CCQE-like events (since they will have contributions from both true CCQE and non-CCQE events). A CCQE-like event is one that has at least one outgoing muon, any number of protons or neutrons, and no mesons as final-state particles. (Mesons, like protons and neutrons, are made of quarks. Protons and neutrons have three quarks; mesons have two.)
MINERvA has measured the likelihood of CCQE-like neutrino interactions using Fermilab’s medium-energy neutrino beam, with the neutrino flux peaking at 6 GeV. Compared to MINERvA’s earlier measurements, which were conducted with a low-energy beam (3 GeV peak neutrino flux), this measurement has the advantage of a broader energy reach and much larger statistics: 1,318,540 CCQE-like events compared to 109,275 events in earlier low-energy runs.
MINERvA made these CCQE interaction probability measurements as a function of the square of the momentum transferred by the neutrino to the nucleus, which scientists denote as Q2. The plot shows discrepancies between the data and most predictions in low-Q2 and high-Q2 regions. By comparing MINERvA’s measurement with various models, scientists can refine them and better explain the physics inside the nuclear environment.
This plot shows the ratio of cross-section as a function of Q2 of data and various predictions with respectto one commonly used interaction model. Image: MINERvA
MINERvA has also made more detailed measurements of the probability of neutrino interaction based on the outgoing muon’s momentum. They take into account the muon’s momentum both in the direction of the incoming neutrino’s trajectory and in the direction perpendicular to its trajectory. This work helps current and future neutrino experiments understand their own data over a wide range of muon kinematics.
Mateus Carneiro, formerly of the Brazilian Center for Research in Physics and Oregon State University and now at Brookhaven National Laboratory, and Dan Ruterbories of the University of Rochester were the main drivers of this analysis. The results were published in Physical Review Letters.
Amit Bashyal is an Oregon State University scientist on the MINERvA experiment.
This work is supported by the DOE Office of Science, National Science Foundation, Coordination for the Improvement of Higher Education Personnel in Brazil, Brazilian National Council for Scientific and Technological Development, Mexican National Council of Science and Technology, Basal Project in Chile, Chilean National Commission for Scientific and Technological Research, Chilean National Fund for Scientific and Technological Development, Peruvian National Council for Science, Technology and Technological Innovation, Research Management Directorate at the Pontifical Catholic University of Peru, National University of Engineering in Peru, Polish National Science Center and UK Science and Technology Facilities Council.
Fermi National Accelerator Laboratory is supported by the Office of Science of the U.S. Department of Energy. The Office of Science is the single largest supporter of basic research in the physical sciences in the United States and is working to address some of the most pressing challenges of our time. For more information, visit science.energy.gov.
Amit Bashyal (far right) at the Neutrino summer school in Illinois.
5th year grad student Amit Bashyal’s team won a group presentation prize at the International Neutrino Summer School for their study of magnetization of DUNE experiment far detectors. Students were divided into groups given a short time to do an original study and then present it! Congratulations Amit and team!
3 other Schellman group members and alumni are pictured. Laura Fields (Fermilab, 6th from left) and Cheryl Patrick (UCL, 7th from left) were lecturers at the summer school and 2nd year student Sean Gilligan (3rd from right) also attended.
Jen Hobbs worked in the Schellman group as an undergraduate and beginning grad student at Northwestern. She and Howard Budd (Rochester) built the testing system for all 32,000 scintillating fibers for the MINERvA experiment using LabView. I knew she was good when she got frustrated with the slow device drivers and rewrote them during her first summer at Fermilab. When MINERvA was complete she switched to neuroscience so she could continue doing hands-on stuff in the lab.
Her dissertation was on the relation between physical motion and neural signals from rat whiskers and won the Journal of Experimental Biology Outstanding Paper Award in 2015.
Kaseylin Yoke shows her poster on MINERvA data quality at the SURE Science Symposium.
Junior Physics major Kaseylin Yoke was awarded a $5,000 SURE Science Summer Fellowship by the College of Science. The SURE Fellowship supports students to do research over the summer. She spent the summer learning to analyze data from the MINERvA experimentand finished by spending two weeks in Illinois at Fermilab working with graduate students Amit Bashyal and Sean Gilligan and seeing the apparatus in the flesh.
She delivered a very popular poster on her work at the SURE Science Symposium in early September.
Differential QE-like cross section dσ(E_QEν)/dQ^2_QE, in bins of E_QEν. Inner error bars show statistical uncertainties; outer error bars show total (statistical and systematic) uncertainty. The red histogram shows the MINERvA-tuned GENIE model used to estimate smearing and acceptance.
The much anticipated paper version of Cheryl Patrick’s thesis has been accepted by Physical Review D. Check it out at:
