Uproot is a library for reading and writing ROOT files in pure Python and NumPy.

Scikit-HEP Project

Last update: Dec 31, 2022

Related tags

File & Path Utilities python big-data analysis numpy bigdata file-format root hep root-cern hep-ex scikit-hep hep-py

Overview

Uproot is a library for reading and writing ROOT files in pure Python and NumPy.

Unlike the standard C++ ROOT implementation, Uproot is only an I/O library, primarily intended to stream data into machine learning libraries in Python. Unlike PyROOT and root_numpy, Uproot does not depend on C++ ROOT. Instead, it uses Numpy to cast blocks of data from the ROOT file as Numpy arrays.

Installation

Uproot can be installed from PyPI using pip. Awkward Array is optional but highly recommended:

pip install uproot awkward

Uproot is also available using conda (in this case, Awkward Array is automatically installed):

conda install -c conda-forge uproot

If you have already added conda-forge as a channel, the -c conda-forge is unnecessary. Adding the channel is recommended because it ensures that all of your packages use compatible versions (see conda-forge docs):

conda config --add channels conda-forge
conda update --all

Getting help

Start with the tutorials and reference documentation.

Report bugs, request features, and ask for additional documentation on GitHub Issues.
If you have a "How do I...?" question, start a GitHub Discussion with category "Q&A".
Alternatively, ask about it on StackOverflow with the [uproot] tag. Be sure to include tags for any other libraries that you use, such as Pandas or PyTorch.
To ask questions in real time, try the Gitter Scikit-HEP/uproot chat room.

Installation for developers

Uproot is an ordinary Python library; you can get a copy of the code with

git clone https://github.com/scikit-hep/uproot4.git

and install it locally by calling pip install . in the repository directory.

If you need to develop Awkward Array as well, see its installation for developers.

Dependencies

Uproot's only strict dependency is NumPy. This is the only dependency that pip will automatically install.

Awkward Array is highly recommended. It is not a strict dependency to allow Uproot to be used in restrictive environments. If you're using Uproot without Awkward Array, you'll have to use the library="np" option or globally set uproot.default_library to return arrays as NumPy arrays (see documentation).

awkward: be sure to use Awkward Array 1.x.

The following libraries are also useful in conjunction with Uproot, but are not necessary. If you call a function that needs one, you'll be prompted to install it. (Conda installs most of these automatically.)

For ROOT files, compressed different ways:

lz4 and xxhash: only if reading ROOT files that have been LZ4-compressed.
zstandard: only if reading ROOT files that have been ZSTD-compressed.
backports.lzma: only if reading ROOT files that have been LZMA-compressed (in Python 2).

For remote data:

xrootd: only if reading files with root:// URLs.

For exporting data to other libraries:

pandas: only if library="pd".
cupy: only if library="cp" (reads arrays onto GPUs).
boost-histogram: only if converting histograms to boost-histogram with histogram.to_boost().
hist: only if converting histograms to hist with histogram.to_hist().

Acknowledgements

Support for this work was provided by NSF cooperative agreement OAC-1836650 (IRIS-HEP), grant OAC-1450377 (DIANA/HEP) and PHY-1520942 (US-CMS LHC Ops).

Thanks especially to the gracious help of Uproot contributors (including the original repository).

_{Jim Pivarski} 💻 📖 🚇 🚧	_{Pratyush Das} 💻 🚇	_{Chris Burr} 💻 🚇	_{Dmitri Smirnov} 💻	_{Matthew Feickert} 🚇	_{Tamas Gal} 💻	_{Luke Kreczko} 💻 ⚠️
_{Nicholas Smith} 💻	_{Noah Biederbeck} 💻	_{Oksana Shadura} 💻 🚇	_{Henry Schreiner} 💻 🚇 ⚠️	_{Mason Proffitt} 💻 ⚠️	_{Jonas Rembser} 💻	_benkrikler 💻
_{Hans Dembinski} 📖	_{Marcel R.} 💻	_{Ruggero Turra} 💻	_{Jonas Rübenach} 💻	_bfis 💻	_{Raymond Ehlers} 💻	_{Andrzej Novak} 💻
_{Josh Bendavid} 💻	_{Doug Davis} 💻	_{Chao Gu} 💻	_{Lukas Koch} 💻	_{Michele Peresano} 💻	_Edoardo 💻	_{JMSchoeffmann} 💻
_{alexander-held} 💻	_{Giordon Stark} 💻	_{Ryunosuke O'Neil} 💻	_{ChristopheRappold} 📖	_{Cosmin Deaconu} ⚠️ 💻	_{Carlos Pegueros} 📖 💡 ⚠️ ✅	_{Benjamin Tovar} 💻
_{Duncan Macleod} 🚇	_mpad 💻	_{Peter Fackeldey} 💻

💻 : code, 📖 : documentation, 🚇 : infrastructure, 🚧 : maintainance, ⚠ : tests and feedback, 🤔 : foundational ideas.

Comments

uproot4 unable to open ROOT file with colons in the name that uproot3 can

Hi. I've come across an issue where if a ROOT file has a colon in the name of it uproot3 can open the file but uproot4 fails. I can't give you the file, but I can show you a minimal failing example and then a reproducible example with public files.

Minimal Failing Example

$ tree .
.
├── data-tree
│   └──  data16_13TeV:data16_13TeV.periodA.physics_Main.PhysCont.DAOD_JETM1.grp16_v01_p4061.root
├── issue.py
├── requirements.txt

1 directory, 3 files
$ cat requirements.txt 
uproot
uproot4
$ docker run --rm -it -v $PWD:/data -w /data python:3.8 /bin/bash
root@510598a7f4e8:/data# python -m pip install --upgrade pip setuptools wheel
root@510598a7f4e8:/data# python -m pip install -r requirements.txt
root@510598a7f4e8:/data# python --version
Python 3.8.5
root@510598a7f4e8:/data# python -m pip list
Package        Version
-------------- -------
awkward        0.13.0
cachetools     4.1.1
numpy          1.19.1
pip            20.2.2
setuptools     49.6.0
uproot         3.12.0
uproot-methods 0.7.4
uproot4        0.0.18
wheel          0.35.1
root@510598a7f4e8:/data# cp data-tree/data16_13TeV\:data16_13TeV.periodA.physics_Main.PhysCont.DAOD_JETM1.grp16_v01_p4061.root data-tree/renamed.root
root@510598a7f4e8:/data# python
Python 3.8.5 (default, Aug  5 2020, 08:22:02) 
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import uproot as uproot3
>>> import uproot4
>>> from pathlib import Path
>>> 
>>> uproot3_file = uproot3.open(
...     "data-tree/data16_13TeV:data16_13TeV.periodA.physics_Main.PhysCont.DAOD_JETM1.grp16_v01_p4061.root"
... )
>>> print(f"uproot3 opens file as {uproot3_file}")
uproot3 opens file as <ROOTDirectory b'/home/feickert/workarea/submitDir/data-tree//data16_13TeV:data16_13TeV.periodA.physics_Main.PhysCont.DAOD_JETM1.grp16_v01_p4061.root' at 0x7fce6da5b5e0>
>>> 
>>> # uproot4 fails with the ':' in the filename
>>> uproot4.open(
...     "data-tree/data16_13TeV:data16_13TeV.periodA.physics_Main.PhysCont.DAOD_JETM1.grp16_v01_p4061.root"
... )
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/uproot4/source/file.py", line 74, in __init__
    self._file = numpy.memmap(self._file_path, dtype=self._dtype, mode="r")
  File "/usr/local/lib/python3.8/site-packages/numpy/core/memmap.py", line 225, in __new__
    f_ctx = open(os_fspath(filename), ('r' if mode == 'c' else mode)+'b')
FileNotFoundError: [Errno 2] No such file or directory: 'data-tree/data16_13TeV'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.8/site-packages/uproot4/reading.py", line 78, in open
    file = ReadOnlyFile(
  File "/usr/local/lib/python3.8/site-packages/uproot4/reading.py", line 265, in __init__
    self._source = Source(file_path, **self._options)
  File "/usr/local/lib/python3.8/site-packages/uproot4/source/file.py", line 80, in __init__
    self._fallback = uproot4.source.file.FileSource(file_path, opts)
AttributeError: module 'uproot4.source.file' has no attribute 'FileSource'
>>> # even if that is inside a pathlib object
>>> uproot4.open(
...     Path(
...         "data-tree/data16_13TeV:data16_13TeV.periodA.physics_Main.PhysCont.DAOD_JETM1.grp16_v01_p4061.root"
...     )
... )
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/uproot4/source/file.py", line 74, in __init__
    self._file = numpy.memmap(self._file_path, dtype=self._dtype, mode="r")
  File "/usr/local/lib/python3.8/site-packages/numpy/core/memmap.py", line 225, in __new__
    f_ctx = open(os_fspath(filename), ('r' if mode == 'c' else mode)+'b')
FileNotFoundError: [Errno 2] No such file or directory: 'data-tree/data16_13TeV'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.8/site-packages/uproot4/reading.py", line 78, in open
    file = ReadOnlyFile(
  File "/usr/local/lib/python3.8/site-packages/uproot4/reading.py", line 265, in __init__
    self._source = Source(file_path, **self._options)
  File "/usr/local/lib/python3.8/site-packages/uproot4/source/file.py", line 80, in __init__
    self._fallback = uproot4.source.file.FileSource(file_path, opts)
AttributeError: module 'uproot4.source.file' has no attribute 'FileSource'
>>> # but the file itself is fine
>>> uproot4.open("data-tree/renamed.root")
<ReadOnlyDirectory '/' at 0x7fce725e7670>

Failing Reproducible Example

# issue.py
import uproot as uproot3
import uproot4
from pathlib import Path


def main():
    # curl -sL https://github.com/scikit-hep/scikit-hep-testdata/raw/master/src/skhep_testdata/data/uproot-HZZ-lz4.root -o uproot-HZZ-lz4.root
    uproot3.open("uproot-HZZ-lz4.root")
    uproot3.open("uproot:HZZ-lz4.root")
    uproot3.open(Path("uproot:HZZ-lz4.root"))

    uproot4.open("uproot-HZZ-lz4.root")
    uproot4.open("uproot:HZZ-lz4.root")


if __name__ == "__main__":
    main()

root@510598a7f4e8:/data# curl -sL https://github.com/scikit-hep/scikit-hep-testdata/raw/master/src/skhep_testdata/data/uproot-HZZ-lz4.root -o uproot-HZZ-lz4.root
root@510598a7f4e8:/data# cp uproot-HZZ-lz4.root uproot:HZZ-lz4.root 
root@510598a7f4e8:/data# python issue.py 
Traceback (most recent call last):
  File "issue.py", line 17, in <module>
    main()
  File "issue.py", line 9, in main
    uproot3.open("uproot:HZZ-lz4.root")
  File "/usr/local/lib/python3.8/site-packages/uproot/rootio.py", line 63, in open
    raise ValueError("URI scheme not recognized: {0}".format(path))
ValueError: URI scheme not recognized: uproot:HZZ-lz4.root

Comments

I realize that this is probably because uproot4's open's path is

https://github.com/scikit-hep/uproot4/blob/b8828069b9ae52c742fb704f07e1a4e179fe7b30/uproot4/reading.py#L43-L46

and it isn't a good idea to have a file with a colon in it in general. However, there are ATLAS files that do, and it would be nice if uproot3 behavior toward filenames could still be supported. If support for this is firmly out of scope it would be great if there could be some huge warning about this in the docs.

policy

opened by matthewfeickert 38

Loading multiple large files (>4GB) onto a lazy array produces deserialization error

Issue description

This is an issue regarding the question posted here: https://stackoverflow.com/questions/64565208/how-can-i-load-many-very-large-files-using-uproot

When loading 3 files of about 9 GB each onto a lazy array, the code crashes when the event loop reaches the borders between the first and second files producing a deserialization error. The code works flawlessly with small (~ 1 GB) files. Way of access (local vs XRootD) also doesn't seem to affect the issue. The error message is the following:

Traceback (most recent call last):
  File "scripts/plotEventShapes_lazy.py", line 44, in <module>
    tracks_x = events['Tracks.fCoordinates.fX'][ievt]
  File "/Users/chrispap/Library/Python/3.8/lib/python/site-packages/awkward1/highlevel.py", line 946, in __getitem__
    self._layout[where], self._behavior, cache=self._cache
  File "/Users/chrispap/Library/Python/3.8/lib/python/site-packages/awkward1/partition.py", line 366, in __getitem__
    return PartitionedArray.from_ext(self._ext.getitem_at(where))
  File "/Users/chrispap/Library/Python/3.8/lib/python/site-packages/uproot4/behaviors/TBranch.py", line 1621, in array
    _ranges_or_baskets_to_arrays(
  File "/Users/chrispap/Library/Python/3.8/lib/python/site-packages/uproot4/behaviors/TBranch.py", line 518, in _ranges_or_baskets_to_arrays
    uproot4.source.futures.delayed_raise(*obj)
  File "/Users/chrispap/Library/Python/3.8/lib/python/site-packages/uproot4/source/futures.py", line 37, in delayed_raise
    raise exception_value.with_traceback(traceback)
  File "/Users/chrispap/Library/Python/3.8/lib/python/site-packages/uproot4/behaviors/TBranch.py", line 444, in chunk_to_basket
    basket = uproot4.models.TBasket.Model_TBasket.read(
  File "/Users/chrispap/Library/Python/3.8/lib/python/site-packages/uproot4/model.py", line 128, in read
    self.read_members(chunk, cursor, context, file)
  File "/Users/chrispap/Library/Python/3.8/lib/python/site-packages/uproot4/models/TBasket.py", line 55, in read_members
    ) = cursor.fields(chunk, _tbasket_format2, context)
  File "/Users/chrispap/Library/Python/3.8/lib/python/site-packages/uproot4/source/cursor.py", line 168, in fields
    return format.unpack(chunk.get(start, stop, self, context))
  File "/Users/chrispap/Library/Python/3.8/lib/python/site-packages/uproot4/source/chunk.py", line 278, in get
    raise uproot4.deserialization.DeserializationError(
uproot4.deserialization.DeserializationError: while reading

    TBasket version None as uproot4.models.TBasket.Model_TBasket (? bytes)
        fNbytes: -1607368158
        fObjlen: -1243566277
        fDatime: 2634931141
        fKeylen: -27664
        fCycle: 21409

attempting to get bytes 483015:483033
outside expected range 510698:538758 for this Chunk
in file /Users/chrispap/QCD/Autumn18.QCD_HT1500to2000_TuneCP5_13TeV-madgraphMLM-pythia8_RA2AnalysisTree.root

Reproduction instructions

The three very large files (9 GB each) are the following:

root://cmseos.fnal.gov//store/user/kdipetri/SUEP/Production_v0.2/2018/NTUP/Autumn18.QCD_HT1000to1500_TuneCP5_13TeV-madgraphMLM-pythia8_RA2AnalysisTree.root
root://cmseos.fnal.gov//store/user/kdipetri/SUEP/Production_v0.2/2018/NTUP/Autumn18.QCD_HT1500to2000_TuneCP5_13TeV-madgraphMLM-pythia8_RA2AnalysisTree.root
root://cmseos.fnal.gov//store/user/kdipetri/SUEP/Production_v0.2/2018/NTUP/Autumn18.QCD_HT2000toInf_TuneCP5_13TeV-madgraphMLM-pythia8_RA2AnalysisTree.root

Two smaller files (1 GB) for which the code works nicely:

root://cmseos.fnal.gov//store/user/kdipetri/SUEP/Production_v0.2/2018/NTUP/PrivateSamples.SUEP_2018_mMed-1000_mDark-2_temp-2_decay-darkPho_13TeV-pythia8_n-100_0_RA2AnalysisTree.root
root://cmseos.fnal.gov//store/user/kdipetri/SUEP/Production_v0.2/2018/NTUP/PrivateSamples.SUEP_2018_mMed-1000_mDark-2_temp-2_decay-darkPhoHad_13TeV-pythia8_n-100_0_RA2AnalysisTree.root

The following code reproduces the issue using the files that I cite above:

import uproot4 as uproot
import uproot_methods
import awkward1 as ak
import numpy as np
import matplotlib.pyplot as plt

# Get the file and import using uproot
base = 'root://cmseos.fnal.gov//store/user/kdipetri/SUEP/Production_v0.2/2018/NTUP/'
#base = '/Users/chrispap/QCD/'
datasets = {
    base + 'Autumn18.QCD_HT1000to1500_TuneCP5_13TeV-madgraphMLM-pythia8_RA2AnalysisTree.root': 'TreeMaker2/PreSelection',
    base + 'Autumn18.QCD_HT1500to2000_TuneCP5_13TeV-madgraphMLM-pythia8_RA2AnalysisTree.root': 'TreeMaker2/PreSelection',
    base + 'Autumn18.QCD_HT2000toInf_TuneCP5_13TeV-madgraphMLM-pythia8_RA2AnalysisTree.root': 'TreeMaker2/PreSelection',
}

events = uproot.lazy(datasets)

multiplicity = np.zeros(len(events['Tracks.fCoordinates.fX']))
for ievt in range(99000,len(events['Tracks.fCoordinates.fX'])):
    if ievt%1000 == 0:
        print("Processing event %d. Progress: %.2f%%"%(ievt,100*ievt/len(events['Tracks.fCoordinates.fX'])))
    if events['HT'][ievt] < 1200:
        multiplicity[ievt] = -1
        continue
    tracks_x = events['Tracks.fCoordinates.fX'][ievt]
    tracks_y = events['Tracks.fCoordinates.fY'][ievt]
    tracks_z = events['Tracks.fCoordinates.fZ'][ievt]
    tracks_E = np.sqrt(tracks_x**2+tracks_y**2+tracks_z**2+0.13957**2)
    tracks = uproot_methods.TLorentzVectorArray.from_cartesian(ak.to_awkward0(tracks_x),
                                                               ak.to_awkward0(tracks_y),
                                                               ak.to_awkward0(tracks_z),
                                                               ak.to_awkward0(tracks_E))
    tracks_fromPV0 = events['Tracks_fromPV0'][ievt]
    tracks_matchedToPFCandidate = events['Tracks_matchedToPFCandidate'][ievt]
    tracks = tracks[(tracks.pt > 1.) & (tracks.eta < 2.5) &
                    (ak.to_awkward0(tracks_fromPV0) >= 2) &
                    (ak.to_awkward0(tracks_matchedToPFCandidate) > 0)]
    multiplicity[ievt] = tracks.size

# Plot results
fig = plt.figure(figsize=(8,8))
ax = plt.gca()

CrossSection = events['CrossSection'][events['HT'] > 1200]
multiplicity = multiplicity[events['HT'] > 1200]

ax.hist(multiplicity, bins=100, density=True, weights=CrossSection, histtype='step', color='b')

plt.show()

bug (unverified)

opened by chrispap95 20

Drawing of histograms created in uproot fails in ROOT
When saving a histogram in a ROOT file using Uproot, if the content of the last bin in the histogram is negative, ROOT will fail to plot it. The error message is

TCanvas::ResizePad:0: RuntimeWarning: Inf/NaN propagated to the pad. Check drawn objects. TCanvas::ResizePad:0: RuntimeWarning: Canvas_1 height changed from 0 to 10

If the last bin's content is non-negative, plotting works.

To reproduce, create the histogram

import numpy as np import uproot with uproot.recreate("test.root") as f: f["hist_fail"] = (np.asarray([-3, -2, -1]), np.asarray([0, 1, 2, 3])) f["hist_pass"] = (np.asarray([-3, -2, +1]), np.asarray([0, 1, 2, 3]))

Opening test.root using rootbrowser (or just root, getting the histograms, and plotting them using Draw()) results in the hist_pass being drawn, while hist_fail won't be.

Furthermore, even the hist_pass isn't drawn properly - the y-axis starts at 0. When creating the same histogram in ROOT, the range includes all the events.

I think the issue is caused by the errors of the histogram bins - ROOT initializes them as sqrt(abs(content)) while uproot uses sqrt(content), leading to nans for negative numbers. Not sure what is special about the last bin, though.

Uproot 4.3.4 ROOT 6.26/04
bug
opened by dcervenkov 17
File access methods performance studies
Data provided by @chrisburr; discussion below

If I'm making a fundamental mistake in these file access methods, then I don't want to design around it. I've been thinking about how I can adequately test that. Do you have a suggestion for a publicly available test file?

I default to using these three open data files for XRootD:

$ xrdfs root://eospublic.cern.ch/ ls -l /eos/opendata/lhcb/AntimatterMatters2017/data -r-- 2017-03-07 13:53:05 666484974 /eos/opendata/lhcb/AntimatterMatters2017/data/B2HHH_MagnetDown.root -r-- 2017-03-07 13:53:08 444723234 /eos/opendata/lhcb/AntimatterMatters2017/data/B2HHH_MagnetUp.root -r-- 2017-03-07 13:53:08 2272072 /eos/opendata/lhcb/AntimatterMatters2017/data/PhaseSpaceSimulation.root

I've just copied them to CERN S3 so they can be accessed using standard HTTP(S) as well:

https://scikit-hep-test-data.s3.cern.ch/B2HHH_MagnetDown.root

https://scikit-hep-test-data.s3.cern.ch/B2HHH_MagnetUp.root

https://scikit-hep-test-data.s3.cern.ch/PhaseSpaceSimulation.root

Would a test through a household ISP mean anything, given that most of these files would be used on university or lab computers or on the GRID?

It's definitely useful as the quality of institute connections varies a lot in my experience and it's more likely to expose issues with latency, especially given you'll be accessing them from the US. I think a good implementation should only be limited by bandwidth when reading large enough files.

If you put together a test I can easily try it at a few different institutes as well as pushing a test job to every LHCb grid site.

Originally posted by @chrisburr in https://github.com/scikit-hep/uproot4/pull/1#issuecomment-627928856
performance
opened by jpivarski 17
fix: missing hist flow bins handling
Currently, when uproot writes hist histograms without flow bins, the property isn't checked and regular edge bins get written as flow bins causing weird issues with reading.

import uproot import hist import numpy as np import mplhep as hep h = hist.new.Reg(20, 0, 20, name='msd', flow=False).Weight().fill(np.random.normal(10, 6, 1000)) fout = uproot.recreate('test.root') fout['test'] = h fout.close() fin = uproot.open('test.root') h_read = fin['test'] fin.close() len(h.values()), len(h_read.values()), len(h.axes[0].edges) >>> (20, 18, 21)

This PR materializes the empty flow bins as Nans when writing and filters them out again when reading to match the current expected behaviour. The caveat is that when only one flow bin exists when storing, it won't be obvious which one it is when reading the TH1 with values(flow=True) Possibly this should return a masked array instead to be clearer, but I didn't want to change the return type here.
opened by andrzejnovak 16

feat: Add TGraphAsymmErrors

here's a test script to work with

import uproot
import os.path

fname = "HEPData-ins1755298-v3-Expected_limit_1lbb.root"

if not os.path.isfile(fname):
    with open(fname, 'wb') as f:
        import requests
        response = requests.get("https://www.hepdata.net/download/table/ins1755298/Expected%20limit%201lbb/3/root")
        response.raise_for_status()
        f.write(response.content)

f = uproot.open(fname)
graph = f['Expected limit 1lbb/Graph1D_y1']
graph.values()
graph.errors()

opened by kratsg 16

refactor: modernize setup

This is an attempt to modernize the setup. This fixes the issue with https://github.com/conda-forge/uproot-feedstock/pull/73 - and is generally much cleaner and nicer being declarative.

opened by henryiii 15
fix: working TList serialization
Closes https://github.com/scikit-hep/uproot5/issues/762.

This PR was motivated because of problems serializing TList.

TList serialization was fixed in 4da653fee0f28789fbb20abcad3a4541acd6701a.

A few other minor improvements were done along the way:

TObject should now correctly serialize fUniqueID member (previously it was using only 0).

TObject __repr__ is fixed.

A method Model_TObject.empty has been added as override to avoid using its parent method in order to simplify code.

TObjString now correctly initializes its bases with a TObject with all its members.

Updated some tojson calls of above mentined classes.

Added / Updated tests for writing and reading TObjString and TList.
opened by lobis 14
Add TGraph, RooCurve, and RooHist behaviours.

This adds a behaviour for TGraph (based on that for TGraphAsymmErrors), and behaviours for RooHist and RooCurve based on these. These make it easier to plot RooFit objects on a plot that's otherwise made with Python.

~~Draft because I haven't added any tests yet. I thought it would be better to discuss first.~~

opened by beojan 14
feat: support categorical axes on boost histograms
Closes https://github.com/scikit-hep/uproot5/issues/672.

The logic for setting the fLabels on the axis is pretty straightforward, however to achieve this we need to serialize THashList which is not currently implemented.

While working on this I found out a problem when serializing TList (which THashList inherits directly from), which is pointed out in https://github.com/scikit-hep/uproot5/issues/762 and is fixed by https://github.com/scikit-hep/uproot5/pull/763.

Another issue related to weighted histograms was also found and fixed at https://github.com/scikit-hep/uproot5/pull/774.

Summary of changes:

Implement THashList serialization and to_HashList factory method.

Add tests for THashList writing. Compare serialization (bytes) to TList as they should be the same except for class name.

The name (fName) of all axes is now xaxis, yaxis or zaxis regardless of user definition, as this field should not be modified by the user since it causes problems on histogram drawing. (The text of the axis is fTitle). Tests have been updated accordingly.

Add tests for histograms with categorical axes.

next-release
opened by lobis 13

Which cycle ReadOnlyDirectory should return when not specified

Hello,

I've tried to use uproot to parse *.root files; At the same time, I used c++ to verify.

In the same root file: For Python:

import uproot
uproot.__version__         # (4.0.11)

Coins = uproot.open(input_file + ':Coincidences')
print(Coins.num_entries)

For C++:

#include "TFile.h"
#include "TTree.h"

TFile *rootFile;
rootFile = TFile::Open(root_filename.c_str());
TTree *Coins = (TTree*)rootFile->Get("Coincidences");

int nevents = Coins->GetEntries();
std::cout << nevents << std::endl;

The results:

Python: 34393000
C++ : 34393400

Why???

Best regards,

Connor,

bug

opened by HeConnor 13

DAOD_PHYSLITE works in pure python interpretation, not in awkward-forth (uproot 5.0.0)

A test in coffea results in:

ValueError: 'skip beyond' in AwkwardForth runtime: tried to skip beyond the bounds of an input (0 or length)

when awkward forth is used, and passes when forcing the pure-python interpretation paths to be used. These tests involve iterating over arrays and fully validating the data contained, so the arrays produced within the test are correct (and in fact the same data-wise as uproot4/awkward1).

To reproduce this:

$ pip install 'git+https://github.com/CoffeaTeam/coffea.git@awkward2_dev'
$ python -c 'from coffea.nanoevents import NanoEventsFactory, PHYSLITESchema; NanoEventsFactory.from_root("https://github.com/CoffeaTeam/coffea/blob/master/tests/samples/DAOD_PHYSLITE_21.2.108.0.art.pool.root?raw=true", treepath="CollectionTree", schemaclass=PHYSLITESchema, use_ak_forth=False).events()'
$ python -c 'from coffea.nanoevents import NanoEventsFactory, PHYSLITESchema; NanoEventsFactory.from_root("https://github.com/CoffeaTeam/coffea/blob/master/tests/samples/DAOD_PHYSLITE_21.2.108.0.art.pool.root?raw=true", treepath="CollectionTree", schemaclass=PHYSLITESchema, use_ak_forth=True).events()'

bug

opened by lgray 1

Problems when saving jagged arrays
I'm a newbie to uproot and encountered some problems when trying to use uproot. Sorry for possible misunderstandings of the usage, though I think my expectation might be reasonable.

Create a jagged array

import uproot import awkward as ak array = ak.Array({'a': [[1.1, 2.2, 3.3], [], [4.4, 5.5]], 'b': [[1.1, 2.2, 3.3], [], [4.4, 5.5]], 'c': [1, 2, 3]})

and save it to a ROOT file

with uproot.recreate('data.root') as output_file: output_file['tree'] = array

I expect that the saved tree has 3 branches

name | typename | interpretation
---------------------+--------------------------+------------------------------- a | double[] | AsJagged(AsDtype('>f8')) b | double[] | AsJagged(AsDtype('>f8')) c | int64_t | AsDtype('>i8')

but actually I got errors

TypeError: fields of a record must be NumPy types, though the record itself may be in a jagged array

field 'a' has type var * float64

This situation easily happens when reading a TTree with variable length arrays and then saving it back after some manipulations.
feature
opened by laf070810 2
Support for TLeafG
I have a tree with TLeafG type. When I try to process it, I get the following:

name | typename | interpretation
---------------------+--------------------------+------------------------------- runNumber | int32_t | AsDtype('>i4') subrunNumber | int32_t | AsDtype('>i4') timestamp | unknown | <UnknownInterpretation 'non...

Would it be possible to add the support for TLeafG type in identify.py?

Thanks, Yuri.

What you're looking for might already be possible as a combination of existing functions, so these requests might get converted into requests for documentation (with a corresponding change in title).

Here are some links to documentation, to help you in your search.

The tutorials site

StackOverflow: [uproot] tag

Gitter: Scikit-HEP/uproot room

feature
opened by oksuzian 1
How to create an empty tree with several var-length branches that share the var length
I want to write several variable length arrays that have the same length to an empty tree. https://uproot.readthedocs.io/en/latest/basic.html#extending-ttrees-with-large-datasets explains

How to generate an empty tree with mktree, which can be extended later, I need this

How to generate a tree with several arrays that share the length and already contain data, with awkward.zip

What I am missing is the combination of the two. I want to start with an empty tree like in 1) that has branches which share the varlength. Then I want to extend this tree subsequently. Is this possible? If not, may I suggest the following intuitive API:

Instead of {"x": "var * float32", "y": "var * float32"}, which generates two extra branches "nx" and "ny", please allow this declaration, where the counting branch is explicit:

{"n": "int32", "x": "n * float32", "y": "n * float32"}

As the Python of Zen says, explicit is better than implicit.
docs
opened by HDembinski 3
uproot.open(filename, "w") and uproot.open(filename, mode="w") fail
uproot 4.3.7

When opening a root file for writing, the call to uproot.open(filename, mode="w") fails with this error message

--------------------------------------------------------------------------- AttributeError Traceback (most recent call last) Cell In [15], line 2 1 with uproot.open("output.root", mode="w") as f: ----> 2 f.mktree("tree", { 3 "px": "var * float32", 4 }) 5 f["tree"].show() AttributeError: 'ReadOnlyDirectory' object has no attribute 'mktree'

It looks like mode="w" is ignored. This is not pythonic. By providing uproot.open you are suggesting that you mimic the interface of the open command, which is used in several places in the Python stdlib (builtins open, gzip.open, lzma.open, ...). But then you should also mimics the signature of the stdlib version as much as possible. Calling uproot.open with mode="w" should forward to uproot.recreate.

uproot.open(filename, "w") fails with a different error, because the next positional argument after path is not mode, but object_cache. To fix this, mode should be added as the second argument after path or uproot.open should reject positional arguments after path with the signature open(path, *, object_cache=...)
bug (unverified)
opened by HDembinski 5

Releases(v5.0.2)

v5.0.2(Dec 22, 2022)
New features

(none!)

Bug-fixes and performance

fix: uproot.dask: Protect against branches=None in project_columns by @douglasdavis in https://github.com/scikit-hep/uproot5/pull/806

fix: AsStridedObjects.awkward_form was still including the '@' members. by @jpivarski in https://github.com/scikit-hep/uproot5/pull/808

Other

ci: [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in https://github.com/scikit-hep/uproot5/pull/807

Full Changelog: https://github.com/scikit-hep/uproot5/compare/v5.0.1...v5.0.2
Source code(tar.gz)
Source code(zip)
v5.0.1(Dec 15, 2022)
New features

(none!)

Bug-fixes and performance

fix: protect Uproot's 'project_columns' from Dask node names. by @jpivarski in https://github.com/scikit-hep/uproot5/pull/801

Other

ci: [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in https://github.com/scikit-hep/uproot5/pull/799

docs: update documentation about Pandas; we don't do MultiIndex anymore. by @jpivarski in https://github.com/scikit-hep/uproot5/pull/802

chore(deps): bump pypa/gh-action-pypi-publish from 1.6.1 to 1.6.4 by @dependabot in https://github.com/scikit-hep/uproot5/pull/797

Full Changelog: https://github.com/scikit-hep/uproot5/compare/v5.0.0...v5.0.1
Source code(tar.gz)
Source code(zip)
v5.0.0(Dec 10, 2022)
Uproot version 5.0.0

Uproot version 5 has a few major new features, one removal (uproot.lazy), and is based on Awkward Array version 2 instead of version 1.

uproot.lazy → uproot.dask

@kkothari2001 upgraded Uproot from Awkward version 1 to version 2, the major part of which was replacing uproot.lazy, which is based on Awkward 1's virtual and partitioned lazy arrays, with the new Dask collection, dask-awkward. The entry point for this function is uproot.dask.

@kkothari2001 also simplified Uproot's Pandas backend, which used to "explode" ragged arrays from ROOT into Pandas DataFrames with a non-trivial MultiIndex. Now, it takes advantage of awkward-pandas to put ragged (and more complex) Awkward Arrays directly into Pandas columns.

If you want the old behavior, you can read data using library="ak" to get an Awkward Array, and use ak.to_dataframe to "explode" the data into a MultiIndex.

TTree-reading with AwkwardForth

@aryan26roy added a new code path to the TTree-reading routines to read them with AwkwardForth instead of pure Python. Users won't see any interface changes due to this code, but the performance of reading TBranches with AsObject or AsStrings Interpretations should be orders of magnitude faster. For example, std::vector<std::vector<float>> reading is now 400× faster.

Reading RNTuples

@Moelf added a complete reader of RNTuple data with most of an RNTuple-writer in an unmerged pull request (#705). Although the RNTuple format is still in development, this is a very good start at reading RNTuple data, whose structure is a close match to Awkward Arrays (so the translation is more one-to-one than it is for TTrees, for instance).

New features

feat: move to hatchling by @henryiii in https://github.com/scikit-hep/uproot5/pull/688

feat: from_map like optimization for dask arrays by @kkothari2001 in https://github.com/scikit-hep/uproot5/pull/679

feat: Finalizing AwkwardForth reader for Uproot by @aryan26roy in https://github.com/scikit-hep/uproot5/pull/644

feat: implemented NON-memberwise deserialization for AsMap. by @jpivarski in https://github.com/scikit-hep/uproot5/pull/746

feat: Added column_projection optimization by @kkothari2001 in https://github.com/scikit-hep/uproot5/pull/755

feat: support categorical axes on boost histograms by @lobis in https://github.com/scikit-hep/uproot5/pull/764

feat: warn about TBranch name, alias name conflict. by @jpivarski in https://github.com/scikit-hep/uproot5/pull/776

feat: any Mapping assigned to a WritableDirectory is interpreted as a TTree or failure, no fall-through. by @jpivarski in https://github.com/scikit-hep/uproot5/pull/779

feat: add 'interp_options' mechanism and ak_add_doc. by @jpivarski in https://github.com/scikit-hep/uproot5/pull/784

feat: Use awkward pandas, instead of the existing code that explodes Pandas Dataframes by @kkothari2001 in https://github.com/scikit-hep/uproot5/pull/734

feat: made 'very optional' arguments keyword-only by @jpivarski in https://github.com/scikit-hep/uproot5/pull/787

feat: adjust for name change in scikit-hep/awkward#1919. by @jpivarski in https://github.com/scikit-hep/uproot5/pull/788

Bug-fixes and performance

fix: depend on packaging, not setuptools vendored packaging by @henryiii in https://github.com/scikit-hep/uproot5/pull/684

fix: Avoid triggering temporary dask-awkward/awkward incompatibility. by @jpivarski in https://github.com/scikit-hep/uproot5/pull/694

fix: Do not write incorrect fSumw2 in histograms (v5). by @jpivarski in https://github.com/scikit-hep/uproot5/pull/698

fix: Fixes uproot.dask bug with empty branches by @kkothari2001 in https://github.com/scikit-hep/uproot5/pull/700

fix: Use from_map optimization for delayed numpy arrays and add tests with empty branches for the same by @kkothari2001 in https://github.com/scikit-hep/uproot5/pull/703

fix: use ctx manager to ensure resources are freed by @agoose77 in https://github.com/scikit-hep/uproot5/pull/713

fix: ReadOnlyDirectory should provide the largest abs(cycle) when cycle is unspecified, not the largest cycle. by @jpivarski in https://github.com/scikit-hep/uproot5/pull/715

fix: regularize ROOT type aliases to C fundamental type names. by @jpivarski in https://github.com/scikit-hep/uproot5/pull/717

fix: avoid empty TBasket issue in embedded TBasket by @jpivarski in https://github.com/scikit-hep/uproot5/pull/751

fix: don't use Awkward in test_0751 that doesn't need it by @jpivarski in https://github.com/scikit-hep/uproot5/pull/753

fix: working TList serialization by @lobis in https://github.com/scikit-hep/uproot5/pull/763

fix: histogram weights not handled correctly in hist / boost conversion by @lobis in https://github.com/scikit-hep/uproot5/pull/774

perf: streamline metadata handling for TBranch name lookup and uproot.dask by @jpivarski in https://github.com/scikit-hep/uproot5/pull/772

fix: ensure AwkwardForth fallback path is tested without history. by @jpivarski in https://github.com/scikit-hep/uproot5/pull/780

fix: all AwkwardForth Forms now agree with awkward_form method output. by @jpivarski in https://github.com/scikit-hep/uproot5/pull/790

fix: Uproot tests now work with Awkward 2.0.0. by @jpivarski in https://github.com/scikit-hep/uproot5/pull/795

Other

Manually add a Model for TMatrixTSym. by @jpivarski in https://github.com/scikit-hep/uproot5/pull/484

Updating docs and test in response to the removal of uproot.lazy in Uproot5 by @kkothari2001 in https://github.com/scikit-hep/uproot5/pull/615

changed arguments for awkward_form by @aryan26roy in https://github.com/scikit-hep/uproot5/pull/617

Completed the Forth based AsStrings reader. by @aryan26roy in https://github.com/scikit-hep/uproot5/pull/616

Actually pass user-specified 'awkward_form' arguments into context. by @jpivarski in https://github.com/scikit-hep/uproot5/pull/622

Fix annoying gaps in test files. by @jpivarski in https://github.com/scikit-hep/uproot5/pull/625

Cleaning up string generation in streamers.py by @aryan26roy in https://github.com/scikit-hep/uproot5/pull/629

chore: add dependabot for actions by @henryiii in https://github.com/scikit-hep/uproot5/pull/631

Bump actions/checkout from 2 to 3 by @dependabot in https://github.com/scikit-hep/uproot5/pull/632

Bump actions/upload-artifact from 2 to 3 by @dependabot in https://github.com/scikit-hep/uproot5/pull/633

Bump pypa/gh-action-pypi-publish from 1.4.2 to 1.5.0 by @dependabot in https://github.com/scikit-hep/uproot5/pull/634

Bump actions/download-artifact from 2 to 3 by @dependabot in https://github.com/scikit-hep/uproot5/pull/635

Forth based ROOT reader (revised) by @aryan26roy in https://github.com/scikit-hep/uproot5/pull/636

Set up tests for AsObjects, for the AwkwardForth reader by @jpivarski in https://github.com/scikit-hep/uproot5/pull/637

Iterate over objects in TDirectory in linear time. by @jpivarski in https://github.com/scikit-hep/uproot5/pull/638

[pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in https://github.com/scikit-hep/uproot5/pull/641

Awkward v2 update by @kkothari2001 in https://github.com/scikit-hep/uproot5/pull/620

docs: add aryan26roy as a contributor for code by @allcontributors in https://github.com/scikit-hep/uproot5/pull/645

docs: add kkothari2001 as a contributor for code by @allcontributors in https://github.com/scikit-hep/uproot5/pull/646

docs: add Moelf as a contributor for code by @allcontributors in https://github.com/scikit-hep/uproot5/pull/647

chore: cleanup flake8 by @henryiii in https://github.com/scikit-hep/uproot5/pull/527

[pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in https://github.com/scikit-hep/uproot5/pull/650

[pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in https://github.com/scikit-hep/uproot5/pull/654

Bump pypa/gh-action-pypi-publish from 1.5.0 to 1.5.1 by @dependabot in https://github.com/scikit-hep/uproot5/pull/656

Primitive Support for RNTuple by @Moelf in https://github.com/scikit-hep/uproot5/pull/630

Set ReadOnlyDirectory attributes when fSeekKeys == 0 by @kakwok in https://github.com/scikit-hep/uproot5/pull/660

docs: add kakwok as a contributor for code by @allcontributors in https://github.com/scikit-hep/uproot5/pull/663

Dask awkward support for uproot.dask by @kkothari2001 in https://github.com/scikit-hep/uproot5/pull/652

pathlib.Path drops '//' (naturally), but it's sometimes used for URLs by @jpivarski in https://github.com/scikit-hep/uproot5/pull/670

Gets the number of overflow bins for hist.axis.IntCategory, at least. by @jpivarski in https://github.com/scikit-hep/uproot5/pull/671

Prevent std::pair from being AsStridedObjects. by @jpivarski in https://github.com/scikit-hep/uproot5/pull/673

Implement transformed axis from boost-histogram/hist. by @jpivarski in https://github.com/scikit-hep/uproot5/pull/675

AsDynamic has no self._header. by @jpivarski in https://github.com/scikit-hep/uproot5/pull/674

Fixed TTree write_anew in a subdirectory (consistent caches). by @jpivarski in https://github.com/scikit-hep/uproot5/pull/677

[pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in https://github.com/scikit-hep/uproot5/pull/657

Implement stl containers for RNTuple by @Moelf in https://github.com/scikit-hep/uproot5/pull/662

Multiple clusters support for RNTuple by @Moelf in https://github.com/scikit-hep/uproot5/pull/682

ci: Get test dependencies from one source by @jpivarski in https://github.com/scikit-hep/uproot5/pull/686

ci: autocancel repeated runs by @henryiii in https://github.com/scikit-hep/uproot5/pull/685

ci: use mamba by @henryiii in https://github.com/scikit-hep/uproot5/pull/683

ci: Lint PR titles according to conventional commits by @jpivarski in https://github.com/scikit-hep/uproot5/pull/689

docs: Installation requirements and error text for dask/dask-awkward in extras.py by @jpivarski in https://github.com/scikit-hep/uproot5/pull/690

ci: use concurrency group for semantic-pr-title by @agoose77 in https://github.com/scikit-hep/uproot5/pull/691

docs: Add image for dask docs by @kkothari2001 in https://github.com/scikit-hep/uproot5/pull/708

ci: [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in https://github.com/scikit-hep/uproot5/pull/706

chore: dask_awkward.test_utils moved in ContinuumIO/dask-awkward#76 by @jpivarski in https://github.com/scikit-hep/uproot5/pull/714

test: adjust for boost-histogram 1.3.2's _storage_type deprecation. by @jpivarski in https://github.com/scikit-hep/uproot5/pull/719

ci: [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in https://github.com/scikit-hep/uproot5/pull/716

add veprbl as a contributor by @allcontributors in https://github.com/scikit-hep/uproot5/pull/728

add nikoladze as a contributor by @allcontributors in https://github.com/scikit-hep/uproot5/pull/729

add klieret as a contributor by @allcontributors in https://github.com/scikit-hep/uproot5/pull/730

add dcervenkov as a contributor by @allcontributors in https://github.com/scikit-hep/uproot5/pull/731

add beojan as a contributor by @allcontributors in https://github.com/scikit-hep/uproot5/pull/732

add agoose77 as a contributor by @allcontributors in https://github.com/scikit-hep/uproot5/pull/733

docs: add a CITATION.cff for Uproot. by @jpivarski in https://github.com/scikit-hep/uproot5/pull/726

chore: drop Python 3.6. by @jpivarski in https://github.com/scikit-hep/uproot5/pull/742

ci: remove GHA workaround for macOS Python 3.11 by @henryiii in https://github.com/scikit-hep/uproot5/pull/743

chore: some cleanup inpsired by refurb by @henryiii in https://github.com/scikit-hep/uproot5/pull/745

chore(deps): update pre-commit hooks by @pre-commit-ci in https://github.com/scikit-hep/uproot5/pull/737

chore(deps): bump amannn/action-semantic-pull-request from 4 to 5 by @dependabot in https://github.com/scikit-hep/uproot5/pull/757

chore: update Uproot to require Awkward 2.0.0rc1. by @jpivarski in https://github.com/scikit-hep/uproot5/pull/765

ci: update to Python 3.11 final by @henryiii in https://github.com/scikit-hep/uproot5/pull/766

docs: add lobis as a contributor for code by @allcontributors in https://github.com/scikit-hep/uproot5/pull/771

refactor: refactor Forth generation by @aryan26roy in https://github.com/scikit-hep/uproot5/pull/710

chore: remove Identifier and "uproot" parameter. by @jpivarski in https://github.com/scikit-hep/uproot5/pull/770

ci: [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in https://github.com/scikit-hep/uproot5/pull/773

refactor: final refactoring for Forth generation by @aryan26roy in https://github.com/scikit-hep/uproot5/pull/749

docs: fix TRef.py doc urls by @veprbl in https://github.com/scikit-hep/uproot5/pull/782

ci: pre-commit autoupdate by @pre-commit-ci in https://github.com/scikit-hep/uproot5/pull/783

docs: uproot.dask docs by @kkothari2001 in https://github.com/scikit-hep/uproot5/pull/702

test: make tests parallelizable (custom_classes in uproot.open). by @jpivarski in https://github.com/scikit-hep/uproot5/pull/786

ci: [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in https://github.com/scikit-hep/uproot5/pull/789

chore(deps): bump pypa/gh-action-pypi-publish from 1.5.1 to 1.6.1 by @dependabot in https://github.com/scikit-hep/uproot5/pull/792

ci: [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in https://github.com/scikit-hep/uproot5/pull/793

New Contributors

@aryan26roy made their first contribution in https://github.com/scikit-hep/uproot5/pull/617

@dependabot made their first contribution in https://github.com/scikit-hep/uproot5/pull/632

@Moelf made their first contribution in https://github.com/scikit-hep/uproot5/pull/630

@agoose77 made their first contribution in https://github.com/scikit-hep/uproot5/pull/691

@lobis made their first contribution in https://github.com/scikit-hep/uproot5/pull/763

Full Changelog: https://github.com/scikit-hep/uproot5/compare/4.2.4...v5.0.0
Source code(tar.gz)
Source code(zip)
v5.0.0rc7(Dec 6, 2022)
New features

feat: support categorical axes on boost histograms by @lobis in https://github.com/scikit-hep/uproot5/pull/764

feat: warn about TBranch name, alias name conflict. by @jpivarski in https://github.com/scikit-hep/uproot5/pull/776

feat: any Mapping assigned to a WritableDirectory is interpreted as a TTree or failure, no fall-through. by @jpivarski in https://github.com/scikit-hep/uproot5/pull/779

feat: add 'interp_options' mechanism and ak_add_doc. by @jpivarski in https://github.com/scikit-hep/uproot5/pull/784

feat: Use awkward pandas, instead of the existing code that explodes Pandas Dataframes by @kkothari2001 in https://github.com/scikit-hep/uproot5/pull/734

feat: made 'very optional' arguments keyword-only by @jpivarski in https://github.com/scikit-hep/uproot5/pull/787

feat: adjust for name change in scikit-hep/awkward#1919. by @jpivarski in https://github.com/scikit-hep/uproot5/pull/788

Bug-fixes and performance

fix: histogram weights not handled correctly in hist / boost conversion by @lobis in https://github.com/scikit-hep/uproot5/pull/774

perf: streamline metadata handling for TBranch name lookup and uproot.dask by @jpivarski in https://github.com/scikit-hep/uproot5/pull/772

fix: ensure AwkwardForth fallback path is tested without history. by @jpivarski in https://github.com/scikit-hep/uproot5/pull/780

fix: all AwkwardForth Forms now agree with awkward_form method output. by @jpivarski in https://github.com/scikit-hep/uproot5/pull/790

Other

refactor: final refactoring for Forth generation by @aryan26roy in https://github.com/scikit-hep/uproot5/pull/749

test: make tests parallelizable (custom_classes in uproot.open). by @jpivarski in https://github.com/scikit-hep/uproot5/pull/786

docs: fix TRef.py doc urls by @veprbl in https://github.com/scikit-hep/uproot5/pull/782

docs: uproot.dask docs by @kkothari2001 in https://github.com/scikit-hep/uproot5/pull/702

ci: [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in https://github.com/scikit-hep/uproot5/pull/773

ci: pre-commit autoupdate by @pre-commit-ci in https://github.com/scikit-hep/uproot5/pull/783

ci: [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in https://github.com/scikit-hep/uproot5/pull/789

ci: [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in https://github.com/scikit-hep/uproot5/pull/793

chore(deps): bump pypa/gh-action-pypi-publish from 1.5.1 to 1.6.1 by @dependabot in https://github.com/scikit-hep/uproot5/pull/792

Full Changelog: https://github.com/scikit-hep/uproot5/compare/v5.0.0rc6...v5.0.0rc7
Source code(tar.gz)
Source code(zip)
v5.0.0rc6(Oct 29, 2022)
This release is coordinated with Awkward 2.0.0rc2 because it involves a change that had to be applied to both (removal of Identifier/Identities).

New features

(none!)

Bug-fixes and performance

fix: working TList serialization by @lobis in https://github.com/scikit-hep/uproot5/pull/763

Other

refactor: refactor Forth generation by @aryan26roy in https://github.com/scikit-hep/uproot5/pull/710

docs: add lobis as a contributor for code by @allcontributors in https://github.com/scikit-hep/uproot5/pull/771

ci: update to Python 3.11 final by @henryiii in https://github.com/scikit-hep/uproot5/pull/766

chore: remove Identifier and "uproot" parameter. by @jpivarski in https://github.com/scikit-hep/uproot5/pull/770

New Contributors

@lobis made their first contribution in https://github.com/scikit-hep/uproot5/pull/763

Full Changelog: https://github.com/scikit-hep/uproot5/compare/v5.0.0rc5...v5.0.0rc6
Source code(tar.gz)
Source code(zip)
v5.0.0rc5(Oct 26, 2022)
This is the first Uproot release that requires awkward>=2.0.0rc1. See Awkward Array v2.0.0rc1.

New features

feat: implemented NON-memberwise deserialization for AsMap. by @jpivarski in https://github.com/scikit-hep/uproot5/pull/746

feat: Added column_projection optimization by @kkothari2001 in https://github.com/scikit-hep/uproot5/pull/755

Bug-fixes and performance

fix: avoid empty TBasket issue in embedded TBasket by @jpivarski in https://github.com/scikit-hep/uproot5/pull/751

fix: don't use Awkward in test_0751 that doesn't need it by @jpivarski in https://github.com/scikit-hep/uproot5/pull/753

Other

ci: remove GHA workaround for macOS Python 3.11 by @henryiii in https://github.com/scikit-hep/uproot5/pull/743

chore: some cleanup inpsired by refurb by @henryiii in https://github.com/scikit-hep/uproot5/pull/745

chore(deps): update pre-commit hooks by @pre-commit-ci in https://github.com/scikit-hep/uproot5/pull/737

chore(deps): bump amannn/action-semantic-pull-request from 4 to 5 by @dependabot in https://github.com/scikit-hep/uproot5/pull/757

chore: update Uproot to require Awkward 2.0.0rc1. by @jpivarski in https://github.com/scikit-hep/uproot5/pull/765

Full Changelog: https://github.com/scikit-hep/uproot5/compare/v5.0.0rc4...v5.0.0rc5
Source code(tar.gz)
Source code(zip)
v4.3.7(Oct 7, 2022)
New features

(none)

Bug-fixes and performance

fix: avoid empty TBasket issue in embedded TBasket (v4) by @jpivarski in https://github.com/scikit-hep/uproot5/pull/752

Other

(none)

Full Changelog: https://github.com/scikit-hep/uproot5/compare/v4.3.6...v4.3.7
Source code(tar.gz)
Source code(zip)
v4.3.6(Oct 5, 2022)
New features

feat: implemented NON-memberwise deserialization for AsMap (v4). by @jpivarski in https://github.com/scikit-hep/uproot5/pull/747

Bug-fixes and performance

fix: ReadOnlyDirectory should provide the largest abs(cycle) when cycle is unspecified, not the largest cycle (v4). by @jpivarski in https://github.com/scikit-hep/uproot5/pull/721

fix: regularize ROOT type aliases to C fundamental type names (v4). by @jpivarski in https://github.com/scikit-hep/uproot5/pull/718

Other

test: adjust for boost-histogram 1.3.2's _storage_type deprecation (v4). by @jpivarski in https://github.com/scikit-hep/uproot5/pull/720

Full Changelog: https://github.com/scikit-hep/uproot5/compare/v4.3.5...v4.3.6
Source code(tar.gz)
Source code(zip)
v5.0.0rc4(Sep 30, 2022)
New features

(none)

Bug-fixes and performance

(none)

Other

chore: drop Python 3.6. by @jpivarski in https://github.com/scikit-hep/uproot5/pull/742

Full Changelog: https://github.com/scikit-hep/uproot5/compare/v5.0.0rc3...v5.0.0rc4
Source code(tar.gz)
Source code(zip)
v5.0.0rc3(Sep 23, 2022)
New features

feat: from_map like optimization for dask arrays by @kkothari2001 in https://github.com/scikit-hep/uproot5/pull/679

feat: Finalizing AwkwardForth reader for Uproot by @aryan26roy in https://github.com/scikit-hep/uproot5/pull/644

Bug-fixes and performance

fix: Avoid triggering temporary dask-awkward/awkward incompatibility. by @jpivarski in https://github.com/scikit-hep/uproot5/pull/694

fix: Do not write incorrect fSumw2 in histograms (v5). by @jpivarski in https://github.com/scikit-hep/uproot5/pull/698

fix: Fixes uproot.dask bug with empty branches by @kkothari2001 in https://github.com/scikit-hep/uproot5/pull/700

fix: Use from_map optimization for delayed numpy arrays and add tests with empty branches for the same by @kkothari2001 in https://github.com/scikit-hep/uproot5/pull/703

fix: use ctx manager to ensure resources are freed by @agoose77 in https://github.com/scikit-hep/uproot5/pull/713

fix: ReadOnlyDirectory should provide the largest abs(cycle) when cycle is unspecified, not the largest cycle. by @jpivarski in https://github.com/scikit-hep/uproot5/pull/715

fix: regularize ROOT type aliases to C fundamental type names. by @jpivarski in https://github.com/scikit-hep/uproot5/pull/717

Other

test: adjust for boost-histogram 1.3.2's _storage_type deprecation. by @jpivarski in https://github.com/scikit-hep/uproot5/pull/719

ci: Lint PR titles according to conventional commits by @jpivarski in https://github.com/scikit-hep/uproot5/pull/689

ci: use concurrency group for semantic-pr-title by @agoose77 in https://github.com/scikit-hep/uproot5/pull/691

ci: [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in https://github.com/scikit-hep/uproot5/pull/706

docs: Installation requirements and error text for dask/dask-awkward in extras.py by @jpivarski in https://github.com/scikit-hep/uproot5/pull/690

ci: [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in https://github.com/scikit-hep/uproot5/pull/716

docs: Add image for dask docs by @kkothari2001 in https://github.com/scikit-hep/uproot5/pull/708

docs: add veprbl as a contributor by @allcontributors in https://github.com/scikit-hep/uproot5/pull/728

docs: add nikoladze as a contributor by @allcontributors in https://github.com/scikit-hep/uproot5/pull/729

docs: add klieret as a contributor by @allcontributors in https://github.com/scikit-hep/uproot5/pull/730

docs: add dcervenkov as a contributor by @allcontributors in https://github.com/scikit-hep/uproot5/pull/731

docs: add beojan as a contributor by @allcontributors in https://github.com/scikit-hep/uproot5/pull/732

docs: add agoose77 as a contributor by @allcontributors in https://github.com/scikit-hep/uproot5/pull/733

docs: add a CITATION.cff for Uproot. by @jpivarski in https://github.com/scikit-hep/uproot5/pull/726

chore: dask_awkward.test_utils moved in ContinuumIO/dask-awkward#76 by @jpivarski in https://github.com/scikit-hep/uproot5/pull/714

New Contributors

@agoose77 made their first contribution in https://github.com/scikit-hep/uproot5/pull/691

Full Changelog: https://github.com/scikit-hep/uproot5/compare/5.0.0rc2...v5.0.0rc3
Source code(tar.gz)
Source code(zip)
v4.3.5(Sep 2, 2022)
Features

feat: Implement transformed axis from boost-histogram/hist (v4). by @jpivarski in https://github.com/scikit-hep/uproot5/pull/676

Bug-fixes

fix: Fixed TTree write_anew in a subdirectory (consistent caches). (v4) by @jpivarski in https://github.com/scikit-hep/uproot5/pull/678

fix: Do not write incorrect fSumw2 in histograms (v4). by @jpivarski in https://github.com/scikit-hep/uproot5/pull/699

Other

ci: Apply #657 (pre-commit with flake8 v5) to main-v4. by @jpivarski in https://github.com/scikit-hep/uproot5/pull/681

Full Changelog: https://github.com/scikit-hep/uproot5/compare/4.3.4...v4.3.5
Source code(tar.gz)
Source code(zip)
5.0.0rc2(Aug 24, 2022)

Development of Uproot v5:

@Moelf reached a first milestone in implementing an RNTuple reader: basic and jagged arrays from all sorts of STL containers: PR #630, #662. Multiple clusters for RNTuple: PR #682.

@kakwok fixed the RAII state of ReadOnlyDirectory in the corner-case fSeekKeys == 0: PR #660.

@kkothari2001 added uproot.dask support for Awkward arrays using dask-awkward: PR #652.

@jpivarski fixed pathlib.Path handling of double slashes in URLs: PR #670. Fixed number of overflow bins in hist.axis.IntCategory: PR #671. Prevented std::pair from being AsStridedObjects (which doesn't work): PR #673. Properly handled transformed axis from boost-histogram: PR #675. Fixed a bug caused by AsDynamic with self._header: PR #674. Fixed WritableTTree's write_anew when in a subdirectory: PR #677.

@henryiii made many CI updates, concluding with a move to hatchling: PRs #686, #684, #685, #683, #688.
Source code(tar.gz)
Source code(zip)
4.3.4(Aug 4, 2022)

@kakwok fixed uninitialized attributes of ReadOnlyDirectory: PR #660 (v5) and #661 (v4).
Source code(tar.gz)
Source code(zip)
5.0.0rc1(Jul 22, 2022)

This is the first 5.0 (pre) release!

It should not be considered stable, and it strictly requires Awkward 2.0, so you need to have the latest pre-release of that. This may be reflected in the lazy import of awkward; if not, a subsequent 5.0 pre-release will get that right.

Changes relative to any 4.x release:

@kkothari2001 removed the uproot.lazy function (though a stub remains for the sake of keeping the documentation until 5.0 is actually released): PR #615. Updated the entire codebase from Awkward 1.x to Awkward 2.x: PR #620.

@aryan26roy changed arguments in the awkward_form method: PR #617. (@jpivarski fixed a bug in it: PR #622.) @aryan26roy completed an AwkwardForth-based reader for AsStrings: PR #616. Made the Python code generation in streamers.py more readable: #629. Revised the way that the AwkwardForth code generator would pass through the control flow: PR #636.

@henryiii added dependabot for GitHub Actions: PR #631. Cleaned up flake8: PR #527.

@jpivarski fixed a performance bug, to allow linear-time iteration over TDirectory fields: PR #638 (backported to v4 as well). Set up as close as possible to full-coverage tests for AwkwardForth: #637.
Source code(tar.gz)
Source code(zip)
4.3.3(Jul 2, 2022)

@jpivarski fixed an O(n²) scaling bug in getting data from TDirectories. Now it's O(n). (Matters for large TDirectories.)

PR #638 in main and #639 in main-v4. This release is from main-v4.
Source code(tar.gz)
Source code(zip)
4.3.2(Jun 22, 2022)

@jpivarski cleanly ensured all Interpretation.awkward_form signatures have been reverted (even AsJagged): PR #627.
Source code(tar.gz)
Source code(zip)
4.3.1(Jun 22, 2022)

@jpivarski applied this fix: https://github.com/scikit-hep/uproot5/pull/618#discussion_r903623799.
Source code(tar.gz)
Source code(zip)
4.3.0(Jun 21, 2022)

First Uproot 4 release from the bug-fix branch.

@jpivarski added a TMatrixTSym model (long stalled PR): #484.

@aryan26roy restored the argument list of Interpretation.awkward_form, a public function: PR #618.
Source code(tar.gz)
Source code(zip)
4.2.4(Jun 21, 2022)

@kkothari2001 implemented uproot.dask (for library="np" only), a new function to provide Uproot data as Dask arrays: PRs #578, #603. Added another new function, uproot.num_entries, for getting the number of TTree entries from a set of files with minimal reading: PR #609.

@jpivarski removed tests that rely on a ROOT TStreamerInfo behavior: PR #612. Changed awkward_form arguments to (self, file, context) so that new pseudo-arguments can be passed through the context without another major code change: PR #611. This is to prepare for @aryan26roy's AwkwardForth implementation of AsObjects interpretations (see #610, WIP targeting Uproot 5).

Note: this is the last Uproot 4 release before the repo branches into main (for version 5) and main-v4 (for bug-fixes in Uproot 4, starting with 4.3.0). Uproot 5 will incorporate Awkward version 2, as described in the Awkward road map, but it will have minimal interface changes relative to Uproot 4 (mostly just uproot.lazy → uproot.dask).
Source code(tar.gz)
Source code(zip)
4.2.3(May 11, 2022)

@jpivarski fixed the use of Unicode names for objects in ROOT files: PR #577. Made flow=True mean the same thing for Uproot histograms as hist: PR #582. Added an interpretation for ROOT::VecOps::RVec: PR #593. Replaced an error message in HTTPSource with a fallback to the non-multipart-GET case: PR #594. Put an upper limit on the time KeyInFileError takes to print itself: PR #595.
Source code(tar.gz)
Source code(zip)
4.2.2(Mar 14, 2022)

@jpivarski fixed #572: PR #573.
Source code(tar.gz)
Source code(zip)
4.2.1(Mar 8, 2022)

@henryiii removed wheel from pyproject.toml: PR #565.

@jpivarski added a rule to skip parsing Float16/Double32 TBranch titles if the title is not parsable (and just assume default number of bits): PR #561. Removed references to deprecated distutils and Pandas Int64Index: PR #564. Removed the rule that interpreted fBits as 1 byte (it's 4 bytes everywhere except in some branches of some Delphes files): PR #570.
Source code(tar.gz)
Source code(zip)
4.2.0(Feb 14, 2022)

Dropped support for Pythons 2.7 and 3.5: PR #525. From now on, Uproot only supports Python 3.6 and up.

@henryiii dropped Python2isms from the codebase: PR #526.

@jrueb made writing to Python file handles work (fixed a half-finished, forgotten implementation): PR #538.

@jpivarski fixed cut jagged-array corner-case in library="pd". Fixed a case in which the instance version is 0, but the streamer version is not: PR #537. Fixed uproot.WritableTTree.extend when the metadata needs to be rewritten: PR #547. When checking to see if something in file["name"] = something is an Awkward Array or Pandas DataFrame for creating a TTree, also check for superclasses (the whole mro): PR #557.
Source code(tar.gz)
Source code(zip)
4.1.9(Dec 8, 2021)

The 4.1.x series is the last to support Python 2.7 and Python 3.5. From 4.2.0 onward, only Python 3.6 and above will be supported.

(Any bug-fixes that absolutely must be used in an old Python environment will need to be versions 4.1.10, 4.1.11, etc., and these won't include features and regular maintenance that go into the main branch, 4.2.x and beyond.)

In this version:

@pfackeldey removed an unnecessary copy of memory-mapped data.

@veprbl exposed the data property of TTable: PR #506.

@jpivarski added an example of RNTuple header parsing: accidentally as 4bb0f6aedbe7e9a8ba8feb166a16e71274259d26, rather than a formal PR. Added more careful DataFrame-checking code in TTree-writing, to avoid a spurious error message: PR #517. Fixed a picking issue with histograms (just the TAxis) by ensuring that all Behaviors are not Python Abstract Base Classes: PR #521.
Source code(tar.gz)
Source code(zip)
4.1.8(Nov 8, 2021)

@masonproffitt fixed warning message for old versions of XRootD/pyxrootd (before 4.11.1): PR #501.

@jpivarski fixed conflicts between explicitly specified TBranches and automatically generated counter-TBranches when they have the same name: PR #499. Also, uproot.lazy identifies the only TTree in a file even if there are multiple with the same name, different cycle numbers: PR #500.
Source code(tar.gz)
Source code(zip)
4.1.7(Oct 28, 2021)

@mpad added an AsDtypeInPlace interpretation to read data into a preallocated array (in library="np" mode): PR #487.

@jpivarski restricted Uproot 4.x to Awkward 1.x in pip install uproot[dev] as well as at runtime.
Source code(tar.gz)
Source code(zip)
4.1.6(Oct 28, 2021)

@duncanmmacleod included the test suite in the source distribution: PR #477.

@btovar fixed exception handling in XRootD callback threads by passing them on to the main thread: PR #480.

@jpivarski restricted Uproot 4.x to Awkward 1.x: PR #478. Also fixed some bugs when writing TTrees to preexisting files ("update" mode): PR #488.
Source code(tar.gz)
Source code(zip)
4.1.5(Oct 14, 2021)

@jpivarski fixed another Uproot-writing bug: writing TTrees didn't update the FreeSegments record, which means that writing another TTree (with another file handle, with uproot.update or ROOT) could overwrite the first TTree.

Like 4.1.4, this is an important bug-fix for anyone creating a robust dataset of ROOT files. (It doesn't affect you if you consider all files to be write-once only, but if you're ever thinking of updating files, this will matter. And besides, correctness is important in principle!)
Source code(tar.gz)
Source code(zip)
4.1.4(Oct 13, 2021)

@jpivarski added TStreamerInfo for TTrees in files written by Uproot. If you're using an older version, you're producing files without TStreamerInfo! (Which means they won't be readable in some older versions of ROOT, and possibly not new ones, either.) PR #472.

Also corrected the TLeaf fTitle for jagged arrays, which is needed to read jagged arrays back in ROOT's TTree::Draw and TTree::Scan, but not for iteration in PyROOT (which is what we had been using for testing). PR #458.

Added an implementation of non-split TClonesArray, which makes more files readable: PR #467, and added RNTuple to the must_be_attached list, which enables RNTuple objects to read more data (like TTrees): PR #463. That's just a step in developing RNTuple-reading capabilities.
Source code(tar.gz)
Source code(zip)
4.1.3(Sep 29, 2021)

@nsmith- fixed Awkward Forms for objects: PR #441, and implemented regular array support for TClonesArray: PR #442.

@jpivarski fixed Pandas representations of leaf-lists (structs) and non-leaf-lists in the same DataFrame: PR #449. Fixed uproot.lazy for empty TTrees and boost-histogram with no title: #451. Fixed issue #439 by checking to see if an array is an Awkward Array before checking to see if it could be NumPy: #453. Fixed both Delphes bugs in issue #438: PR #454.
Source code(tar.gz)
Source code(zip)