Support for dcm <tag1> <tag2> ...?

[realcundo]

Perhaps it'd be useful for dcm to only parse subset of the dicom input, and output only specified tags. The benefits would be:

• dicom input parsing speedup
• reduced memory use from only extracting data that is passed downstream

Current behaviour is to load entire dicom into memory (only skipping PixelData.)

Questions

1. Currently dcm foo is a shortcut for select foo | dcm. Is it useful to keep current behaviour?
2. Should dcm output only tags that are specified as arguments, or should it output all tags until the highest arg tag?
   • Should this be configurable and if so,
   • what is the default?
3. Should it support nesting inside sequences, or only operate on root-level tags?
4. If no tags match, return null/empty/some default value like in case of -e for errors?
5. When specifying numeric tags, support wildcards? (e.g. extract all elements in a group)
6. Similarly, when specifying tags by name, support regex?

[Enet4]

Just leaving my thoughts, though they are mostly oriented around what I know about DICOM and DICOM-rs, and the little I currently know about nushell.

Perhaps it'd be useful for dcm to only parse subset of the dicom input, and output only specified tags. The benefits would be:

• dicom input parsing speedup
• reduced memory use from only extracting data that is passed downstream

Current behaviour is to load entire dicom into memory (only skipping PixelData.)

There are plans to support this in DICOM-rs directly, first with a separate API that allows consumers to retrieve data sets in chunks, and then with a DICOM object implementation which fetches some parts on demand only as they are needed (i.e. the pixel data).
Until then, it is already possible to set the stop_until option somewhere after the last tag indicated on the requested list. DICOM-rs already expects data elements to be ordered by tag (this is the standard-compliant behavior).

When processing nothing but metadata, fetching and parsing everything at once is already pretty fast, unless we are working with multi-frame files with per-frame functional group containing lots of frames, or structured reports.

Questions

1. Currently dcm foo is a shortcut for select foo | dcm. Is it useful to keep current behaviour?
   Do you mean dcm | select foo? There is one advantage in supporting arguments at dcm, which is to support the attribute selection syntax in DICOM-rs. It supports an attribute path beyond just simple tags, plus it recognizes tags in hexadecimal form.

open file.dcm | dcm SequenceOfUltrasoundRegions.RegionSpatialFormat

2. Should dcm output only tags that are specified as arguments, or should it output all tags until the highest arg tag?>
   • Should this be configurable and if so,
   • what is the default?

I suspect anything other than fetching all metadata by default would be surprising. An option to fetch the pixel data as well (e.g. dcm --pixel-data or dcm --all) could be added in the event that there are use cases for it.

3. Should it support nesting inside sequences, or only operate on root-level tags?

See response to question 2 above, though it is unclear what the output should be if multiple selectors were provided. One would probably flatten all selected attributes into the same level in a non-nested record.

4. If no tags match, return null/empty/some default value like in case of -e for errors?

Given the existence of DICOM data elements of type 2 (can be empty) and type 3 (can be missing), I would rather not see missing or empty attributes as errors, but rather have this reflected in the output: empty elements would appear on the output record with null (of the nothing type), and missing elements would not appear in the output.

5. When specifying numeric tags, support wildcards? (e.g. extract all elements in a group)

OK, that is one use case not covered by the attribute selector syntax (it was only designed to unambiguously select one attribute). It could still be done through a distinct syntax or with other filtering options such as --start-tag «tag» and --read-until.

6. Similarly, when specifying tags by name, support regex?

At a glance I don't see a use case for regex matching DICOM keywords, but it can already be done by other commands if really necessary. The attribute selection syntax seems more useful, on the other hand.

[realcundo]

Thank you for sharing your thoughts!

When processing nothing but metadata, fetching and parsing everything at once is already pretty fast, unless we are working with multi-frame files with per-frame functional group containing lots of frames, or structured reports.

Yes, this is very true, for now it's plenty fast. I think I'd optimise it not necessarily from the CPU perspective but memory consumption: if you pass binary data to dcm, the file is read into memory, then parsed into DicomObject and then transformed to nushell tree, leading to having 2 or 3 copies of data.

I was looking at dicom_parser::dataset::lazy_read::LazyDataSetReader and I think I'll support at at some point, together with more efficient binary streaming into the plugin. Nushell supports it already, but not the dcm plugin. The only time binary data would be read into memory by nushell is when you try to pass a list of binary data, e.g. [(open 1.dcm), (open 2.dcm)] | dcm in which case contents are loaded into memory by nushell since nushell doesn't support list of streams. Only open file.dcm | dcm can utilise a binary stream.

Do you mean dcm | select foo? There is one advantage in supporting arguments at dcm, which is to support the attribute selection syntax in DICOM-rs. It supports an attribute path beyond just simple tags, plus it recognizes tags in hexadecimal form.

Didn't know about dicom_core::ops::AttributeSelector, looks useful! Before this I was thinking of having nushell-like selectors , e.g. your example of

open file.dcm | dcm SequenceOfUltrasoundRegions.RegionSpatialFormat

would become

open file.dcm | dcm SequenceOfUltrasoundRegions.0.RegionSpatialFormat  # first element
open file.dcm | dcm SequenceOfUltrasoundRegions.-1.RegionSpatialFormat # last element
open file.dcm | dcm SequenceOfUltrasoundRegions.*.RegionSpatialFormat  # all elements

I haven't decided on the selector format yet.

One thing I'd like to support is "virtual groups" like fmi, data and pixeldata where each would match all tags belonging to that group. And maybe even expand this to support information modules like PatientModule, StudyModule, etc. I'd like to avoid using flags like --all, so that you can construct all selectors programmatically without having special cases when sometimes you need to pass selectors and sometimes parameters/flags.

Having modules and these "virtual groups" (bad name!) would most likely avoid any need for wildcard tag matching or using tag ranges.

See response to question 2 above, though it is unclear what the output should be if multiple selectors were provided. One would probably flatten all selected attributes into the same level in a non-nested record.

Yes, this is something that's not clear to me yet. The simplest thing would be to keep the original nesting so that to access the list of RegionSpatialFormat you'd have to write

open file.dcm | dcm SequenceOfUltrasoundRegions.0.RegionSpatialFormat | select SequenceOfUltrasoundRegions

and leave flattening to nushell. As above, still undecided on this.

Given the existence of DICOM data elements of type 2 (can be empty) and type 3 (can be missing), I would rather not see missing or empty attributes as errors, but rather have this reflected in the output: empty elements would appear on the output record with null (of the nothing type), and missing elements would not appear in the output.

Would this mean that if you do

open file.dcm | dcm SequenceOfUltrasoundRegions.*.RegionSpatialFormat

but the input doesn't contain SequenceOfUltrasoundRegions, you'd still see SequenceOfUltrasoundRegions in the output (as nothing/empty list)? Or would it be omitted from the output? My initial thinking was that the output only contains data that are present in the input, but I can see how it could be useful to inject missing data for easier processing downstream. Maybe there can be a flag for this. 🤔