Spark 3.5: Set data file sort_order_id in manifest when writing/compacting data files

[jbewing]

What

This PR updates all writes from Spark 3.5 to set the sort_order_id of data files when applicable per the Iceberg Table Spec. I've opened this PR to:

1. Gauge interest in this and validate my approach
2. If the approach is good, port these changes to Spark 4.0 and 3.4 in a follow up PR
3. As a follow up to those PRs, implement the ability to then report this ordering information when applicable to Spark to better inform the query planner of potential optimizations. This is rather useful when used in conjunction with SPJ and I have this running on an Iceberg fork with success

Testing

I've added tests for newer added utility functions and updated existing tests that write data files and compact data files in a sorted manner to verify that we're setting the sort_order_id entry in the manifests to the correct value. Additionally, I've used this patch on an internal fork and verified that it correctly sets this field during compaction and normal writes.

Issue: #13634

[jbewing]

I'm not in love with this approach, but I went through a couple of iterations of this and plumbing the intended Iceberg SortOrder along through with the Spark SortOrder ended up being the cleanest in my opinion (open to any suggestions of course). I initially had an approach which would attempt to re-guess the Iceberg SortOrder from a Spark SortOrder. It worked pretty well, but was pretty naive in my opinion—the algorithm was to essentially reverse the Iceberg -> Spark SortOrder transformation as best as possible and then see if things matched up. It gets complicated when you start to consider Spark Orderings for stuff like say a positionDeltaUpdateMergeOrdering.

[github-actions]

This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.

[github-actions]

This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.

[puchengy]

@jbewing I have same issue and thanks for the fix. What's your plan of this change, do you still plan to drive this to be merged?

[jbewing]

I have same issue and thanks for the fix

OOC, what's your intended usage of the sort order in the manifest files? I'm assuming to better inform a downstream tool about potential optimizations

What's your plan of this change, do you still plan to drive this to be merged?

Sorry, I put this one up and kinda forgot about it and one of my other Iceberg PRs after I got radio silence from a maintainer in another PR after some feedback. Since this code is running perfectly fine in a fork of Iceberg, there wasn't a big motivation to get things upstreamed. However, given that you would find benefit from this (and it also appears like there's another user with the same issue: #14531), I will find some time next week to update my PR on the current master branch and get this reviewed and integrated!

[puchengy]

@jbewing its also for optimization. We need the sort id to be in files table so that we can make some accurate decisions about it. Thanks for the quick response and your continual effort on this.

[jbewing]

Sorry for the delay here @puchengy , but I wanted to let you know that I've re-opened my initial PR. I added support for Spark 4.0 in addition to Spark 3.5 in that PR. You can follow that PR here: #14683