Skip to content

Conversation

@Omswastik-11
Copy link
Contributor

@Omswastik-11 Omswastik-11 commented Dec 6, 2025

Summary

This PR brings the skpro probabilistic metrics module into alignment with sktime, as discussed in issue #367.
It adds sample_weight support across metrics, ensures consistent output types, and fixes weighted multioutput behavior.


Changes

1. Sample Weight Support

  • Added a sample_weight parameter to:

    • BaseProbaMetric.evaluate
    • BaseDistrMetric.evaluate
  • Updated input validation methods:

    • _check_consistent_input
    • _check_ys
      to correctly propagate sample_weight
  • Implemented proper weighted averaging using np.average in:

    • _evaluate (Proba metrics)
    • evaluate (Distribution metrics)

2. Output Consistency Improvements

  • BaseDistrMetric.evaluate now returns a pd.Series when:

    • multioutput="raw_values"
    • the metric is univariate (non-multivariate)
  • This matches:

    • BaseProbaMetric behavior
    • sktime conventions
  • Fixes the previous inconsistency where a 1-row DataFrame was returned.


3. Weighted Multioutput Fixes

  • Improved handling of array-like multioutput weights in BaseDistrMetric.evaluate
  • Ensures correct weighted aggregation across variables
  • Fixes incorrect averaging when users pass custom weights.

Tests Added / Updated

  • test_sample_weight_pinball (in test_probabilistic_metrics.py)
    Verifies sample_weight support for quantile and interval metrics.

  • test_sample_weight_logloss (in test_distr_metrics.py)
    Verifies sample_weight support for distribution-based metrics.

  • test_multioutput_weights_logloss (in test_distr_metrics.py)
    Ensures weighted multioutput aggregation behaves correctly.

  • Updated test_distr_evaluate to assert that raw-value outputs are returned as pd.Series.

Fixes #367

@Omswastik-11 Omswastik-11 changed the title [ENH] sync skpro probabilistic modules with sktime [ENH] sync skpro probabilistic metric modules with sktime Dec 6, 2025
@Omswastik-11 Omswastik-11 changed the title [ENH] sync skpro probabilistic metric modules with sktime [ENH] Sync skpro and sktime probabilistic metrics modules: sample_weight and output consistency* Dec 6, 2025
@Omswastik-11 Omswastik-11 changed the title [ENH] Sync skpro and sktime probabilistic metrics modules: sample_weight and output consistency* [ENH] Sync skpro and sktime probabilistic metrics modules: sample_weight and output consistency Dec 6, 2025
@Omswastik-11 Omswastik-11 marked this pull request as ready for review December 6, 2025 18:30
Copy link
Collaborator

@fkiraly fkiraly left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry if the description of the task has not been precise enough, I think you did not quite catch what should happen in the TestAll classes.

The TestAll classes run their tests on all the metrics, and should check API elements that are common to all. So, if you add sample_weight as an argument, the TestAll classes should pass examples where sample_weight is being passed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[ENH] sync skpro and sktime probabilistic metrics modules

2 participants