feat: add unified SolverMetrics #583
Conversation
- 7 optional fields: solver_name, solve_time, objective_value, best_bound, mip_gap, node_count, iteration_count — all default to None - Custom __repr__ that only shows non-None fields - Added as metrics field on Result (backward-compatible — defaults to None) Solver-specific metric extraction (linopy/solvers.py) - Base Solver class: _extract_metrics() returns solver_name + objective_value - Gurobi: extracts Runtime, ObjBound, MIPGap, NodeCount, IterCount - HiGHS: extracts getRunTime(), mip_node_count, simplex_iteration_count, mip_gap, mip_objective_bound - SCIP: extracts getSolvingTime(), getDualbound(), getGap(), getNNodes(), getNLPIterations() - CBC: uses already-parsed mip_gap and runtime from log output - All other solvers (GLPK, Cplex, Xpress, Mosek, COPT, MindOpt, cuPDLPx): use base class default - All 12 return Result(...) sites updated to pass metrics - Every attribute access is wrapped in try/except so extraction never breaks the solve Model integration (linopy/model.py) - _solver_metrics slot, initialized to None - solver_metrics property - Stored from result.metrics after solve() - Set to basic metrics in _mock_solve() - Reset to None in reset_solution() Package export (linopy/__init__.py) - SolverMetrics added to imports and __all__ Tests (test/test_solver_metrics.py) - 13 tests covering: dataclass defaults, partial values, repr, Result backward compat, Model integration (before/after solve, reset), parametrized solver-specific tests for both direct and file-IO solvers
- Added mock-based unit tests for all 10 solver overrides (CBC, Highs, Gurobi, SCIP, Cplex, Xpress, Mosek, COPT, MindOpt, cuPDLPx) - Added test_extract_metrics_graceful_on_missing_attr — verifies _safe_get degrades gracefully - Tests skip for unavailable solvers using @pytest.mark.skipif
Remove all mock/patch-based _extract_metrics tests. The parametrized integration tests (test_solver_metrics_direct, test_solver_metrics_file_io) now assert solve_time >= 0 for every available solver, ensuring attribute names are correct against real solver objects. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
FBumann
changed the title
Only parametrize over solvers with _extract_metrics overrides (gurobi, highs, scip, cplex, xpress, mosek), so solvers with base-only metrics (glpk, copt, cbc) don't fail on solve_time.
|
Covereage fails, as not all solvers are in CI. But this is a change which might affect CI in general. |
lkstrp
left a comment
lkstrp
left a comment
There was a problem hiding this comment.
I would really like to see something like this. Not sure though if this would be enough to reliable benchmark the model in automated runs, but with a fixed node/ environment I don't see why this wouldn't already be helpful. Any thoughts on getting memory usage as well? I think gurobi gives you peak memory, not sure for the other solvers
I tried to design this in a way to be extensible, add new attributes and populate them for solvers that provide them. |
2 participants
Summary
Add a unified
SolverMetricsdataclass that provides solver-independent access to performance metrics after solving. Accessible viaModel.solver_metrics.Fields:
solver_name,solve_time,objective_value,dual_bound,mip_gap— all default toNone; solvers populate what they can.Design:
Solver._extract_metrics()populatessolver_nameandobjective_valuedataclasses.replace()_safe_get()with debug logging so extraction never breaks the solve_extract_metrics()and use_safe_get(). PRs adding metrics for additional solvers are welcome!Solver coverage:
*CBC parses
solve_timeandmip_gapfrom log output via regex. These fields depend on the CBC log format, which varies across versions.Solvers without a tested
_extract_metricsoverride still getsolver_nameandobjective_valuefrom the base class. We intentionally did not add solver-specific overrides for untested solvers — incorrect attribute names would silently returnNone, giving a false sense of coverage.Bugs fixed along the way:
mip_objective_bound— fixed tomip_dual_bound. Also fixed status comparison (== 0→== highspy.HighsStatus.kOk).miprelgapattribute doesn't exist — compute gap manually frommipbestobjvalandbestbound.m.solution.progress.get_time()doesn't exist — usetime.perf_counter()around the solve call.Test plan
SolverMetricsdataclass unit tests (defaults, partial, repr, frozen)Resultbackward compatibility tests (with/without metrics)Modelintegration tests (metrics before solve, after mock solve, after reset)mip_gapanddual_boundare populatedCloses #428