[runtime] Fix Python awaitable lost during checkpoint restore

[weiqingy]

Linked issue: #508

Purpose of change

Fix a crash that occurs when restoring a Python async action from checkpoint.

Root cause: Python coroutines cannot be serialized. When a checkpoint captures state while an async action is in progress (e.g., waiting for LLM response), the coroutine object is lost. On restore, the awaitable reference is None, causing:
AttributeError: 'NoneType' object has no attribute 'send'

Fix:

• Detect None awaitable in PythonActionExecutor.callPythonAwaitable()
• Throw AwaitableLostException to signal the awaitable was lost
• PythonGeneratorActionTask catches this and re-executes the action from the beginning
• Durable execution cache ensures already-completed calls are skipped

Tests

• Manual testing: Run react_agent_example.py, trigger LLM timeout, verify job recovers instead of crashing
• No unit test added - these classes depend on Pemja (Python interpreter) which is difficult to mock; the fix is better validated via e2e testing

API

No public API changes.

Documentation

• doc-needed
• doc-not-needed
• doc-included

[weiqingy]

Hey @Sxnan, wondering if there’s a way to re-run the failed CI? It seems I don’t have access to trigger it. This looks like a pre-existing flaky test, not related to the PR.

[Sxnan]

@weiqingy , Thanks for identifying this bug and submitting the fix! The issue you found is real and important.
After reviewing the PR, I made some adjustments to align the Python async handling with how we handle Java continuations. The current approach uses a transient map in the operator (similar to continuationContexts for Java), which makes the checkpoint restore detection more explicit and consistent across both languages.
I've pushed a fixup commit with these changes. Could you please take a look and let me know what you think?

[weiqingy]

@weiqingy , Thanks for identifying this bug and submitting the fix! The issue you found is real and important. After reviewing the PR, I made some adjustments to align the Python async handling with how we handle Java continuations. The current approach uses a transient map in the operator (similar to continuationContexts for Java), which makes the checkpoint restore detection more explicit and consistent across both languages. I've pushed a fixup commit with these changes. Could you please take a look and let me know what you think?

Thanks for the quick review and changes, @Sxnan! It LGTM.

[Sxnan]

@xintongsong Could you take a look at this PR?

[xintongsong]

LGTM

[Sxnan]

Ported to release-0.2 in 5d9fb77