Real-time collaboration: Add new REST endpoints, setting, and registered post meta

[chriszarate]

Description

Trac ticket: https://core.trac.wordpress.org/ticket/64622

In Gutenberg, we have added support for real-time collaboration using CRDT documents (via the Yjs library). This work has suggested the following additions to WordPress:

1. A default "sync provider" based on HTTP polling that allows collaborators to share updates with each other. Previously, we relied on WebRTC connections between collaborators for this purpose, but it proved unreliable under many network conditions.

   • Our solution is designed to work on any WordPress installation.
   • HTTP polling is the transport we identified as most likely to work universally.
   • Given the isolation and lifecycle of PHP processes, updates must be stored centrally in order to be shared among peers. We have chosen to store updates in post meta against a special post type, but alternate storage mechanisms are possible.
   • Collaborative editing can involve syncing multiple CRDT documents. To limit the number of connections consumed by this provider, requests are batched.
   • To prevent unbounded linear growth, updates are periodically compacted.
   • To avoid excessive load on lower-resourced hosts, this provider will benefit from usage limits (e.g., a maximum of three connected collaborators) enforced by the client (Gutenberg).

2. A new registered post meta that allows Gutenberg to persist CRDT documents alongside posts.

   • This provides all collaborators with a "shared starting point" for the collaborative session, which avoids duplicate updates.
   • Content stored in the WordPress database always remains the source of truth. If the content differs from the persisted CRDT document, the CRDT document is updated to match the database.

3. A new Writing setting that allows users to opt-in to real-time collaboration.

   • Enabling real-time collaboration disables post lock functionality and connects users to the sync provider.

4. A behavior change to autosaves is needed. When the the original author is editing a draft post (post_status == 'draft' OR 'auto-draft') and they hold the post lock, the autosave targets the actual post instead of an autosave revision. This puts the post data and the persisted CRDT document out of sync and leads to duplicate updates. When real-time collaboration is enabled, all collaborators must autosave in the same way.

This PR provides a proposed implementation of the changes above. This corresponding Gutenberg PR moves the work from the experimental directory to lib/compat:

WordPress/gutenberg#75366

Cumulative work to add this functionality can be found using this label:

https://github.com/WordPress/gutenberg/issues?q=label%3A%22%5BFeature%5D%20Real-time%20Collaboration%22%20is%3Apr

Testing instructions

1. Check out this branch.
2. Follow the setup instructions in the README.
3. Settings > Writing > Enable real-time collaboration.
4. Open a post for editing in two browsers / browser tabs.

[github-actions]

Hi @chriszarate! 👋

Thank you for your contribution to WordPress! 💖

It looks like this is your first pull request to wordpress-develop. Here are a few things to be aware of that may help you out!

No one monitors this repository for new pull requests. Pull requests must be attached to a Trac ticket to be considered for inclusion in WordPress Core. To attach a pull request to a Trac ticket, please include the ticket's full URL in your pull request description.

Pull requests are never merged on GitHub. The WordPress codebase continues to be managed through the SVN repository that this GitHub repository mirrors. Please feel free to open pull requests to work on any contribution you are making.

More information about how GitHub pull requests can be used to contribute to WordPress can be found in the Core Handbook.

Please include automated tests. Including tests in your pull request is one way to help your patch be considered faster. To learn about WordPress' test suites, visit the Automated Testing page in the handbook.

If you have not had a chance, please review the Contribute with Code page in the WordPress Core Handbook.

The Developer Hub also documents the various coding standards that are followed:

• PHP Coding Standards
• CSS Coding Standards
• HTML Coding Standards
• JavaScript Coding Standards
• Accessibility Coding Standards
• Inline Documentation Standards

Thank you,
The WordPress Project

[github-actions]

Test using WordPress Playground

The changes in this pull request can previewed and tested using a WordPress Playground instance.

WordPress Playground is an experimental project that creates a full WordPress instance entirely within the browser.

Some things to be aware of

• The Plugin and Theme Directories cannot be accessed within Playground.
• All changes will be lost when closing a tab with a Playground instance.
• All changes will be lost when refreshing the page.
• A fresh instance is created each time the link below is clicked.
• Every time this pull request is updated, a new ZIP file containing all changes is created. If changes are not reflected in the Playground instance,
  it's possible that the most recent build failed, or has not completed. Check the list of workflow runs to be sure.

For more details about these limitations and more, check out the Limitations page in the WordPress Playground documentation.

Test this pull request with WordPress Playground.

[pkevan]

Is there any tests we can add for the default provider and new functionality?

[pkevan]

Can we also provide some documentation on how to override the provider given it's known limitation listed above?

[TimothyBJacobs]

Why isn't each room a post?

[TimothyBJacobs]

This seems like it should be a responsibility of the storage layer. If I want to swap with something that can retrieve results ordered by time, we shouldn't have to load all updates always to do that.

[chriszarate]

I've updated the storage interface and class to take this approach in 9e5cf96. It was a little trickier than I thought, because compaction is an external concern and shouldn't be implemented by storage.

[chriszarate]

Why isn't each room a post?

No strong reason. Mostly to avoid creating a large number of posts. Each room corresponds to a synced CRDT document (representing a WordPress entity). Currently supported entity types:

• posts (any post type)
• taxonomy collections

In the future, this could be expanded to include all database entities, including individual taxonomy terms. We might also create rooms to share awareness data at the page or screen level.

We intend to enforce limits on this default provider, but probably applied to simultaneous collaborators and not on the overall number of synced entities. Sharing a single post felt prudent but happy to reconsider! I know wp_postmeta#meta_key is indexed but I don't have a grasp on the performance tradeoff at large scale.

[pkevan]

Fix one of the failing unit tests here: chriszarate#1 since I couldn't PR against this.

[chriszarate]

Note that I've removed the commit that updates Gutenberg to the required version to allow unit tests to pass. The testing instructions have been updated accordingly.

[github-actions]

The following accounts have interacted with this PR and/or linked issues. I will continue to update these lists as activity occurs. You can also manually ask me to refresh this list by adding the props-bot label.

Unlinked Accounts

The following contributors have not linked their GitHub and WordPress.org accounts: @gclapps0612-cmd.

Contributors, please read how to link your accounts to ensure your work is properly credited in WordPress releases.

Core Committers: Use this line as a base for the props when committing in SVN:

Props czarate, paulkevan, timothyblynjacobs, westonruter, jorgefilipecosta, mindctrl, ellatrix.

To understand the WordPress project's expectations around crediting contributors, please review the Contributor Attribution page in the Core Handbook.

[westonruter]

The process_awareness_update method never returns a WP_Error, yeah?

Suggested change

	if ( is_wp_error( $merged_awareness ) ) {
	return $merged_awareness;
	}

[chriszarate]

Fixed in 466a3f4

[westonruter]

This fixes a PHPStan complaint:

phpstan: Method WP_Sync_Post_Meta_Storage::get_updates_after_cursor() return type has no value type specified in iterable type array.

Suggested change

	* @return array Sync updates.
	* @return mixed[] Sync updates.

[westonruter]

Suggested change

	* @return array Sync updates.
	* @return array<int, mixed> Sync updates.

[chriszarate]

I've addressed all of these in 3d55d27. Thanks!

[pkevan]

Suggested change

	'enable_real_time_collaboration' => 0,
	'enable_real_time_collaboration' => 1,

switch on for beta1

[chriszarate]

I've implemented this in a separate branch that can be merged in this one or separately into trunk:

chriszarate#2

[jorgefilipecosta]

Could we avoid registering the post type if the enable_real_time_collaboration setting is not enabled?

[chriszarate]

I think that makes sense. Addressed in d1d3040

[mindctrl]

One of my first questions, which was already asked by @TimothyBJacobs, is related to the use of a single wp_sync_storage post.

Unless I'm misreading it, when add_post_meta, update_post_meta, and delete_post_meta are called, they invalidate all the cached meta for that post ID, but because every room shares the same wp_sync_storage post, that means all cached sync data across all rooms is invalidated. It's hard to understand why an editing session on post ID 1 should impact the caching and performance of a session on post ID 2.

In testing with 3 clients, I'm seeing thousands of post meta function calls within minutes. I think some of that could be alleviated by giving each room its own wp_sync_storage post, so that meta cache invalidation is scoped to a single session rather than all of them.

[chriszarate]

In testing with 3 clients, I'm seeing thousands of post meta function calls within minutes. I think some of that could be alleviated by giving each room its own wp_sync_storage post, so that meta cache invalidation is scoped to a single session rather than all of them.

That's great feedback, thank you. I'll explore using separate posts in a new PR based on this one.

[chriszarate]

@mindctrl, I've implemented your suggestion in this PR. cc @TimothyBJacobs. Now merged into this PR

[westonruter]

I increased PHPStan level from 8 to 9 and found a couple other nits:

 ------ --------------------------------------------------------------------------------------------------------------------------------------------- 
  Line   class-wp-sync-post-meta-storage.php                                                                                                          
 ------ --------------------------------------------------------------------------------------------------------------------------------------------- 
  104    Cannot access offset 'timestamp' on mixed.                                                                                                   
         🪪  offsetAccess.nonOffsetAccessible                                                                                                         
         at src/wp-includes/collaboration/class-wp-sync-post-meta-storage.php:104                                                                     
  104    Cannot access offset 'value' on mixed.                                                                                                       
         🪪  offsetAccess.nonOffsetAccessible                                                                                                         
         at src/wp-includes/collaboration/class-wp-sync-post-meta-storage.php:104                                                                     
  110    Method WP_Sync_Post_Meta_Storage::get_all_updates() should return array<int, array{timestamp: int, value: mixed}> but returns array<mixed>.  
         🪪  return.type                                                                                                                              
         at src/wp-includes/collaboration/class-wp-sync-post-meta-storage.php:110                                                                     
  134    Method WP_Sync_Post_Meta_Storage::get_awareness_state() should return array<int, mixed> but returns array<mixed, mixed>.                     
         🪪  return.type                                                                                                                              
         at src/wp-includes/collaboration/class-wp-sync-post-meta-storage.php:134                                                                     
 ------ --------------------------------------------------------------------------------------------------------------------------------------------- 


 [ERROR] Found 4 errors