Comments/questions

[UkoeHB]

Hey, I spent some time studying this repo to evaluate requirements for an edgegap integration in my project. Had some questions and comments.

Comments + questions about impl

• bevygap_matchmaker
  • Potentially-fragile security decision: lightyear ConnectTokens always use the same auth key. More robust would be new auth key per game server.
  • kv_cert_digests is never cleaned up.
• bevygap_matchmaker_httpd
  • What is matchmaker/wannaplay endpoint for? It's not used anywhere in the crate.
• bevygap_shared
  • Why does NATS_CA_CONTENTS require writing to a temp file? Can't you just extract the string directly?
• bevygap_server_plugin
  • Why does this wait until NatsConnected to get ArbitriumContext?
• bevygap_webhook_sink
  • Nothing listens to hook messages. Is this just prepped for future use?
• edgegap_async
  • Why is this auto-generated and not on crates.io?

Questions

• Bevygap does not require the webtransport client have the same SocketAddr as the websocket client used to connect to the matchmaker backend (which is then passed to EdgeGap when creating a session).
  • Possibilities:
    • EdgeGap allows connections from other addresses (but then why need the client IP?).
    • The bevygap architecture assumes the WebTransport address will 'end up' having the same IP as the websocket client even when a wildcard is used (or browser auto-generates an address).
• Why does bevygap only support WebTransport connections?
  • Does EdgeGap require TLS on all connections to deployments?
• Is NATS efficient for game servers? Storage persistence seems overkill, since all game servers would presumably participate. Or do NATS clients not store anything extra, and always do networked requests for reads?
  • NATS allows game servers to connect to backend without needing a specific address, which is a limitation of websockets.
• It looks like connecting clients are thrown into a random deployment. How does EdgeGap decide to make a new deployment, and how can you assign a specific deployment to a predefined client list?
  • Maybe use SessionModel::selectors to 'choose' what deployment a session is assigned to? Or is there a separate API for custom deployments (also, can you send data into a deployment as CLI arg?)?
• Lightyear ConnectTokens are constructed from the deployment public IP and first port. If a new UDP socket (or webtransport or websocket server) is made inside the deployment, will its address not match the 'deployment public IP'? If it doesn't match, will clients not be able to connect?

[RJ]

hi! answering inline –

Hey, I spent some time studying this repo to evaluate requirements for an edgegap integration in my project. Had some questions and comments.

Comments + questions about impl

* bevygap_matchmaker
  
  * Potentially-fragile security decision: lightyear ConnectTokens always use the same auth key. More robust would be new auth key per game server.

true - the key is read from the LIGHTYEAR_PRIVATE_KEY env at the moment, and i'm not changing it per deployment.
it's set in the edgegap dashboard. this is an artefact of porting the traditional lightyear setup to edgegap.

one option would be to randomly assign the key on server boot, and write to NATS kv so the matchmaker can access it for generating tokens. that would be better, and less config.. will do something like that.

  * kv_cert_digests is never cleaned up.

yep, need to do that

* bevygap_matchmaker_httpd
  
  * What is matchmaker/wannaplay endpoint for? It's not used anywhere in the crate.

older pre-websockets endpoint, i need to just delete that. i experimented with long-running http / sse before switching to websockets.

* bevygap_shared
  
  * Why does NATS_CA_CONTENTS require writing to a temp file? Can't you just extract the string directly?

this was a work-around while testing ways to provide the CA to an edgegap deployment iirc, and i just wanted to reuse the read-from-file codepath. edgegap told me they have a ticket to increase the size of the ENVs you can set, and when that happens i'll probably make reading the NATS_CA_CONTENTS env the main way to set the CA data and get rid of the temp file thing.

* bevygap_server_plugin
  
  * Why does this wait until NatsConnected to get ArbitriumContext?

next thing to happen after reading the context was to write some of it to nats iirc, so i wanted to ensure the server crashes asap if nats isn't available on boot

* bevygap_webhook_sink
  
  * Nothing listens to hook messages. Is this just prepped for future use?

yes. the matchmaker polls the get session http endpoint at the mo. the idea is in future to arrange it so an incoming webhook can interrupt the polling and return instantly, or even avoid polling altogether. but unimlemented since polling hasn't caused any issues yet.

* edgegap_async
  
  * Why is this auto-generated and not on crates.io?

i've not got around to publishing the matchmaker crate to crates yet, i'll need to publish edgegap_sync when i do that.
i've published edgegap which is a sync version of the client, for my arbctl tool

Questions

* Bevygap does not require the webtransport client have the same SocketAddr as the websocket client used to connect to the matchmaker backend (which is then passed to EdgeGap when creating a session).

the lightyear connect token includes this IP i believe, but the check is disabled for now. see lightyear/692

it should require the client ip that makes the matchmaking request and the eventual connection to the game server to be the same, i don't think there's any reason not to require this. it's something the lightyear connecttoken code will hopefully take care of, although i'm not necessarily against adding checks into bevygap too.

  * Possibilities:
    
    * EdgeGap allows connections from other addresses (but then why need the client IP?).
    * The bevygap architecture assumes the WebTransport address will 'end up' having the same IP as the websocket client even when a wildcard is used (or browser auto-generates an address).

not sure what you mean here

* Why does bevygap only support WebTransport connections?

no reason other than i wanted to make a native+wasm game and i only need webtransport for that. shouldn't be a stretch to support whatever lightyear supports. or to abstract out lightyear and support whatever i think.

  * Does EdgeGap require TLS on all connections to deployments?

afaik edgegap doesn't care, it just lets you expose ports so you can do what you want

* Is NATS efficient for game servers? Storage persistence seems overkill, since all game servers would presumably participate. Or do NATS clients not store anything extra, and always do networked requests for reads?

the gameservers themselves aren't doing any nats lookups in the game loop or anything, i think the storage bit only happens on the remote nats server itself, where there's a configured storage directory. so clients would always be reading over the network. unsure if there's any caching. the gameservers don't read anything from nats, just writing some metadata on boot. so that nats client is just holding a tcp connection to the nats server until needed.

i imagined gameservers writing events onto nats for aggregate stats collection in future, like writing player joined, player made a kill, etc. (only read by a central non-gameserver aggregator system, and i doubt those events would be persistent anyway).

  * NATS allows game servers to connect to backend without needing a specific address, which is a limitation of websockets.

i'm using a star topology for nats where each gameserver connects directly to my single nats server, although other setups are possible. i considered writing my own websocket server to broker the various messages needed, but it would just mean reinventing a lot of what nats/redis/pubsub-systems can already do (and redis shouldn't really be directly exposed to the internet, but nats can be).

* It looks like connecting clients are thrown into a random deployment. How does EdgeGap decide to make a new deployment, and how can you assign a specific deployment to a predefined client list?

edgegap is very flexible in this regard, but the bevygap matchmaker is only doing something simplistic with their API for now. how i'm using it now, edgegap gets to decide, based on the client ip, if there's an existing deployment close enough to the player (with capacity) to join them to, or if it should autostart a new one.

edgegap's api allows you to manage deployments yourself, direct players to specific deployements, matchmake groups of client ips together for "join with a party" setups, and lots more.

for example the bg matchmaker could get the list of active deployments to send to the client, display them in a server browser, and the client could request to join a specific active deployment.

edgegap also have their own fancy matchmaker that supports a lot of more complex scenarios but i haven't tried it.

  * Maybe use SessionModel::selectors to 'choose' what deployment a session is assigned to? Or is there a separate API for custom deployments (also, can you send data into a deployment as CLI arg?)?

you can provide the deployment_request_id to the get session api call to assign a client to an existing deployment (after presumably doing get-deployemnts to let the client pick from active ones)

you can create/stop/update deployments via edgegap api too, which i haven't tried.

* Lightyear ConnectTokens are constructed from the deployment public IP and first port. If a new UDP socket (or webtransport or websocket server) is made inside the deployment, will its address not match the 'deployment public IP'? If it doesn't match, will clients not be able to connect?

each deployment is a single instance of a gameserver (ie, one running docker container), so its public ip can't change while it's running.

you can configure edgegap to expose multiple ports on your deployement, and you'd know the external port numbers via the arbitrium context / api call. if you wanted to dynamically start listening on anther socket at runtime, you've have to make sure you have configured an exposed port for it in edgegap, but it should be fine.

[UkoeHB]

Thanks for the reply, informative :)

• edgegap_async
• Why is this auto-generated and not on crates.io?

I guess my question was more: why didn't the edgegap_async creator publish it to crates.io?

• bevygap_server_plugin

• Why does this wait until NatsConnected to get ArbitriumContext?

The reason I asked was the impl is fairly convoluted, compared to just requesting the ArbitriumContext in-line in the tokio task that sets up NATS. NATS not being available should be quite rare (since spawning deployments relies on services that need NATS), so I imagine you wouldn't lose much in terms of EdgeGap CCU by failing < 0.5s slower.

it should require the client ip that makes the matchmaking request and the eventual connection to the game server to be the same, i don't think there's any reason not to require this.

Right, so this would be The bevygap architecture [currently] assumes the WebTransport address will 'end up' having the same IP as the websocket client even when a wildcard is used [to construct the webtransport client] (or [when the] browser auto-generates an address). Because presumably EdgeGap will deny network messages from anything other than session-linked IPs.

the gameservers don't read anything from nats, just writing some metadata on boot.

Actually they do read from kv_c2s to get the session id on client connect.

edgegap's api allows you to manage deployments yourself, direct players to specific deployements, matchmake groups of client ips together for "join with a party" setups, and lots more.

Ok this is reassuring.

each deployment is a single instance of a gameserver (ie, one running docker container), so its public ip can't change while it's running.

I guess my question is kind of 'basic'. As in, if you use a wildcard address to bind a UdpSocket inside a deployment, will the socket's local_addr always be the public IP of the deployment? What about wildcard ports, will it auto-select a port that was 'enabled' for the deployment? Or would it be more correct to get the deployment IP + available ports and manually assign them to servers inside the deployment? Like you'd need one for NATS, webtransport, and potentially UdpSocket and websocket (in the case of renet2, which can do cross-platform setups).

[RJ]

Thanks for the reply, informative :)

• edgegap_async
• Why is this auto-generated and not on crates.io?

I guess my question was more: why didn't the edgegap_async creator publish it to crates.io?

i generated this crate from edgegap's api defs swagger file, see https://github.com/RJ/bevygap/blob/main/utils/gen-edgegap-client.sh

• bevygap_server_plugin
• Why does this wait until NatsConnected to get ArbitriumContext?

The reason I asked was the impl is fairly convoluted, compared to just requesting the ArbitriumContext in-line in the tokio task that sets up NATS. NATS not being available should be quite rare (since spawning deployments relies on services that need NATS), so I imagine you wouldn't lose much in terms of EdgeGap CCU by failing < 0.5s slower.

yes looking at this again it could definitely be simplified and just requested inline like you said.

it should require the client ip that makes the matchmaking request and the eventual connection to the game server to be the same, i don't think there's any reason not to require this.

Right, so this would be The bevygap architecture [currently] assumes the WebTransport address will 'end up' having the same IP as the websocket client even when a wildcard is used [to construct the webtransport client] (or [when the] browser auto-generates an address). Because presumably EdgeGap will deny network messages from anything other than session-linked IPs.

yes i'm assuming that the websocket client and webtransport client ips will match. a fairly safe assumption i think since both connect out over http (although you never know). and its not strictly checked/enforced yet as previously mentioned.

i'm actually not sure if edgegap deny packets from non-linked session IPs.

the gameservers don't read anything from nats, just writing some metadata on boot.

Actually they do read from kv_c2s to get the session id on client connect.

the gameservers don't know/care about edgegap session ids iirc. the matchmaker maintains the mapping between edgegap session ids and lightyear client ids in nats kv.

in the bevygap_server_plugin the gameserver publishes a connected/disconnected message to nats including the lightyear client id, and the matchmaker listens and does the edgegap api cleanup.

        app.add_observer(handle_lightyear_client_connect);
        app.add_observer(handle_lightyear_client_disconnect);
// ...
fn handle_lightyear_client_connect(
    trigger: Trigger<ConnectEvent>,
    nats_sender: ResMut<NatsSender>,
) {
    let client_id = trigger.event().client_id;
    info!("Lightyear connect event for client_id {}", client_id);
    nats_sender.client_connected(client_id.to_bits()); // this is non blocking - via channel to another thread.
}

the lightyear ConnectTokens have space for some arbitrary user data in their signed payload, so how i imagine it working in future is when the matchmaker creates the ConnectToken, it embeds a unique user identifier, and the gameserver uses that to lookup that user's info (stats, character data, whatever) from a database service on connect.

there's no matchmaker auth requirement yet. but if you had signin somehow to the matchmaker, that would be the account id to embed in the ConnectToken. I don't think the gameservers need to know the edgegap session ids.

edgegap's api allows you to manage deployments yourself, direct players to specific deployements, matchmake groups of client ips together for "join with a party" setups, and lots more.

Ok this is reassuring.

each deployment is a single instance of a gameserver (ie, one running docker container), so its public ip can't change while it's running.

I guess my question is kind of 'basic'. As in, if you use a wildcard address to bind a UdpSocket inside a deployment, will the socket's local_addr always be the public IP of the deployment? What about wildcard ports, will it auto-select a port that was 'enabled' for the deployment? Or would it be more correct to get the deployment IP + available ports and manually assign them to servers inside the deployment? Like you'd need one for NATS, webtransport, and potentially UdpSocket and websocket (in the case of renet2, which can do cross-platform setups).

i see - since the deployments are hosted on AWS or whatever cloud, how they usually work is the underlying server doesn't have a public IP in the traditional sense (ie, you can't just see it with ip addr or ifconfig). so you can't really enumerate your public ip address like you would on a regular machine.

there is some network address translation going on to route the assigned public ip to the server. so you would listen on the wildcard interface and your chosen port, so for IPv4: 0.0.0.0:1234 and then your service would be accessible via the public IP assigned to you and port 1234. you can use the aws dashboard to buy more public ips at any point, that also get routed to your machine.

however with edgegap, they also do network translation on the port to expose your port using a high port number.

For example, my gameserver listens on 0.0.0.0:5000 for UDP not TCP (because webtransport is UDP) and edgegap is configured like this:

and the API response for get session gives me the public ip and port number (usually in the 33000 range) to use when connecting. my gameserver sees all traffic arriving on port 5000, but the game client is sending to port 33000ish

amongst other reasons, this setup allows edgegap to host gameserver containers from multiple customers on the same machine, each "listening" on the same port, oblivious to whatever network plumbing is happening before packets reach them.

if i wanted the gameserver to listen on multiple ports, like if i was supporting webtransport, and raw udp, and steam etc. i'd have the gameserver listen on 0.0.0.0:5000 and 0.0.0.0:5001 etc. then i'd create more port definitions in edgegap with a useful name like "steam" "raw udp" etc. then the api call would give you the ip:port to use for each named port definition.

edgegap gameservers can connect out to the internet fine, so no special rules needed for nats, since the nats connection is from the gameserver to the nats server, and the nats server has a fixed, known public ip and port (and tls cert, hopefully).

[UkoeHB]

i generated this crate from edgegap's api defs swagger file

Oh got it, cool. Didn't realize that was a custom solution for bevygap.

i'm actually not sure if edgegap deny packets from non-linked session IPs.

Hmm if it doesn't deny packets, that would simplify some of the workflows since you wouldn't actually need sessions in order to run a game server. Or maybe you'd only need sessions temporarily when deciding what physical location the deployment should spawn at.

the gameservers don't know/care about edgegap session ids iirc. the matchmaker maintains the mapping between edgegap session ids and lightyear client ids in nats kv.

Right now you need the session id to update active_connections, that's why you're accessing kv_c2s in the server plugin.

the lightyear ConnectTokens have space for some arbitrary user data in their signed payload, so how i imagine it working in future is when the matchmaker creates the ConnectToken, it embeds a unique user identifier, and the gameserver uses that to lookup that user's info (stats, character data, whatever) from a database service on connect.

Hmm, what game style are you actually going for? Open attendance or match-based? If it's match-based then you will know the user list ahead of time and can just save that info in NATS. (the current design is open attendance)

In my game's architecture, I just send a 'game launch' package to the game server on startup, which contains all info needed to run the game. Knowing everything on startup simplifies server and client initialization since as soon as clients connect they will receive everything needed from the server.

edgegap gameservers can connect out to the internet fine, so no special rules needed for nats, since the nats connection is from the gameserver to the nats server, and the nats server has a fixed, known public ip and port (and tls cert, hopefully).

Hmm a little confused by this. The NATS connection can receive data so it must have a socket with an IP. So wouldn't there need to be address translation between that socket's local port and a port exposed by the edgegap API? If not, what if your NATS connection accidentally claims one of the 'edgegap mapped' ports that was intended for the gameserver to use?