[Enhancement] GPU selection for multi-GPU setups

[MrDrMcCoy]

Users with multiple GPUs at present do not appear to have a way to choose a GPU to run on with this library. This would be very nice to have, along with an example of creating a pool of workers for each and queuing generation tasks across them. Would this be difficult to implement?

[william-murray1204]

Hi @MrDrMcCoy, there are a few scenarios where multi-GPU setups work and are limited, depending on your hardware. The simplest and currently supported scenario is using multiple GPUs running the same backend, for example two CUDA GPUs. In that case, you can force which GPU is visible via environment variables, as described in this issue. I have an example of how you might go about implementing something like this multi-AMD GPU inference (using a HIP backend) in my Python tests.

If, on the other hand, you want to run multi-backend GPU hardware simultaneously (e.g. a CUDA and an AMD GPU), there is no easy way to do this built into the library, as each build can only support a single backend. It is not impossible, but it would require running separate Python environments with separate installs of stable-diffusion-cpp-python each targeting a different backend. You could then coordinate inference externally, for example, via a task queue that dispatches jobs to each environment.