scaling questions

[sameermahajan]

Currently I notice that for every query to Open AI you send all the pre defined contexts along with the query text to get the results. I think the Open AI API is stateless and that is why you need to send the entire context every time.

In that case how would this solution scale when this system evolves or in context of other cases (I have some in mind that I am currently exploring) where there might be thousands (if not even more) and even complex predefined contexts? Will it have to send all of them every time to get desired results? Are there plans of keeping some state in the cloud, may be with some kind of session id or cookie etc. or some other mechanism so that you don't have to send the entire context every time?

[sameermahajan]

This can be mitigated to some extent by https://learn.microsoft.com/en-gb/training/modules/use-own-data-azure-openai/ on Azure Open AI

However current limitation of this feature restricts the capability to current interactive chat session. We want to be able to do it (may be as a separate model, deployment, endpoint) so that it can be queried externally using APIs.