continuation of deep populate

[atiertant]

here is continuation of PR
relink conversations:
https://github.com/balderdashy/sails-mysql/issues/292
https://github.com/balderdashy/sails-mysql/issues/291
https://github.com/particlebanana/waterline-query-docs/issues/2
https://github.com/balderdashy/waterline/issues/308
balderdashy/sails#3429
balderdashy/waterline-docs#98
https://github.com/balderdashy/sails-mongo/issues/108
https://github.com/balderdashy/waterline/issues/1157
balderdashy/sails#3105
https://github.com/balderdashy/waterline/issues/484
https://github.com/balderdashy/waterline/issues/266

[atiertant]

@mikermcneil

The issue with this approach is specifically when dealing with bidirectional "hasMany" associations that stretch across different databases, and combining that behavior with sort. In order to do that, you have to fetch all records in the junction table/collection -- the in-between model storing the records associating both sides of the bidirectional collection ("hasMany") association. It's a perfectly fine solution unless your database(s) are large. But for certain production use cases (i.e. where you have millions of records), this is not okay; it can overflow RAM and kills your Node process.

where have you seen this deep code ? has i explained you:

this code will
1 - compagyn.find(id).populate('driver')
2 - the deepCursor will collect all drivers pk and index reference to driver objects.
3 - driver.find([cursor previously collected drivers pk]).populate('taxi')
4 - cursor link results in compagny.driver previously indexed
which database doesn't suport this ?

this code will never fetch all records in the junction table/collection and crash node process...
if it does, this is the original waterline populate who do this...

sorry if you can't lock this conversation but be sure i'll don't !

[emahuni]

i have some implementation, but i saw this in the issues of waterline: go thru it in waterline.