Improve memory usage of SendWorkerOneMessageToManyRequest#72
Improve memory usage of SendWorkerOneMessageToManyRequest#72dlogothetis wants to merge 3 commits intoapache:trunkfrom
Conversation
|
|
||
| @Parameterized.Parameter(value = 0) | ||
| public int numVertices; | ||
|
|
There was a problem hiding this comment.
Looks like if you only need a single parameter, you can just use:
@parameters
public static Object[] data() {
return new Object[] { "first test", "second test" };
}
https://github.com/junit-team/junit4/wiki/parameterized-tests
There was a problem hiding this comment.
I'll simplify this then, thanks!
| // Data to send | ||
| int partitionId = 13; | ||
| Partition<IntWritable, IntWritable, IntWritable> partition = | ||
| public static class NonParameterizedTest extends SharedSetup { |
There was a problem hiding this comment.
Any reason for not using Parameterized tests for this two test cases?
There was a problem hiding this comment.
These other tests are a bit different with respect to what the parameter would be and the range of values. For now, I just wanted to make this particular test parameterized.
|
Not sure why the commits show Graph-1185. Do you have to rebase? |
|
I accidentally created a commit with the same message as the previous commit. When this lands, it'll get a new commit message anyway. |
yukselakinci
left a comment
yukselakinci
left a comment
There was a problem hiding this comment.
Needs better unit tests.
2 participants
https://issues.apache.org/jira/browse/GIRAPH-1190
The current implementation takes incoming messages stored as ByteArrayOneMessageToManyIds and prepares them as a map from partition id to a ByteArrayVertexIdMessages, which holds the messages for the corresponding partition. It then adds these to the message store.
However, it is possible that these intermediate lists of message get big before they are added to the message store. If they reach the capacity of the underlying buffers, the job fails. This can be avoided if we push these lists to the message store before the get big. This is mostly beneficial when we use a combiner in which case the message store keeps only one value per vertex.
Tests: