03-05-2014, 06:16 PM
A little update:
I just completely rewrote the chunk dispatching algorithm so it would better handle timeouted chunks.
If a chunk is assigned to an agent and it not finished (or even started), after a certain period of time it will be reassigned to a new agent.
In this case, the amount of work remaining will be stripped of this chunk and used to create a new chunk for new agent. The original chunk will be marked as completed.
If the remaining work is too large for currently served agent, only a portion of this chunk will be dispatched to this agent and another chunk will be created with the original agent. This will repeat untill the remaining work is splitted enough times to vanish. The amount of how much work can each agent be given in one dispatch is dependent on its benchmark for the task. That way, if you have chunk size configured to 300s, you are guaranteed that each agent will have exactly 300s worth of computing except for the "ending" sequence of redispatched chunk, in which case it will be smaller number than 300.
Bottomline - this new systems perfectly accurately tracks how much of work has each agent computed (I'm talking ACTUALLY computed, not just been dispatched).
I just completely rewrote the chunk dispatching algorithm so it would better handle timeouted chunks.
If a chunk is assigned to an agent and it not finished (or even started), after a certain period of time it will be reassigned to a new agent.
In this case, the amount of work remaining will be stripped of this chunk and used to create a new chunk for new agent. The original chunk will be marked as completed.
If the remaining work is too large for currently served agent, only a portion of this chunk will be dispatched to this agent and another chunk will be created with the original agent. This will repeat untill the remaining work is splitted enough times to vanish. The amount of how much work can each agent be given in one dispatch is dependent on its benchmark for the task. That way, if you have chunk size configured to 300s, you are guaranteed that each agent will have exactly 300s worth of computing except for the "ending" sequence of redispatched chunk, in which case it will be smaller number than 300.
Bottomline - this new systems perfectly accurately tracks how much of work has each agent computed (I'm talking ACTUALLY computed, not just been dispatched).