HPCC-33137 Add supports for executing the cost optimizers in Thor #19423
Conversation
shamser
changed the title
Cost optimizer will execute in Thor for containerized deployments. It will continue to execute in EclAgent for non-containerized deployment. The reason for this change is that in containerized deployments EclAgent cannot calculate the cost of the Thor cluster and so cannot not assigned the cost of issues reported by the cost optimizer. It cannot calculate the cost of the Thor cluster because Thor's cost parameters and resource configuration is not available to EclAgent - this information is only available in the Thor's configuration. Note, that as the Thor manager executes on a per job basis, it executes the analyzer after every graph executes. The concept of "end of workunit" does not exist at present in Thor so the analyzeWhenComplete==true has not been implemented for Thor. The changes: * Support for executing cost optimizer in Thor. This the default in containerized. * New 'analyzeInEclAgent' option - if true, the analyzer executes in EclAgent, otherwise it executes in Thor Manager. * By default 'analyzeInEclAgent' is true in bare-metal and false in containerized * The defaults for 'analyzeWhenComplete' has changed for containerized. It is now false by default in containerized. It remains true in bare-metal. * New 'disabled' option is available within analyzerOptions to disable cost optimizer. 'disabled' is false by default. Signed-off-by: Shamser Ahmed <[email protected]>
|
Jira Issue: https://hpccsystems.atlassian.net//browse/HPCC-33137 Jirabot Action Result: |
| if (w->getDebugValueBool("analyzeWorkunit", agentTopology->getPropBool("@analyzeWorkunit", true))) | ||
| if (w->hasDebugValue("analyzeWorkunit") && !w->getDebugValueBool("analyzeWorkunit", true)) | ||
| return; | ||
| if (!getBoolWUOption(nullptr, nullptr, "analyzerOptions/@disabled", false)) |
There was a problem hiding this comment.
Why not pass w and "analyzeWorkunit" to the function?
There was a problem hiding this comment.
Because if these are passed into the function and 'analyzeWorkunit' was true, then the getBoolWUOption would also return true and that would mean the workunit would not be analyzed, which is not what it should do:
if analyzerOptions/@disabled == true, then it shouldn't execute the analysis. However, if analyzeWorkunit == true, it should analyze the workunit.
I am not sure how to produce such a graph - probably utilizing ECL macros and #LOOP perhaps to produce long sequences of chained attributes(?), or a macro that produces a smaller subgraph variant that can be produced many times. As discussed in my meeting with Shamser, if speed (and the incurred additional Thor cost) is an issue we could:
|
Cost optimizer will execute in Thor for containerized deployments. It will continue to execute in EclAgent for non-containerized deployment.
The reason for this change is that in containerized deployments EclAgent cannot calculate the cost of the Thor cluster and so cannot not assigned the cost of issues reported by the cost optimizer. It cannot calculate the cost of the Thor cluster because Thor's cost parameters and resource configuration is not available to EclAgent - this information is only available in the Thor's configuration.
Note, that as the Thor manager executes on a per job basis, it executes the analyzer after every graph executes. The concept of "end of workunit" does not exist at present in Thor so the analyzeWhenComplete==true has not been implemented for Thor.
The changes:
Type of change:
Checklist:
Smoketest:
Testing: