Eagerly write to tuning database #2025
Conversation
mirza-halilcevic
requested review from
dhernandez0,
djramic,
pabloantoniom and
umangyadav
|
|
||
| winners, allData = tuneMLIRKernels(configs, confClass, paths, options) | ||
|
|
||
| if winners is None: |
There was a problem hiding this comment.
this was a weird check, maybe we should still check len(entries ) > 0 inside tuneMLIRKernels()?
There was a problem hiding this comment.
I think we should keep this check if winner is none inside tuneMLIRKernels.
There was a problem hiding this comment.
In your opinion, should we abort the entire loop in case this happens, or just continue with the rest of the configs? Same goes for when verification fails?
There was a problem hiding this comment.
I wonder if it is possible to continue with rest of tuning and print error message about config that is failing to pick any winning perfConfig. That would be better IMO
| if options.tflops: | ||
| winners[testVector] = (winningConfig,maxTFlops) | ||
| if not headerWritten: | ||
| print(f"# arch\tnumCUs\ttestVector\tperfConfig\tTFlops ({options.tuningSpaceKind})", file=outFile) |
There was a problem hiding this comment.
It is better to create list out of header columns and then do pd.DataFrame.to_csv(sep='\t)` instead of directly putting print inside the file itself.
| if not headerWritten: | ||
| print(f"# arch\tnumCUs\ttestVector\tperfConfig\tTFlops ({options.tuningSpaceKind})", file=outFile) | ||
| headerWritten = True | ||
| print(f"{options.arch}\t{options.numCU}\t{testVector}\t{winningConfig}\t{maxTFlops}", file=outFile) |
There was a problem hiding this comment.
It is better to create list and use pd.DataFrame.to_csv here too.
There was a problem hiding this comment.
Pull Request Overview
This PR refactors the tuning process to write winning configurations to the output file immediately after each test vector is tuned, rather than waiting for the entire tuning process to complete. This prevents loss of tuning results if the process crashes or is interrupted.
- Modified
tuneMLIRKernelsto accept output file handles and write results immediately - Changed
getWinningConfigto return collected entries instead of appending to a global list - Restructured main function to open files upfront and use try/finally for proper cleanup
Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
mirza-halilcevic
force-pushed
the
tuning-persistence
branch
from
fa63939 to
82e038c
Compare
3 participants
Motivation
Refactor of tuningRunner.py so that the winning configs are immediately written to the output file instead of waiting for the entire tuning process to finish, so we avoid situations where a crash forces us to start all over.
Technical Details
Code changes to the tuningRunner.py script to implement said functionality.
Resolves https://github.com/ROCm/rocMLIR-internal/issues/2017
Test Plan
Manually tested.
Test Result
Tuning results persist after the process is interrupted.
Submission Checklist