Skip to content
This repository was archived by the owner on Jun 3, 2025. It is now read-only.

Conversation

rahul-tuli
Copy link
Member

@rahul-tuli rahul-tuli commented Apr 12, 2023

This PR majorly enables comparison across common layers and types,
before this comparison was only done at model level

This also does some rename(s):

  • sparsity --> sparsity_percent
  • quantized --> quantized_percent

Slightly improves the logging and print messaging
Additionally adds docstrings to added functions

For the following manual test:

  • rs-pr-95-uni-q.onnx is from zoo:cv/classification/resnet_v1-50/pytorch/sparseml/imagenet/pruned95_uniform_quant-none
  • rs-pr95-q.onnx is from zoo:cv/classification/resnet_v1-50/pytorch/sparseml/imagenet/pruned95_quant-none
sparsezoo.analyze \
  rs-pr-95-uni-q.onnx \
  --by-layers True \
  --by-types True \
  --compare rs-pr95-q.onnx
(sparsezoo3.9) ~ sparsezoo.analyze \
  rs-pr-95-uni-q.onnx \
  --by-layers True \
  --by-types True \
  --compare rs-pr95-q.onnx
INFO:sparsezoo.analyze_cli:Starting Analysis ...
INFO:sparsezoo.analyze_cli:Analysis complete, collating results...
INFO:root:Running analysis `by_layers`
INFO:root:Running analysis `by_types`
Analysis by Layers:
                                    NAME          TOTAL            SIZE        SPARSITY       QUANTIZED
                           Conv_13_quant        9472.00           58679           24.10           99.32
                           Conv_37_quant        4160.00            2252           93.53           98.46
                          Conv_105_quant       16640.00            8982           93.55           98.46
                           Conv_60_quant       36928.00           15234           94.87           99.83
                           Conv_83_quant       16640.00            8912           93.60           98.46
                          Conv_129_quant       16448.00            7148           94.63           99.61
                          Conv_152_quant       36928.00           15293           94.85           99.83
                          Conv_175_quant       16640.00            8968           93.56           98.46
                          Conv_199_quant       16448.00            7135           94.64           99.61
                          Conv_222_quant       36928.00           15293           94.85           99.83
                          Conv_245_quant       16640.00            8898           93.61           98.46
                          Conv_269_quant       32896.00           14270           94.64           99.61
                          Conv_337_quant      131584.00           54951           94.84           99.61
                          Conv_292_quant      147584.00           60015           94.93           99.91
                          Conv_315_quant       66048.00           30872           94.29           99.22
                          Conv_361_quant       65664.00           26789           94.93           99.81
                          Conv_384_quant      147584.00           59660           94.96           99.91
                          Conv_407_quant       66048.00           30764           94.31           99.22
                          Conv_431_quant       65664.00           27264           94.84           99.81
                          Conv_454_quant      147584.00           59779           94.95           99.91
                          Conv_477_quant       66048.00           30926           94.28           99.22
                          Conv_501_quant       65664.00           27264           94.84           99.81
                          Conv_524_quant      147584.00           60015           94.93           99.91
                          Conv_547_quant       66048.00           30926           94.28           99.22
                          Conv_571_quant      131328.00           54740           94.82           99.81
                          Conv_639_quant      525312.00          216426           94.88           99.81
                          Conv_594_quant      590080.00          236811           94.99           99.96
                          Conv_617_quant      263168.00          114163           94.64           99.61
                          Conv_663_quant      262400.00          105477           94.99           99.90
                          Conv_686_quant      590080.00          236339           95.00           99.96
                          Conv_709_quant      263168.00          113737           94.66           99.61
                          Conv_733_quant      262400.00          105267           95.00           99.90
                          Conv_756_quant      590080.00          237284           94.98           99.96
                          Conv_779_quant      263168.00          113524           94.67           99.61
                          Conv_803_quant      262400.00          106319           94.95           99.90
                          Conv_826_quant      590080.00          237284           94.98           99.96
                          Conv_849_quant      263168.00          113737           94.66           99.61
                          Conv_873_quant      262400.00          106319           94.95           99.90
                          Conv_896_quant      590080.00          236811           94.99           99.96
                          Conv_919_quant      263168.00          113737           94.66           99.61
                          Conv_943_quant      262400.00          106530           94.94           99.90
                          Conv_966_quant      590080.00          237284           94.98           99.96
                          Conv_989_quant      263168.00          113737           94.66           99.61
                         Conv_1013_quant      524800.00          214324           94.91           99.90
                         Conv_1081_quant     2099200.00          838769           95.02           99.90
                         Conv_1036_quant     2359808.00          940759           95.02           99.98
                         Conv_1059_quant     1050624.00          436233           94.84           99.81
                         Conv_1105_quant     1049088.00          421930           94.98           99.95
                         Conv_1128_quant     2359808.00          942648           95.01           99.98
                         Conv_1151_quant     1050624.00          434543           94.86           99.81
                         Conv_1175_quant     1049088.00          422771           94.97           99.95
                         Conv_1198_quant     2359808.00          946426           94.99           99.98
                         Conv_1221_quant     1050624.00          436233           94.84           99.81
                               Gemm_1239     2049000.00        65568000            0.00            0.00

Parameters by types:
                                    TYPE           SIZE        SPARSITY       QUANTIZED
                                  Weight       31950722           87.38           91.97
                                    Bias         881920            0.00            0.00
                                   Total       32316077           87.28           91.87

Ops by types:
                                    TYPE           SIZE        SPARSITY       QUANTIZED
                                    Gemm       65568000            0.00            0.00
                             QLinearConv        9580440           94.90          100.00
                                  Gather             32            0.00            0.00
                               Unsqueeze             32            0.00            0.00
                                  Concat             32            0.00            0.00
                                 Softmax             32            0.00            0.00
                          QuantizeLinear            544            0.00            0.00
                                 MaxPool             32            0.00            0.00
                                     Add            512            0.00            0.00
                       GlobalAveragePool             32            0.00            0.00
                        DequantizeLinear           1056            0.00            0.00
                                   Shape             32            0.00            0.00
                                    Relu            416            0.00            0.00
                                 Reshape             32            0.00            0.00
                                   Total       32235349           87.28           91.97

Params:
                                   MODEL       SPARSITY       QUANTIZED           COUNT            SIZE
                     rs-pr-95-uni-q.onnx          87.28           91.87        25530472        32316077

Ops:
                                   MODEL       SPARSITY       QUANTIZED           COUNT            SIZE
                     rs-pr-95-uni-q.onnx          87.28           91.97        25530559        32235349

Overall:
                                   MODEL       SPARSITY       QUANTIZED
                     rs-pr-95-uni-q.onnx          87.28           91.92

Comparison Analysis!!!
INFO:root:Running analysis `by_layers`
INFO:root:Running analysis `by_types`
Comparing rs-pr-95-uni-q.onnx with rs-pr95-q.onnx
Note: comparison analysis displays differences b/w models
Analysis by Layers:
                                    NAME          TOTAL            SIZE        SPARSITY       QUANTIZED
                           Conv_13_quant           0.00           17797          -23.02            0.00
                           Conv_37_quant           0.00          -12395           35.60            0.00
                          Conv_105_quant           0.00          -27783           19.95            0.00
                           Conv_60_quant           0.00          -35754           12.04            0.00
                           Conv_83_quant           0.00          -24483           17.58            0.00
                          Conv_129_quant           0.00          -24321           18.27            0.00
                          Conv_152_quant           0.00          -44395           14.95            0.00
                          Conv_175_quant           0.00          -22073           15.85            0.00
                          Conv_199_quant           0.00          -28740           21.59            0.00
                          Conv_222_quant           0.00          -62094           20.91            0.00
                          Conv_245_quant           0.00          -24567           17.64            0.00
                          Conv_269_quant           0.00          -68264           25.64            0.00
                          Conv_337_quant           0.00          -36103            3.39            0.00
                          Conv_292_quant           0.00         -100855            8.52            0.00
                          Conv_315_quant           0.00          -59474           11.00            0.00
                          Conv_361_quant           0.00           -6340            1.20            0.00
                          Conv_384_quant           0.00          -15507            1.31            0.00
                          Conv_407_quant           0.00          -16490            3.05            0.00
                          Conv_431_quant           0.00          -42059            7.96            0.00
                          Conv_454_quant           0.00          -60726            5.13            0.00
                          Conv_477_quant           0.00          -53472            9.89            0.00
                          Conv_501_quant           0.00          -51095            9.67            0.00
                          Conv_524_quant           0.00          -96002            8.11            0.00
                          Conv_547_quant           0.00          -42929            7.94            0.00
                          Conv_571_quant           0.00         -201843           19.10            0.00
                          Conv_639_quant           0.00           16063           -0.38            0.00
                          Conv_594_quant           0.00          -29779            0.63            0.00
                          Conv_617_quant           0.00         -184451            8.66            0.00
                          Conv_663_quant           0.00           42949           -2.04            0.00
                          Conv_686_quant           0.00          106826           -2.26            0.00
                          Conv_709_quant           0.00          -26411            1.24            0.00
                          Conv_733_quant           0.00           12843           -0.61            0.00
                          Conv_756_quant           0.00           85555           -1.81            0.00
                          Conv_779_quant           0.00          -54739            2.57            0.00
                          Conv_803_quant           0.00          -17264            0.82            0.00
                          Conv_826_quant           0.00           65702           -1.39            0.00
                          Conv_849_quant           0.00          -37274            1.75            0.00
                          Conv_873_quant           0.00          -24843            1.18            0.00
                          Conv_896_quant           0.00          108716           -2.30            0.00
                          Conv_919_quant           0.00           -3621            0.17            0.00
                          Conv_943_quant           0.00          -51370            2.44            0.00
                          Conv_966_quant           0.00           93118           -1.97            0.00
                          Conv_989_quant           0.00          -20021            0.94            0.00
                         Conv_1013_quant           0.00         -585285           13.90            0.00
                         Conv_1081_quant           0.00          517073           -3.07            0.00
                         Conv_1036_quant           0.00          817970           -4.33            0.00
                         Conv_1059_quant           0.00         -135267            1.60            0.00
                         Conv_1105_quant           0.00          246266           -2.93            0.00
                         Conv_1128_quant           0.00          680067           -3.60            0.00
                         Conv_1151_quant           0.00          149639           -1.77            0.00
                         Conv_1175_quant           0.00         -237861            2.83            0.00
                         Conv_1198_quant           0.00          853862           -4.52            0.00
                         Conv_1221_quant           0.00          -45653            0.54            0.00
                               Gemm_1239           0.00        57319546          -87.42            0.00

Parameters by types:
                                    TYPE           SIZE        SPARSITY       QUANTIZED
                                  Weight       19291958           -7.62            0.00
                                    Bias              0            0.00            0.00
                                   Total       19359159           -7.62            0.00

Ops by types:
                                    TYPE           SIZE        SPARSITY       QUANTIZED
                                    Gemm       57319546          -87.42            0.00
                             QLinearConv        1221036           -0.65            0.00
                                  Gather              0            0.00            0.00
                               Unsqueeze              0            0.00            0.00
                                  Concat              0            0.00            0.00
                                 Softmax              0            0.00            0.00
                          QuantizeLinear              0            0.00            0.00
                                 MaxPool              0            0.00            0.00
                                     Add              0            0.00            0.00
                       GlobalAveragePool              0            0.00            0.00
                        DequantizeLinear              0            0.00            0.00
                                   Shape              0            0.00            0.00
                                    Relu              0            0.00            0.00
                                 Reshape              0            0.00            0.00
                                   Total       19310799           -7.62            0.00

Params:
                                   MODEL       SPARSITY       QUANTIZED           COUNT            SIZE
    rs-pr-95-uni-q.onnx - rs-pr95-q.onnx          -7.62            0.00               0        19359159

Ops:
                                   MODEL       SPARSITY       QUANTIZED           COUNT            SIZE
    rs-pr-95-uni-q.onnx - rs-pr95-q.onnx          -7.62            0.00               0        19310799

Overall:
                                   MODEL       SPARSITY       QUANTIZED
    rs-pr-95-uni-q.onnx - rs-pr95-q.onnx          -7.62            0.00

Base automatically changed from by-layers to update-summary-generation April 13, 2023 08:56
 * sparsity --> sparsity_percent
 * quantized --> quantized_percent

 Update: Size calculation to include sparsity
 Add: __name__ to avoid using root logger
 Added: slight improvements to print messaging, and logs
 Style Updates
* _get_entries_to_compare --> _get_comparable_entries
@rahul-tuli rahul-tuli marked this pull request as ready for review April 13, 2023 11:59
@rahul-tuli rahul-tuli self-assigned this Apr 13, 2023
@rahul-tuli rahul-tuli merged commit 8e734c6 into update-summary-generation Apr 18, 2023
@rahul-tuli rahul-tuli deleted the compare-by-types branch April 18, 2023 13:21
rahul-tuli added a commit that referenced this pull request Apr 18, 2023
* Feature: Add compare across types and layers

* Renames:
 * sparsity --> sparsity_percent
 * quantized --> quantized_percent

 Update: Size calculation to include sparsity
 Add: __name__ to avoid using root logger
 Added: slight improvements to print messaging, and logs
 Style Updates

* Rename:
* _get_entries_to_compare --> _get_comparable_entries

* Add: docstrings
rahul-tuli added a commit that referenced this pull request Apr 19, 2023
* Add:ModuleAnalysisSummary schema for easily displaying comparing summaries from ModelAnalysis
Add: Tests for yaml serialization of the same

* Move: yaml de-serialization methods to YAMLSerializableBaseModel

* Style

* Add: Tests for serialization, pretty printing, and subtraction

* Get path programmatically

* Add from_analysis method

* Move relevant schemas to models.py
Move ModelAnalysisSummary to analysis.py
Moved analysis tests to a separate dir
Remove extraneous test
Simplify logic
Move Summary generation to a separate class
Delete old code

* Fix failing tests

* Update CLI

* Update CLI

* Add by_types info to analyze api (#292)

* Add by types

* Remove unintended changes

* Style

* Add kwargs

* Fix failing tests

* Fix failing test + `by-layers`

* Add compare functionality

* Add deprecation warning for `ModelAnalysis.pretty_print_summary(...)`
Move pandas import within a pretty_print_summary(...) function that will be deprecated in a future version

* Add `by-layers` support to analyze api(s) (#301)

* Remove Not Implemented Error

* Add: `by-layers` analysis
Add: total property to `ZeroNonZeroParams`

* Propagate `by-layers` to comparison summary

* Rename: LINEAR_OP_TYPES --> TARGETED_LINEAR_OP_TYPES

* BugFix: int32 was ignored from dense ops during analysis by types

* Add support to compare by types and layers (#302)

* Feature: Add compare across types and layers

* Renames:
 * sparsity --> sparsity_percent
 * quantized --> quantized_percent

 Update: Size calculation to include sparsity
 Add: __name__ to avoid using root logger
 Added: slight improvements to print messaging, and logs
 Style Updates

* Rename:
* _get_entries_to_compare --> _get_comparable_entries

* Add: docstrings

* Add: Multiline row printing (#303)

* Feature: Add compare across types and layers

* Renames:
 * sparsity --> sparsity_percent
 * quantized --> quantized_percent

 Update: Size calculation to include sparsity
 Add: __name__ to avoid using root logger
 Added: slight improvements to print messaging, and logs
 Style Updates

* Rename:
* _get_entries_to_compare --> _get_comparable_entries

* Add: docstrings

* Add: Support to print multiline rows

* Connect Deepsparse.Analyze (#304)

* Feature: Add compare across types and layers

* Renames:
 * sparsity --> sparsity_percent
 * quantized --> quantized_percent

 Update: Size calculation to include sparsity
 Add: __name__ to avoid using root logger
 Added: slight improvements to print messaging, and logs
 Style Updates

* Rename:
* _get_entries_to_compare --> _get_comparable_entries

* Add: docstrings

* Add: Support to print multiline rows

* Connect deepsparse.analyze to sparsezoo.analyze
Add: PerformanceEntry to entry types, else it will be converted to a ModelEntry
Fix: model_name while instantiating PerformanceEntry

* Add: supported_graph_percentage to BenchmarkResult

* Add node level timings

* Style
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants