Skip to content

Re-add the compound unique index to catalog_product_index_price_tmp #37701

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Conversation

nicka101
Copy link
Contributor

@nicka101 nicka101 commented Jul 3, 2023

Addresses #37700

Description (*)

Adds a composite unique constraint back to catalog_product_index_price_tmp (and by extension catalog_product_index_price_temp), to make the join used in deleteOutdatedData performant on large stores

Related Pull Requests

Fixed Issues (if relevant)

  1. Fixes Price index performance on very large stores #37700

Manual testing scenarios (*)

  1. Reindex catalog_product_price on a catalog with a very large number of prices and Magento 2.4.5+

Contribution checklist (*)

  • Pull request has a meaningful description of its purpose
  • All commits are accompanied by meaningful commit messages
  • All new or changed code is covered with unit/integration tests (if applicable)
  • README.md files for modified modules are updated and included in the pull request if any README.md predefined sections require an update
  • All automated tests passed successfully (all builds are green)

@m2-assistant
Copy link

m2-assistant bot commented Jul 3, 2023

Hi @nicka101. Thank you for your contribution!
Here are some useful tips on how you can test your changes using Magento test environment.

Add the comment under your pull request to deploy test or vanilla Magento instance:
  • @magento give me test instance - deploy test instance based on PR changes
  • @magento give me 2.4-develop instance - deploy vanilla Magento instance

❗ Automated tests can be triggered manually with an appropriate comment:

  • @magento run all tests - run or re-run all required tests against the PR changes
  • @magento run <test-build(s)> - run or re-run specific test build(s)
    For example: @magento run Unit Tests

<test-build(s)> is a comma-separated list of build names.

Allowed build names are:
  1. Database Compare
  2. Functional Tests CE
  3. Functional Tests EE
  4. Functional Tests B2B
  5. Integration Tests
  6. Magento Health Index
  7. Sample Data Tests CE
  8. Sample Data Tests EE
  9. Sample Data Tests B2B
  10. Static Tests
  11. Unit Tests
  12. WebAPI Tests
  13. Semantic Version Checker

You can find more information about the builds here
ℹ️ Run only required test builds during development. Run all test builds before sending your pull request for review.


For more details, review the Code Contributions documentation.
Join Magento Community Engineering Slack and ask your questions in #github channel.

@nicka101
Copy link
Contributor Author

nicka101 commented Jul 3, 2023

@magento run all tests

@magento-automated-testing
Copy link

The requested builds are added to the queue. You should be able to see them here within a few minutes. Please message the #magento-devops slack channel if they don't show in a reasonable amount of time and a representative will look into any issues.

@nicka101
Copy link
Contributor Author

nicka101 commented Jul 3, 2023

@magento run Functional Tests B2B

@magento-automated-testing
Copy link

The requested builds are added to the queue. You should be able to see them here within a few minutes. Please message the #magento-devops slack channel if they don't show in a reasonable amount of time and a representative will look into any issues.

@m2-community-project m2-community-project bot added the Priority: P2 A defect with this priority could have functionality issues which are not to expectations. label Jul 4, 2023
@nicka101 nicka101 force-pushed the bugfix/catalog_product_index_price_tmp_indexes branch from 73a0c63 to 90e4ecc Compare July 6, 2023 17:42
@nicka101
Copy link
Contributor Author

nicka101 commented Jul 6, 2023

@magento run all tests

@magento-automated-testing
Copy link

The requested builds are added to the queue. You should be able to see them here within a few minutes. Please message the #magento-devops slack channel if they don't show in a reasonable amount of time and a representative will look into any issues.

@kandy
Copy link
Contributor

kandy commented Jul 11, 2023

Hi @nicka101
Adding this index will slow down inserting data to the table.

So, can I recommend implementing other ways to resolve this issue?
Instead of joining temporary table to main table as it done in deleteOutdatedData
method, we can just select ids from temporary table and intersect it from ids of current butch

or similar changes to delete queries, like

    $query = $select->delete('main_table',
    [
        'main_table.entity_id IN (?)', $entityIds, \Zend_Db::INT_TYPE,/
         'main_table.entity_id not in ( select id from  'tmp_table' entity_id IN (?) and tmp_table.customer_group_id  = $customer_group_id and tmp_table.website_id = $website_id)',
         'main_table.customer_group_id  = $customer_group_id'
         'main_table.website_id = $website_id',
       
    );

@sinhaparul sinhaparul added the Project: Community Picked PRs upvoted by the community label Sep 13, 2023
Copy link
Contributor

@engcom-Hotel engcom-Hotel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hello @nicka101,

Thanks for the contribution!

I agree with the @kandy here, we need to think of some other way to resolve this issue.

This solution contains backward incompatibility changes.

Thanks

@engcom-Hotel
Copy link
Contributor

Hello @nicka101,

Thank you for your contribution!

As we have been unable to get any reply from your side for a long time, we are closing this PR for now.

Whenever you are ready to take this up please free to reopen it or ask us to reopen the same. We will be happy to do that.

Thanks again!

@dudzio12
Copy link
Member

Well, the downtime for inserts is FAR more marginal than speed up on delete query. I understand it is not ideal, but having those indexes is a real improvement.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Priority: P2 A defect with this priority could have functionality issues which are not to expectations. Project: Community Picked PRs upvoted by the community
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Price index performance on very large stores
5 participants