Speed up Star Ratings recalculation and User Tags population processes

[cyperdark]

a month ago i noticed that populating user tags takes a very long time and i got curious if it because of web requests or not, and only today i was able to check it out. Turns out it was writing to realm of each loop iteration which very very slow process considering that some people have more than 10_000 beatmaps.

As a solution, I moved realmAccess.Write and realmAccess.Run outside of the loop to speed up process. I dont know much about realm and not entirely sure if it's good solution

Star Ratings recalculation (2.3x speed up)

For benchmarking i waited until 5k beatmaps were recalculated, and speed up on my machine was by

Before	After
5074 beatmaps completed in 124_127ms	5077 beatmaps completed in 52_934ms
YouTube video	YouTube video

Populating User Tags (69.3x speed up)

I waited for all 25709 user tags to be populated

Before	After
391_438ms	5_647ms
YouTube video	YouTube video

Tests done offline on 7000mb/s read and 5000mb/s write SSD

Notes and questions

i applied the same changes to processBeatmapsWithMissingObjectCounts, processScoresWithMissingStatistics, convertLegacyTotalScoreToStandardised, upgradeScoreRanks and backpopulateMissingSubmissionAndRankDates, but didnt benchmark them.

I didnt touch processOnlineBeatmapSetsWithNoUpdate because it making web requests to server(?) and im not sure if i should touch this function.

I have a question about processBeatmapsWithMissingObjectCounts: this function calls beatmapUpdater.ProcessObjectCounts to update beatmap object count, but this function has beatmapInfo.Realm!.Write in it which will slow recalculation process, in that case can this function be moved to processBeatmapsWithMissingObjectCounts function instead? since it's the only place it's used in.

Also i'm not so sure about backpopulateMissingSubmissionAndRankDates is it only reads data from local storage without making requests to server?

Another question i have: how realm.write works, does it finishes after file is fully saved and then completeNotification called?

A concern i have with this approach is runtime.log size (803Kb without fix, 5.4Mb with fix)

[peppy]

Did you try performing any operations when testing this? It should deadlock. Realm doesn't support multiple concurrent writes, so holding writes open for long periods as you are doing here, would under my presumptions, cause something like running gameplay (and triggering an update-thread-bound write) to freeze the game.

[cyperdark]

when it was recalculating beatmaps i was scrolling though song select and it was working fine, let me test it gameplay and song select once again. If it wont work i'll try another approach i have in mind

[peppy]

Please air the other solution before you work on it, because if it's what I think you're planning it likely has other implications.

[cyperdark]

Okay i've tested and it's not adding maps to collection, doesnt switch from gameplay to result screen and i assume it wont allow/will freeze if i try to download new map

Another approach is writing once at the end of loop or let's say every 500 beatmaps, like this:

[peppy]

Doing intermittent writes like that will delay the updates from being visible. It also increases the size of individual transactions. Maybe okay, but will need due consideration. And it likely will not fix the "abysmal performance" group of users, who actually see things taking seconds per realm write.

What we really need there is to find someone who gets very low performances on realm transactions and figure out what is causing that, rather than working around the issue.

[cyperdark]

hmm, best candidates for testing that people who have their lazer files/ and realm.client on HDD.

So using bulk update by 100 beatmaps in populateMissingStarRatings is questionable fix but what about backpopulateUserTags, it's less resource demanding task where this approach will be better than updating each beatmap, especially when people like me have over 10k beatmaps and it takes 6 minutes to update all of them

[peppy]

Does it add overhead while updating? If so, that's my concern.

Put another way: we want to be able to run background operations like this without the user being concerned they are running. Making it run faster is fine and all, but it's not fixing the actual issue we are dealing with.

So this is super low priority as long as all that's happening is "batching more".

[cyperdark]

idk how to measure overhead, but here's video with performance graph with calculating star ratings and scrolling in song select. Closer to the end i rerun calculations with small chunk of maps and then started updating tags (both used bulk, star ratings 100, user tags 500): https://youtu.be/m_7wohmkI5U

with only user tags updating: https://youtu.be/6X7xrV1VsyA
but keep in mind that this tests made on high end pc

recalculating 5k maps:
bulk 500 (noticable lags): 64549ms
bulk 100 (slightly lags): 63501ms
normal: 154688ms

user tags (doesnt lag in song select for me):
bulk 500: 6794ms
normal (was): 391438ms