[TRY] perf: spawn (parallelize) io tasks related to tarball

[kdy1]

This PR uses the threadpool of rayon to perform actual blocking IO operations. It's done by using rayon::scope API.

Blocked by:

• perf: Use faster malloc #188
• perf: Increase the size of rayon threadpool for syscalls #189

[kdy1]

Oops my commits are gone. I'll use git reflog to find it..

[KSXGitHub]

Thank you for your works.

I have a few change requests.

[KSXGitHub]

This seems to be the only IO operation in the entire scope. And none of the operations that follow depend on this IO operation being done either. Meaning, we could potentially defer this operation after building the cas_paths (i.e. do the IO on the created cas_paths). See if this suggestion can improve the code quality and performance.

[KSXGitHub]

A new PR has been merged into main. It allows pacquet to work with a big lockfile. You may pull from main (or rebase) and benchmark with this lockfile: https://gist.github.com/KSXGitHub/ca8076a18b384b6cb6eea86d4ec7c675 (The --fixture-dir/-D flag would allow you to change lockfile without copy pasting)

[codecov]

Codecov Report

Attention: 9 lines in your changes are missing coverage. Please review.

Comparison is base (c62a631) 92.65% compared to head (4569c7f) 92.50%.

❗ Current head 4569c7f differs from pull request most recent head 339b83a. Consider uploading reports for the commit 339b83a to get more accurate results

Files	Patch %	Lines
crates/tarball/src/lib.rs	86.15%	9 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #190      +/-   ##
==========================================
- Coverage   92.65%   92.50%   -0.16%     
==========================================
  Files          57       57              
  Lines        2847     2881      +34     
==========================================
+ Hits         2638     2665      +27     
- Misses        209      216       +7     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[KSXGitHub]

@kdy1 main is faster when run just integrated-benchmark -s frozen-lockfile main HEAD -V -D ./crates/testing-utils/src/fixtures/big/ on this branch:

[kdy1]

It's expected because you are using a beast machine and the lockfile is for a very small project.

The PR is about limiting the number of threads used for syscalls.

[KSXGitHub]

the lockfile is for a very small project

I added -D ./crates/testing-utils/src/fixtures/big/ which contains a big lockfile.

The PR is about limiting the number of threads used for syscalls.

Is there a mechanism to scale the number of syscalls limit based on number of CPUs? Yesterday, there was a merged PR that scale the concurrent network requests limit based on number of CPUs.

Also, is the perf: in the title misleading?

[kdy1]

rayon has an issue with a beast machine. rayon-rs/rayon#394

Can you try limiting the number of rayon threads by configuring RAYON_NUM_THREADS? It's documented at https://docs.rs/rayon/latest/rayon/struct.ThreadPoolBuilder.html#method.num_threads

And the perf is intentional, as I expected it to improve the performance. (rayon is not a good fit for syscalls, but we should not mix rayon and spawn_blocking for syscalls anyway)

[KSXGitHub]

@kdy1 Should the RAYON_NUM_THREADS be low or high? Since I/O are blocking operations, which won't be benefited from having many parallel threads, it would be counterproductive to hog all CPUs for themselves. So counterintuitively, the more CPUs the machine has, the slower this code will be? Am I correct? If this is the case, then would it be more appropriate to have a dedicated thread reserved for only I/O operations?

[kdy1]

It should be lower.

So counterintuitively, the more CPUs the machine has, the slower this code will be

This is partially correct, but this is not related to IO but rather a bug of rayon.
But as it's about communication cost, it goes like

         .
      .    .  
   .          .
.                .

The performance graph will look like this because n is very small here. The communication cost is O(n^2), which is still small if the CPU core is in small range

[KSXGitHub]

@kdy1 Is RAYON_NUM_THREADS=16 just integrated-benchmark -s frozen-lockfile main HEAD -V the correct way to run it? main was always 3 times faster than HEAD regardless of the number.

[kdy1]

Hmmm... Yeah, it's correct. So we should move syscalls to spawn_blocking, not rayon

[KSXGitHub]

@kdy1 Actually, my hypothesis regarding I/O operations is proven incorrect.

I have created a test repo which benchmark 4 cases. The result was parallel I/O + serial CPU is the fastest.