ci: add pip/pre-commit cache and fix mirror override in NPU CI#9320
ci: add pip/pre-commit cache and fix mirror override in NPU CI#9320JavaPythonAIForBAT wants to merge 1 commit into
Conversation
- lint.yaml: upgrade setup-python to v5 with pip cache enabled; add pre-commit environment cache keyed on .pre-commit-config.yaml - citest_npu.yaml: add actions/cache@v4 for ~/.cache/pip keyed on requirements/framework.txt and requirements/tests.txt hashes - ci_container_test.sh: remove explicit -i mirror flags so all pip installs use the internal cache server configured in Config mirrors Co-Authored-By: wuhejun <wuhejun@h-partners.com>
There was a problem hiding this comment.
Code Review
This pull request removes the Aliyun PyPI mirror from the pip install commands within the CI container test script. The reviewer suggests enhancing the script's reliability by adding error handling to the installation steps to prevent the CI from proceeding if dependencies fail to install. Furthermore, combining multiple pip install calls into a single command was recommended for better efficiency and dependency resolution.
| if [ "$MODELSCOPE_SDK_DEBUG" == "True" ]; then | ||
| # pip config set global.index-url https://pypi.tuna.tsinghua.edu.cn/simple | ||
| pip install -r requirements/tests.txt -i https://mirrors.aliyun.com/pypi/simple/ | ||
| pip install -r requirements/tests.txt |
There was a problem hiding this comment.
It is recommended to add error handling to the pip install command. Without it, if the installation of test dependencies fails, the script will continue to subsequent steps (like pre-commit), which may lead to misleading failures or unreliable test results in CI.
| pip install -r requirements/tests.txt | |
| pip install -r requirements/tests.txt || exit 1 |
| pip install -r requirements/framework.txt -U | ||
| pip install decord einops -U |
There was a problem hiding this comment.
These two pip install commands can be combined into a single invocation. This is more efficient as it reduces the overhead of starting the pip process multiple times and allows the dependency resolver to handle all packages together, ensuring consistency. Additionally, adding || exit 1 ensures the script terminates immediately if the installation fails.
| pip install -r requirements/framework.txt -U | |
| pip install decord einops -U | |
| pip install -r requirements/framework.txt decord einops -U || exit 1 |
| # test with install | ||
| pip install . | ||
| pip install auto_gptq bitsandbytes deepspeed -U -i https://mirrors.aliyun.com/pypi/simple/ | ||
| pip install auto_gptq bitsandbytes deepspeed -U |
There was a problem hiding this comment.
Adding error handling here ensures that the script does not proceed to run the tests if these dependencies fail to install, which helps in identifying environment setup issues early and prevents running tests in an incomplete environment.
| pip install auto_gptq bitsandbytes deepspeed -U | |
| pip install auto_gptq bitsandbytes deepspeed -U || exit 1 |
Summary
actions/setup-pythonto v5 with built-incache: pipenabled; addactions/cache@v4for pre-commit environments keyed on.pre-commit-config.yamlhashactions/cache@v4for~/.cache/pipkeyed onrequirements/framework.txtandrequirements/tests.txthashes, reducing redundant pip downloads across runs-i https://mirrors.aliyun.com/pypi/simple/flags from allpip installcommands so they use the internal cache server already configured in theConfig mirrorsstep, rather than overriding itMotivation
The
Config mirrorsstep incitest_npu.yamlcorrectly sets the internal K8s pip cache service viapip config set global.index-url, butci_container_test.shwas overriding this with explicit-iflags pointing to the public Aliyun mirror. This PR fixes the inconsistency and adds standard pip/pre-commit caching to speed up CI.Test plan
ci_container_test.shuses the configured index URL