Skip to content

feat: [CI] Introduce uv to accelerate pip install #9169

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Zherphy wants to merge 1 commit into
base: main
Choose a base branch
from
+32 −11
Open

feat: [CI] Introduce uv to accelerate pip install #9169

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 9 additions & 9 deletions .dev_scripts/ci_container_test.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
if [ "$MODELSCOPE_SDK_DEBUG" == "True" ]; then
# pip config set global.index-url https://pypi.tuna.tsinghua.edu.cn/simple
pip install -r requirements/tests.txt -i https://mirrors.aliyun.com/pypi/simple/
pip install uv
Copy link
Copy Markdown
Contributor

@gemini-code-assist gemini-code-assist Bot Apr 21, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The pip install uv command does not use the mirror index that was previously configured for package installations. In environments where the default PyPI index is slow or unreachable (as suggested by the previous use of the Aliyun mirror), this bootstrap step may fail or be significantly slower. Consider using the mirror for this step as well.

Suggested change
pip install uv
pip install uv ${UV_INDEX_URL:+-i $UV_INDEX_URL}

uv pip install -r requirements/tests.txt
Copy link
Copy Markdown
Contributor

@gemini-code-assist gemini-code-assist Bot Apr 21, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The migration to uv appears incomplete. While this shell script now uses uv pip install, the test runner tests/run.py (called on line 45) still contains functions like install_packages and install_requirements that use pip. Additionally, the pip uninstall command on line 25 was not migrated. For consistent performance gains and to avoid mixing package managers, all package operations should be moved to uv.

git config --global --add safe.directory /ms-swift
git config --global user.email tmp
git config --global user.name tmp.com
Expand All @@ -20,17 +20,17 @@ if [ "$MODELSCOPE_SDK_DEBUG" == "True" ]; then
fi
fi

pip install -r requirements/framework.txt -U -i https://mirrors.aliyun.com/pypi/simple/
pip install decord einops -U -i https://mirrors.aliyun.com/pypi/simple/
uv pip install -r requirements/framework.txt -U
uv pip install decord einops -U
pip uninstall autoawq -y
pip install optimum
pip install diffusers
pip install "transformers<5.0"
uv pip install optimum
uv pip install diffusers
uv pip install "transformers<5.0"
# pip install autoawq -U --no-deps

# test with install
pip install .
pip install auto_gptq bitsandbytes deepspeed -U -i https://mirrors.aliyun.com/pypi/simple/
uv pip install .
uv pip install auto_gptq bitsandbytes deepspeed -U
else
echo "Running case in release image, run case directly!"
fi
Expand Down
10 changes: 10 additions & 0 deletions .dev_scripts/dockerci.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,11 @@ do
-e MODEL_TAG_URL=$MODEL_TAG_URL \
-e MODELSCOPE_API_TOKEN=$MODELSCOPE_API_TOKEN \
-e PR_CHANGED_FILES=$PR_CHANGED_FILES \
-e UV_INDEX_URL=$UV_INDEX_URL \
-e UV_EXTRA_INDEX_URL=$UV_EXTRA_INDEX_URL \
-e UV_INDEX_STRATEGY=$UV_INDEX_STRATEGY \
-e UV_NO_CACHE=$UV_NO_CACHE \
-e UV_SYSTEM_PYTHON=$UV_SYSTEM_PYTHON \
Copy link
Copy Markdown
Contributor

@gemini-code-assist gemini-code-assist Bot Apr 21, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

By default, uv prevents installation into the system Python environment unless the UV_SYSTEM_PYTHON environment variable is set to 1 (or the --system flag is used). If this variable is not set in the host environment, the uv pip install commands inside the container will fail. It is safer to provide a default value of 1 here to ensure the CI works correctly in containerized environments.

Suggested change
-e UV_SYSTEM_PYTHON=$UV_SYSTEM_PYTHON \
-e UV_SYSTEM_PYTHON=${UV_SYSTEM_PYTHON:-1} \

--workdir=$CODE_DIR_IN_CONTAINER \
${IMAGE_NAME}:${IMAGE_VERSION} \
$CI_COMMAND
Expand All @@ -73,6 +78,11 @@ do
-e MODEL_TAG_URL=$MODEL_TAG_URL \
-e MODELSCOPE_API_TOKEN=$MODELSCOPE_API_TOKEN \
-e PR_CHANGED_FILES=$PR_CHANGED_FILES \
-e UV_INDEX_URL=$UV_INDEX_URL \
-e UV_EXTRA_INDEX_URL=$UV_EXTRA_INDEX_URL \
-e UV_INDEX_STRATEGY=$UV_INDEX_STRATEGY \
-e UV_NO_CACHE=$UV_NO_CACHE \
-e UV_SYSTEM_PYTHON=$UV_SYSTEM_PYTHON \
Copy link
Copy Markdown
Contributor

@gemini-code-assist gemini-code-assist Bot Apr 21, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

Similar to the change above, providing a default value for UV_SYSTEM_PYTHON ensures that uv can install packages into the system Python environment within the container without requiring explicit configuration in the host environment.

Suggested change
-e UV_SYSTEM_PYTHON=$UV_SYSTEM_PYTHON \
-e UV_SYSTEM_PYTHON=${UV_SYSTEM_PYTHON:-1} \

--workdir=$CODE_DIR_IN_CONTAINER \
${IMAGE_NAME}:${IMAGE_VERSION} \
$CI_COMMAND
Expand Down
6 changes: 6 additions & 0 deletions .github/workflows/citest.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -71,6 +71,12 @@ jobs:
run: git lfs checkout
- name: Run unittest
shell: bash
env:
UV_INDEX_URL: "http://cache-service.nginx-pypi-cache.svc.cluster.local/pypi/simple"
UV_EXTRA_INDEX_URL: "https://mirrors.huaweicloud.com/ascend/repos/pypi"
UV_INDEX_STRATEGY: "unsafe-best-match"
UV_NO_CACHE: 1
UV_SYSTEM_PYTHON: 1
run: |
set -e
source /mnt/modelscope/ci_env.sh
Expand Down
9 changes: 7 additions & 2 deletions .github/workflows/citest_npu.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,12 +43,17 @@ jobs:
timeout-minutes: 240
container:
image: 'ascendai/cann:8.3.rc2-910b-ubuntu22.04-py3.11'
env:
UV_INDEX_URL: "http://cache-service.nginx-pypi-cache.svc.cluster.local/pypi/simple"
UV_EXTRA_INDEX_URL: "https://mirrors.huaweicloud.com/ascend/repos/pypi"
UV_INDEX_STRATEGY: "unsafe-best-match"
UV_NO_CACHE: 1
UV_SYSTEM_PYTHON: 1
steps:
- name: Config mirrors
run: |
sed -Ei 's@(ports|archive).ubuntu.com@cache-service.nginx-pypi-cache.svc.cluster.local:8081@g' /etc/apt/sources.list
pip config set global.index-url http://cache-service.nginx-pypi-cache.svc.cluster.local/pypi/simple
pip config set global.trusted-host cache-service.nginx-pypi-cache.svc.cluster.local
pip install uv

- name: Checkout
uses: actions/checkout@v3
Expand Down