fix:vision_api.template.py cannot handle <thinking> and Add url support#420
Open
tqjason wants to merge 2 commits into
Open
fix:vision_api.template.py cannot handle <thinking> and Add url support#420tqjason wants to merge 2 commits into
tqjason wants to merge 2 commits into
Conversation
b8c8813 to
e34079a
Compare
Contributor
Author
|
图片理解除了能够处理本地图片文件以外,增加了支持网络 URL。这个场景是用浏览器上网查找资料,然后发现图片需要理解。这个时候呢,直接把图片的链接发给 ask_vision 函数就可以了,不用下载到本地再提交。很多多模态模型是支持提交 URL 的。 同时呢,我还写了一个 简单的 SOP,用来指导怎么用图像识别模型?但考虑不是所有人都需要,我没有提交上来。需要的自己创建这个 SOP 吧。
# Image API SOP 图片理解(image understanding) API 用法,网络URL可直传,不用下载到本地 库 memory/vision_api.py ## 快速用法 ```python from vision_api import ask_vision result = ask_vision(image) ``` ## ask_vision 函数签名 ``` def ask_vision(image_input, prompt="详细描述这张图片的内容", backend="claude", timeout=60, max_pixels=1_440_000): # image_input: URL(str) 或 本地文件路径(str/Path) 或 PIL Image # backend: 'claude'(默认) | 'openai' | 'modelscope' # 返回 str:成功为模型回复,失败为 'Error: ...' ``` 相应的在 L1 记忆里边,需要加一条索引。
# [Global Memory Insight] ... L3: memory_cleanup_sop | subagent | plan_sop | tmwebdriver_sop | autonomous_operation_sop | scheduled_task_sop | image_sop ... ... Image understanding: image_sop ... ... [RULES] ... |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
vision_api.py 调用的时候,如果模型返回的有 thinking 块,就不能处理,修复了这个问题。同时,现在逻辑应该处理不了流式输出,所以强制 stream=False