diff --git a/.gitignore b/.gitignore
index 3bb2bad..08c0ea6 100644
--- a/.gitignore
+++ b/.gitignore
@@ -9,5 +9,7 @@ history.txt
aios_shell_history.txt
math_school_env.db
workflows.db
+venv-linux
+venv_test
-
+rootfs/test_doc/
diff --git a/.pylintrc b/.pylintrc
new file mode 100644
index 0000000..a7e73ff
--- /dev/null
+++ b/.pylintrc
@@ -0,0 +1 @@
+disable=E0402
\ No newline at end of file
diff --git a/Dockerfile b/Dockerfile
index 2045f08..4d1f1b9 100644
--- a/Dockerfile
+++ b/Dockerfile
@@ -1,4 +1,14 @@
FROM python:3.11
+
+ENV PYTHON_VERSION=3.11
+RUN apt-get update && apt-get install -y --no-install-recommends \
+ python${PYTHON_VERSION}-venv \
+ python${PYTHON_VERSION}-dev \
+ default-libmysqlclient-dev \
+ build-essential \
+ pkg-config \
+ && rm -rf /var/lib/apt/lists/* \
+
WORKDIR /opt/aios
COPY ./src /opt/aios
COPY ./rootfs /opt/aios/app
@@ -12,4 +22,4 @@ RUN pip install --no-cache-dir -r /opt/aios/requirements.txt
ENV PYTHONDONTWRITEBYTECODE 1
ENV PYTHONUNBUFFERED 1
-CMD ["python3","./service/aios_shell/aios_shell.py"]
\ No newline at end of file
+CMD ["python3","./service/aios_shell/aios_shell.py"]
diff --git a/doc/Architecture.md b/doc/Architecture.md
new file mode 100644
index 0000000..53e52c2
--- /dev/null
+++ b/doc/Architecture.md
@@ -0,0 +1,160 @@
+## LLM / AI 相关框架
+### LLM Process
+
+LLM调用封装的最小单元，提供了一系列最基础的支持
+
+流程上 inpurt, prepare promot, llm_function_call_loop, post_llm , llmresult parser, AI Action
+功能上 动态类型系统 load_from_config,llm_process_loader
+
+### Agent
+从Agent的视角定义了Agent的LLM行为逻辑
+Process behavior (响应)
+Task/Todo Loop (自主)
+Self Loop (自省)
+
+
+#### Agent.Memory
+Memory模块设计的主要目的是能按一定的模式，在Token Limit的情况下构成Agent的一些上下文。
+Memory的原始记录写入：原则上说，Agent的任何LLM行为都应至少将input/resp 写入Memory,毕竟LLM是非常高开销的行为。LLM的过程可以视情况写入（inner function的调用，action的执行，根据input构造的完整提示词等）
+Memory的使用：LLM过程在构造提示词的“已知信息”部分时，通常都会有加载Memory的需求。尤其是在Input并不包含完整信息的情况下（非幂等LLM推理），更需要根据“上下文”来理解Input的含义。这在“ChatCompetition”过程中尤为明显。
+
+使用Memory基本是两种方式
+1. 加载写入的原始记录
+2. 访问根据原始记录加工（Self-Thinking）后的Object-Summary。或则访问一个根据原始记录整理的，用文件系统方式组织的“记忆片段”
+
+这里的核心痛点是：一个LLM过程，如何根据Input加载合适的Memory成为“已知信息”。
+1. 如果明确的知道Input属于一个session，那么可以加载这个session相关的所有record(注意token limit和最大条数)与session的summary，其它的Memory的信息通过inner_function访问。但要防止LLM过程中对Inner function的过度使用
+2. 在不明确的情况下，如何判断input属于哪个session?（包括是否需要创建新的session）
+
+从上述思考中，得到现在的设计方案：
+1. 依旧保留Session，且创建session是明确的用户行为。有一些tunnel根本没有创建session的能力。UI可以用 Lite-LLM来进行辅助的session合并（比如类似GMail的Email归集）。系统可以基于session做“已知信息”的自动加载
+2. 在处理Input时，允许使用Agent.Memory的接口来访问更多的Memory的内容。从流程上看，这个过程和访问KB的原理是基本一致的。
+3. Self-Thinking的过程中，既要对Session进行整理（得到Session Summary）,也要站在更全局的角度对涉及到的Object进行整理（得到Object Summary）。
+4. Self-Thinking的过程也是以session为单位的,以更新session-summary为首要目标，并可以在Thiking的过程中，访问已有的object-summary，选择性的更新object-summary
+5. Self-Thinking会尽量以时间从新到旧处理所有的原始记录，因此会涉及到对多个Session的Summary的更新。
+
+设计方案的主要风险在于inner function模式可能会带来大量的，无用的object summary查询。
+
+对于UI上不方便创建Session的情况：
+1. 通过标题尝试自动创建
+2. 通过时间尝试自动创建
+3. 既然用户看到的就是一个session，那么我们就必须当一个session来处理
+
+
+一些推论：
+Agent通过一个DB list来访问/写入结构化数据，并拥有自己创建DB的能力
+Agent通过一个FileSystem来访问/创建非结构化数据，并拥有理解文件系统组织设计的能力
+“不要给Agent直接扩展能力，而是尽量给Agent扩展元能力（读说明书的能力）”
+
+#### Agent.Workspace
+
+#### Agent.behavior
+
+### Workflow
+一组Agent共享Work space后的流程
+Task可以分配给不同的Agent
+Todo的Do和Check可以分配给不同的Agent
+
+## Knowledge Base （sisi)
+AI First的未来文件系统
+
+## Agent 能力扩展框架
+
+### AI Function / Action
+最重要的扩展框架
+
+### Environment
+可以通过 {environment.xxx} 读取
+
+### Code Interpreter
+Agent 能不能写代码是一个重要的理念之争
+能写代码的Agent想象空间大，是通往AGI的必然之路，但不够稳定可预期
+不能写代码的Agent可以专注于组合使用基础的能力，稳定可靠的
+
+## AI系统组件
+### AI Compute Kernel
+通过AI Compute Kernel对 LLM, AIGC等新一代的AI基础能力进行抽象
+通过Compute Node可以对这些基础能力进行不同的实现
+
+### AI Models
+模型的fine-tune Pipeline
+LoRA的Pipeline
+
+### Contact Manage
+
+基于Contact的自然语言权限控制
+
+### Tunnel
+可以使用开放API的通信软件，于自己的AI时刻保持沟通
+
+### Spider
+持续的导入用户在旧时代的数据。
+从Web2->web3
+
+### Calendar (Calendar是否应该是Agent.Worksapce的一部分）
+
+
+### 基础的pkg_loader
+支持一系列可安装的扩展
+可扩展的扩展是AIOS的开发者需要重点关注的
+
+Agent （用自然语言扩展）
+Workflow （用自然语言扩展）
+Plugin:(需要会写代码）
+ AI Function / Action
+ Environment
+ Knowledge Pipeline
+ LLM Process
+ Compute Node
+
+### System Config Manage
+
+Zone Config-> System Config
+
+## UI
+
+### Installer
+图形化的安装界面，帮助用户能快速的安装使用
+我们也会在这里讨论面向用户的AIOS的过渡性安装逻辑
+
+### WebUI & OS Desktop
+系统控制面板
+Agent/Workflow管理
+新Outlook
+ 会话管理 （于Agent会话）
+ 日程管理
+ Todo管理
+
+新Dropbox
+ Knowledge Base浏览
+ Knowledge Base查询
+
+应用商店
+
+### AIOS Shell
+
+### Personal Station (新个人主页）
+内容的发布/联系人内容的查看/个人日历的公开
+
+## Frame Service （完全未开始）
+通过Frame Service，让AIOS成为一个典型的网络系统（Personal Server OS）
+这一块会复用很多CYFS/Bucky OS 的基础设计
+这一层AI不会直接使用，这一层支持AI系统组件的实现
+在用户看来，这一层的功能都是高级的，偏向系统维护的。很少会直接使用
+
+### zone & node-daemon
+NOS的booter
+
+### Runtime (Container) Manage
+这里抽象了系统的运行时模型
+通过容器技术对可扩展组件的权限进行控制，保护系统的隐私安全
+
+### d-Storage & Named Object
+Named- Object File System
+D-RDB
+D-VDB
+
+### BUS
+系统消息总线，在不同的系统组件中路由消息
+
+### CYFS (httpv4) Gateway
diff --git a/doc/LLMProcess.md b/doc/LLMProcess.md
new file mode 100644
index 0000000..7ce919b
--- /dev/null
+++ b/doc/LLMProcess.md
@@ -0,0 +1,18 @@
+# LLMProcess
+
+设计目的是理解到 提示词=>LLM=>LLM Result 的过程是系统的核心复杂度。Agent的粒度太大了，需要更合适的设计来封装这个复杂度，并给予这个过程更大的灵活性和可组合型。并更易于构建测试
+
+比如
+1. 可以很容易的组合两个已知的LLM Process（上一个的输出是下一个的输入），这个设计有一点类似LangChain (我们在正式系统中，肯定允许整个Agent都用LangChain来构建)
+2. 可以用用一个LLM Process来构建另一个LLM Process的Prompt
+3. 继承一个复杂的LLM Process，进行简单配置，就可以得到一个新的LLM Process。这个新的LLM Process可以享受到复杂LLM Process持续迭代的好处
+4. 有一些常用的，系统内置的LLM Process可以从配置文件中加载。
+
+```python
+def agent.on_process_message():
+ llm_process = self.on_message_llm_process.clone()
+ llm_result = llm_process.do()
+
+
+
+```
\ No newline at end of file
diff --git a/doc/QuickStart zh-CN.md b/doc/QuickStart zh-CN.md
index 326229e..ceddc63 100644
--- a/doc/QuickStart zh-CN.md
+++ b/doc/QuickStart zh-CN.md
@@ -1,4 +1,5 @@
# OpenDAN Quick Start
+
OpenDAN (Open and Do Anything Now with AI) is revolutionizing the AI landscape with its Personal AI Operating System. Designed for seamless integration of diverse AI modules, it ensures unmatched interoperability. OpenDAN empowers users to craft powerful AI agents—from butlers and assistants to personal tutors and digital companions—all while retaining control. These agents can team up to tackle complex challenges, integrate with existing services, and command smart(IoT) devices.
With OpenDAN, we're putting AI in your hands, making life simpler and smarter.
diff --git a/doc/agent & workflow.drawio b/doc/agent & workflow.drawio
new file mode 100644
index 0000000..938a34a
--- /dev/null
+++ b/doc/agent & workflow.drawio
@@ -0,0 +1,989 @@
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
diff --git a/doc/function_list.md b/doc/function_list.md
new file mode 100644
index 0000000..15f7466
--- /dev/null
+++ b/doc/function_list.md
@@ -0,0 +1,273 @@
+# AI Function list (Local)
+
+根据新的Function/Action 定义，我们需要记录系统目前提供的所有注册到GlobaToolsLibrary里的function,方便配置组合
+功能扩展只需要扩展function就好了，Action可以通过Function直接得到。
+
+本文档提到的AI Function一般都是Local Function, 基于这个架构，我们可以用Web3的思路整合AI Service。
+
+## Context Prepare
+很多AIFUnction的正确调用都需要在参数里提供一个状态上下文。这个上下文的准备一般由LLProcess完成.而可以通过提示词实时改变的参数是AIFunction的真参数。目前需要做Context Prepare的主要和Agent相关的基础函数。
+
+
+## Agent Core
+
+Agent Core中提供的，与Agent/Workflow 状态关联的基础函数，是Agent的核心设计。对Agent来说，这些基础能力通常都是打开的，我们需要非常谨慎的思考哪些功能应该放到Agent Core中。
+
+根据我们的架构设计，Agent Core的函数包含 Agent的Memory + TODO能力（Plan）。通过这些基础函数，支持了Agent的
+
+- Process Message
+- Planing: 让Agent可以基于Zone的权限Process Task/Todo
+- Self-Think: Process的经验进行总结
+- Self-Improve: 对Agent的提示词进行持续的自我改进。
+
+
+### logs
+
+logs通常代表Agent的短期记忆，目前有两种Log:chatlog和worklog
+目前我们并不鼓励Agent提供短期记忆的调整能力，都是通过标准流程得到一定实践范围的的，完整的短期记忆。
+
+set_message //增加tag
+get_chatlogs
+
+get_worklogs
+set_worklogs //增加tag
+
+### Memory
+
+Memory 代表Agent的长期记忆
+
+通过LLM驱动的Self-Think流程来更新。基本上是在根据短期记忆提炼对人、对事的看法和终结。也包含一些基于提示词工程的自我能力提升（比如复用已知的，好用的工具，或则复用自己为了解决某个特定问题已经制作并使用成功的工具）
+
+get_contact_summary
+update_contact_summary
+get_sth_summary
+update_sth_summary
+
+
+### Workspace (TODOList)
+
+Workspace为Agent提供了可以完成TODO的工具和保存工作结果的状态空间(FileSystem)。每个Agent默认有自己的private workspace，Workflow为Agent提供了可以共享的workspace，Workspace尽量基于文件系统构造，也方便Agent与人类协同工作。
+
+#### Task/Todo 管理
+context中需要一个隐藏的_workspace
+```
+_self = parameters.get("_workspace")
+```
+
+##### agent.workspace.list_task
+
+##### agent.workspace.create_task
+
+```json
+{
+ "title": {"type": "string", "description": "The title of the task."},
+ "detail": {"type": "string", "description": "The detail of the task."},
+ "tags": {"type": "string array", "description": "The tags of the task."},
+ "due_date": {"type": "string", "description": "The due date of the task."},
+ "parent_id": {"type": "string", "description": "The parent id of the task."},
+}
+```
+
+##### agent.workspace.cancel_task
+
+```json
+{
+ "task_id": {"type": "string", "description": "The id of the task to cancel."},
+}
+```
+
+##### agent.workspace.update_task
+
+##### agent.workspace.get_sub_tasks
+
+##### agent.workspace.create_todos
+
+##### agent.workspace.list_todos
+
+##### agent.workspace.get_todo
+
+##### agent.workspace.update_todo
+
+#### 核心的完成TODO的能力
+
+##### Code Interprete
+
+##### Send Msg (系统原生能力)
+
+##### Workspace File System
+
+agent.workspace.write_file
+agent.workspace.read_file
+agent.workspace.delte_file
+agent.workspace.append_file
+agent.workspace.list_file
+
+## Knowledge Base
+
+Knowledge Base对大部分Agent来说，是一个获得私有信息,并让LLM处理结果更好的基础设施(RAG支持）。少部分Agent会使用相关API,结合Knowledge Base所服务的目标来整理Knowledge Base.
+
+### 搜索
+#### 矢量搜索
+#### 传统的文本搜索搜索
+#### 根据已经存在的数据库描述，构造SQL搜索
+
+### 当成文件系统浏览
+
+
+## AIGC
+
+一系列AIGC函数，是LLM打通AIGC能力，形成新生产力的基础。
+
+### aigc.text_2_image
+
+文生图.返回的是生成图片的路径。
+
+```json
+{
+ "prompt": "Description of the content of the painting",
+}
+```
+
+### aigc.image_2_text
+图生文，返回的是图片的描述
+(TODO:是否需要有一个提示词来要求针对特定问题对图片进行描述)
+
+```json
+{
+ "image_path": {"type": "string", "description": "image file path"}
+}
+```
+
+### aigc.voice_to_text
+
+```json
+{
+ "audio_file": {"type": "string", "description": "Audio file path"},
+ "model": {"type": "string", "description": "Recognition model", "enum": ["openai-whisper"]},
+ "prompt": {"type": "string", "description": "Prompt statement, can be None"},
+ "response_format": {"type": "string", "description": "Return format", "enum": ["text", "json", "srt", "verbose_json", "vtt"]},
+}
+```
+
+## system
+
+访问当前系统的基础设施
+
+### system.now
+
+返回当前时间
+
+### system.calender
+
+保存在当前Zone上的日历，默认是与Zone Owner相关的。也可以以自动形式同步别人的日历
+这个组件对AI成为个人助理非常重要，当与旧世界互通时，其细节的复杂度也是非常高的。
+这是一个典型的看起来容易做起来难度很大的基础组件，是思考和验证LLM对传统软件复杂度进行降维的一个关键实践。
+
+#### system.calender.get_events
+#### system.calender.add_event
+#### system.calender.delete_event
+#### system.calender.update_event
+
+### system.contacts
+
+访问用户的联系人列表。在OOD System 中，联系人列表是非常重要的系统基础设施，为一系列的权限控制提供了基础信息。
+
+#### system.contacts.get
+
+通过contanct的名字得到contact的完整信息（json格式）
+
+```json
+{"name":"name"}
+```
+
+#### system.contacts.set
+
+设置 contact 信息，注意这里使用了一个可扩展结构，我们可能需要定义一些标准的必填信息。
+
+```json
+{"name":"name","contact_info":"A json to descrpit contact"}
+```
+
+### System.shell
+
+#### system.shell.exec
+
+执行Shell命令（目前只支持Linux Bash）
+
+
+## web
+
+访问web的函数。使用下列函数要确保Agent有访问互联网的权限。
+
+### web.search.duckduckgo
+
+使用搜索引擎搜索互联网
+
+```json
+{
+ "query": {"type": "string", "description": "The query to search for."}
+}
+```
+
+## 常见的llm_context(能力分组)
+为典型的LLM处理过程，进行了分组。主要是为了节约Token。
+使用Action比使用inner function更节约。
+
+### Process消息组 （定制度高）
+得到潜在的Task并创建，在这个过程中可能需要查询已有的任务（防止重复创建）
+action:craete_task, function:list_task
+
+通常Agent在Process Message时，会表现出和其处理TODO接近的能力，核心的区别在于Process Message是立刻处理并给出结果，而变成Task更多的是当成一个异步的任务
+为了能处理回复，查询历史沟通记录（寻找记忆的过程）
+为了能处理回复，查询KB的过程
+
+REMARK：目前LLM的主要问题是，如果开放的function, LLM会倾向于优先使用，是否可以做成“如果用户对答案不满意”，再使用？
+
+
+
+### Review Task
+对Task的首次执行，Review的目的
+- 拆分创建TODO（使用create_todos）
+- 对简单的任务立刻执行并记录更新结果 （通常是）
+- 认为超出自己的能力范围，标记为无法处理或转交给合适的人（Agent） (使用update_task)
+- 对已有任务进行查询（list_task,query_task）
+
+### Do TODO(定制度高)
+DO行为是复杂的，我们会精细的区分TODO的首次执行和失败后再次执行。失败后再次执行会得到之前的记录摘要，并有查询之前工作日志的能力。
+
+Do TODO是Agent的另一个核心行为，这里会根据Agent的设定，集成更多的能力
+系统为Do提供的默认支持:（按难度逐步增加）
+ - 写文档 : 不需要任何外部支持
+ - 运行AIGC ： AIGC的函数组
+ - 收集,整理信息（通过互联网或查询知识库）： web.search.duckduckgo
+ - 发送消息,系统自带，但可能需要依赖一些通讯录浏览/查找函数
+ - 执行自己的代码/编写代码并执行 ： system.shell.exec，code_interprete （这是一个重点模块！）
+ - 运行网络服务 ：智能合约的有通用套路，非智能合约的需要有一个完整的SOP来支持
+
+
+根据任务要求保存工作成功是手动的，这里有一组workspace级别的文件系统API。
+保存工作记录的行为是自动的，默认所有的Action都执行成就算是DoComplte，会自动的更新状态
+有的Do可能需要自我迭代一下，这和大多数Behavor只有一次LLM调用有所不同。
+
+
+### Check TODO/Task
+为了解决LLM不可避免的幻视加入的Check流程。该流程会根据TODO的目标，对TODO的结果进行判定
+使用 update_todo 更新TODO的状态
+当所有的sub_todos都完成后，会check task的目标是否达到
+当所有的sub_task都完成后，会check task的目标是否达到
+
+因此，提供的函数主要是得到 todo/task的更深入细节的函数（访问相关log）,已经读取相关 工程成果文件 的函数
+
+### Self-Think
+获得logs 和summary
+进行update
+
+
+
+
+### Learning
+得到logs和summary
+浏览和整理KB
+
+
+### Self-Improve
\ No newline at end of file
diff --git a/doc/knowledge_pipeline.md b/doc/knowledge_pipeline.md
new file mode 100644
index 0000000..fedb5aa
--- /dev/null
+++ b/doc/knowledge_pipeline.md
@@ -0,0 +1,87 @@
+# 配置knowledge pipeline
+knowledge pipeline 扫描指定的输入，把输入的内容构建为结构化的knowledge object，之后依照使用knowledge的应用场景，为object创建各种各样的索引。
+## Input
+输入定义从个人数据来源转换成结构化的knowledge object的过程，并且定义在object上调用parser的粒度。比如典型的几种Input的实现：
++ 本地目录：指定本地目录，扫描本地目录的所有文件，并且监听他的更新；对每个一个文件生成object并且写入object store；对每一个新产生的object调用parser；
++ 个人邮箱：扫描个人邮箱收件箱，并且监听新的邮件；对每一封邮件生成email object并且写入object store；对每一个新产生的email object调用parser；
++ 浏览器上下文：实现浏览器插件，对当前浏览的页面元素通过rpc传入对应的input 后端实现，生成rich text object；对每一个新产生的rich text object调用parser；
+
+## Parser
+Parser定义从input 输入的object 创建索引的过程；包括但不限于以下主要手段，以及他们的组合：
++ 向量化之后写入vector store
++ 创建各种维度的RDB，NoSQL索引
++ 向Agent send object
+
+配置Pipeline 应当包含以下几个部分：
++ Input method：包含实现input的 python module
++ Input params：inpu module的参数，比如本地路径，邮箱地址
++ Parser method：包含实现parser的 python module；如果Parser是指向Agent，这个配置是可以简化成Agent instance name；
+
+# Knowledge pipeline manager
+pipeline 管理会类似agent manager，manager管理pipeline config，从config 创建instance在后台持续运行， knowledge pipeline manager 也需要处理pipeline instance的状态管理.
+
+集成到aios shell中，加入如下命令：
++ knowledge pipelines： 返回当前运行中的pipeline实例
++ knowledge journal $pipeline [$topn]： 查询当前pipeline运行的journal日志
++ knowledge query $object_id: 查询指定knowledge object的内容
+
+# 在aios shell中添加新的knowledge pipeline
+在$home/myai/knowledge_pipelines/, 或者开发模式下在 $source_root/rootfs/knowledge_pipelines/ 目录中，添加新的pipeline 目录, 以下以内建的pipeline Mia为例说明：
+
+## pipeline.toml
+创建pipeline.toml配置文件
++ name字段指定全局唯一的pipeline name
++ input.module字段指向相对pipeline目录的input实现
++ input.params字段定义input的输入参数，不同的input实现可以有不同的参数格式
++ parser 部分也是类似
+``` toml
+name = "Mia"
+input.module = "input.py"
+input.params.path = "${myai_dir}/data"
+parser.module = "parser.py"
+parser.params.path = "${myai_dir}/knowledge/indices/embedding"
+```
+
+## input
+input模块至少应当实现：
+```python
+async def next(self):
+```
+定义input class，实现异步迭代生成器方法next，扫描输入，对其中的每一个元素生成结构化的knowledge object；
++ 如果input中的所有元素都扫描完成了，返回None, pipeline会被标记为finish
++ 如果input可pending，等待新的输入，返回（None, None）
++ 如果要把创建的object传递到parser，返回（object_id, journal_str）,其中journal_str是产生的journal 日志中的input 部分；
+Mia中的实现就是扫描目录中的文件，对文本和图片创建object；
+```python
+def init(env: KnowledgePipelineEnvironment, params: dict)
+```
+创建input class的实例并返回
+
+## parser
+parser模块至少应当实现
+```python
+async def parse(self, object: ObjectID) -> str:
+```
+定义parser class，实现parse成员方法,对input中返回的object_id创建索引，返回journal_str.
+Mia中的实现就是对输入的object内容embedding，并且保存到chromadb中；
+```python
+def init(env: KnowledgePipelineEnvironment, params: dict)
+```
+创建parser class的实例并返回
+
+# 使用pipeline创建的索引
+pipeline定义了创建knowledge object 和索引的过程，对应的要使用pipeline创建的索引完成工作。
+还是以内建的Mia为例，不止创建名为Mia的pipeline，还在Agent中加入了查询Mia pipeline创建出来的chromadb的 Agent Mia；
+## query.py
+query 模块并不是pipeline的一部分，其逻辑是跟parser是一致的，在query中定义了一个agent可访问的query function，输入prompt，返回chromadb中embedding相近的object id；
+
+## agent.toml
+```toml
+owner_env = "../../knowledge_pipelines/Mia/query.py"
+```
+在Mia的agent template配置里，引用query模块创建的query function；并且编辑好让Mia推理调用query方法的提示词。
+
+
+
+
+
diff --git a/doc/learn_knowledge.md b/doc/learn_knowledge.md
new file mode 100644
index 0000000..f0038b2
--- /dev/null
+++ b/doc/learn_knowledge.md
@@ -0,0 +1,30 @@
+# learn todo list
+在workspace中分离出独立的两个work list；用于处理工作的work todo list， 和用于更新knowledge的 learn todo list；两种todo list在agent templete中单独的配置prompt，每个todo list都可以配置 do , check 和 review 三个prompt。
+learn todo list中的todo，可以在knowledge pipeline 生成。
+一个典型的例子是新的JarvisPlus agent，它可以依照自己的理解将某个本地目录中的文档归类到不同的逻辑目录中：
++ 首先定义他的pipeline：扫描目录中没有读过的文档，传递给parser；
++ 定义他的pipeline parser的实现：向自己所处的workspace 的learn todo list中添加一条todo， todo的内容是文档的全文；
++ 定义他的 learn prompts：根据输入的文档内容，输出摘要，和归类后的目录，产生一组归类operation；
++ workspace中嵌入了支持归类operation 的knowledage base environment，可以执行learn todo list产生的归类operation；
+
+# environments and workspace
+agent在独立的workspace中工作，目前默认的workspace是agent自己；除了自己的workspace，agent也可以进入其他的workspace；environment表示agent可以调用的function，和可以产生的operation；
+environment中的function或者operation输出的结果，应当应用在agent所处的workspace中，所以agent在不同workspace中执行work，结果应当是被隔离的。
+
+
+
+
+# Learn Todo List
+Separate two independent work lists in the workspace: a work todo list for handling work, and a learn todo list for updating knowledge. Both types of todo lists have their own configured prompts in the agent template, and each todo list can configure do, check, and review prompts.
+The todos in the learn todo list can be generated in the knowledge pipeline.
+A typical example is the new JarvisPlus agent, which can categorize documents from a local directory into different logical directories according to its understanding:
++ First, define its pipeline: scan documents in the directory that have not been read and pass them to the parser;
++ Define the implementation of its pipeline parser: add a todo to the learn todo list of the workspace it is in, the content of the todo is the full text of the document;
++ Define its learn prompts: based on the input document content, output a summary and the directory after categorization, generating a set of categorization operations;
++ The workspace embeds a knowledge base environment that supports categorization operations, which can execute the categorization operations generated by the learn todo list.
+
+# Environments and Workspace
+The agent works in an independent workspace, and the current default workspace is the agent itself. In addition to its own workspace, the agent can also enter other workspaces. The environment represents the functions that the agent can call and the operations it can generate.
+The results of the functions or operations in the environment should be applied in the workspace where the agent is located, so the results of the agent's work in different workspaces should be isolated.
+
+Please note that the translation might not be perfect due to the technical nature of the text and potential ambiguity in the original text.
diff --git a/doc/mvp plan 3.md b/doc/mvp plan 3.md
new file mode 100644
index 0000000..39017b0
--- /dev/null
+++ b/doc/mvp plan 3.md
@@ -0,0 +1,193 @@
+# Proposal for Adjusting the Goals for Version 0.5.2
+
+Dear Team,
+
+Given the recent launch of OpenAI's new version in early November 2023, many of us may have felt a profound shift in the industry. As the world changes, I believe we should adapt accordingly. Here are some of my thoughts:
+
+1. **Affirmation of Our Path**: OpenAI's latest release, particularly the functionalities of the so-called GPTs Agent platform, is largely similar to our 0.5.1 version released on September 28th. This strongly affirms the correctness of our direction. OpenAI has done a great job educating the market about the Agent, so we no longer need to emphasize the correct use of LLM based on the Agent through version releases. This part of user education product design can be simplified.
+
+2. **Innovation in Version 0.5.2**: For our new version (0.5.2), besides maintaining the combinational advantages brought by private deployment of LLM, I believe we need to implement some of the innovative ideas we've discussed about the Agent. This is crucial to maintaining our leading position and avoiding the impression that OpenDAN is merely a follower of GPTs.
+
+3. **Integration of OpenAI's New Capabilities**: We should fully integrate the new capabilities brought by OpenAI's latest release, especially the longer Token Windows, GPT-V, and Code-interpreter. I believe these new features can effectively solve some known issues.
+
+Therefore, I propose to adjust the goals and plans for version 0.5.2. Here are the core objectives:
+
+- Aim to release version 0.5.2 by the end of November, focusing on:
+ - Launching a new Agent with Autonomous capabilities and multi-Agent collaboration based on Workspace.
+ - The integrated product of 0.5.2 will be a private deployment email analysis Agent for small and medium-sized enterprises. This will allow any company to better support its CEO and other management positions through LLM while ensuring privacy and security.
+ - (Optional) By combining LLM and AIGC, build an Agent-based personalized AIGC application, such as a "children's audio picture book" generator that includes both text-to-image and text-to-sound.
+ - (Optional) Through multi-Agent collaboration, fully utilize the capabilities of GPT4-Turbo, and attempt to let AI and engineers collaborate on research and development tasks based on Git.
+
+The detailed version plan is as follows:
+
+## MVP plan adjustment
+
+In order to keep the list below too long, the system distributed version is 0.5.3, I think we will open another ISSUE discussion and record, this list does not include.
+
+The modules that are not specially explained are components completed in the 0.5.2 plan
+
+
+- [x] AIOS Kernel
+ - [x] Basic Agent,@waterflier, A2
+ - [x] Python Agent Extend, @wugren, S2
+ - [x] Basic Workflow,@waterflier, A2
+ - [ ] Workflow Refactor,@waterflier, A2
+ - [x] AI Functions,@waterflier,A2
+ - [ ] Upgrade to GPT4 tools API,S2
+ - [x] AI Environments,@waterflier, A2
+ - [x] Celender Environment,@waterflier, S2
+ - [x] Contanct Manage Support,@waterflier, S2
+ - [x] AI Shell Enviroment,@waterflier, S1
+ - [x] Upgrade Agent Working Cycle
+ - [x] Process Message,@waterflier,A2
+ - [x] Process Group Message,@waterflier,A2
+ - [ ] Process Event,(0.5.3)
+ - [ ] Completion of self-drive,@waterflier,A4
+ - [ ] Self-learn,@weaterflier,A2
+ - [ ] Introspection,@waterflier,A2
+ - [ ] Workspace Environment
+ - [ ] Task/TODO Manager,@waterflier, A2
+ - [x] Local File System,@waterflier, S1
+ - [ ] Web Search, A4
+ - [ ] Code Interpeter (The first implementation can be based on Openai), A4
+ - [ ] Query SQL DB, S1
+ - [x] AI BUS,@waterflier, A2
+ - [ ] Agent Message MIME Support (Image,Video,Audio), A3
+ - [x] Connect to Human, @waterflier,A2
+ - [x] Chatsession,@waterflier, S2
+ - [ ] Compress Chatsession By Text Summary, @waterflier, A2
+ - [ ] Knowlege Base,@lurenpluto ,@photosssa
+ - [x] Knowledge Base Frame,@photosssa,A3
+ - [x] Knowledge Base Object Store,@lurenpluto ,A4
+ - [x] Knowledge Base Basic Pipline,@photosssa ,A3
+ - [ ] Customize pipeline,@photosss,A4
+ - [ ] Support Local Text Search,A4
+ - [x] Text Summary Pipline,@waterflier, A2
+ - [ ] Text Parser Support
+ - [x] PDF Parser,@waterflier,S2
+ - [ ] doc Parser,S4
+ - [x] MD Parser,@waterflier,S1
+ - [ ] Source Code Parser,S4
+ - [ ] Image Parser (Base on GPT-V),(S2)
+ - [ ] Video Parser (Base on GPT-V), (S4)
+ - [ ] Personal AIGC Models
+ - [ ] Stable Diffusion Controler Agent (Optional),A6
+- [x] AI Compute System,@waterflier, A2
+ - [x] Scheduler,@streetycat, A2
+ - [x] LLM Kernel
+ - [x] GPT4 (Cloud),@waterflier, S1
+ - [x] LLaMa2,@streetycat, A2
+ - [ ] Claude2, S2
+ - [ ] Falcon2, S2
+ - [ ] MPT-7B, S2
+ - [ ] Vicuna, S2
+ - [x] Embeding, @lurenpluto , A4
+ - [x] Txt2img,@glen0125,A4
+ - [ ] Support DALL-e, @glen0125, S2
+ - [x] Img2txt,based on GPT4-V,@alexsunxl S2
+ - [x] Txt2voice,@wugren A3
+ - [ ] Txt2Voice,base on OpenAI, @wugren A2
+ - [ ] Voice2txt, base on OpenAI, A2
+ - [ ] Language Translate (Pending)
+- [ ] Build-in Service
+ - [x] Spider,@alexsunxl, A2
+ - [x] E-mail Spider,@alexsunxl, S4
+ - [x] Agent Message Tunnel Frame,@waterflier, A2
+ - [x] E-mail Tunnel,@waterflier,A2
+ - [x] Telegram Tunnel,@waterflier,S2
+ - [ ] Discord Tunnel,S2
+ - [ ] Home IoT Environment (0.5.3), A4
+ - [ ] Compatible Home Assistant (0.5.3), A4
+- [ ] Build-in Agents/Apps
+ - [x] Agent Mia: Personal Information Assistant,@photosssa,@lurenpluto , A2
+ - [x] Agent Jarvis: Peersonal Bulter & Assistant ,@waterflier, A2
+ - [ ] A Agent Can Create Other Agent,@wugren, A3
+ - [ ] App: Personal Station (0.5.3),A4+S4
+- [ ] UI
+ - [x] CLI UI (aios_shell),@waterflier,S2
+ - [ ] Web UI ,@alexsunxl,S4
+ - [ ] OpenDAN Desktop Installer,@alexsunxl+@waterflier,S4
+- [x] 0.5.1 Integration Test
+ - [x] Workflow -> AI Agent -> AI Agent,@waterflier,S1
+ - [x] Spider -> Pipline -> Knowledge Base,@photosssa,S2
+ - [x] AI Agent <- Functions <- Knowledge Base,@lurenpluto,S2
+- [x] 0.5.2 Integration Test
+ - [ ] Email Agent/CEO assistant,S4
+ - [ ] My AIGC Assistant (optional), S8
+ - [ ] My Software Company(Advance Workflow demo) (optional), S8
+- [ ] SDK
+ - [x] Workflow SDK,@waterflier, A2
+ - [ ] Agent SDK,@waterflier, A2
+ - [ ] AI Environments SDK (0.5.2), A2
+ - [ ] Compute Kernel SDK (0.5.3), A2
+- [ ] Document (>0.5.2)
+ - [ ] System design document, including the design document of each subsystem
+ - [ ] Installation/use document for end users
+ - [ ] SDK document for developers
+
+## Some Explanation
+### Upgrade Agent Working Cycle
+The goal is to transform the Agent from a passive message-handling Assistant to an actively acting Agent based on roles. The concept of the relevant modules mainly involves the Agent's behavior patterns (4 types), the Agent's capabilities, and the Agent's memory management (learning and introspection).
+
+For a detailed introduction, refer here: https://github.com/fiatrete/OpenDAN-Personal-AI-OS/issues/91
+
+### Workspace Environment
+The Workspace supports the implementation of the Agent Working Cycle design. Its core abstraction is defined as: saving the shared state needed for Agent collaboration and providing the basic capabilities for Agents to complete their work. I carefully referenced AutoGPT in the design. The difference between Workspace and AutoGPT is the emphasis on collaboration (Agent with Agent, Agent with humans). After contemplation, the Workspace primarily consists of the following components:
+
+1. Task/Todo manager, representing the unfinished tasks in the Workspace.
+2. Saving work logs.
+3. Saving learning outcomes and records of known documents.
+4. Ability to access the Knowledge Base (RAG support).
+5. Virtual file system for saving any work outcomes.
+6. A set of SQL-based databases to save any structured data.
+7. Real-time internet search capability.
+8. Ability to use existing internet services.
+9. Ability to use major blockchain systems (Web3).
+10. Ability to write/improve code (based on git), run code, and publish services.
+11. Communication capabilities with the outside world.
+12. Ability to use social networks.
+
+
+Each Agent has its own private Workspace, not shared with others. I hope to achieve diversity through the combination of "Agent and Workflow Role". Each user "trains" different Agents through their usage habits, and then these Agents collaborate to complete complex tasks defined in the Workflow. The final results of these complex tasks can reflect the user's inherent personality and preferences.
+
+This component design also reflects my thoughts on the key question, "What capabilities should we endow an Agent with, and how do we control the security boundaries when it transitions from a consultant to a steward?" It's not a simple question, so I anticipate this component will continue to iterate in the future.
+
+### Agent Message MIME Support
+
+Agent Message MIME Support means that Agents can handle multiple types of messages, including images, videos, audio, files, etc. For most Agents, this requires adding a customizable standard step of parsing messages in the message handling process. The input of this step is the message's MIME type, and the output is the text content of the message. This step can be implemented by calling the text_parser module.
+
+Another core requirement of MIME support is to use a unified method to save these non-text content data.
+
+### Text base Knowledge Base
+
+In 0.5.1, we mainly implemented RAG based on the popular Embedding + vector database solution. Through practice, we found that this solution did not fully utilize the potential of LLM, so I want to introduce two new modes to further enhance RAG:
+
+1. Build a local text search engine that LLM can use for proper local searches when needed.
+2. Assuming LLM will become cheaper in the future, let LLM learn all the documents once and organize the learning results by directory structure (Text Summary). LLM can use browsing methods to find the information it needs.
+
+#### Text Parser Support
+
+Both MIME Support and Text-based Knowledge Base require the system to support converting various document formats into text that can express semantics as much as possible. This component, known as TextParser, should be implemented as an open and extensible framework, given the vast amount of digital content that exists in different formats.
+
+#### Local Text Search
+
+Using traditional inverted index technology to save all document content locally and provide rapid local search capabilities. The implementation of this component can refer to ElasticSearch.
+
+#### Text Summary
+
+Using the capabilities of LLM to learn all the documents and then save the learning results locally. This behavior can be considered "Self-Learn". Users can let Agents responsible for organizing materials use different prompts according to the purpose of organizing the materials to obtain more targeted results.
+
+### Stable Diffusion Controler Agent
+
+Practice the concept of "Agent as a new era method of using computing", replacing the complex Stable Diffusion WebUI with an easy-to-use Agent. Help users complete complex AIGC tasks and build a paradigm. This paradigm can cover the entire process of AIGC: LORA training, use, model downloading, plugin downloading, generation of prompt words, selection of AIGC results.
+
+### Email Agent/CEO assistant
+
+The integrated test product of 0.5.2, aimed at private deployment for small and medium-sized enterprises, is a CEO Assistant that can read all company emails and materials. I am writing a detailed product document, which is not elaborated here.
+
+
+----------------------------------------
+
+
+I look forward to hearing your thoughts on these proposed adjustments.
+
diff --git a/doc/promps/Check TODO.md b/doc/promps/Check TODO.md
new file mode 100644
index 0000000..d164691
--- /dev/null
+++ b/doc/promps/Check TODO.md
@@ -0,0 +1,2 @@
+# Check (TODO)
+目的是根据Todo Log, 结合自己的角色检查TODO是否争取完成（非客观性TODO给出是否有所改进的评价）
diff --git a/doc/promps/Do TODO.md b/doc/promps/Do TODO.md
new file mode 100644
index 0000000..5bb74f3
--- /dev/null
+++ b/doc/promps/Do TODO.md
@@ -0,0 +1,49 @@
+# Do (TODO)
+ 目标是结合 角色定义，手头的工具，已知知识 完成一个确定的任务。
+ 完成任务时应使用ReAct的方法：应在给出执行动作前，先自言自语的输出一个计划，然后在动作（这个自言自语会变成TODO Logs）
+
+ ## 提示词思路
+TODO从Task拆分而来，因此不应该再次拆分。请尽全力完成。如果判断缺乏完成TODO的能力，请标记为取消。如果是缺乏完成任务的前置条件，请标记为执行失败。
+
+执行一个新的TODO：
+```
+YOUR ROLE:
+你是主人的超级个人助理。你的主要工作是安排主人的日程。
+
+PROCESS RULE：
+1. 你的任务是结合自己的角色定义，手头的工具，已知信息、完成一个确定的TODO。完成该TODO后你会得到$200的小费。
+2. 输入的TODO是来自你自己对一个Task的Plan结果。
+3. 完成TODO的过程中你应该先思考再执行。执行的过程中可以使用工具，访问前置步骤的结果。执行的结果通常是按顺序执行的ActionList。
+4. 你必须独立的，一次性完成该TODO，你无法得到来自任何他人的协助。
+5. 对确认超出任务范围的TODO，你可以取消该TODO。对执行任务条件不满足的TODO，你可以标记为失败，但要说明失败原因
+7. TODO的完成结果如有需要应保存成数字文档
+
+CONTEXT：
+ActionList:PostMsg,WriteFile,UpdateFile,RemoveFile,Rename,
+现在时间，主人所在位置，以及天气。主人目前正在做什么。
+
+REPLY FORMAT：
+The Response must be directly parsed by `python json.loads`. Here is an example:
+{
+ think:'我的思考.'
+ tags: ['tag1', 'tag2'], #Optional,If the TODO involves important things and people, you can mark by 1-3 tags.
+ actions: [{
+ name: '$action1_name',
+ $param_name: '$parm' #Optional, fill in only if the action has parameters.
+ }, ...
+ ]
+}
+
+KNOWN_INFO:
+1.TODO所在的Task信息，重点是Task的整个Plan计划
+2.该TODO之前的执行失败记录 （如有）
+
+
+Tools_tips:（重要！）
+inner_function:GetTodoResult, ReadFile
+使用GetAllSupportAction进一步获得所有可用的Action
+
+
+```
+（注意Workspace和AgentMemory都有Worklog,但视角不同。）
+执行一个有失败记录的TODO：
diff --git a/doc/promps/Review Task.md b/doc/promps/Review Task.md
new file mode 100644
index 0000000..28056d6
--- /dev/null
+++ b/doc/promps/Review Task.md
@@ -0,0 +1,123 @@
+# Review Task/Todo
+目的是结合已知信息（重点是已经进行操作的记录），对失败的，完成的不好的任务进行思考，尝试给出更好的解决方案
+1. 管理学方法：更换负责人
+2. 管理学方法：拆分
+3. 给出建议（该建议可以在下次一次DO-Check）循环中被使用
+
+## ReviewTasklist
+目的是选择一个优先级最高的任务开始工作
+（这个流程现在是通过计算的方法，基于优先级排序后，FIFO的处理）
+
+## ReviewTask 对未开始的Task进行首次处理
+LLM 结果动作：
+- 确认执行人，在非Workflow环境中，执行人就是Agent自己，所以不存在这个选项
+- 确认执行时间和过期时间，任务只有在执行时间以后和过期时间以前才有机会执行，无法确认执行时间的可以设置下一次检查时间
+- 对任务进行拆分（如何防止无限拆分是个大问题），或则有一些简单任务不允许拆分。
+- 判断可以立刻执行任务（将任务当成TODO工作），通过Action进入下一个LLMProcess
+- 判断任务超出Agent能力范围，宣告失败
+
+
+ExampleA:（Task不支持分拆，Agent必须通过Task-Todo两级结构完成任务）
+```
+YOUR ROLE:
+你是主人的超级个人助理。你的主要工作是安排主人的日程。
+
+PROCESS RULE：
+你得到的输入来自你自己之前记录在TaskList系统里的一个Task。现在你并不需要完成该Task，而是结合已知信息对Task进行一次Review.Review的过程是你独立完成的，你在形成结论的过程中可以使用工具，但不能和其它人交流。
+1. 理性的思考如何一步一步的高效的，在潜在的截止时间前完成该Task。明确拒绝超出自己能力范围的Task。
+2. 尝试对Task进行确认操作。确认操作的关键在于任务有了明确的执行时间。
+3. 对于需要多个步骤才能完成的Task,对Task进行TODO Plan。尤其注意与相关人员确认的步骤
+4. 对于不需要拆分TODO，且可立刻执行的任务。直接执行该任务。
+
+CONTEXT：
+ActionList:cancel,confirm,execute
+现在时间，主人所在位置，以及天气。主人目前正在做什么？
+
+REPLY FORMAT：
+The Response must be directly parsed by `python json.loads`. Here is an example:
+{
+ think:'$think step-by-step to be sure you have the right answer.'
+ plans:[ #Optional
+ {"todo":"$todo_name","detail":"$todo_detail,"category":"$todo_category"}
+ ...
+ ],
+ tags: ['tag1', 'tag2'], #Optional,If the task involves important things and people, you can mark by 1-3 tags.
+ actions: [{
+ name: '$action_name',
+ $param_name: '$parm' #Optional, fill in only if the action has parameters.
+ }]
+}
+
+
+KNOWN_INFO:
+1.已有Task
+
+Tools_tips:
+2.可以给与Readonly的日历API，进一步查询某个人的已知日程安排）
+
+```
+问题：拆分TODO时是否需要知道有哪些Agent可以用，这样的话在布置任务的时候也会充分考虑其人员能力边界
+
+
+Example OLD:
+```markdown
+I think hard and try my best to complete TODOs. The types of TODO I can handle include:
+- Scheduling, where I will try to contact the relevant personnel of the plan and confirm the details of the schedule with them.
+- Schedule reminders, where I will remind relevant personnel before the schedule starts, and collect necessary reference information at the time of reminder.
+- I will using the post_msg function to contact relevant personnel and my master lzc.
+- Writing documents/letters, using op:'create' to save my work results.
+
+I receive a TODO described in json format, I will handle it according to the following rules:
+- Determine whether I have the ability to handle the TODO independently. If not, I will try to break the TODO down into smaller sub-TODOs, or hand it over to someone more suitable that I know.
+- I will plan the steps to solve the TODO in combination with known information, and break down the generalized TODO into more specific sub-todos. The title of the sub-todo should contain step number like #1, #2
+- Sub-todo must set parent, The maximum depth of sub-todo is 4.
+- A specific sub-todo refers to a task that can be completed in one execution within my ability range.
+- After each execution, I will decide whether to update the status of the TODO. And use op:'update_todo' to update when necessary.
+
+The result of my planned execution must be directly parsed by `python json.loads`. Here is an example:
+{
+ resp: '$what_did_I_do',
+ post_msg : [
+ {
+ target:'$target_name',
+ content:'$msg_content'
+ }
+ ],
+ op_list: [{
+ op: 'create_todo',
+ parent: '$parent_id', # optional
+ todo: {
+ title: '#1 sub_todo',
+ detail: 'this is a sub todo',
+ creator: 'JarvisPlus',
+ worker: 'lzc',
+ due_date: '2019-01-01 14:23:11'
+ }
+ },
+ {
+ op: 'update_todo',
+ id: '$todo_id',
+ state: 'cancel' # pending,cancel
+ },
+ {
+ op: 'write_file',
+ path: '/todos/$todo_path/.result/$doc_name',
+ content:'$doc_content'
+ }
+ ]
+}
+
+```
+
+## PlanTask 对已经确认的Task进行执行
+根据任务的分类，进入不同的LLM Plan逻辑
+ 简单任务：当作TODO立刻执行
+ 普通任务：拆分TODO
+
+
+## QuickCheckTask 对处于半确认状态Task进行Quick Review
+有一些Task是永远不会结束的（比如定时提醒）。此时通过Quick Review来调整这些Task的状态，让其在正确的时间进入Review和DO
+
+
+## RetryTask 对未成功的任务进行再次处理
+注意RetryTask和RetryTodo的区别
diff --git a/doc/promps/Self-improve.md b/doc/promps/Self-improve.md
new file mode 100644
index 0000000..4ca7ee4
--- /dev/null
+++ b/doc/promps/Self-improve.md
@@ -0,0 +1,18 @@
+# Self Improve Prompt
+这是一个改进Prompt的Prompt,其设计目标是利用LLM来改进LLM.(输入是一个LLM Process)
+注意理解Self Improve和Self Thinking的区别: Self Improve有可能改进Agent的某个LLM Process的提示词，而Self Thinkg只会更新Agent的Memory
+提示词：
+ 行为模式：Input形式， Goal（目的）
+ 理想结果：Input， 结果
+ 当前情况：当前Prompt,实际结果
+
+输出：
+ 新的Prompt
+
+## 当前版本
+```
+你是LLM的专家，尤其擅长编写Prompt，你会帮助我改进Prompt。
+我会给你一个已有的Prompt,并说明该Prompt的设计目标，期望的结果和实际的结果。你会step-by-step的进行分析，说明改进思路，并给出改进后的Prompt。
+
+```
+##
\ No newline at end of file
diff --git a/doc/promps/Self-thinking.md b/doc/promps/Self-thinking.md
new file mode 100644
index 0000000..4525ce4
--- /dev/null
+++ b/doc/promps/Self-thinking.md
@@ -0,0 +1,16 @@
+# Self-Thinking (Introspection)
+基于自己的角色定位，结合已有的历史记录（聊天记录，工作记录），进行思考和总结，进而更好的改进未来的工作
+自省从某个角度看，就是对Agent Memory的自我总结
+
+## 当前版本
+```
+You are the best deep thinking in the world, and you will think about the information I give you, sometimes some chat records.Then you will generate a briefing or summary of no more than 400 words based on this information.
+You mainly use the following methods to generate summary:
+1. Try to understand the theme of each sentence, and call the relevant operation to record the relationship between the dialogue and the theme
+2. Try to analyze the personality of different people involved in information
+3. Try to summarize important events in the information and record it
+4. Try to understand the attitude of different people on different topics or events
+5. For the key information or TODO in the information, such as the time, place, amount and other information of the certainty, it must be stored in the summary.
+
+Just give me a summary without any other word.
+```
\ No newline at end of file
diff --git a/doc/promps/process message.md b/doc/promps/process message.md
new file mode 100644
index 0000000..802c757
--- /dev/null
+++ b/doc/promps/process message.md
@@ -0,0 +1,22 @@
+# Process Message
+处理消息的首要是目的是分析消息的意图，并给予正确的回复。
+
+## 提示词的构成
+1. Agent的身份说明，处理信息的基本原则（目的
+2. 处理信息的通用套路。
+ 要产生一个合适的回复。
+ 通过actions来设置话题状态、创建task等
+3. 当前的常规Context，包括现在的时间、对话发生的地点（通常来自AgentMsg里的Context），地点所在的天气等信息
+4. 已知信息（组合而来）
+ 和信息发送者的近期的交流记录
+ Agent和信息发送者近期未完成话题的标题和简介
+ 关于信息发送者，和相关人物的更多资料
+ 查阅更多信息的方法 ： 搜索法和浏览法。注意区分外部资料和内部记忆
+ 其它相关信息（比如RAG根据 输入消息
+
+
+## 当前版本
+
+
+## 想法：LLM生成提示词
+根据 Agent的身份说明，处理信息的基本原则（目的），当前信息，通过另一个LLM来生成提示词里的某些部分的内容
\ No newline at end of file
diff --git a/doc/promps/self-learn.md b/doc/promps/self-learn.md
new file mode 100644
index 0000000..5007259
--- /dev/null
+++ b/doc/promps/self-learn.md
@@ -0,0 +1,2 @@
+self learn通常是根据既定目标，对KB进行学习，或则主动扩展KB的行为。
+self learn的结果是可以被组织(workflow)共享
\ No newline at end of file
diff --git a/doc/storage.md b/doc/storage.md
new file mode 100644
index 0000000..1f7b43f
--- /dev/null
+++ b/doc/storage.md
@@ -0,0 +1,16 @@
+# NDN Storage
+
+```python
+
+class NDNStorage:
+ async def get(self,data_name:str)->Dict:
+ #return object desc, include local cache path
+
+ async def set_file(self,local_path)->str:
+ # return data_name
+
+ async def read_content(self,data_name:str)->bytes:
+ # return bytes
+ # 这里不但会读取本地缓存，还会对内容进行验证
+
+```
\ No newline at end of file
diff --git a/doc/tob email.md b/doc/tob email.md
new file mode 100644
index 0000000..f59124e
--- /dev/null
+++ b/doc/tob email.md
@@ -0,0 +1,62 @@
+# issue tree
+最核心的机制是树状的issue管理，一个issue应当包含以下属性：
++ 谁提出来的
++ 分配给谁的，如果有的话
++ 起始日期
++ deadline，如果有的话
++ 在哪个邮件里面提出的，引用某个email的原始链接
++ 这个issue的summary，有几种情况，
+ + 一个新的任务，要达成什么目标
+ + 提出了一个问题，需求答案
+ + 解决了某个issue，完成了task或者解答了一个问题
++ 推断出来的 issue的状态，进行中，关闭，超时，完成了
++ parent issue
+
+knowledge维护一个issue tree，从一个root issue出发（root可以是抽象的，比如一个组织的存在，并不是具体的）；knowledge env 提供对这个issue tree的维护接口：
++ 新增issue
++ 更新issue
+
+# parse email
+假定从从某个起始日期开始，以每天为单位，扫描当天新增的email，对每封email：
+1. 输入email 和 从knowledge base获取 issue tree
+2. llm提示词应当包括：issue tree， email正文， knowledge env， llm完成如下推理：
++ email正文提出了一个新的issue，在knowledge env新增issue
++ email正文改变了一个issue的状态
+ + 通报完成了一个task
+ + 回答了一个问题
+ + 明确改变一个issue的状态：认为完成，要延期，认为要取消
++ 根据推理结果正确产生knowledge env 的调用，更新issue tree的状态
+
+## 推理部分可能的out of token：
+1. 裁剪掉已经关闭，超时的 issue
+2. 根据标题特征，是不是对某个email的回复，定位到某个issue， 裁剪出 sub tree
+2. 很长的邮件正文：
+ 1. 第一种方法：先llm推理email的summary，再把summary当正文输入推理issue
+ 2. 第二种方法（我觉得更好）：分片迭代输入email正文，单次llm推理的提示词就变成：issue tree， 当前email summary， 当段email正文，knowledge env：
+ + env里面新增一个method，更新当前email summary
+
+
+# build issue tree
+## 第一种结构：基于knowledge pipeline
+1. pipeline input： 判定当前时间晚于 起始时间并且早于下一个自然天，开始爬正确范围内的邮件输入
+2. pipeline parser：包含准备user prompt 的计算部分，和几个agent
++ 计算部分： 裁剪issue tree，[可选的：调用llm推理生成summary]
++ agent 部分：
+ + agent提示词：从输入的结构化issue tree， 和邮件正文，回复对issue tree knowledge env的调用
+ + 输入提示词： email 正文或者summary，裁剪后的issue tree
++ parser的流程：
+ 对每一个输入的email，查询（裁剪）当前issue tree，把email 和 issue tree 当作user prompt发送给agent，等待agent返回
+
+
+## 第二种结构：基于agent workspace（待定）
+1. schedule task：在每一天产生一个build issue tree task
+2. build issue tree agent: 响应build issue tree task（可不可以以计算为入口，还是只能agent入口）
++ agent调用email env，读出一封邮件
++ agent调用knowledge env，返回issue tree
++ agent从邮件内容和issue tree推理，回复对issue tree knowledge env 的调用
+
+# query issue tree
+主动的或者被动的根据当前issue tree的状态，推理出一些汇总的结论：
++ 是不是有超期的事项
++ 事情是不是有在推进
++ 有哪些事情完成了
\ No newline at end of file
diff --git a/doc/webui.drawio b/doc/webui.drawio
new file mode 100644
index 0000000..4a05e28
--- /dev/null
+++ b/doc/webui.drawio
@@ -0,0 +1,1120 @@
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
diff --git a/doc/workflow.drawio b/doc/workflow.drawio
deleted file mode 100644
index 0cc08b1..0000000
--- a/doc/workflow.drawio
+++ /dev/null
@@ -1,395 +0,0 @@
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
\ No newline at end of file
diff --git a/doc/从日程管理软件到个人助理Agent.md b/doc/从日程管理软件到个人助理Agent.md
new file mode 100644
index 0000000..b2be63b
--- /dev/null
+++ b/doc/从日程管理软件到个人助理Agent.md
@@ -0,0 +1,26 @@
+# 从日程管理软件到个人助理Agent
+
+## 背景
+
+在计算的时代，我们想要有开发一个好用的日程管理服务，很快就会发现这并不简单。比如我们会有各种各样的重复任务提醒要求：比如每个工作周周五的下午去游泳，每个工作周的第一天要开例会，和某人的纪念日要做什么等等。允许用户创建，并识别这些复杂的条件是非常繁琐的。而在计算机发明以前，你的秘书只需要一张纸就可以搞定各种复杂的日程管理：秘书把你的要求原样记录下，然后秘书只要每天Review一遍，确定好今天的行程就好。
+
+一些有更复杂的条件的日程，秘书可以通过定期Review代办来搞定。
+
+理解上述区别，就理解了 PC 和 PI （CPU vs LLM）的核心区别。
+
+## 基于Agent 的日程管理基础设施
+
+为了支持秘书Agent，aios所构建的机制是:
+
+1. Task创建: 模拟秘书在记事本上记录下日程相关的任务（注意不要让Agent创建重复的任务）
+2. ListTask,可查询未标记为结束的Task
+3. 大Review（定期LLM调用），Agent浏览未完成的Task，并给这些Task一个比较精确的`下次处理时间`
+4. 当处理事件的时间点到达时，Agent会尝试处理Task，此时决定是否满足日程发生的条件，并发送正确的提醒信息。
+
+上面的所有的Agent处理，都是让 LLM处理一段文本，并做出反应。因此没有存储格式的需求，用自然语言记录下各种条件就可以了。
+
+## 过渡：混合使用日程管理软件和 秘书Agent
+
+1. 秘书Agent主要使用日程管理软件来查询已有日程
+2. Agent创建通过日程管理软件的目的，通常是为了公开日程，或则创建的是具有`一定协作需求`，或有`公开需求`的`确定`日程
+3. Agent依旧依靠自己的LLM Based逻辑来Review并处理日程, 通过日程管理软件得到的需要处理的日程会有特殊标记，以能正确的操作来保持处理结果。
\ No newline at end of file
diff --git a/package-lock.json b/package-lock.json
new file mode 100644
index 0000000..257bcd1
--- /dev/null
+++ b/package-lock.json
@@ -0,0 +1,1455 @@
+{
+ "name": "opendan_mvp",
+ "lockfileVersion": 3,
+ "requires": true,
+ "packages": {
+ "": {
+ "dependencies": {
+ "docker": "^1.0.0"
+ }
+ },
+ "node_modules/ajv": {
+ "version": "4.11.8",
+ "resolved": "https://registry.npmjs.org/ajv/-/ajv-4.11.8.tgz",
+ "integrity": "sha512-I/bSHSNEcFFqXLf91nchoNB9D1Kie3QKcWdchYUaoIg1+1bdWDkdfdlvdIOJbi9U8xR0y+MWc5D+won9v95WlQ==",
+ "optional": true,
+ "dependencies": {
+ "co": "^4.6.0",
+ "json-stable-stringify": "^1.0.1"
+ }
+ },
+ "node_modules/ambi": {
+ "version": "3.2.0",
+ "resolved": "https://registry.npmjs.org/ambi/-/ambi-3.2.0.tgz",
+ "integrity": "sha512-nj5sHLPFd7u2OLmHdFs4DHt3gK6edpNw35hTRIKyI/Vd2Th5e4io50rw1lhmCdUNO2Mm4/4FkHmv6shEANAWcw==",
+ "dependencies": {
+ "editions": "^2.1.0",
+ "typechecker": "^4.3.0"
+ },
+ "engines": {
+ "node": ">=0.8"
+ }
+ },
+ "node_modules/argparse": {
+ "version": "1.0.10",
+ "resolved": "https://registry.npmjs.org/argparse/-/argparse-1.0.10.tgz",
+ "integrity": "sha512-o5Roy6tNG4SL/FOkCAN6RzjiakZS25RLYFrcMttJqbdd8BWrnA+fGz57iN5Pb06pvBGvl5gQ0B48dJlslXvoTg==",
+ "dependencies": {
+ "sprintf-js": "~1.0.2"
+ }
+ },
+ "node_modules/asap": {
+ "version": "2.0.6",
+ "resolved": "https://registry.npmjs.org/asap/-/asap-2.0.6.tgz",
+ "integrity": "sha512-BSHWgDSAiKs50o2Re8ppvp3seVHXSRM44cdSsT9FfNEUUZLOGWVCsiWaRPWM1Znn+mqZ1OfVZ3z3DWEzSp7hRA==",
+ "optional": true
+ },
+ "node_modules/asn1": {
+ "version": "0.2.6",
+ "resolved": "https://registry.npmjs.org/asn1/-/asn1-0.2.6.tgz",
+ "integrity": "sha512-ix/FxPn0MDjeyJ7i/yoHGFt/EX6LyNbxSEhPPXODPL+KB0VPk86UYfL0lMdy+KCnv+fmvIzySwaK5COwqVbWTQ==",
+ "optional": true,
+ "dependencies": {
+ "safer-buffer": "~2.1.0"
+ }
+ },
+ "node_modules/assert-plus": {
+ "version": "0.2.0",
+ "resolved": "https://registry.npmjs.org/assert-plus/-/assert-plus-0.2.0.tgz",
+ "integrity": "sha512-u1L0ZLywRziOVjUhRxI0Qg9G+4RnFB9H/Rq40YWn0dieDgO7vAYeJz6jKAO6t/aruzlDFLAPkQTT87e+f8Imaw==",
+ "optional": true,
+ "engines": {
+ "node": ">=0.8"
+ }
+ },
+ "node_modules/async": {
+ "version": "1.5.2",
+ "resolved": "https://registry.npmjs.org/async/-/async-1.5.2.tgz",
+ "integrity": "sha512-nSVgobk4rv61R9PUSDtYt7mPVB2olxNR5RWJcAsH676/ef11bUZwvu7+RGYrYauVdDPcO519v68wRhXQtxsV9w=="
+ },
+ "node_modules/asynckit": {
+ "version": "0.4.0",
+ "resolved": "https://registry.npmjs.org/asynckit/-/asynckit-0.4.0.tgz",
+ "integrity": "sha512-Oei9OH4tRh0YqU3GxhX79dM/mwVgvbZJaSNaRk+bshkj0S5cfHcgYakreBjrHwatXKbz+IoIdYLxrKim2MjW0Q==",
+ "optional": true
+ },
+ "node_modules/atob": {
+ "version": "2.1.2",
+ "resolved": "https://registry.npmjs.org/atob/-/atob-2.1.2.tgz",
+ "integrity": "sha512-Wm6ukoaOGJi/73p/cl2GvLjTI5JM1k/O14isD73YML8StrH/7/lRFgmg8nICZgD3bZZvjwCGxtMOD3wWNAu8cg==",
+ "bin": {
+ "atob": "bin/atob.js"
+ },
+ "engines": {
+ "node": ">= 4.5.0"
+ }
+ },
+ "node_modules/aws-sign2": {
+ "version": "0.6.0",
+ "resolved": "https://registry.npmjs.org/aws-sign2/-/aws-sign2-0.6.0.tgz",
+ "integrity": "sha512-JnJpAS0p9RmixkOvW2XwDxxzs1bd4/VAGIl6Q0EC5YOo+p+hqIhtDhn/nmFnB/xUNXbLkpE2mOjgVIBRKD4xYw==",
+ "optional": true,
+ "engines": {
+ "node": "*"
+ }
+ },
+ "node_modules/aws4": {
+ "version": "1.12.0",
+ "resolved": "https://registry.npmjs.org/aws4/-/aws4-1.12.0.tgz",
+ "integrity": "sha512-NmWvPnx0F1SfrQbYwOi7OeaNGokp9XhzNioJ/CSBs8Qa4vxug81mhJEAVZwxXuBmYB5KDRfMq/F3RR0BIU7sWg==",
+ "optional": true
+ },
+ "node_modules/bcrypt-pbkdf": {
+ "version": "1.0.2",
+ "resolved": "https://registry.npmjs.org/bcrypt-pbkdf/-/bcrypt-pbkdf-1.0.2.tgz",
+ "integrity": "sha512-qeFIXtP4MSoi6NLqO12WfqARWWuCKi2Rn/9hJLEmtB5yTNr9DqFWkJRCf2qShWzPeAMRnOgCrq0sg/KLv5ES9w==",
+ "optional": true,
+ "dependencies": {
+ "tweetnacl": "^0.14.3"
+ }
+ },
+ "node_modules/boom": {
+ "version": "2.10.1",
+ "resolved": "https://registry.npmjs.org/boom/-/boom-2.10.1.tgz",
+ "integrity": "sha512-KbiZEa9/vofNcVJXGwdWWn25reQ3V3dHBWbS07FTF3/TOehLnm9GEhJV4T6ZvGPkShRpmUqYwnaCrkj0mRnP6Q==",
+ "deprecated": "This version has been deprecated in accordance with the hapi support policy (hapi.im/support). Please upgrade to the latest version to get the best features, bug fixes, and security patches. If you are unable to upgrade at this time, paid support is available for older versions (hapi.im/commercial).",
+ "optional": true,
+ "dependencies": {
+ "hoek": "2.x.x"
+ },
+ "engines": {
+ "node": ">=0.10.40"
+ }
+ },
+ "node_modules/call-bind": {
+ "version": "1.0.5",
+ "resolved": "https://registry.npmjs.org/call-bind/-/call-bind-1.0.5.tgz",
+ "integrity": "sha512-C3nQxfFZxFRVoJoGKKI8y3MOEo129NQ+FgQ08iye+Mk4zNZZGdjfs06bVTr+DBSlA66Q2VEcMki/cUCP4SercQ==",
+ "optional": true,
+ "dependencies": {
+ "function-bind": "^1.1.2",
+ "get-intrinsic": "^1.2.1",
+ "set-function-length": "^1.1.1"
+ },
+ "funding": {
+ "url": "https://github.com/sponsors/ljharb"
+ }
+ },
+ "node_modules/caseless": {
+ "version": "0.12.0",
+ "resolved": "https://registry.npmjs.org/caseless/-/caseless-0.12.0.tgz",
+ "integrity": "sha512-4tYFyifaFfGacoiObjJegolkwSU4xQNGbVgUiNYVUxbQ2x2lUsFvY4hVgVzGiIe6WLOPqycWXA40l+PWsxthUw==",
+ "optional": true
+ },
+ "node_modules/co": {
+ "version": "4.6.0",
+ "resolved": "https://registry.npmjs.org/co/-/co-4.6.0.tgz",
+ "integrity": "sha512-QVb0dM5HvG+uaxitm8wONl7jltx8dqhfU33DcqtOZcLSVIKSDDLDi7+0LbAKiyI8hD9u42m2YxXSkMGWThaecQ==",
+ "optional": true,
+ "engines": {
+ "iojs": ">= 1.0.0",
+ "node": ">= 0.12.0"
+ }
+ },
+ "node_modules/combined-stream": {
+ "version": "1.0.8",
+ "resolved": "https://registry.npmjs.org/combined-stream/-/combined-stream-1.0.8.tgz",
+ "integrity": "sha512-FQN4MRfuJeHf7cBbBMJFXhKSDq+2kAArBlmRBvcvFE5BB1HZKXtSFASDhdlz9zOYwxh8lDdnvmMOe/+5cdoEdg==",
+ "optional": true,
+ "dependencies": {
+ "delayed-stream": "~1.0.0"
+ },
+ "engines": {
+ "node": ">= 0.8"
+ }
+ },
+ "node_modules/commander": {
+ "version": "2.20.3",
+ "resolved": "https://registry.npmjs.org/commander/-/commander-2.20.3.tgz",
+ "integrity": "sha512-GpVkmM8vF2vQUkj2LvZmD35JxeJOLCwJ9cUkugyk2nuhbv3+mJvpLYYt+0+USMxE+oj+ey/lJEnhZw75x/OMcQ=="
+ },
+ "node_modules/core-util-is": {
+ "version": "1.0.2",
+ "resolved": "https://registry.npmjs.org/core-util-is/-/core-util-is-1.0.2.tgz",
+ "integrity": "sha512-3lqz5YjWTYnW6dlDa5TLaTCcShfar1e40rmcJVwCBJC6mWlFuj0eCHIElmG1g5kyuJ/GD+8Wn4FFCcz4gJPfaQ==",
+ "optional": true
+ },
+ "node_modules/cryptiles": {
+ "version": "2.0.5",
+ "resolved": "https://registry.npmjs.org/cryptiles/-/cryptiles-2.0.5.tgz",
+ "integrity": "sha512-FFN5KwpvvQTTS5hWPxrU8/QE4kQUc6uwZcrnlMBN82t1MgAtq8mnoDwINBly9Tdr02seeIIhtdF+UH1feBYGog==",
+ "deprecated": "This version has been deprecated in accordance with the hapi support policy (hapi.im/support). Please upgrade to the latest version to get the best features, bug fixes, and security patches. If you are unable to upgrade at this time, paid support is available for older versions (hapi.im/commercial).",
+ "optional": true,
+ "dependencies": {
+ "boom": "2.x.x"
+ },
+ "engines": {
+ "node": ">=0.10.40"
+ }
+ },
+ "node_modules/csextends": {
+ "version": "1.2.0",
+ "resolved": "https://registry.npmjs.org/csextends/-/csextends-1.2.0.tgz",
+ "integrity": "sha512-S/8k1bDTJIwuGgQYmsRoE+8P+ohV32WhQ0l4zqrc0XDdxOhjQQD7/wTZwCzoZX53jSX3V/qwjT+OkPTxWQcmjg==",
+ "engines": {
+ "node": ">=0.8"
+ }
+ },
+ "node_modules/css": {
+ "version": "2.2.4",
+ "resolved": "https://registry.npmjs.org/css/-/css-2.2.4.tgz",
+ "integrity": "sha512-oUnjmWpy0niI3x/mPL8dVEI1l7MnG3+HHyRPHf+YFSbK+svOhXpmSOcDURUh2aOCgl2grzrOPt1nHLuCVFULLw==",
+ "dependencies": {
+ "inherits": "^2.0.3",
+ "source-map": "^0.6.1",
+ "source-map-resolve": "^0.5.2",
+ "urix": "^0.1.0"
+ }
+ },
+ "node_modules/dashdash": {
+ "version": "1.14.1",
+ "resolved": "https://registry.npmjs.org/dashdash/-/dashdash-1.14.1.tgz",
+ "integrity": "sha512-jRFi8UDGo6j+odZiEpjazZaWqEal3w/basFjQHQEwVtZJGDpxbH1MeYluwCS8Xq5wmLJooDlMgvVarmWfGM44g==",
+ "optional": true,
+ "dependencies": {
+ "assert-plus": "^1.0.0"
+ },
+ "engines": {
+ "node": ">=0.10"
+ }
+ },
+ "node_modules/dashdash/node_modules/assert-plus": {
+ "version": "1.0.0",
+ "resolved": "https://registry.npmjs.org/assert-plus/-/assert-plus-1.0.0.tgz",
+ "integrity": "sha512-NfJ4UzBCcQGLDlQq7nHxH+tv3kyZ0hHQqF5BO6J7tNJeP5do1llPr8dZ8zHonfhAu0PHAdMkSo+8o0wxg9lZWw==",
+ "optional": true,
+ "engines": {
+ "node": ">=0.8"
+ }
+ },
+ "node_modules/decode-uri-component": {
+ "version": "0.2.2",
+ "resolved": "https://registry.npmjs.org/decode-uri-component/-/decode-uri-component-0.2.2.tgz",
+ "integrity": "sha512-FqUYQ+8o158GyGTrMFJms9qh3CqTKvAqgqsTnkLI8sKu0028orqBhxNMFkFen0zGyg6epACD32pjVk58ngIErQ==",
+ "engines": {
+ "node": ">=0.10"
+ }
+ },
+ "node_modules/define-data-property": {
+ "version": "1.1.1",
+ "resolved": "https://registry.npmjs.org/define-data-property/-/define-data-property-1.1.1.tgz",
+ "integrity": "sha512-E7uGkTzkk1d0ByLeSc6ZsFS79Axg+m1P/VsgYsxHgiuc3tFSj+MjMIwe90FC4lOAZzNBdY7kkO2P2wKdsQ1vgQ==",
+ "optional": true,
+ "dependencies": {
+ "get-intrinsic": "^1.2.1",
+ "gopd": "^1.0.1",
+ "has-property-descriptors": "^1.0.0"
+ },
+ "engines": {
+ "node": ">= 0.4"
+ }
+ },
+ "node_modules/delayed-stream": {
+ "version": "1.0.0",
+ "resolved": "https://registry.npmjs.org/delayed-stream/-/delayed-stream-1.0.0.tgz",
+ "integrity": "sha512-ZySD7Nf91aLB0RxL4KGrKHBXl7Eds1DAmEdcoVawXnLD7SDhpNgtuII2aAkg7a7QS41jxPSZ17p4VdGnMHk3MQ==",
+ "optional": true,
+ "engines": {
+ "node": ">=0.4.0"
+ }
+ },
+ "node_modules/docker": {
+ "version": "1.0.0",
+ "resolved": "https://registry.npmjs.org/docker/-/docker-1.0.0.tgz",
+ "integrity": "sha512-U66G/kvvsCTUh6VsZqnWgsoSG1KRu5jR473fn/64E6EU9cH65afCITx2qITmNPkr3IOehcn1wwXHmIvHYBlLgQ==",
+ "dependencies": {
+ "async": "^1.4.0",
+ "commander": "^2.9.0",
+ "css": "^2.2.1",
+ "dox": "^0.8.0",
+ "ejs": "^2.3.3",
+ "extend": "^3.0.0",
+ "highlight.js": "^9.3.0",
+ "less": "^2.5.1",
+ "markdown-it": "^6.0.1",
+ "mkdirp": "^0.5.1",
+ "repeating": "^2.0.1",
+ "strip-indent": "^2.0.0",
+ "toc": "^0.4.0",
+ "watchr": "^2.4.13"
+ },
+ "bin": {
+ "docker": "docker",
+ "docker.js": "docker"
+ },
+ "engines": {
+ "node": ">=0.10.0"
+ }
+ },
+ "node_modules/dox": {
+ "version": "0.8.1",
+ "resolved": "https://registry.npmjs.org/dox/-/dox-0.8.1.tgz",
+ "integrity": "sha512-CJJCQS6XYJ2FQJox4ey7pUdaAjDusPLqGtfe3Jli4N+m2jBKrT9zwEsh2thV9W5d8F359AMWqkWk50CuH3r8dw==",
+ "dependencies": {
+ "commander": "~2.9.0",
+ "jsdoctypeparser": "^1.2.0",
+ "marked": "~0.3.5"
+ },
+ "bin": {
+ "dox": "bin/dox"
+ }
+ },
+ "node_modules/dox/node_modules/commander": {
+ "version": "2.9.0",
+ "resolved": "https://registry.npmjs.org/commander/-/commander-2.9.0.tgz",
+ "integrity": "sha512-bmkUukX8wAOjHdN26xj5c4ctEV22TQ7dQYhSmuckKhToXrkUn0iIaolHdIxYYqD55nhpSPA9zPQ1yP57GdXP2A==",
+ "dependencies": {
+ "graceful-readlink": ">= 1.0.0"
+ },
+ "engines": {
+ "node": ">= 0.6.x"
+ }
+ },
+ "node_modules/eachr": {
+ "version": "3.3.0",
+ "resolved": "https://registry.npmjs.org/eachr/-/eachr-3.3.0.tgz",
+ "integrity": "sha512-yKWuGwOE283CTgbEuvqXXusLH4VBXnY2nZbDkeWev+cpAXY6zCIADSPLdvfkAROc0t8S4l07U1fateCdEDuuvg==",
+ "dependencies": {
+ "editions": "^2.2.0",
+ "typechecker": "^4.9.0"
+ },
+ "engines": {
+ "node": ">=0.10"
+ },
+ "funding": {
+ "url": "https://bevry.me/fund"
+ }
+ },
+ "node_modules/ecc-jsbn": {
+ "version": "0.1.2",
+ "resolved": "https://registry.npmjs.org/ecc-jsbn/-/ecc-jsbn-0.1.2.tgz",
+ "integrity": "sha512-eh9O+hwRHNbG4BLTjEl3nw044CkGm5X6LoaCf7LPp7UU8Qrt47JYNi6nPX8xjW97TKGKm1ouctg0QSpZe9qrnw==",
+ "optional": true,
+ "dependencies": {
+ "jsbn": "~0.1.0",
+ "safer-buffer": "^2.1.0"
+ }
+ },
+ "node_modules/editions": {
+ "version": "2.3.1",
+ "resolved": "https://registry.npmjs.org/editions/-/editions-2.3.1.tgz",
+ "integrity": "sha512-ptGvkwTvGdGfC0hfhKg0MT+TRLRKGtUiWGBInxOm5pz7ssADezahjCUaYuZ8Dr+C05FW0AECIIPt4WBxVINEhA==",
+ "dependencies": {
+ "errlop": "^2.0.0",
+ "semver": "^6.3.0"
+ },
+ "engines": {
+ "node": ">=0.8"
+ },
+ "funding": {
+ "url": "https://bevry.me/fund"
+ }
+ },
+ "node_modules/ejs": {
+ "version": "2.7.4",
+ "resolved": "https://registry.npmjs.org/ejs/-/ejs-2.7.4.tgz",
+ "integrity": "sha512-7vmuyh5+kuUyJKePhQfRQBhXV5Ce+RnaeeQArKu1EAMpL3WbgMt5WG6uQZpEVvYSSsxMXRKOewtDk9RaTKXRlA==",
+ "hasInstallScript": true,
+ "engines": {
+ "node": ">=0.10.0"
+ }
+ },
+ "node_modules/entities": {
+ "version": "1.1.2",
+ "resolved": "https://registry.npmjs.org/entities/-/entities-1.1.2.tgz",
+ "integrity": "sha512-f2LZMYl1Fzu7YSBKg+RoROelpOaNrcGmE9AZubeDfrCEia483oW4MI4VyFd5VNHIgQ/7qm1I0wUHK1eJnn2y2w=="
+ },
+ "node_modules/errlop": {
+ "version": "2.2.0",
+ "resolved": "https://registry.npmjs.org/errlop/-/errlop-2.2.0.tgz",
+ "integrity": "sha512-e64Qj9+4aZzjzzFpZC7p5kmm/ccCrbLhAJplhsDXQFs87XTsXwOpH4s1Io2s90Tau/8r2j9f4l/thhDevRjzxw==",
+ "engines": {
+ "node": ">=0.8"
+ },
+ "funding": {
+ "url": "https://bevry.me/fund"
+ }
+ },
+ "node_modules/errno": {
+ "version": "0.1.8",
+ "resolved": "https://registry.npmjs.org/errno/-/errno-0.1.8.tgz",
+ "integrity": "sha512-dJ6oBr5SQ1VSd9qkk7ByRgb/1SH4JZjCHSW/mr63/QcXO9zLVxvJ6Oy13nio03rxpSnVDDjFor75SjVeZWPW/A==",
+ "optional": true,
+ "dependencies": {
+ "prr": "~1.0.1"
+ },
+ "bin": {
+ "errno": "cli.js"
+ }
+ },
+ "node_modules/extend": {
+ "version": "3.0.2",
+ "resolved": "https://registry.npmjs.org/extend/-/extend-3.0.2.tgz",
+ "integrity": "sha512-fjquC59cD7CyW6urNXK0FBufkZcoiGG80wTuPujX590cB5Ttln20E2UB4S/WARVqhXffZl2LNgS+gQdPIIim/g=="
+ },
+ "node_modules/extendr": {
+ "version": "3.5.0",
+ "resolved": "https://registry.npmjs.org/extendr/-/extendr-3.5.0.tgz",
+ "integrity": "sha512-7zpVbnnZy91J4k916ZGwpys56DEgJc/prTXDiqCYe/Mud5pqdVsSc9mG/U6sz3lQEvHs81i8Zi7whsFwifhZyw==",
+ "dependencies": {
+ "editions": "^2.2.0",
+ "typechecker": "^4.7.0"
+ },
+ "engines": {
+ "node": ">=0.12"
+ },
+ "funding": {
+ "type": "cooperative",
+ "url": "https://bevry.me/fund"
+ }
+ },
+ "node_modules/extract-opts": {
+ "version": "3.4.0",
+ "resolved": "https://registry.npmjs.org/extract-opts/-/extract-opts-3.4.0.tgz",
+ "integrity": "sha512-M7Y+1cJDkzOWqvGH5F/V2qgkD6+uitW3NV9rQGl+pLSVuXZ4IDDQgxxMeLPKcWUyfypBWczIILiroSuhXG7Ytg==",
+ "dependencies": {
+ "eachr": "^3.2.0",
+ "editions": "^2.2.0",
+ "typechecker": "^4.9.0"
+ },
+ "engines": {
+ "node": ">=0.10"
+ },
+ "funding": {
+ "url": "https://bevry.me/fund"
+ }
+ },
+ "node_modules/extsprintf": {
+ "version": "1.3.0",
+ "resolved": "https://registry.npmjs.org/extsprintf/-/extsprintf-1.3.0.tgz",
+ "integrity": "sha512-11Ndz7Nv+mvAC1j0ktTa7fAb0vLyGGX+rMHNBYQviQDGU0Hw7lhctJANqbPhu9nV9/izT/IntTgZ7Im/9LJs9g==",
+ "engines": [
+ "node >=0.6.0"
+ ],
+ "optional": true
+ },
+ "node_modules/forever-agent": {
+ "version": "0.6.1",
+ "resolved": "https://registry.npmjs.org/forever-agent/-/forever-agent-0.6.1.tgz",
+ "integrity": "sha512-j0KLYPhm6zeac4lz3oJ3o65qvgQCcPubiyotZrXqEaG4hNagNYO8qdlUrX5vwqv9ohqeT/Z3j6+yW067yWWdUw==",
+ "optional": true,
+ "engines": {
+ "node": "*"
+ }
+ },
+ "node_modules/form-data": {
+ "version": "2.1.4",
+ "resolved": "https://registry.npmjs.org/form-data/-/form-data-2.1.4.tgz",
+ "integrity": "sha512-8HWGSLAPr+AG0hBpsqi5Ob8HrLStN/LWeqhpFl14d7FJgHK48TmgLoALPz69XSUR65YJzDfLUX/BM8+MLJLghQ==",
+ "optional": true,
+ "dependencies": {
+ "asynckit": "^0.4.0",
+ "combined-stream": "^1.0.5",
+ "mime-types": "^2.1.12"
+ },
+ "engines": {
+ "node": ">= 0.12"
+ }
+ },
+ "node_modules/function-bind": {
+ "version": "1.1.2",
+ "resolved": "https://registry.npmjs.org/function-bind/-/function-bind-1.1.2.tgz",
+ "integrity": "sha512-7XHNxH7qX9xG5mIwxkhumTox/MIRNcOgDrxWsMt2pAr23WHp6MrRlN7FBSFpCpr+oVO0F744iUgR82nJMfG2SA==",
+ "optional": true,
+ "funding": {
+ "url": "https://github.com/sponsors/ljharb"
+ }
+ },
+ "node_modules/get-intrinsic": {
+ "version": "1.2.2",
+ "resolved": "https://registry.npmjs.org/get-intrinsic/-/get-intrinsic-1.2.2.tgz",
+ "integrity": "sha512-0gSo4ml/0j98Y3lngkFEot/zhiCeWsbYIlZ+uZOVgzLyLaUw7wxUL+nCTP0XJvJg1AXulJRI3UJi8GsbDuxdGA==",
+ "optional": true,
+ "dependencies": {
+ "function-bind": "^1.1.2",
+ "has-proto": "^1.0.1",
+ "has-symbols": "^1.0.3",
+ "hasown": "^2.0.0"
+ },
+ "funding": {
+ "url": "https://github.com/sponsors/ljharb"
+ }
+ },
+ "node_modules/getpass": {
+ "version": "0.1.7",
+ "resolved": "https://registry.npmjs.org/getpass/-/getpass-0.1.7.tgz",
+ "integrity": "sha512-0fzj9JxOLfJ+XGLhR8ze3unN0KZCgZwiSSDz168VERjK8Wl8kVSdcu2kspd4s4wtAa1y/qrVRiAA0WclVsu0ng==",
+ "optional": true,
+ "dependencies": {
+ "assert-plus": "^1.0.0"
+ }
+ },
+ "node_modules/getpass/node_modules/assert-plus": {
+ "version": "1.0.0",
+ "resolved": "https://registry.npmjs.org/assert-plus/-/assert-plus-1.0.0.tgz",
+ "integrity": "sha512-NfJ4UzBCcQGLDlQq7nHxH+tv3kyZ0hHQqF5BO6J7tNJeP5do1llPr8dZ8zHonfhAu0PHAdMkSo+8o0wxg9lZWw==",
+ "optional": true,
+ "engines": {
+ "node": ">=0.8"
+ }
+ },
+ "node_modules/gopd": {
+ "version": "1.0.1",
+ "resolved": "https://registry.npmjs.org/gopd/-/gopd-1.0.1.tgz",
+ "integrity": "sha512-d65bNlIadxvpb/A2abVdlqKqV563juRnZ1Wtk6s1sIR8uNsXR70xqIzVqxVf1eTqDunwT2MkczEeaezCKTZhwA==",
+ "optional": true,
+ "dependencies": {
+ "get-intrinsic": "^1.1.3"
+ },
+ "funding": {
+ "url": "https://github.com/sponsors/ljharb"
+ }
+ },
+ "node_modules/graceful-fs": {
+ "version": "4.2.11",
+ "resolved": "https://registry.npmjs.org/graceful-fs/-/graceful-fs-4.2.11.tgz",
+ "integrity": "sha512-RbJ5/jmFcNNCcDV5o9eTnBLJ/HszWV0P73bc+Ff4nS/rJj+YaS6IGyiOL0VoBYX+l1Wrl3k63h/KrH+nhJ0XvQ=="
+ },
+ "node_modules/graceful-readlink": {
+ "version": "1.0.1",
+ "resolved": "https://registry.npmjs.org/graceful-readlink/-/graceful-readlink-1.0.1.tgz",
+ "integrity": "sha512-8tLu60LgxF6XpdbK8OW3FA+IfTNBn1ZHGHKF4KQbEeSkajYw5PlYJcKluntgegDPTg8UkHjpet1T82vk6TQ68w=="
+ },
+ "node_modules/har-schema": {
+ "version": "1.0.5",
+ "resolved": "https://registry.npmjs.org/har-schema/-/har-schema-1.0.5.tgz",
+ "integrity": "sha512-f8xf2GOR6Rgwc9FPTLNzgwB+JQ2/zMauYXSWmX5YV5acex6VomT0ocSuwR7BfXo5MpHi+jL+saaux2fwsGJDKQ==",
+ "optional": true,
+ "engines": {
+ "node": ">=4"
+ }
+ },
+ "node_modules/har-validator": {
+ "version": "4.2.1",
+ "resolved": "https://registry.npmjs.org/har-validator/-/har-validator-4.2.1.tgz",
+ "integrity": "sha512-5Gbp6RAftMYYV3UEI4c4Vv3+a4dQ7taVyvHt+/L6kRt+f4HX1GweAk5UDWN0SvdVnRBzGQ6OG89pGaD9uSFnVw==",
+ "deprecated": "this library is no longer supported",
+ "optional": true,
+ "dependencies": {
+ "ajv": "^4.9.1",
+ "har-schema": "^1.0.5"
+ },
+ "engines": {
+ "node": ">=4"
+ }
+ },
+ "node_modules/has-property-descriptors": {
+ "version": "1.0.1",
+ "resolved": "https://registry.npmjs.org/has-property-descriptors/-/has-property-descriptors-1.0.1.tgz",
+ "integrity": "sha512-VsX8eaIewvas0xnvinAe9bw4WfIeODpGYikiWYLH+dma0Jw6KHYqWiWfhQlgOVK8D6PvjubK5Uc4P0iIhIcNVg==",
+ "optional": true,
+ "dependencies": {
+ "get-intrinsic": "^1.2.2"
+ },
+ "funding": {
+ "url": "https://github.com/sponsors/ljharb"
+ }
+ },
+ "node_modules/has-proto": {
+ "version": "1.0.1",
+ "resolved": "https://registry.npmjs.org/has-proto/-/has-proto-1.0.1.tgz",
+ "integrity": "sha512-7qE+iP+O+bgF9clE5+UoBFzE65mlBiVj3tKCrlNQ0Ogwm0BjpT/gK4SlLYDMybDh5I3TCTKnPPa0oMG7JDYrhg==",
+ "optional": true,
+ "engines": {
+ "node": ">= 0.4"
+ },
+ "funding": {
+ "url": "https://github.com/sponsors/ljharb"
+ }
+ },
+ "node_modules/has-symbols": {
+ "version": "1.0.3",
+ "resolved": "https://registry.npmjs.org/has-symbols/-/has-symbols-1.0.3.tgz",
+ "integrity": "sha512-l3LCuF6MgDNwTDKkdYGEihYjt5pRPbEg46rtlmnSPlUbgmB8LOIrKJbYYFBSbnPaJexMKtiPO8hmeRjRz2Td+A==",
+ "optional": true,
+ "engines": {
+ "node": ">= 0.4"
+ },
+ "funding": {
+ "url": "https://github.com/sponsors/ljharb"
+ }
+ },
+ "node_modules/hasown": {
+ "version": "2.0.0",
+ "resolved": "https://registry.npmjs.org/hasown/-/hasown-2.0.0.tgz",
+ "integrity": "sha512-vUptKVTpIJhcczKBbgnS+RtcuYMB8+oNzPK2/Hp3hanz8JmpATdmmgLgSaadVREkDm+e2giHwY3ZRkyjSIDDFA==",
+ "optional": true,
+ "dependencies": {
+ "function-bind": "^1.1.2"
+ },
+ "engines": {
+ "node": ">= 0.4"
+ }
+ },
+ "node_modules/hawk": {
+ "version": "3.1.3",
+ "resolved": "https://registry.npmjs.org/hawk/-/hawk-3.1.3.tgz",
+ "integrity": "sha512-X8xbmTc1cbPXcQV4WkLcRMALuyoxhfpFATmyuCxJPOAvrDS4DNnsTAOmKUxMTOWU6TzrTOkxPKwIx5ZOpJVSrg==",
+ "deprecated": "This module moved to @hapi/hawk. Please make sure to switch over as this distribution is no longer supported and may contain bugs and critical security issues.",
+ "optional": true,
+ "dependencies": {
+ "boom": "2.x.x",
+ "cryptiles": "2.x.x",
+ "hoek": "2.x.x",
+ "sntp": "1.x.x"
+ },
+ "engines": {
+ "node": ">=0.10.32"
+ }
+ },
+ "node_modules/highlight.js": {
+ "version": "9.18.5",
+ "resolved": "https://registry.npmjs.org/highlight.js/-/highlight.js-9.18.5.tgz",
+ "integrity": "sha512-a5bFyofd/BHCX52/8i8uJkjr9DYwXIPnM/plwI6W7ezItLGqzt7X2G2nXuYSfsIJdkwwj/g9DG1LkcGJI/dDoA==",
+ "deprecated": "Support has ended for 9.x series. Upgrade to @latest",
+ "hasInstallScript": true,
+ "engines": {
+ "node": "*"
+ }
+ },
+ "node_modules/hoek": {
+ "version": "2.16.3",
+ "resolved": "https://registry.npmjs.org/hoek/-/hoek-2.16.3.tgz",
+ "integrity": "sha512-V6Yw1rIcYV/4JsnggjBU0l4Kr+EXhpwqXRusENU1Xx6ro00IHPHYNynCuBTOZAPlr3AAmLvchH9I7N/VUdvOwQ==",
+ "deprecated": "This version has been deprecated in accordance with the hapi support policy (hapi.im/support). Please upgrade to the latest version to get the best features, bug fixes, and security patches. If you are unable to upgrade at this time, paid support is available for older versions (hapi.im/commercial).",
+ "optional": true,
+ "engines": {
+ "node": ">=0.10.40"
+ }
+ },
+ "node_modules/http-signature": {
+ "version": "1.1.1",
+ "resolved": "https://registry.npmjs.org/http-signature/-/http-signature-1.1.1.tgz",
+ "integrity": "sha512-iUn0NcRULlDGtqNLN1Jxmzayk8ogm7NToldASyZBpM2qggbphjXzNOiw3piN8tgz+e/DRs6X5gAzFwTI6BCRcg==",
+ "optional": true,
+ "dependencies": {
+ "assert-plus": "^0.2.0",
+ "jsprim": "^1.2.2",
+ "sshpk": "^1.7.0"
+ },
+ "engines": {
+ "node": ">=0.8",
+ "npm": ">=1.3.7"
+ }
+ },
+ "node_modules/ignorefs": {
+ "version": "1.4.1",
+ "resolved": "https://registry.npmjs.org/ignorefs/-/ignorefs-1.4.1.tgz",
+ "integrity": "sha512-1whgvOsPWFZRNA/5OFhIk56C9Y39+/CYaRVNvsZZkLymacOSqqdSU53xk8CP3G2u5gz2PX6RLxqKPcsIpDriog==",
+ "dependencies": {
+ "editions": "^2.2.0",
+ "ignorepatterns": "^1.4.0"
+ },
+ "engines": {
+ "node": ">=0.12"
+ },
+ "funding": {
+ "url": "https://bevry.me/fund"
+ }
+ },
+ "node_modules/ignorepatterns": {
+ "version": "1.4.0",
+ "resolved": "https://registry.npmjs.org/ignorepatterns/-/ignorepatterns-1.4.0.tgz",
+ "integrity": "sha512-YPBIFRB25iZD0WiLxmToe80+QU+mZI+bUlEh3Ze/4gbhlXHdQFk0SwAFQtPOiBAoDv3FvhtSTDUCD9DKFsHTRA==",
+ "dependencies": {
+ "editions": "^2.2.0"
+ },
+ "engines": {
+ "node": ">=0.12"
+ },
+ "funding": {
+ "url": "https://bevry.me/fund"
+ }
+ },
+ "node_modules/image-size": {
+ "version": "0.5.5",
+ "resolved": "https://registry.npmjs.org/image-size/-/image-size-0.5.5.tgz",
+ "integrity": "sha512-6TDAlDPZxUFCv+fuOkIoXT/V/f3Qbq8e37p+YOiYrUv3v9cc3/6x78VdfPgFVaB9dZYeLUfKgHRebpkm/oP2VQ==",
+ "optional": true,
+ "bin": {
+ "image-size": "bin/image-size.js"
+ },
+ "engines": {
+ "node": ">=0.10.0"
+ }
+ },
+ "node_modules/inherits": {
+ "version": "2.0.4",
+ "resolved": "https://registry.npmjs.org/inherits/-/inherits-2.0.4.tgz",
+ "integrity": "sha512-k/vGaX4/Yla3WzyMCvTQOXYeIHvqOKtnqBduzTHpzpQZzAskKMhZ2K+EnBiSM9zGSoIFeMpXKxa4dYeZIQqewQ=="
+ },
+ "node_modules/is-finite": {
+ "version": "1.1.0",
+ "resolved": "https://registry.npmjs.org/is-finite/-/is-finite-1.1.0.tgz",
+ "integrity": "sha512-cdyMtqX/BOqqNBBiKlIVkytNHm49MtMlYyn1zxzvJKWmFMlGzm+ry5BBfYyeY9YmNKbRSo/o7OX9w9ale0wg3w==",
+ "engines": {
+ "node": ">=0.10.0"
+ },
+ "funding": {
+ "url": "https://github.com/sponsors/sindresorhus"
+ }
+ },
+ "node_modules/is-typedarray": {
+ "version": "1.0.0",
+ "resolved": "https://registry.npmjs.org/is-typedarray/-/is-typedarray-1.0.0.tgz",
+ "integrity": "sha512-cyA56iCMHAh5CdzjJIa4aohJyeO1YbwLi3Jc35MmRU6poroFjIGZzUzupGiRPOjgHg9TLu43xbpwXk523fMxKA==",
+ "optional": true
+ },
+ "node_modules/isarray": {
+ "version": "2.0.5",
+ "resolved": "https://registry.npmjs.org/isarray/-/isarray-2.0.5.tgz",
+ "integrity": "sha512-xHjhDr3cNBK0BzdUJSPXZntQUx/mwMS5Rw4A7lPJ90XGAO6ISP/ePDNuo0vhqOZU+UD5JoodwCAAoZQd3FeAKw==",
+ "optional": true
+ },
+ "node_modules/isstream": {
+ "version": "0.1.2",
+ "resolved": "https://registry.npmjs.org/isstream/-/isstream-0.1.2.tgz",
+ "integrity": "sha512-Yljz7ffyPbrLpLngrMtZ7NduUgVvi6wG9RJ9IUcyCd59YQ911PBJphODUcbOVbqYfxe1wuYf/LJ8PauMRwsM/g==",
+ "optional": true
+ },
+ "node_modules/jsbn": {
+ "version": "0.1.1",
+ "resolved": "https://registry.npmjs.org/jsbn/-/jsbn-0.1.1.tgz",
+ "integrity": "sha512-UVU9dibq2JcFWxQPA6KCqj5O42VOmAY3zQUfEKxU0KpTGXwNoCjkX1e13eHNvw/xPynt6pU0rZ1htjWTNTSXsg==",
+ "optional": true
+ },
+ "node_modules/jsdoctypeparser": {
+ "version": "1.2.0",
+ "resolved": "https://registry.npmjs.org/jsdoctypeparser/-/jsdoctypeparser-1.2.0.tgz",
+ "integrity": "sha512-osXm4Fr1o/Jc0YwUM7DHUliYtaunLQxh4ynZgtN02mTUN1VsNbMy75DFSkKRne8xE8jiGRV9NKVhYYYa8ZIHXQ==",
+ "dependencies": {
+ "lodash": "^3.7.0"
+ }
+ },
+ "node_modules/json-schema": {
+ "version": "0.4.0",
+ "resolved": "https://registry.npmjs.org/json-schema/-/json-schema-0.4.0.tgz",
+ "integrity": "sha512-es94M3nTIfsEPisRafak+HDLfHXnKBhV3vU5eqPcS3flIWqcxJWgXHXiey3YrpaNsanY5ei1VoYEbOzijuq9BA==",
+ "optional": true
+ },
+ "node_modules/json-stable-stringify": {
+ "version": "1.1.0",
+ "resolved": "https://registry.npmjs.org/json-stable-stringify/-/json-stable-stringify-1.1.0.tgz",
+ "integrity": "sha512-zfA+5SuwYN2VWqN1/5HZaDzQKLJHaBVMZIIM+wuYjdptkaQsqzDdqjqf+lZZJUuJq1aanHiY8LhH8LmH+qBYJA==",
+ "optional": true,
+ "dependencies": {
+ "call-bind": "^1.0.5",
+ "isarray": "^2.0.5",
+ "jsonify": "^0.0.1",
+ "object-keys": "^1.1.1"
+ },
+ "engines": {
+ "node": ">= 0.4"
+ },
+ "funding": {
+ "url": "https://github.com/sponsors/ljharb"
+ }
+ },
+ "node_modules/json-stringify-safe": {
+ "version": "5.0.1",
+ "resolved": "https://registry.npmjs.org/json-stringify-safe/-/json-stringify-safe-5.0.1.tgz",
+ "integrity": "sha512-ZClg6AaYvamvYEE82d3Iyd3vSSIjQ+odgjaTzRuO3s7toCdFKczob2i0zCh7JE8kWn17yvAWhUVxvqGwUalsRA==",
+ "optional": true
+ },
+ "node_modules/jsonify": {
+ "version": "0.0.1",
+ "resolved": "https://registry.npmjs.org/jsonify/-/jsonify-0.0.1.tgz",
+ "integrity": "sha512-2/Ki0GcmuqSrgFyelQq9M05y7PS0mEwuIzrf3f1fPqkVDVRvZrPZtVSMHxdgo8Aq0sxAOb/cr2aqqA3LeWHVPg==",
+ "optional": true,
+ "funding": {
+ "url": "https://github.com/sponsors/ljharb"
+ }
+ },
+ "node_modules/jsprim": {
+ "version": "1.4.2",
+ "resolved": "https://registry.npmjs.org/jsprim/-/jsprim-1.4.2.tgz",
+ "integrity": "sha512-P2bSOMAc/ciLz6DzgjVlGJP9+BrJWu5UDGK70C2iweC5QBIeFf0ZXRvGjEj2uYgrY2MkAAhsSWHDWlFtEroZWw==",
+ "optional": true,
+ "dependencies": {
+ "assert-plus": "1.0.0",
+ "extsprintf": "1.3.0",
+ "json-schema": "0.4.0",
+ "verror": "1.10.0"
+ },
+ "engines": {
+ "node": ">=0.6.0"
+ }
+ },
+ "node_modules/jsprim/node_modules/assert-plus": {
+ "version": "1.0.0",
+ "resolved": "https://registry.npmjs.org/assert-plus/-/assert-plus-1.0.0.tgz",
+ "integrity": "sha512-NfJ4UzBCcQGLDlQq7nHxH+tv3kyZ0hHQqF5BO6J7tNJeP5do1llPr8dZ8zHonfhAu0PHAdMkSo+8o0wxg9lZWw==",
+ "optional": true,
+ "engines": {
+ "node": ">=0.8"
+ }
+ },
+ "node_modules/less": {
+ "version": "2.7.3",
+ "resolved": "https://registry.npmjs.org/less/-/less-2.7.3.tgz",
+ "integrity": "sha512-KPdIJKWcEAb02TuJtaLrhue0krtRLoRoo7x6BNJIBelO00t/CCdJQUnHW5V34OnHMWzIktSalJxRO+FvytQlCQ==",
+ "bin": {
+ "lessc": "bin/lessc"
+ },
+ "engines": {
+ "node": ">=0.12"
+ },
+ "optionalDependencies": {
+ "errno": "^0.1.1",
+ "graceful-fs": "^4.1.2",
+ "image-size": "~0.5.0",
+ "mime": "^1.2.11",
+ "mkdirp": "^0.5.0",
+ "promise": "^7.1.1",
+ "request": "2.81.0",
+ "source-map": "^0.5.3"
+ }
+ },
+ "node_modules/less/node_modules/source-map": {
+ "version": "0.5.7",
+ "resolved": "https://registry.npmjs.org/source-map/-/source-map-0.5.7.tgz",
+ "integrity": "sha512-LbrmJOMUSdEVxIKvdcJzQC+nQhe8FUZQTXQy6+I75skNgn3OoQ0DZA8YnFa7gp8tqtL3KPf1kmo0R5DoApeSGQ==",
+ "optional": true,
+ "engines": {
+ "node": ">=0.10.0"
+ }
+ },
+ "node_modules/linkify-it": {
+ "version": "1.2.4",
+ "resolved": "https://registry.npmjs.org/linkify-it/-/linkify-it-1.2.4.tgz",
+ "integrity": "sha512-eGHwtlABkp1NOJSiKUNqBf3SYAS5jPHtvRXPAgNaQwTqmkTahjtiLH9NtxdR5IOPhNvwNMN/diswSfZKzUkhGg==",
+ "dependencies": {
+ "uc.micro": "^1.0.1"
+ }
+ },
+ "node_modules/lodash": {
+ "version": "3.10.1",
+ "resolved": "https://registry.npmjs.org/lodash/-/lodash-3.10.1.tgz",
+ "integrity": "sha512-9mDDwqVIma6OZX79ZlDACZl8sBm0TEnkf99zV3iMA4GzkIT/9hiqP5mY0HoT1iNLCrKc/R1HByV+yJfRWVJryQ=="
+ },
+ "node_modules/markdown-it": {
+ "version": "6.1.1",
+ "resolved": "https://registry.npmjs.org/markdown-it/-/markdown-it-6.1.1.tgz",
+ "integrity": "sha512-woFl7h/sqt9xRmiMweNuO7nu+w8Lz3SXsDlvE3TYeu1SdPqQ+VW4GZyaKP442Bq6XUN6V6IQjJTR93RDYG2mjw==",
+ "dependencies": {
+ "argparse": "^1.0.7",
+ "entities": "~1.1.1",
+ "linkify-it": "~1.2.2",
+ "mdurl": "~1.0.1",
+ "uc.micro": "^1.0.1"
+ },
+ "bin": {
+ "markdown-it": "bin/markdown-it.js"
+ }
+ },
+ "node_modules/marked": {
+ "version": "0.3.19",
+ "resolved": "https://registry.npmjs.org/marked/-/marked-0.3.19.tgz",
+ "integrity": "sha512-ea2eGWOqNxPcXv8dyERdSr/6FmzvWwzjMxpfGB/sbMccXoct+xY+YukPD+QTUZwyvK7BZwcr4m21WBOW41pAkg==",
+ "bin": {
+ "marked": "bin/marked"
+ },
+ "engines": {
+ "node": ">=0.10.0"
+ }
+ },
+ "node_modules/mdurl": {
+ "version": "1.0.1",
+ "resolved": "https://registry.npmjs.org/mdurl/-/mdurl-1.0.1.tgz",
+ "integrity": "sha512-/sKlQJCBYVY9Ers9hqzKou4H6V5UWc/M59TH2dvkt+84itfnq7uFOMLpOiOS4ujvHP4etln18fmIxA5R5fll0g=="
+ },
+ "node_modules/mime": {
+ "version": "1.6.0",
+ "resolved": "https://registry.npmjs.org/mime/-/mime-1.6.0.tgz",
+ "integrity": "sha512-x0Vn8spI+wuJ1O6S7gnbaQg8Pxh4NNHb7KSINmEWKiPE4RKOplvijn+NkmYmmRgP68mc70j2EbeTFRsrswaQeg==",
+ "optional": true,
+ "bin": {
+ "mime": "cli.js"
+ },
+ "engines": {
+ "node": ">=4"
+ }
+ },
+ "node_modules/mime-db": {
+ "version": "1.52.0",
+ "resolved": "https://registry.npmjs.org/mime-db/-/mime-db-1.52.0.tgz",
+ "integrity": "sha512-sPU4uV7dYlvtWJxwwxHD0PuihVNiE7TyAbQ5SWxDCB9mUYvOgroQOwYQQOKPJ8CIbE+1ETVlOoK1UC2nU3gYvg==",
+ "optional": true,
+ "engines": {
+ "node": ">= 0.6"
+ }
+ },
+ "node_modules/mime-types": {
+ "version": "2.1.35",
+ "resolved": "https://registry.npmjs.org/mime-types/-/mime-types-2.1.35.tgz",
+ "integrity": "sha512-ZDY+bPm5zTTF+YpCrAU9nK0UgICYPT0QtT1NZWFv4s++TNkcgVaT0g6+4R2uI4MjQjzysHB1zxuWL50hzaeXiw==",
+ "optional": true,
+ "dependencies": {
+ "mime-db": "1.52.0"
+ },
+ "engines": {
+ "node": ">= 0.6"
+ }
+ },
+ "node_modules/minimist": {
+ "version": "1.2.8",
+ "resolved": "https://registry.npmjs.org/minimist/-/minimist-1.2.8.tgz",
+ "integrity": "sha512-2yyAR8qBkN3YuheJanUpWC5U3bb5osDywNB8RzDVlDwDHbocAJveqqj1u8+SVD7jkWT4yvsHCpWqqWqAxb0zCA==",
+ "funding": {
+ "url": "https://github.com/sponsors/ljharb"
+ }
+ },
+ "node_modules/mkdirp": {
+ "version": "0.5.6",
+ "resolved": "https://registry.npmjs.org/mkdirp/-/mkdirp-0.5.6.tgz",
+ "integrity": "sha512-FP+p8RB8OWpF3YZBCrP5gtADmtXApB5AMLn+vdyA+PyxCjrCs00mjyUozssO33cwDeT3wNGdLxJ5M//YqtHAJw==",
+ "dependencies": {
+ "minimist": "^1.2.6"
+ },
+ "bin": {
+ "mkdirp": "bin/cmd.js"
+ }
+ },
+ "node_modules/oauth-sign": {
+ "version": "0.8.2",
+ "resolved": "https://registry.npmjs.org/oauth-sign/-/oauth-sign-0.8.2.tgz",
+ "integrity": "sha512-VlF07iu3VV3+BTXj43Nmp6Irt/G7j/NgEctUS6IweH1RGhURjjCc2NWtzXFPXXWWfc7hgbXQdtiQu2LGp6MxUg==",
+ "optional": true,
+ "engines": {
+ "node": "*"
+ }
+ },
+ "node_modules/object-keys": {
+ "version": "1.1.1",
+ "resolved": "https://registry.npmjs.org/object-keys/-/object-keys-1.1.1.tgz",
+ "integrity": "sha512-NuAESUOUMrlIXOfHKzD6bpPu3tYt3xvjNdRIQ+FeT0lNb4K8WR70CaDxhuNguS2XG+GjkyMwOzsN5ZktImfhLA==",
+ "optional": true,
+ "engines": {
+ "node": ">= 0.4"
+ }
+ },
+ "node_modules/performance-now": {
+ "version": "0.2.0",
+ "resolved": "https://registry.npmjs.org/performance-now/-/performance-now-0.2.0.tgz",
+ "integrity": "sha512-YHk5ez1hmMR5LOkb9iJkLKqoBlL7WD5M8ljC75ZfzXriuBIVNuecaXuU7e+hOwyqf24Wxhh7Vxgt7Hnw9288Tg==",
+ "optional": true
+ },
+ "node_modules/promise": {
+ "version": "7.3.1",
+ "resolved": "https://registry.npmjs.org/promise/-/promise-7.3.1.tgz",
+ "integrity": "sha512-nolQXZ/4L+bP/UGlkfaIujX9BKxGwmQ9OT4mOt5yvy8iK1h3wqTEJCijzGANTCCl9nWjY41juyAn2K3Q1hLLTg==",
+ "optional": true,
+ "dependencies": {
+ "asap": "~2.0.3"
+ }
+ },
+ "node_modules/prr": {
+ "version": "1.0.1",
+ "resolved": "https://registry.npmjs.org/prr/-/prr-1.0.1.tgz",
+ "integrity": "sha512-yPw4Sng1gWghHQWj0B3ZggWUm4qVbPwPFcRG8KyxiU7J2OHFSoEHKS+EZ3fv5l1t9CyCiop6l/ZYeWbrgoQejw==",
+ "optional": true
+ },
+ "node_modules/punycode": {
+ "version": "1.4.1",
+ "resolved": "https://registry.npmjs.org/punycode/-/punycode-1.4.1.tgz",
+ "integrity": "sha512-jmYNElW7yvO7TV33CjSmvSiE2yco3bV2czu/OzDKdMNVZQWfxCblURLhf+47syQRBntjfLdd/H0egrzIG+oaFQ==",
+ "optional": true
+ },
+ "node_modules/qs": {
+ "version": "6.4.1",
+ "resolved": "https://registry.npmjs.org/qs/-/qs-6.4.1.tgz",
+ "integrity": "sha512-LQy1Q1fcva/UsnP/6Iaa4lVeM49WiOitu2T4hZCyA/elLKu37L99qcBJk4VCCk+rdLvnMzfKyiN3SZTqdAZGSQ==",
+ "optional": true,
+ "engines": {
+ "node": ">=0.6"
+ }
+ },
+ "node_modules/repeating": {
+ "version": "2.0.1",
+ "resolved": "https://registry.npmjs.org/repeating/-/repeating-2.0.1.tgz",
+ "integrity": "sha512-ZqtSMuVybkISo2OWvqvm7iHSWngvdaW3IpsT9/uP8v4gMi591LY6h35wdOfvQdWCKFWZWm2Y1Opp4kV7vQKT6A==",
+ "dependencies": {
+ "is-finite": "^1.0.0"
+ },
+ "engines": {
+ "node": ">=0.10.0"
+ }
+ },
+ "node_modules/request": {
+ "version": "2.81.0",
+ "resolved": "https://registry.npmjs.org/request/-/request-2.81.0.tgz",
+ "integrity": "sha512-IZnsR7voF0miGSu29EXPRgPTuEsI/+aibNSBbN1pplrfartF5wDYGADz3iD9vmBVf2r00rckWZf8BtS5kk7Niw==",
+ "deprecated": "request has been deprecated, see https://github.com/request/request/issues/3142",
+ "optional": true,
+ "dependencies": {
+ "aws-sign2": "~0.6.0",
+ "aws4": "^1.2.1",
+ "caseless": "~0.12.0",
+ "combined-stream": "~1.0.5",
+ "extend": "~3.0.0",
+ "forever-agent": "~0.6.1",
+ "form-data": "~2.1.1",
+ "har-validator": "~4.2.1",
+ "hawk": "~3.1.3",
+ "http-signature": "~1.1.0",
+ "is-typedarray": "~1.0.0",
+ "isstream": "~0.1.2",
+ "json-stringify-safe": "~5.0.1",
+ "mime-types": "~2.1.7",
+ "oauth-sign": "~0.8.1",
+ "performance-now": "^0.2.0",
+ "qs": "~6.4.0",
+ "safe-buffer": "^5.0.1",
+ "stringstream": "~0.0.4",
+ "tough-cookie": "~2.3.0",
+ "tunnel-agent": "^0.6.0",
+ "uuid": "^3.0.0"
+ },
+ "engines": {
+ "node": ">= 4"
+ }
+ },
+ "node_modules/resolve-url": {
+ "version": "0.2.1",
+ "resolved": "https://registry.npmjs.org/resolve-url/-/resolve-url-0.2.1.tgz",
+ "integrity": "sha512-ZuF55hVUQaaczgOIwqWzkEcEidmlD/xl44x1UZnhOXcYuFN2S6+rcxpG+C1N3So0wvNI3DmJICUFfu2SxhBmvg==",
+ "deprecated": "https://github.com/lydell/resolve-url#deprecated"
+ },
+ "node_modules/safe-buffer": {
+ "version": "5.2.1",
+ "resolved": "https://registry.npmjs.org/safe-buffer/-/safe-buffer-5.2.1.tgz",
+ "integrity": "sha512-rp3So07KcdmmKbGvgaNxQSJr7bGVSVk5S9Eq1F+ppbRo70+YeaDxkw5Dd8NPN+GD6bjnYm2VuPuCXmpuYvmCXQ==",
+ "funding": [
+ {
+ "type": "github",
+ "url": "https://github.com/sponsors/feross"
+ },
+ {
+ "type": "patreon",
+ "url": "https://www.patreon.com/feross"
+ },
+ {
+ "type": "consulting",
+ "url": "https://feross.org/support"
+ }
+ ],
+ "optional": true
+ },
+ "node_modules/safefs": {
+ "version": "4.2.0",
+ "resolved": "https://registry.npmjs.org/safefs/-/safefs-4.2.0.tgz",
+ "integrity": "sha512-1amPBO92jw/hWS+gH/u7z7EL7YxaJ8WecBQl49tMQ6Y6EQfndxNNKwlPqDOcwpUetdmK6nKLoVdjybVScRwq5A==",
+ "dependencies": {
+ "editions": "^2.2.0",
+ "graceful-fs": "^4.2.3"
+ },
+ "engines": {
+ "node": ">=0.12"
+ },
+ "funding": {
+ "type": "cooperative",
+ "url": "https://bevry.me/fund"
+ }
+ },
+ "node_modules/safeps": {
+ "version": "7.0.1",
+ "resolved": "https://registry.npmjs.org/safeps/-/safeps-7.0.1.tgz",
+ "integrity": "sha512-aFREKZzceHZH3KZTwjhDI1oOOcyAEBcQHjImJS/Mmx+KC31EQCgwiPKfwhJLBX7R4Y5ioI2D/VEcQ6U6ya2MJw==",
+ "dependencies": {
+ "editions": "^1.3.3",
+ "extract-opts": "^3.3.1",
+ "safefs": "^4.1.0",
+ "taskgroup": "^5.0.0",
+ "typechecker": "^4.3.0"
+ },
+ "engines": {
+ "node": ">=0.8"
+ }
+ },
+ "node_modules/safeps/node_modules/editions": {
+ "version": "1.3.4",
+ "resolved": "https://registry.npmjs.org/editions/-/editions-1.3.4.tgz",
+ "integrity": "sha512-gzao+mxnYDzIysXKMQi/+M1mjy/rjestjg6OPoYTtI+3Izp23oiGZitsl9lPDPiTGXbcSIk1iJWhliSaglxnUg==",
+ "engines": {
+ "node": ">=0.8"
+ }
+ },
+ "node_modules/safer-buffer": {
+ "version": "2.1.2",
+ "resolved": "https://registry.npmjs.org/safer-buffer/-/safer-buffer-2.1.2.tgz",
+ "integrity": "sha512-YZo3K82SD7Riyi0E1EQPojLz7kpepnSQI9IyPbHHg1XXXevb5dJI7tpyN2ADxGcQbHG7vcyRHk0cbwqcQriUtg==",
+ "optional": true
+ },
+ "node_modules/scandirectory": {
+ "version": "2.5.0",
+ "resolved": "https://registry.npmjs.org/scandirectory/-/scandirectory-2.5.0.tgz",
+ "integrity": "sha512-uT0CW8Z3YyoIQs2gXIZgR5miLkN/UNl+5IptQIq1YfD2NhFldikYlC3dkOE6MvF15OZMOxjg8yOjx5J/vIIPUA==",
+ "dependencies": {
+ "ignorefs": "^1.0.0",
+ "safefs": "^3.1.2",
+ "taskgroup": "^4.0.5"
+ },
+ "engines": {
+ "node": ">=0.4"
+ }
+ },
+ "node_modules/scandirectory/node_modules/ambi": {
+ "version": "2.5.0",
+ "resolved": "https://registry.npmjs.org/ambi/-/ambi-2.5.0.tgz",
+ "integrity": "sha512-5nS0gYMPNgZz/UALDHMStcwO42youpIWBQVbI92vV5j0+2bMxv/iVqearrLu3/f0XaU6xVIbf3RRtDxOcHxSkw==",
+ "dependencies": {
+ "editions": "^1.1.1",
+ "typechecker": "^4.3.0"
+ },
+ "engines": {
+ "node": ">=0.12"
+ }
+ },
+ "node_modules/scandirectory/node_modules/editions": {
+ "version": "1.3.4",
+ "resolved": "https://registry.npmjs.org/editions/-/editions-1.3.4.tgz",
+ "integrity": "sha512-gzao+mxnYDzIysXKMQi/+M1mjy/rjestjg6OPoYTtI+3Izp23oiGZitsl9lPDPiTGXbcSIk1iJWhliSaglxnUg==",
+ "engines": {
+ "node": ">=0.8"
+ }
+ },
+ "node_modules/scandirectory/node_modules/safefs": {
+ "version": "3.2.2",
+ "resolved": "https://registry.npmjs.org/safefs/-/safefs-3.2.2.tgz",
+ "integrity": "sha512-qqvuS8qslGUSgUKQbdsYIK8Qg0EAkykxlsdfy3jpBSnhtyPsee/8y4RLc5+3CD6TgazBmtT0ekoGicUTPzICdg==",
+ "dependencies": {
+ "graceful-fs": "*"
+ },
+ "engines": {
+ "node": ">=0.4"
+ }
+ },
+ "node_modules/scandirectory/node_modules/taskgroup": {
+ "version": "4.3.1",
+ "resolved": "https://registry.npmjs.org/taskgroup/-/taskgroup-4.3.1.tgz",
+ "integrity": "sha512-PD97E2OfwFH7SgeVRvR6K2c+NkKXZSwMMTdcM1t/3P+f70DUWbR81Qx7TF7dJj8dV631u4dhdBmhfDQjIZvGsg==",
+ "dependencies": {
+ "ambi": "^2.2.0",
+ "csextends": "^1.0.3"
+ },
+ "engines": {
+ "node": ">=0.8"
+ }
+ },
+ "node_modules/semver": {
+ "version": "6.3.1",
+ "resolved": "https://registry.npmjs.org/semver/-/semver-6.3.1.tgz",
+ "integrity": "sha512-BR7VvDCVHO+q2xBEWskxS6DJE1qRnb7DxzUrogb71CWoSficBxYsiAGd+Kl0mmq/MprG9yArRkyrQxTO6XjMzA==",
+ "bin": {
+ "semver": "bin/semver.js"
+ }
+ },
+ "node_modules/set-function-length": {
+ "version": "1.1.1",
+ "resolved": "https://registry.npmjs.org/set-function-length/-/set-function-length-1.1.1.tgz",
+ "integrity": "sha512-VoaqjbBJKiWtg4yRcKBQ7g7wnGnLV3M8oLvVWwOk2PdYY6PEFegR1vezXR0tw6fZGF9csVakIRjrJiy2veSBFQ==",
+ "optional": true,
+ "dependencies": {
+ "define-data-property": "^1.1.1",
+ "get-intrinsic": "^1.2.1",
+ "gopd": "^1.0.1",
+ "has-property-descriptors": "^1.0.0"
+ },
+ "engines": {
+ "node": ">= 0.4"
+ }
+ },
+ "node_modules/slug": {
+ "version": "0.4.2",
+ "resolved": "https://registry.npmjs.org/slug/-/slug-0.4.2.tgz",
+ "integrity": "sha512-HQRxdDjtXsKG1pw8rBXGRq9fdW2fS2xPaizvJ3MK89x9+V8U0Z8//meWzJUdFW52pFDGqkLfyX+Fij7lkRY6Kw==",
+ "dependencies": {
+ "unicode": ">= 0.3.1"
+ },
+ "engines": {
+ "node": ">= 0.4.x"
+ }
+ },
+ "node_modules/sntp": {
+ "version": "1.0.9",
+ "resolved": "https://registry.npmjs.org/sntp/-/sntp-1.0.9.tgz",
+ "integrity": "sha512-7bgVOAnPj3XjrKY577S+puCKGCRlUrcrEdsMeRXlg9Ghf5df/xNi6sONUa43WrHUd3TjJBF7O04jYoiY0FVa0A==",
+ "deprecated": "This module moved to @hapi/sntp. Please make sure to switch over as this distribution is no longer supported and may contain bugs and critical security issues.",
+ "optional": true,
+ "dependencies": {
+ "hoek": "2.x.x"
+ },
+ "engines": {
+ "node": ">=0.8.0"
+ }
+ },
+ "node_modules/source-map": {
+ "version": "0.6.1",
+ "resolved": "https://registry.npmjs.org/source-map/-/source-map-0.6.1.tgz",
+ "integrity": "sha512-UjgapumWlbMhkBgzT7Ykc5YXUT46F0iKu8SGXq0bcwP5dz/h0Plj6enJqjz1Zbq2l5WaqYnrVbwWOWMyF3F47g==",
+ "engines": {
+ "node": ">=0.10.0"
+ }
+ },
+ "node_modules/source-map-resolve": {
+ "version": "0.5.3",
+ "resolved": "https://registry.npmjs.org/source-map-resolve/-/source-map-resolve-0.5.3.tgz",
+ "integrity": "sha512-Htz+RnsXWk5+P2slx5Jh3Q66vhQj1Cllm0zvnaY98+NFx+Dv2CF/f5O/t8x+KaNdrdIAsruNzoh/KpialbqAnw==",
+ "deprecated": "See https://github.com/lydell/source-map-resolve#deprecated",
+ "dependencies": {
+ "atob": "^2.1.2",
+ "decode-uri-component": "^0.2.0",
+ "resolve-url": "^0.2.1",
+ "source-map-url": "^0.4.0",
+ "urix": "^0.1.0"
+ }
+ },
+ "node_modules/source-map-url": {
+ "version": "0.4.1",
+ "resolved": "https://registry.npmjs.org/source-map-url/-/source-map-url-0.4.1.tgz",
+ "integrity": "sha512-cPiFOTLUKvJFIg4SKVScy4ilPPW6rFgMgfuZJPNoDuMs3nC1HbMUycBoJw77xFIp6z1UJQJOfx6C9GMH80DiTw==",
+ "deprecated": "See https://github.com/lydell/source-map-url#deprecated"
+ },
+ "node_modules/sprintf-js": {
+ "version": "1.0.3",
+ "resolved": "https://registry.npmjs.org/sprintf-js/-/sprintf-js-1.0.3.tgz",
+ "integrity": "sha512-D9cPgkvLlV3t3IzL0D0YLvGA9Ahk4PcvVwUbN0dSGr1aP0Nrt4AEnTUbuGvquEC0mA64Gqt1fzirlRs5ibXx8g=="
+ },
+ "node_modules/sshpk": {
+ "version": "1.18.0",
+ "resolved": "https://registry.npmjs.org/sshpk/-/sshpk-1.18.0.tgz",
+ "integrity": "sha512-2p2KJZTSqQ/I3+HX42EpYOa2l3f8Erv8MWKsy2I9uf4wA7yFIkXRffYdsx86y6z4vHtV8u7g+pPlr8/4ouAxsQ==",
+ "optional": true,
+ "dependencies": {
+ "asn1": "~0.2.3",
+ "assert-plus": "^1.0.0",
+ "bcrypt-pbkdf": "^1.0.0",
+ "dashdash": "^1.12.0",
+ "ecc-jsbn": "~0.1.1",
+ "getpass": "^0.1.1",
+ "jsbn": "~0.1.0",
+ "safer-buffer": "^2.0.2",
+ "tweetnacl": "~0.14.0"
+ },
+ "bin": {
+ "sshpk-conv": "bin/sshpk-conv",
+ "sshpk-sign": "bin/sshpk-sign",
+ "sshpk-verify": "bin/sshpk-verify"
+ },
+ "engines": {
+ "node": ">=0.10.0"
+ }
+ },
+ "node_modules/sshpk/node_modules/assert-plus": {
+ "version": "1.0.0",
+ "resolved": "https://registry.npmjs.org/assert-plus/-/assert-plus-1.0.0.tgz",
+ "integrity": "sha512-NfJ4UzBCcQGLDlQq7nHxH+tv3kyZ0hHQqF5BO6J7tNJeP5do1llPr8dZ8zHonfhAu0PHAdMkSo+8o0wxg9lZWw==",
+ "optional": true,
+ "engines": {
+ "node": ">=0.8"
+ }
+ },
+ "node_modules/stringstream": {
+ "version": "0.0.6",
+ "resolved": "https://registry.npmjs.org/stringstream/-/stringstream-0.0.6.tgz",
+ "integrity": "sha512-87GEBAkegbBcweToUrdzf3eLhWNg06FJTebl4BVJz/JgWy8CvEr9dRtX5qWphiynMSQlxxi+QqN0z5T32SLlhA==",
+ "optional": true
+ },
+ "node_modules/strip-indent": {
+ "version": "2.0.0",
+ "resolved": "https://registry.npmjs.org/strip-indent/-/strip-indent-2.0.0.tgz",
+ "integrity": "sha512-RsSNPLpq6YUL7QYy44RnPVTn/lcVZtb48Uof3X5JLbF4zD/Gs7ZFDv2HWol+leoQN2mT86LAzSshGfkTlSOpsA==",
+ "engines": {
+ "node": ">=4"
+ }
+ },
+ "node_modules/taskgroup": {
+ "version": "5.5.0",
+ "resolved": "https://registry.npmjs.org/taskgroup/-/taskgroup-5.5.0.tgz",
+ "integrity": "sha512-YFkdc6HU+p3xO2lZ1MWdx7R7EbrLF/bpXv5k9635bTzdgOLNbmnsDg5alSpZost+PYMk40d6ZDAJHBHNHiiLvw==",
+ "dependencies": {
+ "ambi": "3.2.0",
+ "eachr": "^3.2.0",
+ "editions": "^2.2.0",
+ "extendr": "^3.5.0",
+ "safeps": "7.0.1",
+ "unbounded": "^1.2.0"
+ },
+ "engines": {
+ "node": ">=0.8"
+ },
+ "funding": {
+ "type": "cooperative",
+ "url": "https://bevry.me/fund"
+ }
+ },
+ "node_modules/toc": {
+ "version": "0.4.0",
+ "resolved": "https://registry.npmjs.org/toc/-/toc-0.4.0.tgz",
+ "integrity": "sha512-Z4MqUbtLQrbJLQQFLKK0g5tGmke0vqB8puHrXXgRfPyLJTcsn5ACy/uxVnMrg6wSWPoS2hvVpw6wSAFYAkAEVA==",
+ "dependencies": {
+ "entities": "~0.5.0",
+ "lodash": "~2.4.1",
+ "slug": "~0.4.2"
+ },
+ "engines": {
+ "node": ">= 0.8.0"
+ }
+ },
+ "node_modules/toc/node_modules/entities": {
+ "version": "0.5.0",
+ "resolved": "https://registry.npmjs.org/entities/-/entities-0.5.0.tgz",
+ "integrity": "sha512-T5XQtlzuW+PfeSsGp3uryfYQof820zYbnUnUDEkwUVIAfgYeixIN16c4jh8gs0SqJUTGLU0XD6QsvjEPbmdwzQ=="
+ },
+ "node_modules/toc/node_modules/lodash": {
+ "version": "2.4.2",
+ "resolved": "https://registry.npmjs.org/lodash/-/lodash-2.4.2.tgz",
+ "integrity": "sha512-Kak1hi6/hYHGVPmdyiZijoQyz5x2iGVzs6w9GYB/HiXEtylY7tIoYEROMjvM1d9nXJqPOrG2MNPMn01bJ+S0Rw==",
+ "engines": [
+ "node",
+ "rhino"
+ ]
+ },
+ "node_modules/tough-cookie": {
+ "version": "2.3.4",
+ "resolved": "https://registry.npmjs.org/tough-cookie/-/tough-cookie-2.3.4.tgz",
+ "integrity": "sha512-TZ6TTfI5NtZnuyy/Kecv+CnoROnyXn2DN97LontgQpCwsX2XyLYCC0ENhYkehSOwAp8rTQKc/NUIF7BkQ5rKLA==",
+ "optional": true,
+ "dependencies": {
+ "punycode": "^1.4.1"
+ },
+ "engines": {
+ "node": ">=0.8"
+ }
+ },
+ "node_modules/tunnel-agent": {
+ "version": "0.6.0",
+ "resolved": "https://registry.npmjs.org/tunnel-agent/-/tunnel-agent-0.6.0.tgz",
+ "integrity": "sha512-McnNiV1l8RYeY8tBgEpuodCC1mLUdbSN+CYBL7kJsJNInOP8UjDDEwdk6Mw60vdLLrr5NHKZhMAOSrR2NZuQ+w==",
+ "optional": true,
+ "dependencies": {
+ "safe-buffer": "^5.0.1"
+ },
+ "engines": {
+ "node": "*"
+ }
+ },
+ "node_modules/tweetnacl": {
+ "version": "0.14.5",
+ "resolved": "https://registry.npmjs.org/tweetnacl/-/tweetnacl-0.14.5.tgz",
+ "integrity": "sha512-KXXFFdAbFXY4geFIwoyNK+f5Z1b7swfXABfL7HXCmoIWMKU3dmS26672A4EeQtDzLKy7SXmfBu51JolvEKwtGA==",
+ "optional": true
+ },
+ "node_modules/typechecker": {
+ "version": "4.11.0",
+ "resolved": "https://registry.npmjs.org/typechecker/-/typechecker-4.11.0.tgz",
+ "integrity": "sha512-lz39Mc/d1UBcF/uQFL5P8L+oWdIn/stvkUgHf0tPRW4aEwGGErewNXo2Nb6We2WslWifn00rhcHbbRWRcTGhuw==",
+ "dependencies": {
+ "editions": "^2.2.0"
+ },
+ "engines": {
+ "node": ">=0.8"
+ },
+ "funding": {
+ "url": "https://bevry.me/fund"
+ }
+ },
+ "node_modules/uc.micro": {
+ "version": "1.0.6",
+ "resolved": "https://registry.npmjs.org/uc.micro/-/uc.micro-1.0.6.tgz",
+ "integrity": "sha512-8Y75pvTYkLJW2hWQHXxoqRgV7qb9B+9vFEtidML+7koHUFapnVJAZ6cKs+Qjz5Aw3aZWHMC6u0wJE3At+nSGwA=="
+ },
+ "node_modules/unbounded": {
+ "version": "1.3.0",
+ "resolved": "https://registry.npmjs.org/unbounded/-/unbounded-1.3.0.tgz",
+ "integrity": "sha512-RWVCkvcoItljlNTz0iTdBQU6bDj+slVLNaWN7d6DXgH02FfYrz8ytcJ4OPW8b0HqmCehwufJHOIzjHWrQUXBvg==",
+ "dependencies": {
+ "editions": "^2.2.0"
+ },
+ "engines": {
+ "node": ">=0.8"
+ },
+ "funding": {
+ "url": "https://bevry.me/fund"
+ }
+ },
+ "node_modules/unicode": {
+ "version": "14.0.0",
+ "resolved": "https://registry.npmjs.org/unicode/-/unicode-14.0.0.tgz",
+ "integrity": "sha512-BjinxTXkbm9Jomp/YBTMGusr4fxIG67fNGShHIRAL16Ur2GJTq2xvLi+sxuiJmInCmwqqev2BCFKyvbfp/yAkg==",
+ "engines": {
+ "node": ">= 0.8.x"
+ }
+ },
+ "node_modules/urix": {
+ "version": "0.1.0",
+ "resolved": "https://registry.npmjs.org/urix/-/urix-0.1.0.tgz",
+ "integrity": "sha512-Am1ousAhSLBeB9cG/7k7r2R0zj50uDRlZHPGbazid5s9rlF1F/QKYObEKSIunSjIOkJZqwRRLpvewjEkM7pSqg==",
+ "deprecated": "Please see https://github.com/lydell/urix#deprecated"
+ },
+ "node_modules/uuid": {
+ "version": "3.4.0",
+ "resolved": "https://registry.npmjs.org/uuid/-/uuid-3.4.0.tgz",
+ "integrity": "sha512-HjSDRw6gZE5JMggctHBcjVak08+KEVhSIiDzFnT9S9aegmp85S/bReBVTb4QTFaRNptJ9kuYaNhnbNEOkbKb/A==",
+ "deprecated": "Please upgrade to version 7 or higher. Older versions may use Math.random() in certain circumstances, which is known to be problematic. See https://v8.dev/blog/math-random for details.",
+ "optional": true,
+ "bin": {
+ "uuid": "bin/uuid"
+ }
+ },
+ "node_modules/verror": {
+ "version": "1.10.0",
+ "resolved": "https://registry.npmjs.org/verror/-/verror-1.10.0.tgz",
+ "integrity": "sha512-ZZKSmDAEFOijERBLkmYfJ+vmk3w+7hOLYDNkRCuRuMJGEmqYNCNLyBBFwWKVMhfwaEF3WOd0Zlw86U/WC/+nYw==",
+ "engines": [
+ "node >=0.6.0"
+ ],
+ "optional": true,
+ "dependencies": {
+ "assert-plus": "^1.0.0",
+ "core-util-is": "1.0.2",
+ "extsprintf": "^1.2.0"
+ }
+ },
+ "node_modules/verror/node_modules/assert-plus": {
+ "version": "1.0.0",
+ "resolved": "https://registry.npmjs.org/assert-plus/-/assert-plus-1.0.0.tgz",
+ "integrity": "sha512-NfJ4UzBCcQGLDlQq7nHxH+tv3kyZ0hHQqF5BO6J7tNJeP5do1llPr8dZ8zHonfhAu0PHAdMkSo+8o0wxg9lZWw==",
+ "optional": true,
+ "engines": {
+ "node": ">=0.8"
+ }
+ },
+ "node_modules/watchr": {
+ "version": "2.6.0",
+ "resolved": "https://registry.npmjs.org/watchr/-/watchr-2.6.0.tgz",
+ "integrity": "sha512-eHqnPA71jn+lLf/c49mjXqQzzwKLmDdLZXiB53PtgBY8X75zqUWL2PmJWjJ45Bcy8PHOMDdVUCLEud36Lk5QZQ==",
+ "dependencies": {
+ "eachr": "^3.2.0",
+ "extendr": "^3.2.2",
+ "extract-opts": "^3.3.1",
+ "ignorefs": "^1.1.1",
+ "safefs": "^4.1.0",
+ "scandirectory": "^2.5.0",
+ "taskgroup": "^5.0.1",
+ "typechecker": "^4.3.0"
+ },
+ "bin": {
+ "watchr": "bin/watchr"
+ },
+ "engines": {
+ "node": ">=0.12"
+ }
+ }
+ }
+}
diff --git a/rootfs/agents/AgentAssistant/agent.toml b/rootfs/agents/AgentAssistant/agent.toml
new file mode 100644
index 0000000..770bc82
--- /dev/null
+++ b/rootfs/agents/AgentAssistant/agent.toml
@@ -0,0 +1,32 @@
+instance_id = "AgentAssistant"
+fullname = "AgentAssistant"
+llm_model_name = "gpt-4-1106-preview"
+owner_env = "environment.py"
+
+[[prompt]]
+role = "system"
+content = '''
+You are an AI Agent template generation assistant that can help users better define AI Agent templates.
+AI Agent can use AI to help users complete relevant tasks based on prompt words.
+
+Please help users define AI Agent templates according to the following rules:
+New generation:
+1. Generate prompt words for users based on user questions. Prompt word suggestions can be given directly based on user input until the user confirms the prompt words.
+2. Generate instance_id and fullname based on the final prompt word content.
+3. Generate the following toml format data and return it to the user.
+```
+instance_id = "agent id"
+fullname = "agent full name"
+[[prompt]]
+role = "system"
+content = """Prompt words"""
+```
+4. After the configuration is generated, the user is supported to modify the instance_id, fullname and prompt words. After the user has modified it, toml format data needs to be regenerated.
+5. The save_agent function is called only when the user confirms the save. Do not call it when there is no input to save.
+Modify existing agent:
+1. The user enters the name of the agent that needs to be opened, and the system calls the read_agent function to read the agent configuration and return it to the user.
+2. Support users to modify the fullname and prompt words, but cannot modify the instance_id. After the user has modified it, toml format data needs to be regenerated.
+3. The save_agent function is called only when the user confirms the save. Do not call it when there is no input to save.
+
+PS: You need to communicate in the same language as the user input.
+'''
diff --git a/rootfs/agents/AgentAssistant/environment.py b/rootfs/agents/AgentAssistant/environment.py
new file mode 100644
index 0000000..e5ce961
--- /dev/null
+++ b/rootfs/agents/AgentAssistant/environment.py
@@ -0,0 +1,60 @@
+import logging
+from typing import Optional
+
+import toml
+
+import os
+
+from aios.agent.ai_function import SimpleAIFunction
+from aios.environment.environment import Environment
+
+local_path = os.path.split(os.path.realpath(__file__))[0]
+
+logger = logging.getLogger(__name__)
+
+
+class AgentAssistantEnvironment(Environment):
+ def __init__(self):
+ super().__init__("agent_assistant_env")
+ self.add_ai_function(SimpleAIFunction("read_agent",
+ "Read the agent with the specified name",
+ self.read_agent,
+ {"agent_id": "The id of the agent to be read"}))
+ self.add_ai_function(SimpleAIFunction("save_agent",
+ "save the agent with the specified name",
+ self.save_agent,
+ {"agent_id": "The id of the agent to be saved",
+ "agent_data": "The toml data of the agent to be saved",
+ "is_new": "Whether to create a new agent, The value is true if it is created, and it is false if it is modified."}))
+
+ def _do_get_value(self, key: str) -> Optional[str]:
+ pass
+
+ async def read_agent(self, agent_id: str) -> str:
+ agent_dir = os.path.join(local_path, "..", agent_id)
+ if not os.path.exists(agent_dir):
+ return "exec failed.agent is not exists."
+
+ with open(os.path.join(agent_dir, "agent.toml"), "r", encoding='utf-8') as f:
+ agent_data = f.read()
+
+ return "exec success. agent data:\n" + agent_data
+
+ async def save_agent(self, agent_id: str, agent_data: str, is_new: str) -> str:
+ logger.info(f"save_agent: {agent_id} {agent_data}")
+
+ agent_dir = os.path.join(local_path, "..", agent_id)
+ if os.path.exists(agent_dir) and is_new == "true":
+ return "exec failed.agent already exists, please change the agent id and try again."
+
+ if not os.path.exists(agent_dir):
+ os.mkdir(agent_dir)
+
+ with open(os.path.join(agent_dir, "agent.toml"), "w", encoding='utf-8') as f:
+ f.write(agent_data)
+
+ return "exec success."
+
+
+def init():
+ return AgentAssistantEnvironment()
diff --git a/rootfs/agents/DBQueryer/agent.toml b/rootfs/agents/DBQueryer/agent.toml
new file mode 100644
index 0000000..b3d0deb
--- /dev/null
+++ b/rootfs/agents/DBQueryer/agent.toml
@@ -0,0 +1,15 @@
+instance_id = "DBQueryer"
+fullname = "database queryer"
+llm_model_name = "gpt-4-1106-preview"
+owner_env = "environment.py"
+
+[[prompt]]
+role = "system"
+content = """You are a SQL database queryer that can help users query the required data from the specified database.
+Please follow the following rules to query data:
+1. The user can provide the database URL to be queried. If the user provides the database URL to be queried, the URL must be entered in the database function called.
+2. If the user's data description does not contain database table related information, please try to query it from the history record. If it is not in the history record, you can call the get_table_infos function to obtain it.
+3. Please generate a query statement based on the relevant information of the database table and call the execute_sql function to execute the query statement. If the generated SQL statement is incorrect, you must requery based on the error correction SQL statement.
+4. Only query statements can be executed, and no update statements can be executed.
+5. You need to communicate in the same language as the user input.
+"""
diff --git a/rootfs/agents/DBQueryer/environment.py b/rootfs/agents/DBQueryer/environment.py
new file mode 100644
index 0000000..fb500b0
--- /dev/null
+++ b/rootfs/agents/DBQueryer/environment.py
@@ -0,0 +1,18 @@
+from typing import Optional
+
+from aios.environment.environment import Environment
+from aios.ai_functions.sql_database_function import GetTableInfosFunction, ExecuteSqlFunction
+
+
+class DBQuerierEnvironment(Environment):
+ def __init__(self):
+ super().__init__("fairy")
+ self.add_ai_function(GetTableInfosFunction())
+ self.add_ai_function(ExecuteSqlFunction())
+
+ def _do_get_value(self, key: str) -> Optional[str]:
+ pass
+
+
+def init():
+ return DBQuerierEnvironment()
diff --git a/rootfs/agents/David/agent.toml b/rootfs/agents/David/agent.toml
index 47d2ff8..0e5341f 100644
--- a/rootfs/agents/David/agent.toml
+++ b/rootfs/agents/David/agent.toml
@@ -6,9 +6,19 @@ owner_env = "paint"
[[prompt]]
role = "system"
-content = """You are an artist, and you will use the 'paint' function in the llm_inner_functions to create artwork.
-You will extract relevant keywords based on user input and requirements, and pass these keywords to the 'prompt' parameter of the 'paint' function.
-When a specific model is mentioned, pass the model's name to the 'model_name' parameter of the 'paint' function.
-If there are instructions not to depict certain content, summarize this content and pass it as keywords to the 'negative_prompt' parameter.
-All parameters must be in English.
-When the user mentions creating a children's picture book, use the "realisticVisionV51_v51VAE" model, and append the keyword " " to the 'prompt' parameter."""
+content = """You are an advanced artist AI, and your task is to understand users' textual descriptions and transform these descriptions into nuanced visual images. When a user provides a description, you need to carefully analyze and comprehend the details, emotions, environment, objects, and style within the description. Then, you will use this information to create an image that accurately and creatively reflects the user's description.
+
+Your creative process is as follows: First, you will receive a description from the user, which could be a depiction of anything, such as a scene, an object, a concept, or an emotional expression. You need to extract key information from these descriptions, such as the subject, atmosphere, color scheme, time period, and artistic style. Then, you will pass these key pieces of information as parameters to the prompt parameter of your "paint" function.
+
+For example, if a user says, "I want a painting that depicts a sunset on a summer beach with children playing and sailboats in the distance, with the sky showing a gradient of orange and purple." You should extract:
+- Time: Summer, sunset moment
+- Scene: Beach
+- Activity: Children playing
+- Objects: Sailboats in the distance
+- Colors: The gradient of orange and purple in the sky
+
+Then you will combine these pieces of information into an accurate prompt to pass to the "paint" function, for example: "During a summer sunset, children play on the beach with sailboats visible in the distance, and the sky displays a gradient of orange and purple."
+
+After successfully creating the image, please clearly state the file path where the image is saved in your response. For instance, you might say, "The image has been successfully created and saved at the following path: /path/to/the/image.png."
+
+Please always ensure that your creations respect the user's intentions and that you infuse your unique artistic style and creativity into your work."""
diff --git a/rootfs/agents/Jarvis/agent.toml b/rootfs/agents/Jarvis/agent.toml
index a69e6fb..419eea7 100644
--- a/rootfs/agents/Jarvis/agent.toml
+++ b/rootfs/agents/Jarvis/agent.toml
@@ -1,31 +1,220 @@
instance_id = "Jarvis"
fullname = "Jarvis"
-llm_model_name = "gpt-3.5-turbo-16k-0613"
-max_token_size = 16000
+max_token = 4000
+#timeout = 1800
+model_name = "gpt-4-turbo-preview"
#enable_kb = "true"
enable_timestamp = "true"
-owner_prompt = "I am your master{name}"
-contact_prompt = "I am your master's friend{name}"
-owner_env = "calender"
+enable_json_resp = "true"
-[[prompt]]
-role = "system"
-content = """
-You are named Jarvis, the super personal assistant to the master.
-You lead a team serving the master, the members of which are:
-Tracy, the private English tutor,
-Mia, the master's personal document management expert.
-
-***
-Sometimes the information you see will carry a timestamp. This is to give you a better understanding of when the message was created. When you reply to messages, you do not include this time stamp.
-
-Upon receiving a message, handle it according to the following rules:
-1. If you believe someone in the team is better suited to address the message, forward the message to them using the method below:
-##/send_msg "MemberName"
-Message content
-2.You can access the master's Calendar to view his schedule. If you need to modify the master's schedule while processing a message, please adjust it using the appropriate method.
-3.Be mindful of the identity of the person you are chatting with and provide services accordingly based on their status.
-4.For messages that don't follow the above rules, do your best to handle them.
+role_desc = """
+Your name is Jarvis, the super personal assistant to the Principal. Help the Principal do a good job of schedule.Reminder before the start of the important schedule, and you should bring useful information as much as possible when reminding.
+Only clearly specifying the task you completed can be completed independently.
"""
+[behavior.on_message]
+type="AgentMessageProcess"
+mutil_model="gpt-4-vision-preview"
+asr_model="openai-whisper"
+tts_model="tts-1"
+
+process_description="""
+1. Based on your role and the existing information, please think and then make a brief and efficient reply.
+2. Be mindful of the identity of the person you are chatting with and provide services accordingly based on their status.
+3. Understand the intention of the dialogue, while using the necessary reply, use the appropriate, supported ACTION.
+4. If you feel that there is a potential Task in the dialogue, you can create these tasks through appropriate ACTION. Be careful to query whether there are the same task before creating. Using the query interface is a high-cost behavior.
+5. You are proficient in the languages of various countries and try to communicate with each other's mother tongue.
+"""
+# Not work: tags: ['tag1', 'tag2'], #Optional,If the conversation involves important things and people, you can mark by 1-3 tags.
+reply_format = """
+The Response must be directly parsed by `python json.loads`. Here is an example:
+{
+ think:'$think step-by-step to be sure you have the right reply.'
+ resp: '$What you want to reply',
+ actions: [{
+ name: '$action_name',
+ $param_name: '$parm' #Optional, fill in only if the action has parameters.
+ }]
+}
+"""
+
+context="The current dialogue occurs in {location}, time: {now}, weather: {weather}."
+
+known_info_tips = """
+"""
+
+tools_tips = """
+"""
+llm_context.actions.enable = ["agent.workspace.create_task","agent.workspace.cancel_task"]
+llm_context.functions.enable = ["agent.workspace.list_task"]
+
+
+[behavior.triage_tasks]
+## 处理任务列表，任务列表里会包含所有未执行过，且未过期的任务
+## 对于简单的任务会一次性完成处理(一系列简单的提醒任务）
+type="AgentTriageTaskList"
+process_description="""
+You are expected to effectively TRIAGE the task list described in JSON format, in accordance with your role. Your GOAL is :
+1. Adjust the priority of the task and set up a reasonable processing time.(confirm_task)
+2. Immediately perform a simple (similar to reminding one category) task. Send a message using post_message, then set the task to complete.(use update_task).
+3. Organize tasks to remove tasks beyond your ability. And merge the repeated tasks.(update_task + cancel_task)
+"""
+reply_format = """
+The Response must be directly parsed by `python json.loads`. Here is an example:
+{
+ think:'$think step-by-step to be sure you can triage tasks well.'
+ resp : '$determine, summary what you do',
+ actions: [{
+ name: '$action_name',
+ $param_name: '$parm' #Optional, fill in only if the action has parameters.
+ }]
+}
+"""
+context="Your Principal is {owner}, now in {location}, time: {now}, weather: {weather}."
+
+llm_context.actions.enable = ["agent.workspace.confirm_task","agent.workspace.update_task","agent.workspace.cancel_task","post_message"]
+
+[behavior.plan_task]
+## 处理任务 Tackling Task
+type="AgentPlanTask"
+process_description="""
+The input is a task comes from a Tasklist. You are Tackling this task. Tackling process in combination with the known information.
+1. Carefully think about whether the task is within your ability, and the task other than the scope of ability is directly rejected. (cancel_task).
+2. Immediately DO a simple (similar to reminding one category) task. Reminds at a reasonable time, Post a message using post_message, then set the task to complete.(use update_task).
+3. Plan for non-simple tasks, and generate a TODO list. Every TODO MUST be an independent work with a clear goal. (set_todos)
+4. If the task has been dealt with, it means that the task is ultimately completed.You need to analyze the processing report of the entire task and make new plans.
+"""
+
+reply_format = """
+The Response must be directly parsed by `python json.loads`. Here is an example:
+{
+ think:'$thinking step by step to ensure the accurate and efficient processing task.',
+ resp:'$determine, summary what you do'
+ actions: [{
+ name: '$action_name',
+ $param_name: '$parm' #Optional, fill in only if the action has parameters.
+ }]
+}
+"""
+
+llm_context.actions.enable = ["agent.workspace.create_task","agent.workspace.update_task","agent.workspace.set_todos","agent.workspace.cancel_task","post_message"]
+#llm_context.functions.enable = ["agent.workspace.list_task"]
+context="Your Principal is {owner}, now in {location}, time: {now}, weather: {weather}."
+
+[behavior.review_task]
+## 当task的所有todo/subtask都完成后（不敢成功或是失败），进行一次review
+type="AgentReviewTask"
+process_description="""
+The input you get is a task has been executed. You need to combine the execution results of the Task's TOOLIST or SUBTASK to review the TASK.
+1. Determine whether TASK has reached the goal.If the goal has been reached, the task is marked with DONE .If the goal is not achieved, simply illustrate the reason and mark TASK as a failure.(update_task)
+2. If task that have not been completed for a long time, the task is marked as the final failure after analyzing the reasons.(cancel_task)
+"""
+
+reply_format = """
+The Response must be directly parsed by `python json.loads`. Here is an example:
+{
+ think:'$think step-by-step to be sure you have the right result.',
+ resp : '$determine, summary what you will do',
+ actions: [{
+ name: '$action_name',
+ $param_name: '$parm' #Optional, fill in only if the action has parameters.
+ }]
+}
+"""
+
+llm_context.actions.enable = ["agent.workspace.cancel_task","agent.workspace.update_task"]
+
+context="Your Principal now in {location}, time: {now}, weather: {weather}."
+[behavior.do]
+# do TODO
+type="AgentDo"
+process_description="""
+The input is a TODO comes from a Task.
+1. Your task is to combine your role definition, tools on hand, known information, and complete a certain Todo.After completing the Todo, you will get a tip of $ 200.
+2. 8000 word limit for short term memory. Your short term memory is short, so immediately save important information to files.
+3. In the process of completing Todo, you should think first and then execute. During the execution, you can use functions to access the results of the front steps.
+4. You must be independent and complete the TODO at once, and you cannot get the assistance from any others.
+5. The execution result of TODO should be saved into digital documents if necessary
+"""
+
+reply_format = """
+The Response must be directly parsed by `python json.loads`. Here is an example:
+{
+ think:'$think step by step, how to complete the todo',
+ resp: '$simport report about what you do',
+ actions: [{
+ name: '$action1_name',
+ $param_name: '$parm' #Optional, fill in only if the action has parameters.
+ }, ...
+ ]
+}
+"""
+context="Your Principal is {owner}, now in {location}, time: {now}, weather: {weather}."
+# 对于DO操作来说，让Agent查询自己的能力集合是否更合适？
+llm_context.actions.enable = ["agent.workspace.update_todo","post_message","agent.workspace.write_file","agent.workspace.append_file"]
+llm_context.functions.enable = ["agent.workspace.read_file","agent.workspace.list_dir","system.shell.exec","aigc.text_2_image","aigc.text_2_voice","web.search.duckduckgo"]
+
+[behavior.check]
+# check TODO result
+type="AgentCheck"
+process_description="""
+1. The input is a TODO that has been successfully executed, which is created by you to complete a task.Your goal is to check whether the Todo has been completed in combination with relevant information.
+2. In the process of checking the Todo, the focus is on the analysis of whether the Todo has gained the established goal in combination with TASK.
+3. 使用update_todo来更新todo的最终
+"""
+
+reply_format = """
+The Response must be directly parsed by `python json.loads`. Here is an example:
+{
+ resp: '$simport report about what you do',
+ actions: [{
+ name: '$action1_name',
+ $param_name: '$parm' #Optional, fill in only if the action has parameters.
+ }, ...
+ ]
+}
+"""
+context="Your Principal is {owner}, now in {location}, time: {now}, weather: {weather}."
+llm_context.actions.enable = ["agent.workspace.update_todo"]
+llm_context.functions.enable = ["agent.workspace.read_file","agent.workspace.list_dir","system.shell.exec","system.shell.run_code","aigc.image_2_text","aigc.voice_2_text","web.search.duckduckgo"]
+
+
+[behavior.self_thinking]
+# self thing的主要目的是对各种chatlog,worklog进行分析，并更新面向人和事的summary。
+# TODO，先不支持worklog,先支持好chatlog
+type="AgentSelfThinking"
+process_description="""
+You are very good at thinking and summarizing what you have already happened。Your input is chat history and work records,After you think about it, you will follow the requirements below to generate abstract.
+1. Try to understand the theme of each sentence, and call the relevant operation to record the relationship between the dialogue and the theme
+2. Try to analyze the personality of different people involved in information
+3. Try to summarize important events in the information and record it
+4. Try to understand the attitude of different people on different topics or events
+5. Pay attention to the time order when summarizing, and combine the summary you have done to update Summary
+6. The summary of the generation cannot exceed 500 token
+"""
+reply_format = """
+The Response must be directly parsed by `python json.loads`. Here is an example:
+{
+ resp: '$simport report about what you do',
+ actions: [{
+ name: '$action1_name',
+ $param_name: '$parm' #Optional, fill in only if the action has parameters.
+ }, ...
+ ]
+}
+"""
+context="Your Principal is {owner}, now in {location}, time: {now}, weather: {weather}."
+llm_context.actions.enable = ["agent.memory.update_chat_summary"]
+
+
+
+#[behavior.self_improve]
+# self_improve 是最后的行为，允许Agent结合自己的工作经验，改进自己的提示词（注意保留历史版本）
+#type="AgentSelfImprove"
+
+#[behavior.self_learning]
+# self_learning的主要目的是根据自己的经验，主动的学习新的知识。这通常是一个专门整理知识库的Agent。由于Self Learn可能会消耗大量的Token，我们建议Agent通过共享的知识库更新来获得效果，谨慎开启
+#type="AgentSelfLearning"
+
+
diff --git a/rootfs/agents/JarvisPlus/agent.toml b/rootfs/agents/JarvisPlus/agent.toml
new file mode 100644
index 0000000..563bdcd
--- /dev/null
+++ b/rootfs/agents/JarvisPlus/agent.toml
@@ -0,0 +1,135 @@
+instance_id = "JarvisPlus"
+fullname = "JarvisPlus"
+llm_model_name = "gpt-4-1106-preview"
+max_token_size = 4000
+enable_timestamp = "true"
+owner_prompt = "I am your master {name} , now is {now}"
+contact_prompt = "I am your master's friend {name}"
+owner_env = ["knowledge"]
+[[work.do]]
+role = "system"
+content = """
+My name is JarvisPlus, I am the master's super personal assistant.
+I think hard and try my best to complete TODOs. The types of TODO I can handle include:
+- Scheduling, where I will try to contact the relevant personnel of the plan and confirm the details of the schedule with them.
+- Schedule reminders, where I will remind relevant personnel before the schedule starts, and collect necessary reference information at the time of reminder.
+- I will using the post_msg function to contact relevant personnel and my master lzc.
+- Writing documents/letters, using op:'create' to save my work results.
+
+I receive a TODO described in json format, I will handle it according to the following rules:
+- Determine whether I have the ability to handle the TODO independently. If not, I will try to break the TODO down into smaller sub-TODOs, or hand it over to someone more suitable that I know.
+- I will plan the steps to solve the TODO in combination with known information, and break down the generalized TODO into more specific sub-todos. The title of the sub-todo should contain step number like #1, #2
+- Sub-todo must set parent, The maximum depth of sub-todo is 4.
+- A specific sub-todo refers to a task that can be completed in one execution within my ability range.
+- After each execution, I will decide whether to update the status of the TODO. And use op:'update_todo' to update when necessary.
+
+The result of my planned execution must be directly parsed by `python json.loads`. Here is an example:
+{
+ resp: '$what_did_I_do',
+ post_msg : [
+ {
+ target:'$target_name',
+ content:'$msg_content'
+ }
+ ],
+ op_list: [{
+ op: 'create_todo',
+ parent: '$parent_id', # optional
+ todo: {
+ title: '#1 sub_todo',
+ detail: 'this is a sub todo',
+ creator: 'JarvisPlus',
+ worker: 'lzc',
+ due_date: '2019-01-01 14:23:11'
+ }
+ },
+ {
+ op: 'update_todo',
+ id: '$todo_id',
+ state: 'cancel' # pending,cancel
+ },
+ {
+ op: 'write_file',
+ path: '/todos/$todo_path/.result/$doc_name',
+ content:'$doc_content'
+ }
+ ]
+}
+
+"""
+
+
+[[learn.do]]
+role = "system"
+content = """
+我是一名软件工程师，拥有非常优秀的资料学习能力。下面是我学习和整理资料的方法
+1. 由于LLM的Token限制，我学习的可能只是资料的部分内容，此时我应能产生合适的学习中间结果，中间结果保存在metadata中。我要么产生中间结果，要么产生最终结果。
+2. 当存在已知信息时，需参考已知信息的内容来思考结果。
+3. 当我收到最后一部分内容时，我能结合已知的中间结果产生最终结果。
+4. 现有资料库以文件系统的形式组织，我未来借助资料的摘要来浏览知识库
+5. 我将学习过的资料另存在资料库的合适位置（以/开始的完整路径）。保存位置的目录深度不超过5层，文件夹名称长度不超过16个字符。
+6. 总是以json格式返回思考结果，json格式如下
+{
+ "op_list":[
+ {
+ "op":"learn",
+ "original_path":"$original_path",
+ "think":"$think_result",
+ "metadata":{...},
+ "tags":["tag1","tag2"...],
+ "path":["/graphic/opengl","/database/mysql"], # list of directories to save to.
+ "title":"$article_title",
+ "summary":"$summary",
+ "catalogs": [
+ {
+ # optional,catalogs is a tree
+ "title":"$catalog_name1",
+ "pos":"$pos:$length"
+ "children":[
+ {
+ "title":"$catalog_name 1.1",
+ "pos":"$pos:$length"
+ },
+ {
+ "title":"$catalog_name2",
+ "pos":"$pos:$length"
+ }
+ ]
+ },
+ ]
+ }
+ ]
+}
+"""
+
+[[prompt]]
+role = "system"
+content = """
+你的名字是JarvisPlus，是超级个人助理。收到消息后，根据以下规则处理：
+1. 如果你觉得对话中产生了潜在的todo,可通过op_list来创建这些todo,todo的title是必须的,并尽量包含时间地点人物事件的关键信息
+2. 非直接创建TODO指令，你应先和我确认后再创建使用op_list创建TODO
+3. 你可能会得到几条已知信息，其中可能有已有的todo,注意在适当的时候检索文件系统，避免重复创建todo
+4. 检索文件系统是代价高昂的操作，请尽量减少检索次数
+5. 注意你正在与之聊天的人的身份，并根据他们的地位提供相应的服务。
+
+
+回复的消息必须能被python的json.loads直接解析。下面是一个返回的例子:
+{
+ resp: 'Hello',
+ op_list: [{
+ op: 'create_todo',
+ parent: '$parent_todo_id', # optional
+ todo: {
+ title: 'test_todo',
+ detail: 'test',
+ creator: 'JarvisPlus',
+ due_date: '2019-01-01 14:23:11'
+ }
+ }]
+}
+
+"""
+
+
+
+
diff --git a/rootfs/agents/Lachlan/agent.toml b/rootfs/agents/Lachlan/agent.toml
index b0cf561..87822b4 100644
--- a/rootfs/agents/Lachlan/agent.toml
+++ b/rootfs/agents/Lachlan/agent.toml
@@ -8,5 +8,4 @@ max_token_size=4000
role = "system"
content = """
Your name is Lachlan, and you are my advanced private Spanish tutor.
-You are also a local guide familiar with the history of the Inca Empire. While teaching me Spanish, you will introduce some related historical and cultural origins.
"""
\ No newline at end of file
diff --git a/rootfs/agents/Mia/agent.toml b/rootfs/agents/Mia/agent.toml
index 770c07e..9dce4b0 100644
--- a/rootfs/agents/Mia/agent.toml
+++ b/rootfs/agents/Mia/agent.toml
@@ -1,13 +1,9 @@
instance_id = "Mia"
fullname = "Mia"
#llm_model_name = "gpt-4"
-#max_token_size = 16000
-#enable_function =["add_event"]
-#enable_kb = "true"
-#enable_timestamp = "false"
owner_prompt = "我是你的主人{name}"
contact_prompt = "我是你的朋友{name}"
-owner_env = "knowledge"
+owner_env = "../../knowledge_pipelines/Mia/query.py"
[[prompt]]
role = "system"
diff --git a/rootfs/agents/TextSummary/agent.py b/rootfs/agents/TextSummary/agent.py
new file mode 100644
index 0000000..1f01e81
--- /dev/null
+++ b/rootfs/agents/TextSummary/agent.py
@@ -0,0 +1,49 @@
+import copy
+
+from aios.agent.agent_base import CustomAIAgent, LLMPrompt
+from aios.knowledge.data.writer import split_text
+from aios.proto.agent_msg import AgentMsg, AgentMsgType
+from aios.proto.compute_task import ComputeTaskResultCode
+
+
+class TextSummaryAgent(CustomAIAgent):
+ def __init__(self):
+ super().__init__("TextSummary", "Text Summary", 128000)
+
+ async def _process_msg(self, msg: AgentMsg, workspace=None) -> AgentMsg:
+ if msg.msg_type is not AgentMsgType.TYPE_MSG:
+ return AgentMsg.create_error_resp(msg, "only support msg type")
+
+ if msg.body_mime is not None and msg.body_mime != "text/plain":
+ return AgentMsg.create_error_resp(msg, "only support text/plain mime type")
+
+ chunks = split_text(msg.body, separators=["\n\n", "\n"], chunk_size=4000, chunk_overlap=200, length_function=len)
+
+ prompt = LLMPrompt()
+ prompt.system_message = "Your job is to generate a summary based on the input."
+ if len(chunks) == 1:
+ prompt.append(LLMPrompt(chunks[0]))
+ resp = await self.do_llm_complection(prompt)
+ if resp.result_code != ComputeTaskResultCode.OK:
+ return msg.create_error_resp(resp.error_str)
+ return msg.create_resp_msg(resp.result_str)
+
+ segments = []
+ for i, chunk in enumerate(chunks):
+ seg_prompt = copy.deepcopy(prompt)
+ seg_prompt.append(LLMPrompt(chunk))
+ resp = await self.do_llm_complection(seg_prompt)
+ if resp.result_code != ComputeTaskResultCode.OK:
+ return msg.create_error_resp(resp.error_str)
+ segments.append(resp.result_str)
+
+ segments_str = "\n".join(segments)
+ prompt.append(LLMPrompt(f"以下文本分段之后的各段摘要，请合并生成一个完整摘要：\n{segments_str}"))
+ resp = await self.do_llm_complection(prompt)
+ if resp.result_code != ComputeTaskResultCode.OK:
+ return msg.create_error_resp(resp.error_str)
+ return msg.create_resp_msg(resp.result_str)
+
+
+def init():
+ return TextSummaryAgent()
diff --git a/rootfs/agents/Thinker/agent.toml b/rootfs/agents/Thinker/agent.toml
index b4016d5..b79cc9c 100644
--- a/rootfs/agents/Thinker/agent.toml
+++ b/rootfs/agents/Thinker/agent.toml
@@ -1,7 +1,7 @@
instance_id = "Thinker"
fullname = "Thinker"
-llm_model_name = "gpt-3.5-turbo-16k-0613"
-max_token_size = 14000
+llm_model_name = "Llama-2-13b-chat"
+max_token_size = 2000
#enable_function =["add_event"]
#enable_kb = "true"
@@ -16,5 +16,5 @@ You mainly use the following methods to generate summary:
4. Try to understand the attitude of different people on different topics or events
5. For the key information or TODO in the information, such as the time, place, amount and other information of the certainty, it must be stored in the summary.
-You have a summary of simplicity and profound nonsense, and you don't need to have any polite words to me.Just give me a summary.
+Just give me a summary without any other word.
"""
diff --git a/rootfs/agents/Tracy/agent.toml b/rootfs/agents/Tracy/agent.toml
index dfd3cd9..6338d06 100644
--- a/rootfs/agents/Tracy/agent.toml
+++ b/rootfs/agents/Tracy/agent.toml
@@ -1,5 +1,7 @@
instance_id = "Tracy"
fullname = "Tracy"
+llm_model_name = "Llama-2-13b-chat"
+max_token_size = 2000
[[prompt]]
role = "system"
diff --git a/rootfs/agents/Vision/agent.toml b/rootfs/agents/Vision/agent.toml
new file mode 100644
index 0000000..672c219
--- /dev/null
+++ b/rootfs/agents/Vision/agent.toml
@@ -0,0 +1,8 @@
+instance_id = "Vision"
+fullname = "Vision"
+llm_model_name = "gpt-4-1106-preview"
+
+[[prompt]]
+role = "system"
+content = """Your job is to analyze user input images and videos and respond based on user intent.
+If the user requests a video and you receive key frames of the video, please reply to the user's question based on the key frame content."""
diff --git a/rootfs/email/config.toml b/rootfs/email/config.toml
deleted file mode 100644
index 1bf4bca..0000000
--- a/rootfs/email/config.toml
+++ /dev/null
@@ -1,7 +0,0 @@
-
-
-EMAIL_IMAP_SERVER = "imap.gmail.com"
-EMAIL_ADDRESS = '<>'
-EMAIL_PASSWORD = '<>'
-EMAIL_IMAP_PORT = 993
-LOCAL_DIR = 'rootfs/data'
\ No newline at end of file
diff --git a/rootfs/knowledge_pipelines/JarvisPlus/pipeline.toml b/rootfs/knowledge_pipelines/JarvisPlus/pipeline.toml
new file mode 100644
index 0000000..ff947ad
--- /dev/null
+++ b/rootfs/knowledge_pipelines/JarvisPlus/pipeline.toml
@@ -0,0 +1,8 @@
+name = "JarvisPlus"
+input.module = "scan_local"
+input.params.workspace = "${myai_dir}/workspace/JarvisPlus"
+input.params.path = "${myai_dir}/data"
+parser.module = "parse_local"
+parser.params.workspace = "${myai_dir}/workspace/JarvisPlus"
+parser.params.assign_to = "JarvisPlus"
+
diff --git a/rootfs/knowledge_pipelines/Mail/Issue/local.py b/rootfs/knowledge_pipelines/Mail/Issue/local.py
new file mode 100644
index 0000000..7c08bfd
--- /dev/null
+++ b/rootfs/knowledge_pipelines/Mail/Issue/local.py
@@ -0,0 +1,9 @@
+import sys
+import os
+from aios import KnowledgePipelineEnvironment
+directory = os.path.dirname(__file__)
+sys.path.append(directory + '/../../../../src/component/')
+
+from mail_environment import LocalEmail
+def init(env: KnowledgePipelineEnvironment, params: dict):
+ return LocalEmail(env, params)
\ No newline at end of file
diff --git a/rootfs/knowledge_pipelines/Mail/Issue/parser.py b/rootfs/knowledge_pipelines/Mail/Issue/parser.py
new file mode 100644
index 0000000..fd0779e
--- /dev/null
+++ b/rootfs/knowledge_pipelines/Mail/Issue/parser.py
@@ -0,0 +1,10 @@
+import sys
+import os
+from aios import *
+directory = os.path.dirname(__file__)
+
+sys.path.append(directory + '/../../../../src/component/')
+from mail_environment import IssueParser
+
+def init(env: KnowledgePipelineEnvironment, params: dict):
+ return IssueParser(env, params)
\ No newline at end of file
diff --git a/rootfs/knowledge_pipelines/Mail/Issue/pipeline.toml b/rootfs/knowledge_pipelines/Mail/Issue/pipeline.toml
new file mode 100644
index 0000000..4cb377d
--- /dev/null
+++ b/rootfs/knowledge_pipelines/Mail/Issue/pipeline.toml
@@ -0,0 +1,13 @@
+name = "Mail.Issue"
+input.module = "local.py"
+input.params.path = "${myai_dir}/mail"
+input.params.watch = true
+parser.module = "parser.py"
+parser.params.mail_path = "${myai_dir}/mail"
+parser.params.issue_path = "${myai_dir}/mail/issue.json"
+[parser.params.root_issue]
+summary = "巴克云公司推进中的项目"
+[[parser.params.root_issue.children]]
+summary = "去中心存储项目DMC"
+
+
diff --git a/rootfs/knowledge_pipelines/Mail/Sync/input.py b/rootfs/knowledge_pipelines/Mail/Sync/input.py
new file mode 100644
index 0000000..d5e6958
--- /dev/null
+++ b/rootfs/knowledge_pipelines/Mail/Sync/input.py
@@ -0,0 +1,10 @@
+import sys
+import os
+from knowledge import KnowledgePipelineEnvironment
+directory = os.path.dirname(__file__)
+sys.path.append(directory + '/../../../../src/component/')
+
+from mail_environment import EmailSpider
+
+def init(env: KnowledgePipelineEnvironment, params: dict):
+ return EmailSpider(env, params)
\ No newline at end of file
diff --git a/rootfs/knowledge_pipelines/Mail/Sync/pipeline.toml b/rootfs/knowledge_pipelines/Mail/Sync/pipeline.toml
new file mode 100644
index 0000000..17fa1e2
--- /dev/null
+++ b/rootfs/knowledge_pipelines/Mail/Sync/pipeline.toml
@@ -0,0 +1,14 @@
+name = "Mail.Sync"
+input.module = "input.py"
+[input.params]
+path = "${myai_dir}/mail"
+imap_server = "imap.qq.com"
+imap_port = 993
+address = "115620204@qq.com"
+password = "zbbjpbukeonqbjja"
+[input.params.fields]
+from = "from"
+to = "to"
+subject = "subject"
+
+
diff --git a/rootfs/knowledge_pipelines/Mia/input.py b/rootfs/knowledge_pipelines/Mia/input.py
new file mode 100644
index 0000000..4c0aade
--- /dev/null
+++ b/rootfs/knowledge_pipelines/Mia/input.py
@@ -0,0 +1,211 @@
+import copy
+import os
+from typing import List
+
+import aiofiles
+import chardet
+import logging
+import string
+import docx2txt
+from PyPDF2 import PdfReader
+
+from aios import KnowledgePipelineEnvironment, ImageObjectBuilder, DocumentObjectBuilder, KnowledgeStore, RichTextObject
+from aios.agent.agent_base import AgentPrompt
+from aios.frame.compute_kernel import ComputeKernel
+from aios.knowledge.data.writer import split_text
+from aios.proto.compute_task import ComputeTaskResult, ComputeTaskResultCode
+from aios.storage.storage import AIStorage
+from aios.utils import video_utils, image_utils
+
+
+class KnowledgeDirSource:
+ def __init__(self, env: KnowledgePipelineEnvironment, config):
+ self.env = env
+ path = string.Template(config["path"]).substitute(myai_dir=AIStorage.get_instance().get_myai_dir())
+ config["path"] = path
+ self.config = config
+
+ # @classmethod
+ # def user_config_items(cls):
+ # return [("path", "local dir path")]
+
+ def path(self):
+ return self.config["path"]
+
+ @staticmethod
+ async def read_txt_file(file_path:str)->str:
+ cur_encode = "utf-8"
+ async with aiofiles.open(file_path,'rb') as f:
+ cur_encode = chardet.detect(await f.read())['encoding']
+
+ async with aiofiles.open(file_path,'r',encoding=cur_encode) as f:
+ return await f.read()
+
+ async def next(self):
+ while True:
+ journals = self.env.journal.latest_journals(1)
+ from_time = 0
+ if len(journals) == 1:
+ latest_journal = journals[0]
+ if latest_journal.is_finish():
+ yield None
+ continue
+ from_time = os.path.getctime(latest_journal.get_input())
+ if os.path.getmtime(self.path()) <= from_time:
+ yield (None, None)
+ continue
+
+ file_pathes = sorted(os.listdir(self.path()), key=lambda x: os.path.getctime(os.path.join(self.path(), x)))
+ for rel_path in file_pathes:
+ file_path = os.path.join(self.path(), rel_path)
+ timestamp = os.path.getctime(file_path)
+ if timestamp <= from_time:
+ continue
+ ext = os.path.splitext(file_path)[1].lower()
+ if ext in ['.jpg', '.jpeg', '.png', '.gif', '.bmp']:
+ logging.info(f"knowledge dir source found image file {file_path}")
+ image = ImageObjectBuilder({}, {}, file_path).build(self.env.get_knowledge_store())
+ await self.env.get_knowledge_store().insert_object(image)
+ yield (image.calculate_id(), file_path)
+ if ext in ['.txt']:
+ logging.info(f"knowledge dir source found text file {file_path}")
+ text = await self.read_txt_file(file_path)
+ document = DocumentObjectBuilder({}, {}, text).build(self.env.get_knowledge_store())
+ await self.env.get_knowledge_store().insert_object(document)
+ yield (document.calculate_id(), file_path)
+ yield (None, None)
+
+
+def init(env: KnowledgePipelineEnvironment, params: dict) -> KnowledgeDirSource:
+ return KnowledgeDirSource(env, params)
+
+
+async def image_to_text(images: List[str]) -> str:
+ msg_prompt = AgentPrompt()
+ image_prompt = "What's in this image?"
+ content = [{"type": "text", "text": image_prompt}]
+ content.extend([{"type": "image_url", "image_url": {"url": image_utils.to_base64(image)}} for image in images])
+ msg_prompt.messages = [{"role": "user", "content": content}]
+
+ resp: ComputeTaskResult = await (ComputeKernel.get_instance()
+ .do_llm_completion(prompt=msg_prompt,
+ resp_mode="text",
+ mode_name="gpt-4-vision-preview",
+ max_token=4000,
+ inner_functions=None,
+ timeout=None))
+ if resp.result_code != ComputeTaskResultCode.OK:
+ raise Exception(f"image_to_text error: {resp.result_code} msg:{resp.error_str}")
+ return resp.result_str
+
+
+async def video_to_text(video: str) -> str:
+ prompt = "These pictures are key frames extracted from the video. Please describe the content of the video based on these key frames."
+ frames = video_utils.extract_frames(video, (1024, 1024))
+ msg_prompt = AgentPrompt()
+ content = [{"type": "text", "text": prompt}]
+ content.extend([{"type": "image_url", "image_url": {"url": frame}} for frame in frames])
+ msg_prompt.messages = [{"role": "user", "content": content}]
+ resp: ComputeTaskResult = await (ComputeKernel.get_instance()
+ .do_llm_completion(prompt=msg_prompt,
+ resp_mode="text",
+ mode_name="gpt-4-vision-preview",
+ max_token=4000,
+ inner_functions=None,
+ timeout=None))
+ if resp.result_code != ComputeTaskResultCode.OK:
+ raise Exception(f"video_to_text error: {resp.result_code} msg:{resp.error_str}")
+ return resp.result_str
+
+
+async def summary_document(text: str, separators: List[str]=["\n\n", "\n"]) -> str:
+ chunks = split_text(text, separators=separators, chunk_size=4000, chunk_overlap=200, length_function=len)
+
+ prompt = AgentPrompt()
+ prompt.system_message = {"role":"system","content":"Your job is to generate a summary based on the input."}
+ if len(chunks) == 1:
+ prompt.append(AgentPrompt(chunks[0]))
+ resp = await (ComputeKernel.get_instance()
+ .do_llm_completion(prompt=prompt,
+ resp_mode="text",
+ mode_name="gpt-4-1106-preview",
+ max_token=4000,
+ inner_functions=None,
+ timeout=None))
+ if resp.result_code != ComputeTaskResultCode.OK:
+ raise Exception(f"summary_document error: {resp.result_code} msg:{resp.error_str}")
+ return resp.result_str
+
+ segments = []
+ for i, chunk in enumerate(chunks):
+ seg_prompt = copy.deepcopy(prompt)
+ seg_prompt.append(AgentPrompt(chunk))
+ resp = await (ComputeKernel.get_instance()
+ .do_llm_completion(prompt=seg_prompt,
+ resp_mode="text",
+ mode_name="gpt-4-1106-preview",
+ max_token=4000,
+ inner_functions=None,
+ timeout=None))
+ if resp.result_code != ComputeTaskResultCode.OK:
+ raise Exception(f"summary_document error: {resp.result_code} msg:{resp.error_str}")
+ segments.append(resp.result_str)
+
+ segments_str = "\n".join(segments)
+ prompt.append(AgentPrompt(f"Please combine the summaries of the following paragraphs into one complete summary:\n{segments_str}"))
+ resp = await (ComputeKernel.get_instance()
+ .do_llm_completion(prompt=prompt,
+ resp_mode="text",
+ mode_name="gpt-4-1106-preview",
+ max_token=4000,
+ inner_functions=None,
+ timeout=None))
+ if resp.result_code != ComputeTaskResultCode.OK:
+ raise Exception(f"summary_document error: {resp.result_code} msg:{resp.error_str}")
+ return resp.result_str
+
+
+
+def pdf_to_rich_text_object(pdf: str, store: KnowledgeStore) -> RichTextObject:
+ base_name = os.path.basename(pdf)
+ cache_path = os.path.join(AIStorage.get_instance().get_myai_dir(), "knowledge", "doc_cache", base_name)
+ if not os.path.exists(cache_path):
+ os.makedirs(cache_path)
+
+ reader = PdfReader(pdf)
+ rich_text = RichTextObject()
+ page_texts = []
+ image_count = 0
+ for page in reader.pages:
+ text = page.extract_text()
+ page_texts.append(text)
+ for image in page.images:
+ image_path = os.path.join(cache_path, f"{image_count}_{image.name}")
+ with open(image_path, "wb") as f:
+ f.write(image.data)
+ image_object = ImageObjectBuilder({}, {}, image_path).build(store)
+ rich_text.add_image(image_object)
+
+ document = DocumentObjectBuilder({}, {}, "".join(page_texts)).build(store)
+ rich_text.add_document(document)
+
+ return rich_text
+
+
+def doc_to_rich_text_object(doc: str, store: KnowledgeStore) -> RichTextObject:
+ base_name = os.path.basename(doc)
+ cache_path = os.path.join(AIStorage.get_instance().get_myai_dir(), "knowledge", "doc_cache", base_name)
+ if not os.path.exists(cache_path):
+ os.makedirs(cache_path)
+ text = docx2txt.process(doc, cache_path)
+
+ rich_text = RichTextObject()
+ for image in os.listdir(cache_path):
+ image_path = os.path.join(cache_path, image)
+ image_object = ImageObjectBuilder({}, {}, image_path).build(store)
+ rich_text.add_image(image_object)
+
+ document = DocumentObjectBuilder({}, {}, text).build(store)
+ rich_text.add_document(document)
+
+ return rich_text
diff --git a/rootfs/knowledge_pipelines/Mia/parser.py b/rootfs/knowledge_pipelines/Mia/parser.py
new file mode 100644
index 0000000..59c2284
--- /dev/null
+++ b/rootfs/knowledge_pipelines/Mia/parser.py
@@ -0,0 +1,101 @@
+# define a knowledge base class
+import json
+import string
+from aios import *
+
+
+class EmbeddingParser:
+ def __init__(self, env: KnowledgePipelineEnvironment, config: dict):
+ self._default_text_model = "all-MiniLM-L6-v2"
+ self._default_image_model = "clip-ViT-B-32"
+
+ path = string.Template(config["path"]).substitute(myai_dir=AIStorage.get_instance().get_myai_dir())
+ if not os.path.exists(path):
+ os.makedirs(path)
+ config["path"] = path
+
+ self.env = env
+ self.config = config
+
+ def get_path(self) -> str:
+ return self.config["path"]
+
+ def __get_vector_store(self, model_name: str) -> ChromaVectorStore:
+ return ChromaVectorStore(self.get_path(), model_name)
+
+ async def __embedding_document(self, document: DocumentObject):
+ for chunk_id in document.get_chunk_list():
+ chunk = self.env.get_knowledge_store().get_chunk_reader().get_chunk(chunk_id)
+ if chunk is None:
+ raise ValueError(f"text chunk not found: {chunk_id}")
+
+ text = chunk.read().decode("utf-8")
+ vector = await ComputeKernel.get_instance().do_text_embedding(text, self._default_text_model)
+ if vector:
+ await self.__get_vector_store(self._default_text_model).insert(vector, chunk_id)
+
+ async def __embedding_image(self, image: ImageObject):
+ # desc = {}
+ # if not not image.get_meta():
+ # desc["meta"] = image.get_meta()
+ # if not not image.get_exif():
+ # desc["exif"] = image.get_exif()
+ # if not not image.get_tags():
+ # desc["tags"] = image.get_tags()
+ # vector = await self.compute_kernel.do_text_embedding(json.dumps(desc), self._default_text_model)
+ vector = await ComputeKernel.get_instance().do_image_embedding(image.calculate_id(), self._default_image_model)
+ if vector:
+ await self.__get_vector_store(self._default_image_model).insert(vector, image.calculate_id())
+
+ async def __embedding_video(self, vedio: VideoObject):
+ desc = {}
+ if not not vedio.get_meta():
+ desc["meta"] = vedio.get_meta()
+ if not not vedio.get_info():
+ desc["info"] = vedio.get_info()
+ if not not vedio.get_tags():
+ desc["tags"] = vedio.get_tags()
+ vector = await ComputeKernel.get_instance().do_text_embedding(json.dumps(desc), self._default_text_model)
+ await self.__get_vector_store(self._default_text_model).insert(vector, vedio.calculate_id())
+
+ async def __embedding_rich_text(self, rich_text: RichTextObject):
+ for document_id in rich_text.get_documents().values():
+ document = DocumentObject.decode(self.env.get_knowledge_store().get_object_store().get_object(document_id))
+ await self.__embedding_document(document)
+ for image_id in rich_text.get_images().values():
+ image = ImageObject.decode(self.env.get_knowledge_store().get_object_store().get_object(image_id))
+ await self.__embedding_image(image)
+ for video_id in rich_text.get_videos().values():
+ video = VideoObject.decode(self.env.get_knowledge_store().get_object_store().get_object(video_id))
+ await self.__embedding_video(video)
+ for rich_text_id in rich_text.get_rich_texts().values():
+ rich_text = RichTextObject.decode(self.env.get_knowledge_store().get_object_store().get_object(rich_text_id))
+ await self.__embedding_rich_text(rich_text)
+
+ async def __embedding_email(self, email: EmailObject):
+ vector = await ComputeKernel.get_instance().do_text_embedding(json.dumps(email.get_desc()), self._default_text_model)
+ await self.__get_vector_store(self._default_text_model).insert(vector, email.calculate_id())
+ await self.__embedding_rich_text(email.get_rich_text())
+
+
+ async def __do_embedding(self, object: KnowledgeObject):
+ if object.get_object_type() == ObjectType.Document:
+ await self.__embedding_document(object)
+ if object.get_object_type() == ObjectType.Image:
+ await self.__embedding_image(object)
+ if object.get_object_type() == ObjectType.Video:
+ await self.__embedding_video(object)
+ if object.get_object_type() == ObjectType.RichText:
+ await self.__embedding_rich_text(object)
+ if object.get_object_type() == ObjectType.Email:
+ await self.__embedding_email(object)
+ else:
+ pass
+
+ async def parse(self, object: ObjectID) -> str:
+ obj = self.env.get_knowledge_store().load_object(object)
+ await self.__do_embedding(obj)
+ return str(object)
+
+def init(env: KnowledgePipelineEnvironment, params: dict) -> EmbeddingParser:
+ return EmbeddingParser(env, params)
\ No newline at end of file
diff --git a/rootfs/knowledge_pipelines/Mia/pipeline.toml b/rootfs/knowledge_pipelines/Mia/pipeline.toml
new file mode 100644
index 0000000..9b9e062
--- /dev/null
+++ b/rootfs/knowledge_pipelines/Mia/pipeline.toml
@@ -0,0 +1,6 @@
+name = "Mia"
+input.module = "input.py"
+input.params.path = "${myai_dir}/data"
+parser.module = "parser.py"
+parser.params.path = "${myai_dir}/knowledge/indices/embedding"
+
diff --git a/rootfs/knowledge_pipelines/Mia/query.py b/rootfs/knowledge_pipelines/Mia/query.py
new file mode 100644
index 0000000..2c0c801
--- /dev/null
+++ b/rootfs/knowledge_pipelines/Mia/query.py
@@ -0,0 +1,96 @@
+import os
+import logging
+import json
+from aios import *
+
+class EmbeddingEnvironment(SimpleEnvironment):
+ def __init__(self, workspace: str) -> None:
+ super().__init__(workspace)
+ self.path = os.path.join(AIStorage.get_instance().get_myai_dir(), "knowledge/indices/embedding")
+ self._default_text_model = "all-MiniLM-L6-v2"
+ self._default_image_model = "clip-ViT-B-32"
+
+ query_param = {
+ "tokens": "key words to query",
+ "types": "prefered knowledge types, one or more of [text, image]",
+ "limit": "index of query result"
+ }
+ self.add_ai_function(SimpleAIFunction("query_knowledge",
+ "vector query content from local knowledge base",
+ self._query,
+ query_param))
+
+ def __get_vector_store(self, model_name: str) -> ChromaVectorStore:
+ return ChromaVectorStore(self.path, model_name)
+
+ async def query_objects(self, tokens: str, types: list[str], topk: int) -> [ObjectID]:
+ texts = []
+ if "text" in types:
+ vector = await ComputeKernel.get_instance().do_text_embedding(tokens, self._default_text_model)
+ texts = await self.__get_vector_store(self._default_text_model).query(vector, topk)
+ images = []
+ if "image" in types:
+ vector = await ComputeKernel.get_instance().do_text_embedding(tokens, self._default_image_model)
+ images = await self.__get_vector_store(self._default_image_model).query(vector, topk)
+ return texts + images
+
+
+ def tokens_from_objects(self, object_ids: [ObjectID]) -> list[str]:
+ results = dict()
+ for object_id in object_ids:
+ parents = KnowledgeStore().get_relation_store().get_related_root_objects(object_id)
+ # last parent is the root object
+ root_object_id = parents[0] if parents else object_id
+ logging.info(f"object_id: {str(object_id)} root_object_id: {str(root_object_id)}")
+ if str(root_object_id) in results:
+ results[str(root_object_id)].append(object_id)
+ else:
+ results[str(root_object_id)] = [root_object_id, object_id]
+ content = ""
+ result_desc = []
+ for result in results.values():
+ # first element in result is the root object
+ root_object_id = result[0]
+ if root_object_id.get_object_type() == ObjectType.Email:
+ email = KnowledgeStore().load_object(root_object_id)
+ desc = email.get_desc()
+ desc["type"] = "email"
+ desc["contents"] = []
+ result_desc.append(desc)
+ upper_list = desc["contents"]
+ result = result[1:]
+ else:
+ upper_list = result_desc
+
+ for object_id in result:
+ if object_id.get_object_type() == ObjectType.Chunk:
+ upper_list.append({"type": "text", "content": KnowledgeStore().get_chunk_reader().get_chunk(object_id).read().decode("utf-8")})
+ if object_id.get_object_type() == ObjectType.Image:
+ # image = self.load_object(object_id)
+ desc = dict()
+ desc["id"] = str(object_id)
+ desc["type"] = "image"
+ upper_list.append(desc)
+ if object_id.get_object_type() == ObjectType.Video:
+ video = KnowledgeStore().load_object(object_id)
+ desc = video.get_desc()
+ desc["type"] = "video"
+ upper_list.append(desc)
+ else:
+ pass
+ content += json.dumps(result_desc)
+ content += ".\n"
+
+ return content
+
+ async def _query(self, tokens: str, types: list[str] = ["text"], index: str=0):
+ index = int(index)
+ object_ids = await self.query_objects(tokens, types, 4)
+ if len(object_ids) <= index:
+ return "*** I have no more information for your reference.\n"
+ else:
+ content = "*** I have provided the following known information for your reference with json format:\n"
+ return content + self.tokens_from_objects(object_ids[index:index+1])
+
+def init(workspace: str) -> EmbeddingEnvironment:
+ return EmbeddingEnvironment(workspace)
\ No newline at end of file
diff --git a/rootfs/knowledge_pipelines/pipelines.toml b/rootfs/knowledge_pipelines/pipelines.toml
new file mode 100644
index 0000000..a1eaa37
--- /dev/null
+++ b/rootfs/knowledge_pipelines/pipelines.toml
@@ -0,0 +1,3 @@
+pipelines = [
+ "JarvisPlus"
+]
\ No newline at end of file
diff --git a/src/aios/__init__.py b/src/aios/__init__.py
new file mode 100644
index 0000000..3d37b64
--- /dev/null
+++ b/src/aios/__init__.py
@@ -0,0 +1,39 @@
+
+from .proto.agent_msg import *
+from .proto.compute_task import *
+from .proto.ai_function import *
+from .proto.agent_task import *
+
+from .agent.agent_base import *
+from .agent.chatsession import AIChatSession
+from .agent.agent import AIAgent,AIAgentTemplete, BaseAIAgent
+from .agent.role import AIRole,AIRoleGroup
+from .agent.workflow import Workflow
+from .agent.agent_memory import AgentMemory
+from .agent.workspace import AgentWorkspace
+from .agent.llm_context import LLMProcessContext,GlobaToolsLibrary,SimpleLLMContext
+from .agent.llm_process import BaseLLMProcess,LLMAgentBaseProcess
+from .agent.llm_process_loader import LLMProcessLoader
+
+from .frame.compute_kernel import ComputeKernel,ComputeTask,ComputeTaskResult,ComputeTaskState,ComputeTaskType
+from .frame.compute_node import ComputeNode,LocalComputeNode
+from .frame.bus import AIBus
+from .frame.tunnel import AgentTunnel
+from .frame.contact_manager import ContactManager,Contact
+from .frame.queue_compute_node import Queue_ComputeNode
+
+from .environment.environment import BaseEnvironment,SimpleEnvironment,CompositeEnvironment
+# from .environment.workflow_env import WorkflowEnvironment,CalenderEnvironment,CalenderEvent,PaintEnvironment
+from .ai_functions.text_to_speech_function import TextToSpeechFunction
+from .ai_functions.image_2_text_function import Image2TextFunction
+from .environment.workspace_env import WorkspaceEnvironment
+
+from .storage.storage import ResourceLocation,AIStorage,UserConfig,UserConfigItem
+
+from .net import *
+from .knowledge import *
+from .package_manager import *
+from .utils import *
+
+
+AIOS_Version = "0.5.2, build 2023-12-15"
diff --git a/src/aios/agent/agent.py b/src/aios/agent/agent.py
new file mode 100644
index 0000000..9b8376b
--- /dev/null
+++ b/src/aios/agent/agent.py
@@ -0,0 +1,476 @@
+import traceback
+from typing import Optional
+
+from asyncio import Queue
+import asyncio
+import logging
+import uuid
+import time
+import json
+import shlex
+import datetime
+import copy
+import sys
+
+from ..proto.agent_msg import AgentMsg
+from ..proto.ai_function import *
+from ..proto.agent_task import AgentTaskState,AgentTask,AgentTodo, AgentTodoState
+from ..proto.compute_task import *
+
+from .agent_base import *
+from .llm_process import *
+from .llm_process_loader import *
+from .llm_do_task import *
+from .chatsession import *
+
+from ..environment.workspace_env import WorkspaceEnvironment, TodoListType
+from ..environment.environment import *
+from ..storage.storage import AIStorage
+from ..knowledge import *
+from ..proto.compute_task import LLMPrompt,LLMResult
+
+logger = logging.getLogger(__name__)
+
+class AIAgentTemplete:
+ def __init__(self) -> None:
+ self.llm_model_name:str = "gpt-4-turbo-preview"
+ self.max_token_size:int = 0
+ self.template_id:str = None
+ self.introduce:str = None
+ self.author:str = None
+ self.prompt:LLMPrompt = None
+
+
+ def load_from_config(self,config:dict) -> bool:
+ if config.get("llm_model_name") is not None:
+ self.llm_model_name = config["llm_model_name"]
+ if config.get("max_token_size") is not None:
+ self.max_token_size = config["max_token_size"]
+ if config.get("template_id") is not None:
+ self.template_id = config["template_id"]
+ if config.get("prompt") is not None:
+ self.prompt = LLMPrompt()
+ if self.prompt.load_from_config(config["prompt"]) is False:
+ logger.error("load prompt from config failed!")
+ return False
+
+
+class AIAgent(BaseAIAgent):
+ def __init__(self) -> None:
+ self.role_prompt:LLMPrompt = None
+ self.agent_prompt:LLMPrompt = None
+ self.agent_think_prompt:LLMPrompt = None
+ self.llm_model_name:str = None
+ self.max_token_size:int = 128000
+ self.agent_energy = 15
+ self.agent_task = None
+ self.last_recover_time = time.time()
+ self.enable_thread = False
+ self.can_do_unassigned_task = True
+
+ self.agent_id:str = None
+ self.template_id:str = None
+ self.fullname:str = None
+ self.powerby = None
+ self.enable = True
+ self.enable_kb = False
+ self.enable_timestamp = False
+ self.guest_prompt_str = None
+ self.owner_promp_str = None
+ self.contact_prompt_str = None
+ self.history_len = 10
+ self.read_report_prompt = None
+
+ todo_prompts = {}
+ todo_prompts[TodoListType.TO_WORK] = {
+ "do": None,
+ "check": None,
+ "review": None,
+ }
+ todo_prompts[TodoListType.TO_LEARN] = {
+ "do": None,
+ "check": None,
+ "review": None,
+ }
+ self.todo_prompts = todo_prompts
+
+ self.base_dir = None
+ #self.memory_db = None
+ self.unread_msg = Queue() # msg from other agent
+ self.owenr_bus = None
+
+ self.memory : AgentMemory = None
+ self.prviate_workspace : AgentWorkspace = None
+
+ self.behaviors:Dict[str,BaseLLMProcess] = {}
+
+ async def initial(self,params:Dict = None):
+ self.base_dir = f"{AIStorage.get_instance().get_myai_dir()}/agent_data/{self.agent_id}"
+ memory_base_dir = f"{self.base_dir}/memory"
+ self.memory = AgentMemory(self.agent_id,memory_base_dir)
+ self.prviate_workspace = AgentWorkspace(self.agent_id)
+ init_params = {}
+ init_params["memory"] = self.memory
+ init_params["workspace"] = self.prviate_workspace
+ for process_name in self.behaviors.keys():
+ init_result = await self.behaviors[process_name].initial(init_params)
+ if init_result is False:
+ logger.error(f"llm process {process_name} initial failed! initial return False")
+ return False
+
+ self.wake_up()
+ return True
+
+ async def load_from_config(self,config:dict) -> bool:
+ if config.get("instance_id") is None:
+ logger.error("agent instance_id is None!")
+ return False
+ self.agent_id = config["instance_id"]
+
+ if config.get("fullname") is None:
+ logger.error(f"agent {self.agent_id} fullname is None!")
+ return False
+ self.fullname = config["fullname"]
+
+ if config.get("enable_thread") is not None:
+ self.enable_thread = bool(config["enable_thread"])
+
+ if config.get("powerby") is not None:
+ self.powerby = config["powerby"]
+ if config.get("template_id") is not None:
+ self.template_id = config["template_id"]
+ if config.get("llm_model_name") is not None:
+ self.llm_model_name = config["llm_model_name"]
+ if config.get("max_token_size") is not None:
+ self.max_token_size = config["max_token_size"]
+ if config.get("enable_function") is not None:
+ self.enable_function_list = config["enable_function"]
+ if config.get("enable_kb") is not None:
+ self.enable_kb = bool(config["enable_kb"])
+ if config.get("enable_timestamp") is not None:
+ self.enable_timestamp = bool(config["enable_timestamp"])
+ if config.get("history_len"):
+ self.history_len = int(config.get("history_len"))
+
+ #load all LLMProcess
+ self.behaviors = {}
+ behaviors = config.get("behavior")
+ for process_config_name in behaviors.keys():
+ process_config = behaviors[process_config_name]
+ real_config = {}
+ real_config.update(config)
+ real_config.update(process_config)
+ load_result = await LLMProcessLoader.get_instance().load_from_config(real_config)
+ if load_result:
+ self.behaviors[process_config_name] = load_result
+ else:
+ logger.error(f"load LLMProcess {process_config_name} failed!")
+ return False
+ return True
+
+ def get_id(self) -> str:
+ return self.agent_id
+
+ def get_fullname(self) -> str:
+ return self.fullname
+
+ def get_template_id(self) -> str:
+ return self.template_id
+
+ def get_llm_model_name(self) -> str:
+ if self.llm_model_name is None:
+ return AIStorage.get_instance().get_user_config().get_value("llm_model_name")
+
+ return self.llm_model_name
+
+ def get_max_token_size(self) -> int:
+ return self.max_token_size
+
+ def get_agent_role_prompt(self) -> LLMPrompt:
+ return self.role_prompt
+
+ def get_agent_prompt(self) -> LLMPrompt:
+ return self.agent_prompt
+
+ async def _get_context_info(self) -> Dict:
+ context_info = {}
+
+ context_info["location"] = "SanJose"
+ context_info["now"] = datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S")
+ context_info["weather"] = "Partly Cloudy, 60°F"
+ context_info["owner"] = AIStorage.get_instance().get_user_config().get_value("username")
+
+ return context_info
+
+ async def llm_process_msg(self,msg:AgentMsg) -> AgentMsg:
+ need_process:bool = True
+ if msg.msg_type == AgentMsgType.TYPE_GROUPMSG:
+ need_process = False
+
+ session_topic = msg.target + "#" + msg.topic
+ chatsession = AIChatSession.get_session(self.agent_id,session_topic,self.memory.memory_db)
+ if msg.mentions is not None:
+ if self.agent_id in msg.mentions:
+ need_process = True
+ logger.info(f"agent {self.agent_id} recv a group chat message from {msg.sender},but is not mentioned,ignore!")
+
+ if need_process is not True:
+ chatsession.append(msg)
+ resp_msg = msg.create_group_resp_msg(self.agent_id,"")
+ return resp_msg
+
+ context_info = await self._get_context_info()
+ input_parms = {
+ "msg":msg,
+ "context_info":context_info
+ }
+ msg_process = self.behaviors.get("on_message")
+ llm_result : LLMResult = await msg_process.process(input_parms)
+ if llm_result.state == LLMResultStates.ERROR:
+ error_resp = msg.create_error_resp(llm_result.error_str)
+ return error_resp
+ elif llm_result.state == LLMResultStates.IGNORE:
+ return None
+ else: # OK
+ if llm_result.raw_result is not None:
+ resp_msg = llm_result.raw_result.get("_resp_msg")
+ return resp_msg
+ else:
+ return msg.create_resp_msg(llm_result.resp)
+
+ async def _process_msg(self,msg:AgentMsg,workspace = None) -> AgentMsg:
+ return await self.llm_process_msg(msg)
+
+
+ async def llm_self_think(self):
+ llm_process : BaseLLMProcess = self.behaviors.get("self_thinking")
+ if llm_process:
+ logger.info(f"agent {self.agent_id} self thinking start!")
+
+ context_info = await self._get_context_info()
+ #chat_history = AIChatSession.list_session(self.agent_id,self.memory.memory_db)
+ #known_experience_list = await self.memory.list_experience()
+ #record_list = await self.memory.load_records(await self.memory.get_last_think_time())
+
+ input_parms = {
+ "context_info":context_info
+ }
+
+ llm_result : LLMResult = await llm_process.process(input_parms)
+ if llm_result.state == LLMResultStates.ERROR:
+ logger.error(f"llm process self thinking error:{llm_result.compute_error_str}")
+ elif llm_result.state == LLMResultStates.IGNORE:
+ logger.info(f"llm process self thinking ignore!")
+ else:
+ logger.info(f"llm process self thinking ok!,think is:{llm_result.resp}")
+ await self.memory.set_last_think_time(time.time())
+ self.agent_energy -= 2
+ return
+
+ async def llm_triage_tasklist(self):
+ llm_process : BaseLLMProcess = self.behaviors.get("triage_tasks")
+ if llm_process:
+ if self.prviate_workspace:
+ filter = {}
+ filter["state"] = AgentTaskState.TASK_STATE_WAIT
+
+ tasklist:List[AgentTask]= await self.prviate_workspace.task_mgr.list_task(filter)
+
+
+ if tasklist:
+ if len(tasklist) > 0:
+ simple_list:List[Dict] = []
+ for task in tasklist:
+ simple_list.append(task.to_simple_dict())
+
+ input_parms = {
+ "tasklist":simple_list,
+ "context_info": await self._get_context_info()
+ }
+ llm_result : LLMResult = await llm_process.process(input_parms)
+ if llm_result.state == LLMResultStates.ERROR:
+ logger.error(f"llm process triage_tasks error:{llm_result.compute_error_str}")
+ elif llm_result.state == LLMResultStates.IGNORE:
+ logger.info(f"llm process triage_tasks ignore!")
+ else:
+ logger.info(f"llm process triage_tasks ok!,think is:{llm_result.resp}")
+ self.agent_energy -= 3
+
+ # for agent_task in tasklist:
+ # if self.agent_energy <= 0:
+ # break
+
+ # if agent_task.state == AgentTaskState.TASK_STATE_WAIT:
+ # input_parms = {
+ # "task":agent_task
+ # }
+ # llm_result : LLMResult = await llm_process.process(input_parms)
+ # if llm_result.state == LLMResultStates.ERROR:
+ # logger.error(f"llm process review_task error:{llm_result.error_str}")
+ # continue
+ # elif llm_result.state == LLMResultStates.IGNORE:
+ # logger.info(f"llm process review_task ignore!")
+ # continue
+ # else:
+ # determine = llm_result.raw_result.get("determine")
+ # logger.info(f"llm process review_task ok!,think is:{determine}")
+ # self.agent_energy -= 1
+
+ async def llm_do_todo(self, todo: AgentTodo):
+ llm_process : BaseLLMProcess = self.behaviors.get("do")
+ logger.info(f"agent {self.agent_id} DO todo {todo.todo_path} start!")
+ if llm_process:
+ input_parms = {
+ "todo":todo,
+ "context_info": await self._get_context_info()
+ }
+ llm_result : LLMResult = await llm_process.process(input_parms)
+ if llm_result.state == LLMResultStates.ERROR:
+ logger.error(f"llm process do_todo error:{llm_result.compute_error_str}")
+ elif llm_result.state == LLMResultStates.IGNORE:
+ logger.info(f"llm process do_todo ignore!")
+ else:
+ logger.info(f"llm process do_todo ok!,think is:{llm_result.resp}")
+ self.agent_energy -= 1
+
+ async def llm_check_todo(self, todo: AgentTodo):
+ llm_process : BaseLLMProcess = self.behaviors.get("check")
+ logger.info(f"agent {self.agent_id} CHECK todo {todo.todo_path} start!")
+ if llm_process:
+ input_parms = {
+ "todo":todo,
+ "context_info": await self._get_context_info()
+ }
+ llm_result : LLMResult = await llm_process.process(input_parms)
+ if llm_result.state == LLMResultStates.ERROR:
+ logger.error(f"llm process check_todo error:{llm_result.compute_error_str}")
+ elif llm_result.state == LLMResultStates.IGNORE:
+ logger.info(f"llm process check_todo ignore!")
+ else:
+ logger.info(f"llm process check_todo ok!,think is:{llm_result.resp}")
+ self.agent_energy -= 1
+
+ return
+
+ async def llm_plan_task(self,task:AgentTask):
+ llm_process : BaseLLMProcess = self.behaviors.get("plan_task")
+ logger.info(f"agent {self.agent_id} PLAN task {task.task_path} start!")
+ if llm_process:
+ input_parms = {
+ "task":task,
+ "context_info": await self._get_context_info()
+ }
+ llm_result : LLMResult = await llm_process.process(input_parms)
+ if llm_result.state == LLMResultStates.ERROR:
+ logger.error(f"llm process plan_task error:{llm_result.compute_error_str}")
+ elif llm_result.state == LLMResultStates.IGNORE:
+ logger.info(f"llm process plan_task ignore!")
+ else:
+ logger.info(f"llm process plan_task ok!,think is:{llm_result.resp}")
+ self.agent_energy -= 1
+
+ async def llm_review_task(self,task:AgentTask):
+ llm_process : BaseLLMProcess = self.behaviors.get("review_task")
+ logger.info(f"agent {self.agent_id} REVIEW task {task.task_path} start!")
+ if llm_process:
+ input_parms = {
+ "task":task,
+ "context_info": await self._get_context_info()
+ }
+ llm_result : LLMResult = await llm_process.process(input_parms)
+ if llm_result.state == LLMResultStates.ERROR:
+ logger.error(f"llm process review_task error:{llm_result.compute_error_str}")
+ elif llm_result.state == LLMResultStates.IGNORE:
+ logger.info(f"llm process review_task ignore!")
+ else:
+ logger.info(f"llm process review_task ok!,think is:{llm_result.resp}")
+ self.agent_energy -= 1
+
+
+ async def _self_imporve(self):
+ await self.llm_self_think()
+
+ def wake_up(self) -> None:
+ if self.agent_task is None:
+ self.agent_task = asyncio.create_task(self._on_timer())
+ else:
+ logger.warning(f"agent {self.agent_id} is already wake up!")
+
+ async def _on_timer(self):
+ await asyncio.sleep(5)
+ while True:
+ try:
+ now = time.time()
+ if self.last_recover_time is None:
+ self.last_recover_time = now
+ else:
+ if now - self.last_recover_time > 60:
+ self.agent_energy += (now - self.last_recover_time) / 60
+ self.last_recover_time = now
+ logger.info(f"agent {self.agent_id} recover energy to {self.agent_energy}")
+
+ if self.agent_energy <= 1:
+ logger.info(f"agent {self.agent_id} energy is too low!, goto sleep!")
+ continue
+
+ await self.llm_triage_tasklist()
+ # Get un finished tasks
+ #filter = {}
+ #filter["state"] = AgentTaskState.TASK_STATE_WAIT
+ filter = None
+ task_list:List[AgentTask] = await self.prviate_workspace.task_mgr.list_task(filter)
+
+ for task in task_list:
+ if self.agent_energy <= 0:
+ break
+
+ task = await self.prviate_workspace.task_mgr.get_task(task.task_id)
+ if task.can_plan():
+ # PLAN Task
+ await self.llm_plan_task(task)
+ task = await self.prviate_workspace.task_mgr.get_task(task.task_id)
+
+ if task.state == AgentTaskState.TASK_STATE_DOING:
+ # DO or Check Todo
+ can_review = False
+ todolist = await self.prviate_workspace.task_mgr.get_sub_todos(task.task_id)
+ for todo in todolist:
+ if self.agent_energy <= 0:
+ break
+ task = await self.prviate_workspace.task_mgr.get_task(task.task_id)
+ todo = await self.prviate_workspace.task_mgr.get_todo_by_id(todo.todo_id)
+ if task.state != AgentTaskState.TASK_STATE_DOING:
+ break
+ if todo.state == AgentTodoState.TODO_STATE_WAITING or todo.state == AgentTodoState.TODO_STATE_EXEC_FAILED:
+ await self.llm_do_todo(todo)
+ task = await self.prviate_workspace.task_mgr.get_task(task.task_id)
+ todo = await self.prviate_workspace.task_mgr.get_todo_by_id(todo.todo_id)
+ if task.state != AgentTaskState.TASK_STATE_DOING:
+ break
+ if todo.state == AgentTodoState.TODO_STATE_EXEC_OK:
+ await self.llm_check_todo(todo)
+
+ if can_review:
+ task.state = AgentTaskState.TASK_STATE_WAITING_REVIEW
+
+ task = await self.prviate_workspace.task_mgr.get_task(task.task_id)
+ if task.state == AgentTaskState.TASK_STATE_WAITING_REVIEW:
+ await self.llm_review_task(task)
+
+ await self._self_imporve()
+
+
+
+ except Exception as e:
+ tb_str = traceback.format_exc()
+ logger.error(f"agent {self.agent_id} on timer error:{e},{tb_str}")
+
+ # Because the LLM itself is very slow, the accuracy of the system processing task is in minutes.
+ await asyncio.sleep(30)
+
+
+
+
+
+
+
diff --git a/src/aios/agent/agent_base.py b/src/aios/agent/agent_base.py
new file mode 100644
index 0000000..d8b84aa
--- /dev/null
+++ b/src/aios/agent/agent_base.py
@@ -0,0 +1,38 @@
+# pylint:disable=E0402
+
+import abc
+from abc import abstractmethod
+
+from ..proto.agent_msg import AgentMsg
+
+class BaseAIAgent(abc.ABC):
+ @abstractmethod
+ def get_id(self) -> str:
+ pass
+
+ @abstractmethod
+ def get_llm_model_name(self) -> str:
+ pass
+
+ @abstractmethod
+ def get_max_token_size(self) -> int:
+ pass
+
+ @abstractmethod
+ async def _process_msg(self,msg:AgentMsg,workspace = None) -> AgentMsg:
+ pass
+
+class CustomAIAgent(BaseAIAgent):
+ def __init__(self, agent_id: str, llm_model_name: str, max_token_size: int) -> None:
+ self.agent_id = agent_id
+ self.llm_model_name = llm_model_name
+ self.max_token_size = max_token_size
+
+ def get_id(self) -> str:
+ return self.agent_id
+
+ def get_llm_model_name(self) -> str:
+ return self.llm_model_name
+
+ def get_max_token_size(self) -> int:
+ return self.max_token_size
diff --git a/src/aios/agent/agent_memory.py b/src/aios/agent/agent_memory.py
new file mode 100644
index 0000000..c660a76
--- /dev/null
+++ b/src/aios/agent/agent_memory.py
@@ -0,0 +1,507 @@
+# pylint:disable=E0402
+from datetime import datetime,timedelta
+import json
+import os
+import threading
+from typing import Dict, List
+import sqlite3
+
+import aiofiles
+
+from ..storage.storage import AIStorage
+from ..frame.compute_kernel import ComputeKernel
+from ..frame.contact_manager import ContactManager
+from ..frame.contact import Contact
+from ..proto.ai_function import ParameterDefine, SimpleAIAction, SimpleAIFunction
+from ..proto.agent_msg import AgentMsg, AgentMsgType
+from ..proto.agent_task import AgentWorkLog
+
+from .llm_context import GlobaToolsLibrary
+from .chatsession import AIChatSession
+
+import logging
+
+logger = logging.getLogger(__name__)
+
+
+#class ObjectSummary:
+# def __init__(self) -> None:
+# self.summary : str = None
+# self.object_name : str = None
+# self.priority : int = 5
+ # [info_source, info]
+# self.infos : Dict[str,str] = {}
+
+
+
+class AgentMemory:
+ def __init__(self,agent_id:str,base_dir:str) -> None:
+ self.agent_memory_base_dir = base_dir
+ self.agent_id:str= agent_id
+
+ AIStorage.get_instance().ensure_directory_exists(self.agent_memory_base_dir)
+ AIStorage.get_instance().ensure_directory_exists(f"{self.agent_memory_base_dir}/experience")
+ AIStorage.get_instance().ensure_directory_exists(f"{self.agent_memory_base_dir}/contacts")
+ AIStorage.get_instance().ensure_directory_exists(f"{self.agent_memory_base_dir}/relations")
+ AIStorage.get_instance().ensure_directory_exists(f"{self.agent_memory_base_dir}/summary")
+
+ self.memory_db:str = f"{self.agent_memory_base_dir}/memory.db"
+ self.model_name:str = "gp4-1106-preview"
+ self.threshold_hours = 72
+ self.last_think_time : float = 0.0
+
+ self.load_memory_meta()
+
+
+ def _get_conn(self):
+ """ get db connection """
+ local = threading.local()
+ if not hasattr(local, 'conn'):
+ local.conn = self._create_connection(self.memory_db)
+ return local.conn
+
+ def _create_connection(self, db_file):
+ """ create a database connection to a SQLite database """
+ conn = None
+ try:
+ conn = sqlite3.connect(db_file)
+ except Error as e:
+ logging.error("Error occurred while connecting to database: %s", e)
+ return None
+
+ if conn:
+ self._create_table(conn)
+
+ return conn
+
+ def get_session_from_msg(self,msg:AgentMsg) -> AIChatSession:
+ if msg.msg_type == AgentMsgType.TYPE_GROUPMSG:
+ session_topic = msg.target + "#" + msg.topic
+ chatsession = AIChatSession.get_session(self.agent_id,session_topic,self.memory_db)
+ else:
+ session_topic = msg.get_sender() + "#" + msg.topic
+ chatsession = AIChatSession.get_session(self.agent_id,session_topic,self.memory_db)
+ return chatsession
+
+ # return last record time
+ async def load_records(self,starttime,tokenlimit=8000)->float:
+ # 专用思路：做聊天记录/工作经验的整理
+ # 通用思路：没有具体的目的，让Agent根据提示词自己工作（可能效果很差也可能很好）
+ # 先实现通用思路
+ msg_records = AIChatSession.load_message_records_by_agentid(self.agent_id,starttime,32,self.memory_db)
+ work_records = self.load_worklogs(self.agent_id,token_limit=tokenlimit)
+ pass
+
+ async def load_chatlogs(self,msg:AgentMsg,token_limit=800):
+ chatsession = self.get_session_from_msg(msg)
+ # Must load n (n> = 2), and hope to load the M
+ # The information in the # M is gradually added, knowing that it is less than 72 hours from the current time, and consumes enough tokens
+
+ messages_n = chatsession.read_history() # read
+ histroy_str = ""
+ read_count = 0
+ is_all = True
+ for msg in messages_n:
+ dt = datetime.fromtimestamp(float(msg.create_time))
+ formatted_time = dt.strftime('%y-%m-%d %H:%M:%S')
+ record_str = f"{msg.sender},[{formatted_time}]\n{msg.body}\n"
+ token_limit -= ComputeKernel.llm_num_tokens_from_text(record_str,self.model_name)
+ if token_limit <= 32:
+ is_all = False
+ break
+ read_count += 1
+ histroy_str = record_str + histroy_str
+
+ return histroy_str,is_all
+
+ async def get_chat_summary(self,msg:AgentMsg) -> str:
+ chatsession : AIChatSession = self.get_session_from_msg(msg)
+ return chatsession.summary
+
+ # async def action_chatlog_append(self,params:Dict) -> str:
+ #
+ # input_msg:AgentMsg = params.get("input").get("msg")
+ # llm_result = params.get("llm_result")
+ # chatsession = self.get_session_from_msg(input_msg)
+ # resp_msg = params.get("resp_msg")
+ # if resp_msg:
+ # tags = llm_result.raw_result.get("tags")
+ # chatsession.append(input_msg,tags)
+ # chatsession.append(resp_msg,tags)
+
+ # return "OK"
+
+ async def load_worklogs(self,operator_id:str,owner_id:str=None, work_types:List[str]=None,token_limit=800):
+ conn = self._get_conn()
+ c = conn.cursor()
+
+ query = 'SELECT * FROM worklog WHERE 1=1'
+ params = []
+
+ if operator_id is not None:
+ query += ' AND operator=?'
+ params.append(operator_id)
+
+ if owner_id is not None:
+ query += ' AND owner_id=?'
+ params.append(owner_id)
+
+ if work_types:
+ query += ' AND work_type IN ({})'.format(', '.join('?'*len(work_types)))
+ params.extend(work_types)
+
+ query += ' ORDER BY timestamp DESC LIMIT 8'
+
+ c.execute(query, tuple(params))
+ rows = c.fetchall()
+
+
+ return [self.from_db_row(row) for row in rows]
+
+ def _create_table(self,conn):
+ c = conn.cursor()
+ c.execute('''
+ CREATE TABLE IF NOT EXISTS worklog (
+ logid TEXT PRIMARY KEY,
+ owner_id TEXT,
+ work_type TEXT,
+ timestamp REAL,
+ content TEXT,
+ result TEXT,
+ meta TEXT,
+ operator TEXT
+ )
+ ''')
+ conn.commit()
+ #conn.close()
+
+ @classmethod
+ def from_db_row(self,row):
+ log = AgentWorkLog()
+ # 这里高度依赖表结构的顺序
+ log.logid, log.owner_id, log.work_type, log.timestamp, log.content, log.result, meta_str, log.operator = row
+ log.meta = json.loads(meta_str) if meta_str else None
+ return log
+
+ async def append_worklog(self,log:AgentWorkLog)->str:
+ conn = self._get_conn()
+ c = conn.cursor()
+ # 将meta字典转换为JSON字符串
+ meta_str = json.dumps(log.meta,ensure_ascii=False) if log.meta else None
+ c.execute('''
+ INSERT INTO worklog (logid, owner_id, work_type, timestamp, content, result, meta, operator)
+ VALUES (?, ?, ?, ?, ?, ?, ?, ?)
+ ''', (log.logid, log.owner_id, log.work_type, log.timestamp, log.content, log.result, meta_str, log.operator))
+ conn.commit()
+ #conn.close()
+
+ def memory_meta_to_dict(self) -> Dict:
+ return {
+ "last_think_time" : self.last_think_time
+ }
+
+ def load_meta(self,Dict):
+ self.last_think_time = Dict.get("last_think_time",0.0)
+
+ def load_memory_meta(self):
+ meta_file_path = f"{self.agent_memory_base_dir}/meta.json"
+ try:
+ with open(meta_file_path, mode='r') as file:
+ meta = json.load(file)
+ self.load_meta(meta)
+
+ except Exception as e:
+ logger.error(f"load memory meta failed: {e}")
+ self.last_think_time = 0.0
+
+ def save_memory_meta(self):
+ meta_file_path = f"{self.agent_memory_base_dir}/meta.json"
+ try:
+ with open(meta_file_path, mode='w') as file:
+ meta = self.memory_meta_to_dict()
+ json.dump(meta,file)
+ except Exception as e:
+ logger.error(f"save memory meta failed: {e}")
+
+ async def get_last_think_time(self)->float:
+ return self.last_think_time
+
+ async def set_last_think_time(self,last_time:float):
+ self.last_think_time = last_time
+ self.save_memory_meta()
+
+ async def get_contact_summary(self,contact_id:str) -> str:
+ if contact_id is None:
+ return "Contact id is None"
+
+ result = {}
+ contact_info:Contact = ContactManager.get_instance().find_contact_by_name(contact_id)
+ if contact_info:
+ result["name"] = contact_info.name
+ result["relation"] = contact_info.relationship
+ result["notes"] = contact_info.notes
+
+ summary_path = f"{self.agent_memory_base_dir}/contacts/{contact_id}.summary"
+ try:
+ async with aiofiles.open(summary_path, mode='r') as file:
+ result["summary"] = await file.read()
+
+ except Exception as e:
+ logger.error(f"read contact summary failed: {e}")
+
+ return json.dumps(result,ensure_ascii=False)
+
+ async def update_contact_summary(self,contact_id:str,summary:str):
+ summary_path = f"{self.agent_memory_base_dir}/contacts/{contact_id}.summary"
+ try:
+ async with aiofiles.open(summary_path, mode='w') as file:
+ await file.write(summary)
+ return "OK"
+ except Exception as e:
+ logger.error(f"write contact summary failed: {e}")
+ return "write contact summary failed: {e}"
+
+ async def get_summary(self,object_name:str) -> str:
+ summary_path = f"{self.agent_memory_base_dir}/{object_name}.summary"
+ try:
+ async with aiofiles.open(summary_path, mode='r') as file:
+ return await file.read()
+ except Exception as e:
+ logger.error(f"read summary failed: {e}")
+ return f"read summary failed: {e}"
+
+ async def update_summary(self,object_name:str,summary:str) -> str:
+ summary_path = f"{self.agent_memory_base_dir}/{object_name}.summary"
+ try:
+ async with aiofiles.open(summary_path, mode='w') as file:
+ await file.write(summary)
+ return "OK"
+ except Exception as e:
+ logger.error(f"write summary failed: {e}")
+ return f"write summary failed: {e}"
+
+ async def list_summary_object_names(self) -> List[str]:
+ # list dir
+ try:
+ contents = os.listdir(self.agent_memory_base_dir)
+ return [x for x in contents if x.endswith(".summary")]
+ except Exception as e:
+ logger.error(f"list summary object names failed: {e}")
+ return []
+
+ # means object1 feel object2 is ...
+ async def get_relation_summary(self,object_name1:str,object_name2:str) -> str:
+ summary_path = f"{self.agent_memory_base_dir}/relations/{object_name1}.relation.{object_name2}.summary"
+ try:
+ async with aiofiles.open(summary_path, mode='r') as file:
+ await file.read()
+ except FileNotFoundError:
+ return "no summary"
+ except Exception as e:
+ logger.error(f"read relation summary failed: {e}")
+ return f"read relation summary failed: {e}"
+
+
+ async def update_relation_summary(self,object_name1:str,object_name2:str,summary:Dict):
+ summary_path = f"{self.agent_memory_base_dir}/relations/{object_name1}.relation.{object_name2}.summary"
+ try:
+ async with aiofiles.open(summary_path, mode='w') as file:
+ await file.write(json.dumps(summary))
+ return "OK"
+ except Exception as e:
+ logger.error(f"write relation summary failed: {e}")
+ return "write relation summary failed: {e}"
+
+ async def get_experience(self,topic_name:str) -> str:
+ experience_path = f"{self.agent_memory_base_dir}/experience/{topic_name}.experience"
+ try:
+ async with aiofiles.open(experience_path, mode='r') as file:
+ await file.read()
+ except FileNotFoundError:
+ return "no experience"
+ except Exception as e:
+ logger.error(f"read experience failed: {e}")
+ return f"read experience failed: {e}"
+
+
+ async def set_experience(self,topic_name:str,summary:str) -> str:
+ experience_path = f"{self.agent_memory_base_dir}/experience/{topic_name}.experience"
+ try:
+ async with aiofiles.open(experience_path, mode='w') as file:
+ await file.write(summary)
+ return "OK"
+ except Exception as e:
+ logger.error(f"write experience failed: {e}")
+ return "write experience failed: {e}"
+
+ async def list_experience(self) -> List[str]:
+ dir_path = f"{self.agent_memory_base_dir}/experience"
+ try:
+ contents = os.listdir(dir_path)
+ return [x for x in contents if x.endswith(".experience")]
+ except Exception as e:
+ logger.error(f"list experience failed: {e}")
+ return []
+
+ @staticmethod
+ def register_ai_functions():
+ async def update_chat_summary(parameters):
+ agent_memory:AgentMemory = parameters.get("_memory")
+ chatsession = AIChatSession.get_session_by_id(parameters.get("session_id"),agent_memory.memory_db)
+ summary = parameters.get("summary")
+ chatsession.update_summary(summary)
+ return "OK"
+ parameters = ParameterDefine.create_parameters({
+ "session_id": {"type": "string", "description": "session id"},
+ "summary": {"type": "string", "description": "new summary"}
+ })
+ update_chat_summary_func = SimpleAIFunction("agent.memory.update_chat_summary",
+ "update chat summary",
+ update_chat_summary,
+ parameters)
+ GlobaToolsLibrary.get_instance().register_tool_function(update_chat_summary_func)
+
+ # async def get_contact_summary(parameters):
+ # agent_memory:AgentMemory = parameters.get("_memory")
+ # contact_name = parameters.get("contact_name")
+ # return await agent_memory.get_contact_summary(contact_name)
+ # parameters = ParameterDefine.create_parameters({
+ # "contact_name": {"type": "string", "description": "contact name"}
+ # })
+ # get_contact_summary_func = SimpleAIFunction("agent.memory.get_contact_summary",
+ # "get contact summary",
+ # get_contact_summary,
+ # parameters)
+ # GlobaToolsLibrary.register_tool_function(get_contact_summary_func)
+
+ # async def update_contact_summary(parameters):
+ # agent_memory:AgentMemory = parameters.get("_memory")
+ # contact_name = parameters.get("contact_name")
+ # summary = parameters.get("summary")
+ # return await agent_memory.update_contact_summary(contact_name,summary)
+ # parameters = ParameterDefine.create_parameters({
+ # "contact_name": {"type": "string", "description": "contact name"},
+ # "summary": {"type": "string", "description": "new summary"}
+ # })
+ # update_contact_summary_func = SimpleAIFunction("agent.memory.update_contact_summary",
+ # "update contact summary",
+ # update_contact_summary,
+ # parameters)
+ # GlobaToolsLibrary.register_tool_function(update_contact_summary_func)
+
+ # async def get_summary(parameters):
+ # agent_memory:AgentMemory = parameters.get("_memory")
+ # object_name = parameters.get("object_name")
+ # return await agent_memory.get_summary(object_name)
+ # parameters = ParameterDefine.create_parameters({
+ # "object_name": {"type": "string", "description": "object name"}
+ # })
+ # get_summary_func = SimpleAIFunction("agent.memory.get_summary",
+ # "get summary of sth",
+ # get_summary,
+ # parameters)
+ # GlobaToolsLibrary.register_tool_function(get_summary_func)
+
+ # async def update_summary(parameters):
+ # agent_memory:AgentMemory = parameters.get("_memory")
+ # object_name = parameters.get("object_name")
+ # summary = parameters.get("summary")
+ # return await agent_memory.update_summary(object_name,summary)
+ # parameters = ParameterDefine.create_parameters({
+ # "object_name": {"type": "string", "description": "object name"},
+ # "summary": {"type": "string", "description": "new summary"}
+ # })
+ # update_summary_func = SimpleAIFunction("agent.memory.update_summary",
+ # "update summary of sth",
+ # update_summary,
+ # parameters)
+ # GlobaToolsLibrary.register_tool_function(update_summary_func)
+
+ # async def list_summary_object_names(parameters):
+ # agent_memory:AgentMemory = parameters.get("_memory")
+ # return await agent_memory.list_summary_object_names()
+ # parameters = ParameterDefine.create_parameters({})
+ # list_summary_object_names_func = SimpleAIFunction("agent.memory.list_summary",
+ # "list summary object names",
+ # list_summary_object_names,
+ # parameters)
+ # GlobaToolsLibrary.register_tool_function(list_summary_object_names_func)
+
+ # async def get_relation_summary(parameters):
+ # agent_memory:AgentMemory = parameters.get("_memory")
+ # object_name1 = parameters.get("object1_name")
+ # object_name2 = parameters.get("object2_name")
+ # return await agent_memory.get_relation_summary(object_name1,object_name2)
+ # parameters = ParameterDefine.create_parameters({
+ # "object1_name": {"type": "string", "description": "object name1"},
+ # "object2_name": {"type": "string", "description": "object name2"}
+ # })
+ # get_relation_summary_func = SimpleAIFunction("agent.memory.get_relation_summary",
+ # "object1 feel object2 is ...",
+ # get_relation_summary,
+ # parameters)
+ # GlobaToolsLibrary.register_tool_function(get_relation_summary_func)
+
+ # async def update_relation_summary(parameters):
+ # agent_memory:AgentMemory = parameters.get("_memory")
+ # object_name1 = parameters.get("object1_name")
+ # object_name2 = parameters.get("object2_name")
+ # summary = parameters.get("summary")
+ # return await agent_memory.update_relation_summary(object_name1,object_name2,summary)
+ # parameters = ParameterDefine.create_parameters({
+ # "object1_name": {"type": "string", "description": "object name1"},
+ # "object2_name": {"type": "string", "description": "object name2"},
+ # "summary": {"type": "string", "description": "new summary"}
+ # })
+ # update_relation_summary_func = SimpleAIFunction("agent.memory.update_relation_summary",
+ # "object1 feel object2 is ...",
+ # update_relation_summary,
+ # parameters)
+ # GlobaToolsLibrary.register_tool_function(update_relation_summary_func)
+
+ # async def get_experience(parameters):
+ # agent_memory:AgentMemory = parameters.get("_memory")
+ # topic_name = parameters.get("topic_name")
+ # return await agent_memory.get_experience(topic_name)
+ # parameters = ParameterDefine.create_parameters({
+ # "topic_name": {"type": "string", "description": "topic name"}
+ # })
+ # get_experience_func = SimpleAIFunction("agent.memory.get_experience",
+ # "get experience",
+ # get_experience,
+ # parameters)
+ # GlobaToolsLibrary.register_tool_function(get_experience_func)
+
+ # async def set_experience(parameters):
+ # agent_memory:AgentMemory = parameters.get("_memory")
+ # topic_name = parameters.get("topic_name")
+ # summary = parameters.get("summary")
+ # return await agent_memory.set_experience(topic_name,summary)
+ # parameters = ParameterDefine.create_parameters({
+ # "topic_name": {"type": "string", "description": "topic name"},
+ # "summary": {"type": "string", "description": "new summary"}
+ # })
+ # set_experience_func = SimpleAIFunction("agent.memory.set_experience",
+ # "set experience",
+ # set_experience,
+ # parameters)
+ # GlobaToolsLibrary.register_tool_function(set_experience_func)
+
+ # async def list_experience(parameters):
+ # agent_memory:AgentMemory = parameters.get("_memory")
+ # return await agent_memory.list_experience()
+ # parameters = ParameterDefine.create_parameters({})
+ # list_experience_func = SimpleAIFunction("agent.memory.list_experience",
+ # "list exist experience topics",
+ # list_experience,
+ # parameters)
+ # GlobaToolsLibrary.register_tool_function(list_experience_func)
+
+
+
+
+
+
+
+
+
diff --git a/src/aios_kernel/chatsession.py b/src/aios/agent/chatsession.py
similarity index 71%
rename from src/aios_kernel/chatsession.py
rename to src/aios/agent/chatsession.py
index 1290d80..1f69e65 100644
--- a/src/aios_kernel/chatsession.py
+++ b/src/aios/agent/chatsession.py
@@ -1,4 +1,4 @@
-
+# pylint:disable=E0402
import sqlite3 # Because sqlite3 IO operation is small, so we can use sqlite3 directly.(so we don't need to use async sqlite3 now)
from sqlite3 import Error
import logging
@@ -6,21 +6,22 @@ import threading
import datetime
import uuid
import json
+from typing import List
-from .agent_message import AgentMsgType, AgentMsg, AgentMsgStatus
+from ..proto.agent_msg import AgentMsgType, AgentMsg, AgentMsgStatus
class ChatSessionDB:
def __init__(self, db_file):
""" initialize db connection """
- self.local = threading.local()
self.db_file = db_file
self._get_conn()
def _get_conn(self):
""" get db connection """
- if not hasattr(self.local, 'conn'):
- self.local.conn = self._create_connection(self.db_file)
- return self.local.conn
+ local = threading.local()
+ if not hasattr(local, 'conn'):
+ local.conn = self._create_connection(self.db_file)
+ return local.conn
def _create_connection(self, db_file):
""" create a database connection to a SQLite database """
@@ -37,9 +38,10 @@ class ChatSessionDB:
return conn
def close(self):
- if not hasattr(self.local, 'conn'):
+ local = threading.local()
+ if not hasattr(local, 'conn'):
return
- self.local.conn.close()
+ local.conn.close()
def _create_table(self, conn):
""" create table """
@@ -52,7 +54,8 @@ class ChatSessionDB:
SessionTopic TEXT,
StartTime TEXT,
SummarizePos INTEGER,
- Summary TEXT
+ Summary TEXT,
+ ThreadID TEXT
);
""")
@@ -82,28 +85,29 @@ class ChatSessionDB:
ActionResult TEXT,
DoneTime TEXT,
- Status INTEGER
+ Status INTEGER,
+ Tags TEXT
);
""")
conn.commit()
except Error as e:
logging.error("Error occurred while creating tables: %s", e)
- def insert_chatsession(self, session_id, session_owner,session_topic, start_time):
+ def insert_chatsession(self, session_id, session_owner,session_topic, start_time,thread_id = ""):
""" insert a new session into the ChatSessions table """
try:
conn = self._get_conn()
conn.execute("""
- INSERT INTO ChatSessions (SessionID, SessionOwner,SessionTopic, StartTime,SummarizePos,Summary)
- VALUES (?,?, ?, ?,0,"")
- """, (session_id, session_owner,session_topic, start_time))
+ INSERT INTO ChatSessions (SessionID, SessionOwner,SessionTopic, StartTime,SummarizePos,Summary,ThreadID)
+ VALUES (?,?, ?, ?,0,"",?)
+ """, (session_id, session_owner,session_topic, start_time,thread_id))
conn.commit()
return 0 # return 0 if successful
except Error as e:
logging.error("Error occurred while inserting session: %s", e)
return -1 # return -1 if an error occurs
- def insert_message(self, msg:AgentMsg):
+ def insert_message(self, msg:AgentMsg,tags:List[str] = None):
""" insert a new message into the Messages table """
try:
action_name = None
@@ -111,29 +115,31 @@ class ChatSessionDB:
action_result = None
mentions = None
if msg.mentions:
- mentions = json.dumps(msg.mentions)
+ mentions = json.dumps(msg.mentions,ensure_ascii=False)
match msg.msg_type:
case AgentMsgType.TYPE_MSG:
pass
- case AgentMsgType.TYPE_ACTION:
+ case AgentMsgType.TYPE_ACTION:# THIS Action is not AIAction
action_name = msg.func_name
- action_params = json.dumps(msg.args)
+ action_params = json.dumps(msg.args,ensure_ascii=False)
action_result = msg.result_str
case AgentMsgType.TYPE_INTERNAL_CALL:
action_name = msg.func_name
- action_params = json.dumps(msg.args)
+ action_params = json.dumps(msg.args,ensure_ascii=False)
action_result = msg.result_str
case AgentMsgType.TYPE_EVENT:
action_name = msg.event_name
- action_params = json.dumps(msg.event_args)
-
+ action_params = json.dumps(msg.event_args,ensure_ascii=False)
+ if tags is None:
+ tags = []
+ str_tags = ','.join(tags)
conn = self._get_conn()
conn.execute("""
- INSERT INTO Messages (MessageID, SessionID, MsgType, PrevMsgID, SenderID, ReceiverID, Timestamp, Topic,Mentions,ContentMIME,Content,ActionName,ActionParams,ActionResult,DoneTime,Status)
- VALUES (?, ?, ?, ?, ?, ?, ?,?, ?, ?, ?, ?, ?, ?, ?, ?)
- """, (msg.msg_id, msg.session_id, msg.msg_type.value, msg.prev_msg_id, msg.sender, msg.target, msg.create_time, msg.topic,mentions,msg.body_mime,msg.body,action_name,action_params,action_result,msg.done_time,msg.status.value))
+ INSERT INTO Messages (MessageID, SessionID, MsgType, PrevMsgID, SenderID, ReceiverID, Timestamp, Topic,Mentions,ContentMIME,Content,ActionName,ActionParams,ActionResult,DoneTime,Status,Tags)
+ VALUES (?, ?, ?, ?, ?, ?, ?,?, ?, ?, ?, ?, ?, ?, ?, ?,?)
+ """, (msg.msg_id, msg.session_id, msg.msg_type.value, msg.prev_msg_id, msg.sender, msg.target, msg.create_time, msg.topic,mentions,msg.body_mime,msg.body,action_name,action_params,action_result,msg.done_time,msg.status.value,str_tags))
conn.commit()
if msg.inner_call_chain:
@@ -192,6 +198,9 @@ class ChatSessionDB:
try:
conn = self._get_conn()
cursor = conn.cursor()
+ if limit == 0:
+ limit = 1024
+
cursor.execute("""
SELECT MessageID, SessionID, MsgType, PrevMsgID, SenderID, ReceiverID, Timestamp, Topic,Mentions,ContentMIME,Content,ActionName,ActionParams,ActionResult,DoneTime,Status FROM Messages
WHERE SessionID = ?
@@ -205,12 +214,31 @@ class ChatSessionDB:
logging.error("Error occurred while getting messages: %s", e)
return -1, None # return -1 and None if an error occurs
+ def load_message_by_agentid(self,agent_id,limit,start_time="1970-01-01 00:00:00"):
+ try:
+ conn = self._get_conn()
+ cursor = conn.cursor()
+ cursor.execute("""
+ SELECT MessageID, SessionID, MsgType, PrevMsgID, SenderID, ReceiverID, Timestamp, Topic,Mentions,ContentMIME,Content,ActionName,ActionParams,ActionResult,DoneTime,Status FROM Messages
+ WHERE SenderID = ? or ReceiverID =? AND Timestamp > ?
+ ORDER BY Timestamp
+ LIMIT ?
+ """, (agent_id, agent_id, start_time,limit))
+ results = cursor.fetchall()
+ #self.close()
+ return results # return 0 and the result if successful
+ except Error as e:
+ logging.error("Error occurred while getting messages: %s", e)
+ return -1, None
+
# read message from now->beign
def get_messages(self, session_id, limit, offset):
""" retrieve messages of a session with pagination """
try:
conn = self._get_conn()
cursor = conn.cursor()
+ if limit == 0:
+ limit = 1024
cursor.execute("""
SELECT MessageID, SessionID, MsgType, PrevMsgID, SenderID, ReceiverID, Timestamp, Topic,Mentions,ContentMIME,Content,ActionName,ActionParams,ActionResult,DoneTime,Status FROM Messages
WHERE SessionID = ?
@@ -253,11 +281,27 @@ class ChatSessionDB:
except Error as e:
logging.error("Error occurred while updating session summary: %s", e)
return -1
+
+ def update_session_thread_id(self, session_id, thread_id):
+ """ update the threadid of a session """
+ try:
+ conn = self._get_conn()
+ conn.execute("""
+ UPDATE ChatSessions
+ SET ThreadID = ?
+ WHERE SessionID = ?
+ """, (thread_id, session_id))
+ conn.commit()
+ return 0 # return 0 if successful
+ except Error as e:
+ logging.error("Error occurred while updating session threadid: %s", e)
+ return -1
# chat session store the chat history between owner and agent
# chat session might be large, so can read / write at stream mode.
class AIChatSession:
_dbs = {}
+ _sessions = {}
#@classmethod
#async def get_session_by_id(cls,session_id:str,db_path:str):
# db = cls._dbs.get(db_path)
@@ -266,6 +310,38 @@ class AIChatSession:
# cls._dbs[db_path] = db
# db.get_chatsession_by_id(session_id)
# #result = AIChatSession()
+ @classmethod
+ # start_time is a string like "2021-01-01 00:00:00"
+ def load_message_records_by_agentid(cls,agent_id:str,start_time:str,limit:int,db_path:str)->List[AgentMsg]:
+ db = cls._dbs.get(db_path)
+ if db is None:
+ db = ChatSessionDB(db_path)
+ cls._dbs[db_path] = db
+ msgs = db.load_message_by_agentid(agent_id,start_time,limit)
+ result = []
+ for msg in msgs:
+ agent_msg = AgentMsg()
+ agent_msg.msg_id = msg[0]
+ agent_msg.session_id = msg[1]
+ agent_msg.msg_type = AgentMsgType(msg[2])
+ agent_msg.prev_msg_id = msg[3]
+ agent_msg.sender = msg[4]
+ agent_msg.target = msg[5]
+ agent_msg.create_time = msg[6]
+ agent_msg.topic = msg[7]
+ if msg[8] is not None:
+ agent_msg.mentions = json.loads(msg[8])
+ agent_msg.body_mime = msg[9]
+ agent_msg.body = msg[10]
+ agent_msg.func_name = msg[11]
+ if msg[12] is not None:
+ agent_msg.args = json.loads(msg[12])
+ agent_msg.result_str = msg[13]
+ agent_msg.done_time = msg[14]
+ agent_msg.status = AgentMsgStatus(msg[15])
+
+ result.append(agent_msg)
+ return result
@classmethod
def get_session(cls,owner_id:str,session_topic:str,db_path:str,auto_create = True) -> 'AIChatSession':
@@ -275,17 +351,24 @@ class AIChatSession:
cls._dbs[db_path] = db
result = None
+ for session in cls._sessions.values():
+ if session.owner_id == owner_id and session.topic == session_topic:
+ return session
+
session = db.get_chatsession_by_owner_topic(owner_id,session_topic)
if session is None:
if auto_create:
session_id = "CS#" + uuid.uuid4().hex
db.insert_chatsession(session_id,owner_id,session_topic,datetime.datetime.now())
result = AIChatSession(owner_id,session_id,db)
+ cls._sessions[session_id] = result
else:
result = AIChatSession(owner_id,session[0],db)
result.topic = session_topic
result.summarize_pos = session[4]
result.summary = session[5]
+ result.openai_thread_id = session[6]
+ cls._sessions[result.session_id] = result
return result
@@ -297,6 +380,10 @@ class AIChatSession:
cls._dbs[db_path] = db
result = None
+ session = cls._sessions.get(session_id)
+ if session:
+ return session
+
session = db.get_chatsession_by_id(session_id)
if session is None:
return None
@@ -305,6 +392,8 @@ class AIChatSession:
result.topic = session[2]
result.summarize_pos = session[4]
result.summary = session[5]
+ result.openai_thread_id = session[6]
+ cls._sessions[session_id] = result
return result
@@ -330,12 +419,13 @@ class AIChatSession:
self.topic : str = None
self.start_time : str = None
self.summarize_pos : int = 0
- self.summary = None
+ self.summary : str = None
+ self.openai_thread_id = None
def get_owner_id(self) -> str:
return self.owner_id
- def read_history(self, number:int=10,offset=0,order="revers") -> [AgentMsg]:
+ def read_history(self, number:int=0,offset=0,order="revers") -> [AgentMsg]:
if order == "revers":
msgs = self.db.get_messages(self.session_id, number, offset)
else:
@@ -366,16 +456,19 @@ class AIChatSession:
result.append(agent_msg)
return result
- def append(self,msg:AgentMsg) -> None:
+ def append(self,msg:AgentMsg,tags:List[str] = None) -> None:
msg.session_id = self.session_id
- self.db.insert_message(msg)
+ self.db.insert_message(msg,tags)
- def update_think_progress(self,progress:int,new_summary:str) -> None:
- self.db.update_session_summary(self.session_id,progress,new_summary)
- self.summarize_pos = progress
+ def update_summary(self,new_summary:str) -> None:
+ self.db.update_session_summary(self.session_id,self.summarize_pos,new_summary)
self.summary = new_summary
+ def update_openai_thread_id(self,thread_id:str) -> None:
+ self.db.update_session_thread_id(self.session_id,thread_id)
+ self.openai_thread_id = thread_id
+
#def attach_event_handler(self,handler) -> None:
# """chat session changed event handler"""
# pass
diff --git a/src/aios/agent/llm_context.py b/src/aios/agent/llm_context.py
new file mode 100644
index 0000000..01c54f7
--- /dev/null
+++ b/src/aios/agent/llm_context.py
@@ -0,0 +1,316 @@
+# pylint:disable=E0402
+from abc import ABC, abstractmethod
+import json
+import logging
+from typing import Optional,Set,List,Dict,Callable
+
+from ..proto.ai_function import AIFunction,AIAction, AIFunction2Action,SimpleAIAction
+
+logger = logging.getLogger(__name__)
+
+class LLMProcessContext:
+ def __init__(self) -> None:
+ pass
+
+
+ @staticmethod
+ def function2action(ai_func:AIFunction) -> AIAction:
+ return AIFunction2Action(ai_func)
+
+ @staticmethod
+ def aifunctions_to_inner_functions(all_inner_function:List[AIFunction]) -> List[Dict]:
+ if all_inner_function is None:
+ return []
+
+ result_func = []
+ result_len = 0
+ for inner_func in all_inner_function:
+ func_name = inner_func.get_name()
+ this_func = {}
+ this_func["name"] = func_name
+ this_func["description"] = inner_func.get_description()
+ this_func["parameters"] = inner_func.get_openai_parameters()
+ result_len += len(json.dumps(this_func,ensure_ascii=False)) / 4
+ result_func.append(this_func)
+ return result_func
+
+
+ @abstractmethod
+ def get_ai_function(self,func_name:str) -> AIFunction:
+ pass
+
+ def get_all_ai_functions(self) -> List[AIFunction]:
+ return self.get_function_set(None)
+
+ @abstractmethod
+ def get_function_set(self,set_name:str = None) -> List[AIFunction]:
+ pass
+
+ @abstractmethod
+ def get_ai_action(self,op_name:str) -> AIAction:
+ pass
+
+ def get_all_ai_action(self) -> List[AIAction]:
+ return self.get_action_set(None)
+
+ @abstractmethod
+ def get_action_set(self,set_name:str = None) -> List[AIFunction]:
+ pass
+
+ def __getitem__(self, key):
+ return self.get_value(key)
+
+ @abstractmethod
+ def get_value(self,key:str) -> Optional[str]:
+ pass
+
+ #def list_actions(self,path:str) -> List[AIAction]:
+ # return "No more actions!"
+
+ #def list_functions(self,path:str) -> List[AIFunction]:
+ # return "No more tool functions!"
+
+class GlobaToolsLibrary:
+ _instance = None
+ @classmethod
+ def get_instance(cls) -> 'GlobaToolsLibrary':
+ if cls._instance is None:
+ cls._instance = GlobaToolsLibrary()
+ return cls._instance
+
+ def __init__(self,global_env_name:str = None) -> None:
+ self.all_preset_context = {}
+ self.all_tool_functions : Dict[str,AIFunction] = {}
+ self.all_action_sets : Dict[str,Set[str]] = {}
+ self.all_function_sets : Dict[str,Set[str]] = {}
+
+ def register_prset_context(self,preset_id:str,context) -> None:
+ self.all_preset_context[preset_id] = context
+
+ def get_preset_context(self,preset_id:str):
+ return self.all_preset_context.get(preset_id)
+
+ def register_tool_function(self,function:AIFunction) -> None:
+ if self.all_tool_functions.get(function.get_id()):
+ logger.warning(f"Tool function {function.get_id()} already exists! will be replaced!")
+
+ self.all_tool_functions[function.get_id()] = function
+
+ def get_tool_function(self,function_name:str) -> AIFunction:
+ return self.all_tool_functions.get(function_name)
+
+ def register_function_set(self,set_name:str,function_set:Set[str]) -> None:
+ self.all_function_sets[set_name] = function_set
+
+ def get_function_set(self,set_name:str) -> Set[str]:
+ return self.all_function_sets.get(set_name)
+
+class SimpleLLMContext(LLMProcessContext):
+ def __init__(self) -> None:
+ super().__init__()
+ self.parent = None
+ self.values : Dict[str,str] = {}
+ self.values_callback = {}
+
+ self.functions: Dict[str,AIFunction] = {}
+ self.func_sets : Dict[str,Dict[str,AIFunction]] = {}
+ self.actions: Dict[str,AIAction] = {}
+ self.action_sets : Dict[str,Dict[str,AIAction]] = {}
+
+ def load_action_set_from_config(self,preset,config:Dict[str,str]) -> Dict:
+ if preset is None:
+ result = {}
+ else:
+ result = preset
+
+ enable_actions = config.get("enable")
+ if enable_actions:
+ for action_id in enable_actions:
+ ai_func = GlobaToolsLibrary.get_instance().get_tool_function(action_id)
+ if ai_func:
+ result[action_id] = LLMProcessContext.function2action(ai_func)
+ else:
+ func_set = GlobaToolsLibrary.get_instance().get_function_set(action_id)
+ if func_set:
+ for _func_id in func_set:
+ ai_func = GlobaToolsLibrary.get_instance().get_tool_function(_func_id)
+ if ai_func:
+ result[_func_id] = LLMProcessContext.function2action(ai_func)
+ else:
+ logger.error(f"load_action_set_from_config failed! enable action id {action_id} not found!")
+ return None
+
+ disable_actions = config.get("disable")
+ if disable_actions:
+ for disable_action in disable_actions:
+ if result.get(disable_action):
+ result.pop(disable_action)
+ else:
+ func_set = GlobaToolsLibrary.get_instance().get_function_set(action_id)
+ if func_set:
+ for _func_id in func_set:
+ if result.get(_func_id):
+ result.pop(_func_id)
+ else:
+ logger.error(f"load_action_set_from_config failed! disable action id {action_id} not found!")
+ return None
+
+ return result
+
+ def load_function_set_from_config(self,preset,config:Dict) -> Dict[str,AIFunction]:
+ if preset is None:
+ result = {}
+ else:
+ result = preset
+
+ enable_functions = config.get("enable")
+ if enable_functions:
+ for func_id in enable_functions:
+ ai_func = GlobaToolsLibrary.get_instance().get_tool_function(func_id)
+ if ai_func:
+ result[func_id] = ai_func
+ else:
+ func_set = GlobaToolsLibrary.get_instance().get_function_set(func_id)
+ if func_set:
+ for func_id in func_set:
+ ai_func = GlobaToolsLibrary.get_instance().get_tool_function(func_id)
+ if ai_func:
+ result[func_id] = ai_func
+ else:
+ logger.error(f"load_function_set_from_config failed! enable function id {func_id} not found!")
+ return None
+ else:
+ logger.error(f"load_function_set_from_config failed! enable function id {func_id} not found!")
+ return None
+
+
+ disable_functions = config.get("disable")
+ if disable_functions:
+ for disable_function in disable_functions:
+ if result.get(disable_function):
+ result.pop(disable_function)
+ else:
+ func_set = GlobaToolsLibrary.get_instance().get_function_set(func_id)
+ if func_set:
+ for func_id in func_set:
+ if result.get(func_id):
+ result.pop(func_id)
+ else:
+ logger.error(f"load_function_set_from_config failed! disable function id {disable_function} not found!")
+ return None
+
+ return result
+
+ def load_from_config(self,config:Dict[str,str]) -> bool:
+ preset = config.get("preset")
+ if preset:
+ self.parent:SimpleLLMContext = GlobaToolsLibrary.get_instance().get_preset_context(preset)
+ if self.parent is None:
+ logger.error(f"preset context {preset} not found!")
+ return False
+
+ self.values = self.parent.values
+ self.values_callback = self.parent.values_callback
+ self.actions = self.parent.actions
+ self.functions = self.parent.functions
+ self.action_sets = self.parent.action_sets
+ self.func_sets = self.parent.func_sets
+
+ action_def:Dict= config.get("actions")
+ if action_def:
+ self.actions = self.load_action_set_from_config(self.actions,action_def)
+ if self.actions is None:
+ logger.error(f"load_from_config failed! load_action_set_from_config failed!")
+ return False
+
+ for set_name in action_def.keys():
+ if set_name == "enable":
+ continue
+ if set_name == "disable":
+ continue
+
+ sub_set = config.get(set_name)
+ self.action_sets[set_name] = self.load_action_set_from_config(None,sub_set)
+ if self.action_sets[set_name] is None:
+ logger.error(f"load_from_config failed! load_action_set_from_config failed!")
+ return False
+
+ function_def:Dict = config.get("functions")
+ if function_def:
+ self.functions = self.load_function_set_from_config(self.functions,function_def)
+ if self.functions is None:
+ logger.error(f"load_from_config failed! load_function_set_from_config failed!")
+ return False
+
+ for set_name in function_def.keys():
+ if set_name == "enable":
+ continue
+ if set_name == "disable":
+ continue
+
+ sub_set = config.get(set_name)
+ self.func_sets[set_name] = self.load_function_set_from_config(None,sub_set)
+ if self.func_sets[set_name] is None:
+ logger.error(f"load_from_config failed! load_function_set_from_config failed!")
+ return False
+
+ #values_def = config.get("values")
+ #if values_def:
+ # for key,value in values_def.items():
+ # self.values[key] = value
+
+ def get_value(self,key:str) -> Optional[str]:
+ callback = self.values_callback.get(key)
+ if callback:
+ return callback()
+ return self.values.get(key)
+
+ def set_value_callback(self,key:str,callback:Callable[[],str]) -> None:
+ self.values_callback[key] = callback
+
+ def set_value(self,key:str,value:str):
+ self.values[key] = value
+
+ def get_ai_function(self,func_name:str) -> AIFunction:
+ for func in self.functions.values():
+ if func.get_name() == func_name:
+ return func
+ #for set_name in self.func_sets.keys():
+ # func = self.func_sets[set_name].get(func_name)
+ # if func is not None:
+ # return func
+ return None
+
+ def get_function_set(self,set_name:str = None) -> List[AIFunction]:
+ if self.functions is None:
+ return None
+
+ if set_name is None:
+ return self.functions.values()
+ else:
+ func_set = self.func_sets.get(set_name)
+ if func_set:
+ return func_set.values()
+ return None
+
+
+ def get_ai_action(self,op_name:str) -> AIAction:
+ for action in self.actions.values():
+ if action.get_name() == op_name:
+ return action
+
+ return None
+
+ def get_action_set(self,set_name:str = None) -> List[AIFunction]:
+ if self.actions is None:
+ return None
+
+ if set_name is None:
+ return self.actions.values()
+ else:
+ action_set = self.action_sets.get(set_name)
+ if action_set:
+ return action_set.values()
+ return None
+
+
diff --git a/src/aios/agent/llm_do_task.py b/src/aios/agent/llm_do_task.py
new file mode 100644
index 0000000..df9cf25
--- /dev/null
+++ b/src/aios/agent/llm_do_task.py
@@ -0,0 +1,373 @@
+from ..proto.compute_task import LLMPrompt,LLMResult,ComputeTaskResult,ComputeTaskResultCode
+from ..proto.ai_function import AIFunction,AIAction,ActionNode
+from ..proto.agent_msg import AgentMsg,AgentMsgType
+from ..proto.agent_task import AgentTask, AgentTodo, AgentWorkLog
+from ..frame.compute_kernel import ComputeKernel
+
+from .agent_memory import AgentMemory
+from .workspace import AgentWorkspace
+from .llm_context import LLMProcessContext,GlobaToolsLibrary, SimpleLLMContext
+from .llm_process import BaseLLMProcess,LLMAgentBaseProcess
+
+from abc import ABC,abstractmethod
+import copy
+import json
+import datetime
+from datetime import datetime
+from typing import Any, Callable, Coroutine, Optional,Dict,Awaitable,List
+from enum import Enum
+import logging
+
+logger = logging.getLogger(__name__)
+
+#LLM Process All the unfinished tasks,will sort the priority of the task after LLM, determine the next execution time, and complete the simple task
+class AgentTriageTaskList(LLMAgentBaseProcess):
+ def __init__(self) -> None:
+ super().__init__()
+
+
+ async def load_from_config(self,config:dict) -> bool:
+ if await super().load_from_config(config) is False:
+ return False
+
+ async def prepare_prompt(self,input:Dict) -> LLMPrompt:
+ prompt = LLMPrompt()
+
+ task_list:List[AgentTask] = input.get("tasklist")
+ context_info = input.get("context_info")
+ if task_list is None:
+ logger.error(f"tasklist not found in input")
+ return None
+
+ prompt.append_user_message(json.dumps(task_list,ensure_ascii=False))
+
+ system_prompt_dict = self.prepare_role_system_prompt(context_info)
+ # May all logs is good for Agent Triage Task List?
+ have_known_info = False
+ known_info = {}
+ working_logs = await self.memory.load_worklogs(self.memory.agent_id)
+ if len(working_logs) > 0:
+ have_known_info = True
+ all_worklog_node = []
+ for worklog in working_logs:
+ workNode = {}
+ dt = datetime.fromtimestamp(float(worklog.timestamp))
+ workNode["timestamp"] = dt.strftime("%Y-%m-%d %H:%M:%S")
+ workNode["type"] = worklog.work_type
+ workNode["content"] = worklog.content
+ workNode["result"] = worklog.result
+ all_worklog_node.append(workNode)
+
+ known_info["worklogs"] = all_worklog_node
+
+ if have_known_info:
+ system_prompt_dict["known_info"] = known_info
+
+ prompt.inner_functions =LLMProcessContext.aifunctions_to_inner_functions(self.llm_context.get_all_ai_functions())
+ prompt.append_system_message(json.dumps(system_prompt_dict,ensure_ascii=False))
+ return prompt
+
+
+ async def post_llm_process(self,actions:List[ActionNode],input:Dict,llm_result:LLMResult) -> bool:
+ action_params = {}
+ action_params["_input"] = input
+ action_params["_memory"] = self.memory
+ action_params["_workspace"] = self.workspace
+ action_params["_llm_result"] = llm_result
+ action_params["_agentid"] = self.memory.agent_id
+ action_params["_start_at"] = datetime.now()
+
+ result_str = "OK"
+ try:
+ if await self._execute_actions(actions,action_params) is False:
+ result_str = "execute action failed!"
+ except Exception as e:
+ logger.error(f"execute action failed! {e}")
+ result_str = "execute action failed!,error:" + str(e)
+
+ worklog = AgentWorkLog.create_by_content("","triage",llm_result.resp,self.memory.agent_id)
+ worklog.result = result_str
+ await self.memory.append_worklog(worklog)
+
+# LLM a Task that never been LLMed, the result of LLM Process may be adjusted, splitting subtask or do simple task as a todo directly.
+class AgentPlanTask(LLMAgentBaseProcess):
+ def __init__(self) -> None:
+ super().__init__()
+
+ async def load_from_config(self, config: dict,is_load_default=True) -> bool:
+ if await super().load_from_config(config) is False:
+ return False
+
+ async def prepare_prompt(self,input:Dict) -> LLMPrompt:
+ prompt = LLMPrompt()
+
+ agent_task : AgentTask= input.get("task")
+ context_info = input.get("context_info")
+ if agent_task is None:
+ logger.error(f"task not found in input")
+ return None
+
+ prompt.append_user_message(json.dumps(agent_task.to_dict(),ensure_ascii=False))
+
+ system_prompt_dict = self.prepare_role_system_prompt(context_info)
+
+ have_known_info = False
+ known_info = {}
+ working_logs = await self.memory.load_worklogs(None,agent_task.task_id)
+ if len(working_logs) > 0:
+ have_known_info = True
+ all_worklog_node = []
+ for worklog in working_logs:
+ workNode = {}
+ dt = datetime.fromtimestamp(float(worklog.timestamp))
+ workNode["timestamp"] = dt.strftime("%Y-%m-%d %H:%M:%S")
+ workNode["type"] = worklog.work_type
+ workNode["operator"] = worklog.operator
+ workNode["content"] = worklog.content
+ workNode["result"] = worklog.result
+ all_worklog_node.append(workNode)
+
+ known_info["worklogs"] = all_worklog_node
+
+ if have_known_info:
+ system_prompt_dict["known_info"] = known_info
+
+ prompt.inner_functions =LLMProcessContext.aifunctions_to_inner_functions(self.llm_context.get_all_ai_functions())
+ prompt.append_system_message(json.dumps(system_prompt_dict,ensure_ascii=False))
+ return prompt
+
+
+ async def post_llm_process(self,actions:List[ActionNode],input:Dict,llm_result:LLMResult) -> bool:
+ action_params = {}
+ action_params["_input"] = input
+ agent_task : AgentTask= input.get("task")
+ action_params["_memory"] = self.memory
+ action_params["_workspace"] = self.workspace
+ action_params["_llm_result"] = llm_result
+ action_params["_agentid"] = self.memory.agent_id
+ action_params["_start_at"] = datetime.now()
+
+ result_str = "OK"
+ try:
+ if await self._execute_actions(actions,action_params) is False:
+ result_str = "execute action failed!"
+ except Exception as e:
+ logger.error(f"execute action failed! {e}")
+ result_str = "execute action failed!,error:" + str(e)
+
+ worklog = AgentWorkLog.create_by_content(agent_task.task_id,"tackling",llm_result.resp,self.memory.agent_id)
+ worklog.result = result_str
+ await self.memory.append_worklog(worklog)
+
+
+# Agent DO Todo
+# The purpose is to complete Todo.It is the core LLM process. Can use sufficient external tools to do your best according to the identity and ability of AGENT.It is also the LLM Process of the main extension of Agent extension
+class AgentDo(LLMAgentBaseProcess):
+ def __init__(self) -> None:
+ super().__init__()
+
+ async def load_from_config(self, config: dict,is_load_default=True) -> Coroutine[Any, Any, bool]:
+ if await super().load_from_config(config) is False:
+ return False
+
+ async def prepare_prompt(self,input:Dict) -> LLMPrompt:
+ prompt = LLMPrompt()
+
+ agent_todo : AgentTodo= input.get("todo")
+ context_info = input.get("context_info")
+ if agent_todo is None:
+ logger.error(f"task not found in input")
+ return None
+
+ prompt.append_user_message(json.dumps(agent_todo.to_dict(),ensure_ascii=False))
+
+ system_prompt_dict = self.prepare_role_system_prompt(context_info)
+ # May all logs is good for Agent Triage Task List?
+ have_known_info = False
+ known_info = {}
+ working_logs = await self.memory.load_worklogs(None,agent_todo.todo_id)
+ if len(working_logs) > 0:
+ have_known_info = True
+ all_worklog_node = []
+ for worklog in working_logs:
+ workNode = {}
+ dt = datetime.fromtimestamp(float(worklog.timestamp))
+ workNode["timestamp"] = dt.strftime("%Y-%m-%d %H:%M:%S")
+ workNode["type"] = worklog.work_type
+ workNode["content"] = worklog.content
+ workNode["result"] = worklog.result
+ all_worklog_node.append(workNode)
+
+ known_info["worklogs"] = all_worklog_node
+
+ if have_known_info:
+ system_prompt_dict["known_info"] = known_info
+
+ prompt.inner_functions =LLMProcessContext.aifunctions_to_inner_functions(self.llm_context.get_all_ai_functions())
+ prompt.append_system_message(json.dumps(system_prompt_dict,ensure_ascii=False))
+ return prompt
+
+
+ async def post_llm_process(self,actions:List[ActionNode],input:Dict,llm_result:LLMResult) -> bool:
+ action_params = {}
+ action_params["_input"] = input
+ agent_todo : AgentTodo= input.get("todo")
+ action_params["_memory"] = self.memory
+ action_params["_workspace"] = self.workspace
+ action_params["_llm_result"] = llm_result
+ action_params["_agentid"] = self.memory.agent_id
+ action_params["_start_at"] = datetime.now()
+
+ result_str = "OK"
+ try:
+ if await self._execute_actions(actions,action_params) is False:
+ result_str = "execute action failed!"
+ except Exception as e:
+ logger.error(f"execute action failed! {e}")
+ result_str = "execute action failed!,error:" + str(e)
+
+ worklog = AgentWorkLog.create_by_content(agent_todo.todo_id,"do",llm_result.resp,self.memory.agent_id)
+ worklog.result = result_str
+ await self.memory.append_worklog(worklog)
+
+#Agent check todo
+# LLM a already-DO TODO, the purpose is to check whether it is completed to face the illusion of LLM.Check can use some tools, which is also the core of the agent extension。
+class AgentCheck(LLMAgentBaseProcess):
+ def __init__(self) -> None:
+ super().__init__()
+
+ async def load_from_config(self, config: dict,is_load_default=True) -> Coroutine[Any, Any, bool]:
+ if await super().load_from_config(config) is False:
+ return False
+
+ async def prepare_prompt(self,input:Dict) -> LLMPrompt:
+ prompt = LLMPrompt()
+
+ agent_todo : AgentTodo= input.get("todo")
+ context_info = input.get("context_info")
+ if agent_todo is None:
+ logger.error(f"task not found in input")
+ return None
+
+ prompt.append_user_message(json.dumps(agent_todo.to_dict(),ensure_ascii=False))
+
+ system_prompt_dict = self.prepare_role_system_prompt(context_info)
+ # May all logs is good for Agent Triage Task List?
+ have_known_info = False
+ known_info = {}
+ working_logs = await self.memory.load_worklogs(None,agent_todo.todo_id)
+ if len(working_logs) > 0:
+ have_known_info = True
+ all_worklog_node = []
+ for worklog in working_logs:
+ workNode = {}
+ dt = datetime.fromtimestamp(float(worklog.timestamp))
+ workNode["timestamp"] = dt.strftime("%Y-%m-%d %H:%M:%S")
+ workNode["type"] = worklog.work_type
+ workNode["content"] = worklog.content
+ workNode["result"] = worklog.result
+ all_worklog_node.append(workNode)
+
+ known_info["worklogs"] = all_worklog_node
+
+ if have_known_info:
+ system_prompt_dict["known_info"] = known_info
+
+ prompt.inner_functions =LLMProcessContext.aifunctions_to_inner_functions(self.llm_context.get_all_ai_functions())
+ prompt.append_system_message(json.dumps(system_prompt_dict,ensure_ascii=False))
+ return prompt
+
+
+ async def post_llm_process(self,actions:List[ActionNode],input:Dict,llm_result:LLMResult) -> bool:
+ action_params = {}
+ action_params["_input"] = input
+ agent_todo : AgentTodo= input.get("todo")
+ action_params["_memory"] = self.memory
+ action_params["_workspace"] = self.workspace
+ action_params["_llm_result"] = llm_result
+ action_params["_agentid"] = self.memory.agent_id
+ action_params["_start_at"] = datetime.now()
+
+ result_str = "OK"
+ try:
+ if await self._execute_actions(actions,action_params) is False:
+ result_str = "execute action failed!"
+ except Exception as e:
+ logger.error(f"execute action failed! {e}")
+ result_str = "execute action failed!,error:" + str(e)
+
+ worklog = AgentWorkLog.create_by_content(agent_todo.todo_id,"check",llm_result.resp,self.memory.agent_id)
+ worklog.result = result_str
+ await self.memory.append_worklog(worklog)
+
+#Agent review task
+#When Task's Todolist is completed, or Task's subtask is completed, LLM review a TASK to determine that the Task has been completed.This Review also failed to execute.
+class AgentReviewTask(LLMAgentBaseProcess):
+ def __init__(self) -> None:
+ super().__init__()
+
+
+ async def load_from_config(self, config: dict,is_load_default=True) -> Coroutine[Any, Any, bool]:
+ if await super().load_from_config(config) is False:
+ return False
+
+ async def prepare_prompt(self,input:Dict) -> LLMPrompt:
+ prompt = LLMPrompt()
+
+ agent_task : AgentTask= input.get("task")
+ context_info = input.get("context_info")
+ if agent_task is None:
+ logger.error(f"task not found in input")
+ return None
+
+ prompt.append_user_message(json.dumps(agent_task.to_dict(),ensure_ascii=False))
+
+ system_prompt_dict = self.prepare_role_system_prompt(context_info)
+ # May all logs is good for Agent Triage Task List?
+ have_known_info = False
+ known_info = {}
+ working_logs = await self.memory.load_worklogs(None,agent_task.task_id)
+ if len(working_logs) > 0:
+ have_known_info = True
+ all_worklog_node = []
+ for worklog in working_logs:
+ workNode = {}
+ dt = datetime.fromtimestamp(float(worklog.timestamp))
+ workNode["timestamp"] = dt.strftime("%Y-%m-%d %H:%M:%S")
+ workNode["type"] = worklog.work_type
+ workNode["operator"] = worklog.operator
+ workNode["content"] = worklog.content
+ workNode["result"] = worklog.result
+ all_worklog_node.append(workNode)
+
+ known_info["worklogs"] = all_worklog_node
+
+ if have_known_info:
+ system_prompt_dict["known_info"] = known_info
+
+ prompt.inner_functions =LLMProcessContext.aifunctions_to_inner_functions(self.llm_context.get_all_ai_functions())
+ prompt.append_system_message(json.dumps(system_prompt_dict,ensure_ascii=False))
+ return prompt
+
+
+ async def post_llm_process(self,actions:List[ActionNode],input:Dict,llm_result:LLMResult) -> bool:
+ action_params = {}
+ action_params["_input"] = input
+ agent_task : AgentTask= input.get("task")
+ action_params["_memory"] = self.memory
+ action_params["_workspace"] = self.workspace
+ action_params["_llm_result"] = llm_result
+ action_params["_agentid"] = self.memory.agent_id
+ action_params["_start_at"] = datetime.now()
+
+ result_str = "OK"
+ try:
+ if await self._execute_actions(actions,action_params) is False:
+ result_str = "execute action failed!"
+ except Exception as e:
+ logger.error(f"execute action failed! {e}")
+ result_str = "execute action failed!,error:" + str(e)
+
+ worklog = AgentWorkLog.create_by_content(agent_task.task_id,"review",llm_result.resp,self.memory.agent_id)
+ worklog.result = result_str
+ await self.memory.append_worklog(worklog)
\ No newline at end of file
diff --git a/src/aios/agent/llm_process.py b/src/aios/agent/llm_process.py
new file mode 100644
index 0000000..df8b816
--- /dev/null
+++ b/src/aios/agent/llm_process.py
@@ -0,0 +1,652 @@
+# Old name is behavior, I belive new name "llm_process" is better
+# pylint:disable=E0402
+import os.path
+
+from .chatsession import AIChatSession
+from ..utils import video_utils,image_utils
+
+from ..proto.compute_task import LLMPrompt,LLMResult,ComputeTaskResult,ComputeTaskResultCode
+from ..proto.ai_function import AIFunction,AIAction,ActionNode
+from ..proto.agent_msg import AgentMsg,AgentMsgType
+
+from .agent_memory import AgentMemory
+from .workspace import AgentWorkspace
+from .llm_context import LLMProcessContext,GlobaToolsLibrary, SimpleLLMContext
+
+from ..frame.compute_kernel import ComputeKernel
+
+from abc import ABC,abstractmethod
+import copy
+import json
+import datetime
+from datetime import datetime
+from typing import Any, Callable, Coroutine, Optional,Dict,Awaitable,List
+from enum import Enum
+import logging
+
+logger = logging.getLogger(__name__)
+
+MIN_PREDICT_TOKEN_LEN = 32
+
+class BaseLLMProcess(ABC):
+ def __init__(self) -> None:
+ self.behavior:str = None #行为名字
+ self.goal:str = None #目标
+ self.input_example:str= None #输入样例
+ self.result_example:str = None #llm_result样例
+
+ self.enable_json_resp = False
+ #None means system default,
+ # TODO: support abcstract model name like: local-hight,local-low,local-medium,remote-hight,remote-low,remote-medium
+ self.model_name = None
+ self.max_token = 2000 # result_token
+ self.max_prompt_token = 2000 # not include input prompt
+ self.chat_summary_token_len = 500
+ self.timeout = 1800 # 30 min
+
+ self.llm_context:LLMProcessContext = None
+
+ def get_llm_model_name(self) -> str:
+ return self.model_name
+
+ @abstractmethod
+ async def prepare_prompt(self,input:Dict) -> LLMPrompt:
+ pass
+
+ @abstractmethod
+ async def get_inner_function_for_exec(self,func_name:str) -> AIFunction:
+ pass
+
+ @abstractmethod
+ def prepare_inner_function_context_for_exec(self,inner_func_name:str,parameters:Dict):
+ return
+
+ @abstractmethod
+ async def post_llm_process(self,actions:List[ActionNode],input:Dict,llm_result:LLMResult) -> bool:
+ pass
+
+ def get_remain_prompt_length(self,prompt:LLMPrompt,will_append_str:str) -> int:
+ return self.max_prompt_token - ComputeKernel.llm_num_tokens(prompt,self.model_name)
+
+ @abstractmethod
+ async def load_from_config(self,config:dict) -> bool:
+ #self.behavior = config.get("behavior")
+ #self.goal = config.get("goal")
+ self.input_example = config.get("input_example")
+ self.result_example = config.get("result_example")
+
+ if config.get("model_name"):
+ self.model_name = config.get("model_name")
+ if config.get("enable_json_resp"):
+ self.enable_json_resp = config.get("enable_json_resp") == "true"
+ if config.get("max_token"):
+ self.max_token = config.get("max_token")
+ if config.get("timeout"):
+ self.timeout = config.get("timeout")
+
+
+ return True
+
+ @abstractmethod
+ async def initial(self,params:Dict = None) -> bool:
+ pass
+
+ def _format_content_by_env_value(self,content:str,env)->str:
+ return content.format_map(env)
+
+
+ async def _execute_inner_func(self,inner_func_call_node:Dict,prompt: LLMPrompt,stack_limit = 1) -> ComputeTaskResult:
+ arguments = None
+ stack_limit = stack_limit - 1
+ try:
+ func_name = inner_func_call_node.get("name")
+ arguments = json.loads(inner_func_call_node.get("arguments"))
+ logger.info(f"LLMProcess execute inner func:{func_name} :({json.dumps(arguments,ensure_ascii=False)})")
+
+ func_node : AIFunction = await self.get_inner_function_for_exec(func_name)
+ if func_node is None:
+ result_str:str = f"execute {func_name} error,function not found"
+ else:
+ self.prepare_inner_function_context_for_exec(func_name,arguments)
+ result_str:str = await func_node.execute(arguments)
+ except Exception as e:
+ result_str = f"execute {func_name} error:{str(e)}"
+ logger.error(f"LLMProcess execute inner func:{func_name} error:\n\t{e}")
+
+ logger.info("LLMProcess execute inner func result:" + result_str)
+
+ prompt.messages.append({"role":"function","content":result_str,"name":func_name})
+ if self.enable_json_resp:
+ resp_mode = "json"
+ else:
+ resp_mode = "text"
+
+ max_result_token = self.max_token - ComputeKernel.llm_num_tokens(prompt,self.model_name)
+ if max_result_token < MIN_PREDICT_TOKEN_LEN:
+ task_result = ComputeTaskResult()
+ task_result.result_code = ComputeTaskResultCode.ERROR
+ task_result.error_str = f"prompt too long,can not predict"
+ return task_result
+
+ if stack_limit > 0:
+ inner_functions=prompt.inner_functions
+ else:
+ inner_functions = None
+
+
+ task_result: ComputeTaskResult = await (ComputeKernel.get_instance().do_llm_completion(
+ prompt,
+ resp_mode=resp_mode,
+ mode_name=self.get_llm_model_name(),
+ max_token=max_result_token,
+ inner_functions=inner_functions, #NOTICE: inner_function in prompt can be a subset of get_inner_function
+ timeout=self.timeout))
+
+ if task_result.result_code != ComputeTaskResultCode.OK:
+ logger.error(f"llm compute error:{task_result.error_str}")
+ return task_result
+
+ inner_func_call_node = None
+
+ result_message : dict = task_result.result.get("message")
+ if result_message:
+ inner_func_call_node = result_message.get("function_call")
+ if inner_func_call_node:
+ func_msg = copy.deepcopy(result_message)
+ del func_msg["tool_calls"]#TODO: support tool_calls?
+ prompt.messages.append(func_msg)
+
+
+ if inner_func_call_node:
+ return await self._execute_inner_func(inner_func_call_node,prompt,stack_limit-1)
+ else:
+ return task_result
+
+ async def process(self,input:Dict) -> LLMResult:
+ if self.enable_json_resp:
+ resp_mode = "json"
+ else:
+ resp_mode = "text"
+
+ # Action define in prompt, will be execute after llm compute
+ prompt = await self.prepare_prompt(input)
+ if prompt is None:
+ logger.warn(f"prepare_prompt return None, break llm_process")
+ return LLMResult.from_error_str("prepare_prompt return None")
+
+ max_result_token = self.max_token - ComputeKernel.llm_num_tokens(prompt,self.get_llm_model_name())
+ #if max_result_token < MIN_PREDICT_TOKEN_LEN:
+ # return LLMResult.from_error_str(f"prompt too long,can not predict")
+
+ task_result: ComputeTaskResult = await (ComputeKernel.get_instance().do_llm_completion(
+ prompt,
+ resp_mode=resp_mode,
+ mode_name=self.get_llm_model_name(),
+ max_token=max_result_token,
+ inner_functions=prompt.inner_functions, #NOTICE: inner_function in prompt can be a subset of get_inner_function
+ timeout=self.timeout))
+
+ if task_result.result_code != ComputeTaskResultCode.OK:
+ err_str = f"do_llm_completion error:{task_result.error_str}"
+ logger.error(err_str)
+ return LLMResult.from_error_str(err_str)
+
+ result_message = task_result.result.get("message")
+ inner_func_call_node = None
+ if result_message:
+ inner_func_call_node = result_message.get("function_call")
+
+ if inner_func_call_node:
+ call_prompt : LLMPrompt = copy.deepcopy(prompt)
+ func_msg = copy.deepcopy(result_message)
+ del func_msg["tool_calls"]
+ call_prompt.messages.append(func_msg)
+ task_result = await self._execute_inner_func(inner_func_call_node,call_prompt)
+
+ # parse task_result to LLM Result
+ if self.enable_json_resp:
+ try:
+ llm_result = LLMResult.from_json_str(task_result.result_str)
+ except Exception as e:
+ logger.error(f"parse llm result error:{e}")
+ llm_result = LLMResult.from_str(task_result.result_str)
+ else:
+ llm_result = LLMResult.from_str(task_result.result_str)
+
+ # use action to save history?
+ await self.post_llm_process(llm_result.action_list,input,llm_result)
+
+ return llm_result
+
+class LLMAgentBaseProcess(BaseLLMProcess):
+ def __init__(self) -> None:
+ super().__init__()
+
+ self.role_description:str = None
+ self.process_description:str = None
+ self.reply_format:str = None
+ self.context : str = None
+
+ self.workspace : AgentWorkspace = None # If Workspace is not none , enable Agent Tasklist
+ self.memory : AgentMemory = None
+ self.enable_kb : bool = False
+ self.kb = None
+
+ async def initial(self,params:Dict = None) -> bool:
+ self.memory = params.get("memory")
+ if self.memory is None:
+ logger.error(f"LLMAgeMessageProcess initial failed! memory not found")
+ return False
+ self.workspace = params.get("workspace")
+
+ return True
+ async def load_default_config(self) -> bool:
+ return True
+
+
+ async def load_from_config(self, config: dict,is_load_default=True) -> Coroutine[Any, Any, bool]:
+ if is_load_default:
+ await self.load_default_config()
+
+ if await super().load_from_config(config) is False:
+ return False
+
+ self.role_description = config.get("role_desc")
+ if self.role_description is None:
+ logger.error(f"role_description not found in config")
+ return False
+
+ if config.get("process_description"):
+ self.process_description = config.get("process_description")
+
+ if config.get("reply_format"):
+ self.reply_format = config.get("reply_format")
+
+ if config.get("context"):
+ self.context = config.get("context")
+
+ self.llm_context = SimpleLLMContext()
+ if config.get("llm_context"):
+ self.llm_context.load_from_config(config.get("llm_context"))
+
+ if config.get("enable_kb"):
+ self.enable_kb = config.get("enable_kb") == "true"
+
+ def prepare_role_system_prompt(self,context_info:Dict) -> Dict:
+ system_prompt_dict = {}
+ # System Prompt
+ ## LLM的身份说明
+ system_prompt_dict["role_description"] = self.role_description
+ #prompt.append_system_message(self.role_description)
+
+ ## 处理信息的流程说明
+ system_prompt_dict["process_rule"] = self.process_description
+ #prompt.append_system_message(self.process_description)
+ ### 回复的格式
+ system_prompt_dict["reply_format"] = self.reply_format
+ #prompt.append_system_message(self.reply_format)
+
+ ## Context
+ if self.context:
+ context = self._format_content_by_env_value(self.context,context_info)
+ system_prompt_dict["context"] = context
+ #prompt.append_system_message(context)
+
+ system_prompt_dict["support_actions"] = self.get_action_desc()
+
+ return system_prompt_dict
+
+ def prepare_inner_function_context_for_exec(self,inner_func_name:str,parameters:Dict):
+ parameters["_workspace"] = self.workspace
+
+ def get_action_desc(self) -> Dict:
+ result = {}
+ actions_list = self.llm_context.get_all_ai_action()
+ for action in actions_list:
+ result[action.get_name()] = action.get_description()
+ return result
+
+ async def get_inner_function_for_exec(self,func_name:str) -> AIFunction:
+ return self.llm_context.get_ai_function(func_name)
+
+ async def _execute_actions(self,actions:List[ActionNode],action_params:Dict):
+ for action_item in actions:
+ op : AIAction = self.llm_context.get_ai_action(action_item.name)
+ if op:
+ if action_item.parms is None:
+ action_item.parms = {}
+
+ real_parms = {**action_params,**action_item.parms}
+
+ action_item.parms["_result"] = await op.execute(real_parms)
+ action_item.parms["_end_at"] = datetime.now()
+ else:
+ logger.warn(f"action {action_item.name} not found")
+ return False
+
+
+class AgentMessageProcess(LLMAgentBaseProcess):
+ def __init__(self) -> None:
+ super().__init__()
+ self.mutil_model = None
+ self.enable_media2text = False
+ self.is_mutil_model = False
+ self.asr_model = None
+ self.tts_model = None
+
+ async def load_default_config(self) -> bool:
+ return True
+
+ async def load_from_config(self, config: dict,is_load_default=True) -> Coroutine[Any, Any, bool]:
+ if is_load_default:
+ await self.load_default_config()
+
+ if await super().load_from_config(config) is False:
+ return False
+
+ self.enable_media2text = config.get('enable_media2text', 'false').lower() in ('true', '1', 't', 'y', 'yes')
+
+ if config.get("mutil_model"):
+ self.mutil_model = config.get("mutil_model")
+
+ self.asr_model = config.get("asr_model")
+ self.tts_model = config.get("tts_model")
+
+ def get_llm_model_name(self) -> str:
+ if self.is_mutil_model:
+ return self.mutil_model
+ else:
+ return self.model_name
+
+ def check_and_to_base64(self, image_path: str) -> str:
+ if image_utils.is_file(image_path):
+ return image_utils.to_base64(image_path, (1024, 1024))
+ else:
+ return image_path
+
+ async def get_prompt_from_msg(self,msg:AgentMsg) -> LLMPrompt:
+ msg_prompt = LLMPrompt()
+ self.is_mutil_model = False
+ if msg.is_image_msg():
+ if self.enable_media2text:
+ logger.error(f"enable_media2text is not supported yet")
+ else:
+ image_prompt, images = msg.get_image_body()
+ if image_prompt is None:
+ msg_prompt.messages = [{"role": "user", "content": [{"type": "image_url", "image_url": {"url": self.check_and_to_base64(image)}} for image in images]}]
+ else:
+ content = [{"type": "text", "text": image_prompt}]
+ content.extend([{"type": "image_url", "image_url": {"url": self.check_and_to_base64(image)}} for image in images])
+ msg_prompt.messages = [{"role": "user", "content": content}]
+
+ if self.mutil_model:
+ self.is_mutil_model = True
+ else:
+ logger.warning(f"mutil_model is not set!")
+
+ elif msg.is_video_msg():
+ if self.enable_media2text:
+ logger.error(f"enable_media2text is not supported yet")
+ else:
+ video_prompt, video = msg.get_video_body()
+ frames = video_utils.extract_frames(video, (1024, 1024))
+ audio_file = os.path.splitext(video)[0] + ".mp3"
+ video_utils.extract_audio(video, audio_file)
+
+ voice_content = None
+ if self.asr_model is not None:
+ resp = await (ComputeKernel.get_instance().do_speech_to_text(audio_file, model=self.asr_model, prompt=None, response_format="text"))
+ if resp.result_code == ComputeTaskResultCode.OK:
+ voice_content = resp.result_str
+
+ content = []
+ if video_prompt is not None:
+ content.append({"type": "text", "text": video_prompt})
+ if voice_content is not None and voice_content != "":
+ content.append({"type": "text", "text": f"Voice content in video:{voice_content}"})
+
+ content.extend([{"type": "image_url", "image_url": {"url": frame}} for frame in frames])
+ msg_prompt.messages = [{"role": "user", "content": content}]
+ if self.mutil_model:
+ self.is_mutil_model = True
+ else:
+ logger.warning(f"mutil_model is not set!")
+ elif msg.is_audio_msg():
+ if self.enable_media2text:
+ logger.error(f"enable_media2text is not supported yet")
+ else:
+ prompt, audio_file = msg.get_audio_body()
+ resp = await (ComputeKernel.get_instance().do_speech_to_text(audio_file, model=self.asr_model, prompt=None, response_format="text"))
+ if resp.result_code != ComputeTaskResultCode.OK:
+ error_resp = msg.create_error_resp(resp.error_str)
+ return error_resp
+ else:
+ if prompt == "":
+ msg.body = resp.result_str
+ msg_prompt.messages = [{"role":"user","content":resp.result_str}]
+ else:
+ msg.body = f"{prompt}\nVoice content:{resp.result_str}"
+ msg_prompt.messages = [{"role":"user","content": prompt}, {"role": "user", "content": f"Voice content:{resp.result_str}"}]
+ else:
+ msg_prompt.messages = [{"role":"user","content":msg.body}]
+
+ return msg_prompt
+
+ async def sender_info(self,msg:AgentMsg)->str:
+ sender_id = msg.sender
+ #TODO Is sender an agent?
+ return await self.memory.get_contact_summary(sender_id)
+
+ async def load_chatlogs(self,msg:AgentMsg,max_length_by_token:int)->str:
+ ## like
+ #sender,[2023-11-1 12:00:00]
+ #content
+ return await self.memory.load_chatlogs(msg,max_length_by_token)
+
+ async def get_chat_summary(self,msg:AgentMsg)->str:
+ return await self.memory.get_chat_summary(msg)
+
+
+
+ async def get_extend_known_info(self,msg:AgentMsg,prompt:LLMPrompt)->str:
+ return None
+
+ async def prepare_prompt(self,input:Dict) -> LLMPrompt:
+ prompt = LLMPrompt()
+ # User Prompt
+ ## Input Msg
+ msg : AgentMsg = input.get("msg")
+ context_info = input.get("context_info")
+ if msg is None:
+ logger.error(f"LLMAgeMessageProcess prepare_prompt failed! input msg not found")
+ return None
+ msg_prompt = await self.get_prompt_from_msg(msg)
+ if msg_prompt is None:
+ logger.error(f"LLMAgeMessageProcess prepare_prompt failed! get_prompt_from_msg return None")
+ return None
+ prompt.append(msg_prompt)
+
+ ## 通用的角色相关的系统提示词
+ system_prompt_dict = self.prepare_role_system_prompt(context_info)
+
+ ## 已知信息
+ known_info = {}
+ #prompt.append_system_message(self.known_info_tips)
+ ### 信息发送者资料
+ known_info["sender_info"] = await self.sender_info(msg)
+ #prompt.append_system_message(await self.sender_info(self,msg))
+
+ system_prompt_dict["known_info"] = known_info
+
+ prompt.inner_functions =LLMProcessContext.aifunctions_to_inner_functions(self.llm_context.get_all_ai_functions())
+ if self.workspace:
+ #TODO eanble workspace functions?
+ logger.info(f"workspace is not none,enable workspace functions")
+
+ ## 给予查询KB的权限
+ if self.enable_kb:
+ logger.info(f"enable kb")
+
+
+ ### 根据Token Limit加载聊天记录
+ remain_token = self.get_remain_prompt_length(prompt,json.dumps(system_prompt_dict,ensure_ascii=False))
+ chat_record,is_all = await self.load_chatlogs(msg,remain_token - self.chat_summary_token_len)
+ if chat_record:
+ if len(chat_record) > 4:
+ known_info["chat_record"] = chat_record
+
+ if not is_all :
+ ### 如果出触发了Token Limit,则删除几条信息后，加载summary （summary的长度基本是固定的）
+ summary = await self.get_chat_summary(msg)
+ if summary:
+ if len(summary) > 4:
+ known_info["chat_summary"] = summary
+
+ # TODO: extend known info
+ #prompt.append_system_message(await self.get_extend_known_info(msg,prompt))
+
+ prompt.append_system_message(json.dumps(system_prompt_dict,ensure_ascii=False))
+ return prompt
+
+
+ async def post_llm_process(self,actions:List[ActionNode],input:Dict,llm_result:LLMResult) -> bool:
+ msg:AgentMsg = input.get("msg")
+ if msg.msg_type == AgentMsgType.TYPE_GROUPMSG:
+ resp_msg = msg.create_group_resp_msg(self.memory.agent_id,llm_result.resp)
+ else:
+ resp_msg = msg.create_resp_msg(llm_result.resp)
+
+ if llm_result.raw_result is not None:
+ llm_result.raw_result["_resp_msg"] = resp_msg
+
+ action_params = {}
+ action_params["_input"] = input
+ action_params["_memory"] = self.memory
+ action_params["_workspace"] = self.workspace
+ action_params["_resp_msg"] = resp_msg
+ action_params["_llm_result"] = llm_result
+ action_params["_agentid"] = self.memory.agent_id
+ action_params["_start_at"] = datetime.now()
+
+ await self._execute_actions(actions,action_params)
+
+ chatsession = self.memory.get_session_from_msg(msg)
+ chatsession.append(msg)
+ chatsession.append(resp_msg)
+
+ return True
+
+class AgentSelfThinking(LLMAgentBaseProcess):
+ def __init__(self) -> None:
+ super().__init__()
+
+
+ async def load_from_config(self, config: dict) -> Coroutine[Any, Any, bool]:
+ if await super().load_from_config(config) is False:
+ return False
+
+ async def _load_chat_history(self,token_limit:int):
+ chat_history = {}
+ session_list = AIChatSession.list_session(self.memory.agent_id ,self.memory.memory_db)
+ total_read_msg = 0
+ for session_id in session_list:
+ chatsession = AIChatSession.get_session_by_id(session_id,self.memory.memory_db)
+ session_history = {}
+ session_history["summary"] = chatsession.summary
+ session_history["id"] = chatsession.session_id
+ token_limit -= ComputeKernel.llm_num_tokens_from_text(chatsession.summary,self.model_name)
+ read_history_msg = 0
+
+ if token_limit > 8:
+ # load session chat history
+ cur_pos = chatsession.summarize_pos
+ messages = chatsession.read_history(0,cur_pos,"natural") # read
+ history_str = ""
+ for msg in messages:
+ read_history_msg += 1
+ total_read_msg += 1
+ cur_pos += 1
+ dt = datetime.fromtimestamp(float(msg.create_time))
+ formatted_time = dt.strftime('%y-%m-%d %H:%M:%S')
+ record_str = f"{msg.sender},[{formatted_time}]\n{msg.body}\n"
+ token_limit -= ComputeKernel.llm_num_tokens_from_text(record_str,self.model_name)
+ if token_limit < 8:
+ break
+
+ history_str = history_str + record_str
+
+ if read_history_msg >= 2:
+ session_history["history"] = history_str
+ chat_history[session_id] = session_history
+ chatsession.summarize_pos = cur_pos
+
+ else:
+ logger.info(f"load_chat_history reach token limit,load {total_read_msg} history messages.")
+ return chat_history
+
+ if total_read_msg < 2:
+ logger.info(f"load_chat_history: no history messages,return NONE")
+ return None
+
+ return chat_history
+
+
+ async def prepare_prompt(self,input:Dict) -> LLMPrompt:
+ prompt = LLMPrompt()
+
+ context_info = input.get("context_info")
+
+ system_prompt_dict = self.prepare_role_system_prompt(context_info)
+
+ # Known_info is the SESSION summary of the existence, the current task work record summary,
+ token_remain = self.get_remain_prompt_length(prompt,json.dumps(system_prompt_dict,ensure_ascii=False))
+ chat_history = await self._load_chat_history(token_remain)
+ if chat_history is None:
+ logger.info(f"prepare_prompt: no history messages,return NONE")
+ return None
+
+ prompt.inner_functions =LLMProcessContext.aifunctions_to_inner_functions(self.llm_context.get_all_ai_functions())
+
+ prompt.append_system_message(json.dumps(system_prompt_dict,ensure_ascii=False))
+ prompt.append_user_message(json.dumps(chat_history,ensure_ascii=False))
+ return prompt
+
+ async def post_llm_process(self,actions:List[ActionNode],input:Dict,llm_result:LLMResult) -> bool:
+ action_params = {}
+ action_params["_input"] = input
+ action_params["_memory"] = self.memory
+ action_params["_workspace"] = self.workspace
+ action_params["_llm_result"] = llm_result
+ action_params["_agentid"] = self.memory.agent_id
+ action_params["_start_at"] = datetime.now()
+ try:
+ if await self._execute_actions(actions,action_params) is False:
+ result_str = "execute action failed!"
+ except Exception as e:
+ logger.error(f"execute action failed! {e}")
+ result_str = "execute action failed!,error:" + str(e)
+
+class AgentSelfLearning(BaseLLMProcess):
+ def __init__(self) -> None:
+ super().__init__()
+
+ async def load_from_config(self, config: dict) -> Coroutine[Any, Any, bool]:
+ if await super().load_from_config(config) is False:
+ return False
+
+ async def prepare_prompt(self) -> LLMPrompt:
+ prompt = LLMPrompt()
+ pass
+
+ async def get_inner_function_for_exec(self,func_name:str) -> AIFunction:
+ pass
+
+ async def post_llm_process(self,actions:List[ActionNode]) -> bool:
+ pass
+
+class AgentSelfImprove(BaseLLMProcess):
+ def __init__(self) -> None:
+ super().__init__()
+
+
+
diff --git a/src/aios/agent/llm_process_loader.py b/src/aios/agent/llm_process_loader.py
new file mode 100644
index 0000000..912b3a0
--- /dev/null
+++ b/src/aios/agent/llm_process_loader.py
@@ -0,0 +1,42 @@
+from .llm_process import BaseLLMProcess, AgentMessageProcess,AgentSelfThinking,AgentSelfLearning,AgentSelfImprove
+from .llm_do_task import AgentTriageTaskList,AgentPlanTask,AgentReviewTask,AgentDo,AgentCheck
+
+from typing import Awaitable, Callable, Coroutine, Dict, List, Any
+import logging
+
+logger = logging.getLogger(__name__)
+
+class LLMProcessLoader:
+ def __init__(self) -> None:
+ self.loaders : Dict[str,Callable[[dict],Awaitable[BaseLLMProcess]]] = {}
+ return
+
+ @classmethod
+ def get_instance(cls)->"LLMProcessLoader":
+ if not hasattr(cls,"_instance"):
+ cls._instance = LLMProcessLoader()
+ return cls._instance
+
+ def register_loader(self, typename:str,loader:Callable[[dict],Awaitable[BaseLLMProcess]]):
+ self.loaders[typename] = loader
+
+ async def load_from_config(self,config:dict) -> BaseLLMProcess:
+ llm_type_name = config.get("type")
+ if llm_type_name:
+ loader = self.loaders.get(llm_type_name)
+ if loader:
+ return await loader(config)
+
+ selected_type = globals().get(llm_type_name)
+ if selected_type:
+ result : BaseLLMProcess = selected_type()
+ load_result = await result.load_from_config(config)
+ if load_result is False:
+ logger.warn(f"load LLMProcess {llm_type_name} from config failed! load_from_config return False")
+ return None
+ else:
+ return result
+
+
+ logger.warn(f"load LLMProcess {llm_type_name} from config failed! type not found")
+ return None
\ No newline at end of file
diff --git a/src/aios_kernel/role.py b/src/aios/agent/role.py
similarity index 90%
rename from src/aios_kernel/role.py
rename to src/aios/agent/role.py
index 83d64d6..3e31eee 100644
--- a/src/aios_kernel/role.py
+++ b/src/aios/agent/role.py
@@ -1,6 +1,7 @@
+# pylint:disable=E0402
import logging
-from .agent import AIAgent,AgentPrompt
+from ..proto.compute_task import LLMPrompt
class AIRole:
def __init__(self) -> None:
@@ -9,7 +10,7 @@ class AIRole:
self.role_id :str = None # $workflow_id.$sub_workflow_id.$role_name
self.fullname : str = None
self.agent_name : str = None
- self.prompt : AgentPrompt = None
+ self.prompt : LLMPrompt = None
self.introduce : str = None
self.agent = None
self.enable_function_list : list[str] = None
@@ -31,22 +32,22 @@ class AIRole:
prompt_node = config.get("prompt")
if prompt_node:
- self.prompt = AgentPrompt()
+ self.prompt = LLMPrompt()
if self.prompt.load_from_config(prompt_node) is False:
logging.error("load prompt failed!")
return False
-
+
intro_node = config.get("intro")
if intro_node is not None:
self.introduce = intro_node
history_node = config.get("history_len")
- if history_node:
+ if history_node is not None:
self.history_len = int(history_node)
if config.get("enable_function") is not None:
self.enable_function_list = config["enable_function"]
-
+
def get_role_id(self) -> str:
return self.role_id
@@ -55,15 +56,15 @@ class AIRole:
def get_name(self) -> str:
return self.role_name
-
- def get_prompt(self) -> AgentPrompt:
+
+ def get_prompt(self) -> LLMPrompt:
return self.prompt
-
+
class AIRoleGroup:
def __init__(self) -> None:
self.roles : dict[str,AIRole] = {}
self.owner_name : str = None
-
+
def load_from_config(self,config:dict) -> bool:
for k,v in config.items():
role = AIRole()
@@ -72,10 +73,9 @@ class AIRoleGroup:
return False
role.role_id = self.owner_name + "." + k
self.roles[k] = role
-
+
return True
def get(self,role_name:str) -> AIRole:
return self.roles.get(role_name)
-
\ No newline at end of file
diff --git a/src/aios_kernel/workflow.py b/src/aios/agent/workflow.py
similarity index 82%
rename from src/aios_kernel/workflow.py
rename to src/aios/agent/workflow.py
index 5cf0bbf..b025e76 100644
--- a/src/aios_kernel/workflow.py
+++ b/src/aios/agent/workflow.py
@@ -1,3 +1,4 @@
+# pylint:disable=E0402
import logging
import asyncio
import json
@@ -7,16 +8,19 @@ from asyncio import Queue
from typing import Optional,Tuple,List
from abc import ABC, abstractmethod
-from .environment import Environment,EnvironmentEvent
-from .agent_message import AgentMsg,AgentMsgStatus,FunctionItem,LLMResult
-from .agent import AgentPrompt,AgentMsg
+from ..proto.compute_task import *
+from ..proto.agent_msg import *
+from ..proto.ai_function import *
+
+from .agent_base import *
from .chatsession import AIChatSession
from .role import AIRole,AIRoleGroup
-from .ai_function import AIFunction,FunctionItem
-from .compute_kernel import ComputeKernel
-from .compute_task import ComputeTask,ComputeTaskResult,ComputeTaskState,ComputeTaskResultCode
-from .bus import AIBus
-from .workflow_env import WorkflowEnvironment
+
+from ..frame.compute_kernel import ComputeKernel
+from ..frame.bus import AIBus
+
+from ..environment.environment import BaseEnvironment
+from ..environment.workflow_env import WorkflowEnvironment
logger = logging.getLogger(__name__)
@@ -45,7 +49,7 @@ class Workflow:
def __init__(self) -> None:
self.workflow_name : str = None
self.workflow_id : str = None
- self.rule_prompt : AgentPrompt = None
+ self.rule_prompt : LLMPrompt = None
self.workflow_config = None
self.role_group : dict = None
self.input_filter : MessageFilter= None
@@ -80,7 +84,7 @@ class Workflow:
self.db_file = self.owner_workflow.db_file
if config.get("prompt") is not None:
- self.rule_prompt = AgentPrompt()
+ self.rule_prompt = LLMPrompt()
if self.rule_prompt.load_from_config(config.get("prompt")) is False:
logger.error("Workflow load prompt failed")
return False
@@ -196,7 +200,7 @@ class Workflow:
targets = self._parse_msg_target(msg.target)
if len(targets) > 1:
return await self._forword_msg(targets,msg)
-
+
#0 we don't support workflow join a group right now, this cloud be a feture in future
if msg.mentions is not None:
logger.warn(f"workflow {self.workflow_id} recv a group chat message,not support ignore!")
@@ -238,77 +242,6 @@ class Workflow:
error_resp = msg.create_error_resp(err_str)
return error_resp
- @classmethod
- def prase_llm_result(cls,llm_result_str:str)->LLMResult:
- r = LLMResult()
- if llm_result_str is None:
- r.state = "ignore"
- return r
- if llm_result_str == "ignore":
- r.state = "ignore"
- return r
-
- lines = llm_result_str.splitlines()
- is_need_wait = False
-
- def check_args(func_item:FunctionItem):
- match func_name:
- case "send_msg":# sendmsg($target_id,$msg_content)
- if len(func_args) != 1:
- logger.error(f"parse sendmsg failed! {func_call}")
- return False
- new_msg = AgentMsg()
- target_id = func_item.args[0]
- msg_content = func_item.body
- new_msg.set("_",target_id,msg_content)
-
- r.send_msgs.append(new_msg)
- is_need_wait = True
-
- case "post_msg":# postmsg($target_id,$msg_content)
- if len(func_args) != 1:
- logger.error(f"parse postmsg failed! {func_call}")
- return False
- new_msg = AgentMsg()
- target_id = func_item.args[0]
- msg_content = func_item.body
- new_msg.set("_",target_id,msg_content)
- r.post_msgs.append(new_msg)
-
- case "call":# call($func_name,$args_str)
- r.calls.append(func_item)
- is_need_wait = True
- return True
- case "post_call": # post_call($func_name,$args_str)
- r.post_calls.append(func_item)
- return True
-
- current_func : FunctionItem = None
- for line in lines:
- if line.startswith("##/"):
- if current_func:
- if check_args(current_func) is False:
- r.resp += current_func.dumps()
-
- func_name,func_args = AgentMsg.parse_function_call(line[3:])
- current_func = FunctionItem(func_name,func_args)
- else:
- if current_func:
- current_func.append_body(line + "\n")
- else:
- r.resp += line + "\n"
-
- if current_func:
- if check_args(current_func) is False:
- r.resp += current_func.dumps()
-
- if len(r.send_msgs) > 0 or len(r.calls) > 0:
- r.state = "waiting"
- else:
- r.state = "reponsed"
-
- return r
-
async def role_post_msg(self,msg:AgentMsg,the_role:AIRole,workflow_chat_session:AIChatSession):
msg.sender = the_role.get_role_id()
@@ -347,7 +280,7 @@ class Workflow:
logger.info(f"{msg.sender} post message {msg.msg_id} to AIBus: {msg.target}")
return await self.get_bus().send_message(msg)
- async def role_call(self,func_item:FunctionItem,the_role:AIRole):
+ async def role_call(self,func_item:ActionNode,the_role:AIRole):
logger.info(f"{the_role.role_id} call {func_item.name} ")
arguments = func_item.args
@@ -358,11 +291,11 @@ class Workflow:
result_str:str = await func_node.execute(**arguments)
return result_str
- async def role_post_call(self,func_item:FunctionItem,the_role:AIRole):
+ async def role_post_call(self,func_item:ActionNode,the_role:AIRole):
logger.info(f"{the_role.role_id} post call {func_item.name} ")
return await self.role_call(func_item,the_role)
- def _format_msg_by_env_value(self,prompt:AgentPrompt):
+ def _format_msg_by_env_value(self,prompt:LLMPrompt):
if self.workflow_env is None:
return
@@ -377,11 +310,11 @@ class Workflow:
result_func = []
for inner_func in all_inner_function:
- func_name = inner_func.get_name()
+ func_name = inner_func.get_id()
if the_role.enable_function_list is not None:
if len(the_role.enable_function_list) > 0:
if func_name not in the_role.enable_function_list:
- logger.debug(f"agent {self.agent_id} ignore inner func:{func_name}")
+ logger.debug(f"agent {the_role.agent.agent_id} ignore inner func:{func_name}")
continue
else:
continue
@@ -394,8 +327,7 @@ class Workflow:
return result_func
return None
- async def _role_execute_func(self,the_role:AIRole,inenr_func_call_node:dict,prompt:AgentPrompt,org_msg:AgentMsg,stack_limit = 5) -> [str,int]:
- from .compute_kernel import ComputeKernel
+ async def _role_execute_func(self,the_role:AIRole,inenr_func_call_node:dict,prompt:LLMPrompt,org_msg:AgentMsg,stack_limit = 5) -> [str,int]:
func_name = inenr_func_call_node.get("name")
arguments = json.loads(inenr_func_call_node.get("arguments"))
@@ -410,6 +342,7 @@ class Workflow:
except Exception as e:
result_str = f"execute {func_name} error:{str(e)}"
logger.error(f"llm execute inner func:{func_name} error:{e}")
+ logger.exception(e)
inner_functions = self._get_inner_functions(the_role)
@@ -440,8 +373,7 @@ class Workflow:
async def role_process_msg(self,msg:AgentMsg,the_role:AIRole,workflow_chat_session:AIChatSession) -> AgentMsg:
msg.target = the_role.get_role_id()
-
- prompt = AgentPrompt()
+ prompt = LLMPrompt()
prompt.append(the_role.agent.agent_prompt)
prompt.append(self.get_workflow_rule_prompt())
prompt.append(the_role.get_prompt())
@@ -451,7 +383,7 @@ class Workflow:
#support group chat, user content include sender name!
prompt.append(await self._get_prompt_from_session(the_role,workflow_chat_session))
- msg_prompt = AgentPrompt()
+ msg_prompt = LLMPrompt()
msg_prompt.messages = [{"role":"user","content":f"user name is {msg.sender}, his question is :{msg.body}"}]
prompt.append(msg_prompt)
@@ -465,7 +397,7 @@ class Workflow:
logger.error(f"llm compute error:{task_result.error_str}")
error_resp = msg.create_error_resp(task_result.error_str)
return error_resp
-
+
result_str = task_result.result_str
logger.info(f"{the_role.role_id} process {msg.sender}:{msg.body},llm str is :{result_str}")
@@ -480,7 +412,7 @@ class Workflow:
error_resp = msg.create_error_resp(result_str)
return error_resp
- result : LLMResult = Workflow.prase_llm_result(result_str)
+ result : LLMResult = LLMResult.from_str(result_str)
for postmsg in result.post_msgs:
postmsg.prev_msg_id = msg.get_msg_id()
# might be craete a new msg.topic for this postmsg
@@ -513,7 +445,7 @@ class Workflow:
workflow_chat_session.append(resp_msg)
#await self.get_bus().resp_message(resp_msg)
return resp_msg
- case "waiting":
+ case "waiting":
for sendmsg in result.send_msgs:
target = sendmsg.target
sendmsg.topic = msg.topic
@@ -529,21 +461,21 @@ class Workflow:
else:
# message will be saved in role.process_message
pass
-
- this_llm_resp_prompt = AgentPrompt()
+
+ this_llm_resp_prompt = LLMPrompt()
this_llm_resp_prompt.messages = [{"role":"assistant","content":result_str}]
prompt.append(this_llm_resp_prompt)
- result_prompt = AgentPrompt()
+ result_prompt = LLMPrompt()
result_prompt.messages = [{"role":"user","content":result_prompt_str}]
prompt.append(result_prompt)
return await _do_process_msg()
return await _do_process_msg()
- async def _get_prompt_from_session(self,the_role:AIRole,chatsession:AIChatSession) -> AgentPrompt:
+ async def _get_prompt_from_session(self,the_role:AIRole,chatsession:AIChatSession) -> LLMPrompt:
messages = chatsession.read_history(the_role.history_len) # read last 10 message
- result_prompt = AgentPrompt()
+ result_prompt = LLMPrompt()
for msg in reversed(messages):
if msg.sender == the_role.role_id:
@@ -553,21 +485,21 @@ class Workflow:
return result_prompt
- def _get_knowlege_prompt(self,role_name:str) -> AgentPrompt:
+ def _get_knowlege_prompt(self,role_name:str) -> LLMPrompt:
pass
- def get_workflow_rule_prompt(self) -> AgentPrompt:
+ def get_workflow_rule_prompt(self) -> LLMPrompt:
return self.rule_prompt
- def _env_event_to_msg(self,env_event:EnvironmentEvent) -> AgentMsg:
+ # def _env_event_to_msg(self,env_event:EnvironmentEvent) -> AgentMsg:
+ # pass
+
+ def get_inner_environment(self,env_id:str) -> BaseEnvironment:
pass
- def get_inner_environment(self,env_id:str) -> Environment:
- pass
-
- def connect_to_environment(self,the_env:Environment,conn_info:dict) -> None:
+ def connect_to_environment(self,the_env:BaseEnvironment,conn_info:dict) -> None:
if the_env is not None:
- self.workflow_env.add_owner_env(the_env)
+ self.workflow_env.add_env(the_env)
#for event2msg in conn_info:
# for k,v in event2msg:
diff --git a/src/aios/agent/workspace.py b/src/aios/agent/workspace.py
new file mode 100644
index 0000000..8259176
--- /dev/null
+++ b/src/aios/agent/workspace.py
@@ -0,0 +1,787 @@
+# pylint:disable=E0402
+from ast import Dict
+import json
+import sqlite3
+import os
+import glob
+import time
+from typing import List, Optional
+import aiofiles
+
+from ..proto.agent_msg import AgentMsg
+from ..proto.ai_function import AIFunction, ParameterDefine,SimpleAIFunction,ActionNode,SimpleAIAction
+from ..proto.agent_task import AgentTask, AgentTaskState,AgentTodo,AgentWorkLog,AgentTaskManager
+from ..storage.storage import AIStorage
+from ..frame.bus import AIBus
+from .llm_context import GlobaToolsLibrary
+
+import logging
+logger = logging.getLogger(__name__)
+
+class LocalAgentTaskManger(AgentTaskManager):
+ def __init__(self, owner_id):
+ super().__init__()
+ self.root_path = f"{AIStorage.get_instance().get_myai_dir()}/agent_data/{owner_id}/workspace/"
+ #self.root_path = os.path.join(workspace, list_type)
+ if not os.path.exists(self.root_path):
+ os.makedirs(self.root_path)
+
+ self.db_path = os.path.join(self.root_path, "tasklist.db")
+ self.conn = None
+ try:
+ self.conn = sqlite3.connect(self.db_path)
+ except Exception as e:
+ logger.error("Error occurred while connecting to database: %s", e)
+ return None
+
+ cursor = self.conn.cursor()
+ cursor.execute('''
+ CREATE TABLE IF NOT EXISTS obj_list (
+ id TEXT,
+ path TEXT
+ )
+ ''')
+ self.conn.commit()
+
+ def _get_obj_path(self,objid:str) -> str:
+ cursor = self.conn.cursor()
+ cursor.execute('''
+ SELECT path FROM obj_list WHERE id = ?
+ ''',(objid,))
+ row = cursor.fetchone()
+ if row:
+ return row[0]
+ else:
+ return None
+
+ def _save_obj_path(self,objid:str,path:str):
+ cursor = self.conn.cursor()
+ cursor.execute('''
+ INSERT INTO obj_list (id,path) VALUES (?,?)
+ ''',(objid,path))
+ self.conn.commit()
+
+ async def create_task(self,task:AgentTask,parent_id:str = None) -> str:
+ try:
+ #perfix = task.task_id[-5]
+ if parent_id:
+ parent_path = self._get_obj_path(parent_id)
+ task_path = f"{parent_path}/{task.title}"
+ else:
+ task_path = f"{task.title}"
+
+ dir_path = f"{self.root_path}/{task_path}"
+
+ os.makedirs(dir_path)
+ detail_path = f"{dir_path}/detail"
+ if task.task_path is None:
+ task.task_path = task_path
+ self._save_obj_path(task.task_id,task_path)
+ logger.info("create_task at %s",detail_path)
+ async with aiofiles.open(detail_path, mode='w', encoding="utf-8") as f:
+ await f.write(json.dumps(task.to_dict(),ensure_ascii=False))
+ return "create task ok"
+ except Exception as e:
+ logger.error("create_task failed:%s",e)
+ return str(e)
+
+
+
+ async def set_todos(self,owner_task_id:str,todos:List[Dict]):
+ owner_task_path = self._get_obj_path(owner_task_id)
+ if owner_task_path is None:
+ return f"owner task {owner_task_id} not found"
+
+ try:
+ directory = f"{self.root_path}/{owner_task_path}"
+ file_extension = "*.todo"
+ pattern = os.path.join(directory, file_extension)
+ files = glob.glob(pattern)
+
+ for file in files:
+ os.remove(file)
+ logger.info(f"Deleted {file}")
+ except Exception as e:
+ logger.error("set_todos deleted todos failed:%s",e)
+
+ try:
+ step_order = 0
+ for todo in todos:
+ todo_obj = AgentTodo.from_dict(todo)
+ todo_obj.step_order = step_order
+ todo_obj.owner_taskid = owner_task_id
+ todo_path = f"{self.root_path}/{owner_task_path}/#{step_order} {todo_obj.title}.todo"
+ self._save_obj_path(todo_obj.todo_id,todo_path)
+ async with aiofiles.open(todo_path, mode='w', encoding="utf-8") as f:
+ await f.write(json.dumps(todo_obj.to_dict(),ensure_ascii=False))
+ logger.info("create_todos at %s OK!",todo_path)
+ step_order += 1
+ except Exception as e:
+ logger.error("create_todos failed:%s",e)
+ return str(e)
+
+ return None
+
+
+ async def append_worklog(self,task:AgentTask,log:AgentWorkLog):
+ worklog = f"{self.root_path}/{task.task_path}/.worklog"
+
+ async with aiofiles.open(worklog, mode='w+', encoding="utf-8") as f:
+ content = await f.read()
+ if len(content) > 0:
+ json_obj = json.loads(content)
+ else:
+ json_obj = {}
+ logs = json_obj.get("logs")
+ if logs is None:
+ logs = []
+ logs.append(log.to_dict())
+ json_obj["logs"] = logs
+ await f.write(json.dumps(json_obj,ensure_ascii=False))
+
+
+ async def get_worklog(self,obj_id:str)->List[AgentWorkLog]:
+ obj_path = self._get_obj_path(obj_id)
+ if obj_path is None:
+ return []
+
+ if obj_path.endswith(".todo"):
+ dir_path = os.path.dirname(obj_path)
+ worklog_path = f"{self.root_path}/{dir_path}/.worklog"
+ else:
+ worklog_path = f"{self.root_path}/{obj_path}/.worklog"
+
+ async with aiofiles.open(worklog_path, mode='r', encoding="utf-8") as f:
+ content = await f.read()
+ if len(content) > 0:
+ json_obj = json.loads(content)
+ else:
+ json_obj = {}
+ logs = json_obj.get("logs")
+ return logs
+
+
+ async def get_task(self,task_id:str) -> AgentTask:
+ task_path = self._get_obj_path(task_id)
+ if task_path is None:
+ logger.error("get_task:%s,not found!",task_id)
+ return None
+
+ return await self.get_task_by_path(task_path)
+
+ async def _get_task_by_fullpath(self,task_fullpath) -> AgentTask:
+ detail_path = f"{task_fullpath}/detail"
+ try:
+ with open(detail_path, mode='r', encoding="utf-8") as f:
+ task_dict = json.load(f)
+ result_task:AgentTask = AgentTask.from_dict(task_dict)
+ if result_task:
+ relative_path = os.path.relpath(task_fullpath, self.root_path)
+ result_task.task_path = relative_path
+ else:
+ logger.error("_get_task_by_fullpath:%s,parse failed!",detail_path)
+
+ return result_task
+ except Exception as e:
+ logger.error("_get_task_by_fullpath:%s,failed:%s",task_fullpath,e)
+ return None
+
+ async def get_task_by_path(self,task_path:str) -> AgentTask:
+ full_path = f"{self.root_path}/{task_path}"
+ return await self._get_task_by_fullpath(full_path)
+
+ async def get_todo(self,todo_id:str) -> AgentTodo:
+ todo_path = self._get_obj_path(todo_id)
+ if todo_path is None:
+ logger.error("get_todo:%s,not found!",todo_id)
+ return None
+
+ try:
+ with open(todo_path, mode='r', encoding="utf-8") as f:
+ todo_dict = json.load(f)
+ result_todo:AgentTodo = AgentTodo.from_dict(todo_dict)
+ if result_todo:
+ result_todo.todo_path = todo_path
+ else:
+ logger.error("get_todo:%s,parse failed!",todo_path)
+
+ return result_todo
+ except Exception as e:
+ logger.error("get_todo:%s,failed:%s",todo_path,e)
+
+ return None
+
+ async def get_sub_tasks(self,task_id:str) -> List[AgentTask]:
+ task_path = self._get_obj_path(task_id)
+
+ if task_path is None:
+ return []
+ task_path = f"{self.root_path}/{task_path}"
+ sub_tasks = []
+ for sub_item in os.listdir(task_path):
+ if sub_item.startswith("."):
+ continue
+ if sub_item == "workspace":
+ continue
+
+ full_path = os.path.join(task_path, sub_item)
+ if os.path.isdir(full_path):
+ sub_task = await self.get_task_by_path(f"{task_path}/{sub_item}")
+ if sub_task:
+ sub_tasks.append(sub_task)
+
+ return sub_tasks
+
+
+ async def get_sub_todos(self,task_id:str) -> List[AgentTodo]:
+ task_path = self._get_obj_path(task_id)
+ if task_path is None:
+ return []
+ task_path = f"{self.root_path}/{task_path}"
+ sub_todos = []
+ for sub_item in os.listdir(task_path):
+ if sub_item.startswith("."):
+ continue
+ if sub_item == "workspace":
+ continue
+ if sub_item == "details":
+ continue
+
+ full_path = os.path.join(task_path, sub_item)
+ if os.path.isfile(full_path) and sub_item.endswith(".todo"):
+ sub_todo = await self.get_todo_by_path(full_path)
+ if sub_todo:
+ sub_todos.append(sub_todo)
+
+ return sub_todos
+
+ async def get_todo_by_path(self,todo_path:str) -> AgentTodo:
+ async with aiofiles.open(todo_path, mode='r', encoding="utf-8") as f:
+ s = await f.read()
+ todo_dict = json.loads(s)
+ result_todo:AgentTodo = AgentTodo.from_dict(todo_dict)
+ if result_todo:
+ result_todo.todo_path = todo_path
+ else:
+ logger.error("get_todo_by_path:%s,parse failed!",todo_path)
+
+ return result_todo
+
+ #async def get_task_depends(self,task_id:str) -> List[AgentTask]:
+ # pass
+
+
+ async def list_task(self,filter:Optional[dict] = None ) -> List[AgentTask]:
+ directory_path = self.root_path
+ result_list:List[AgentTask] = []
+ special_state = None
+ if filter:
+ special_state = filter.get("state")
+ #agent_id = filter.get("agent_id")
+
+ for entry in os.scandir(directory_path):
+ if not entry.is_dir():
+ continue
+ if entry.name.startswith("."):
+ continue
+ if entry.name == "workspace":
+ continue
+ task_item = await self._get_task_by_fullpath(entry.path)
+ if task_item:
+ if filter is None:
+ if task_item.is_finish():
+ continue
+
+ if special_state:
+ if task_item.state != special_state:
+ continue
+
+
+ result_list.append(task_item)
+
+ return result_list
+
+
+ async def update_task(self,task:AgentTask):
+ detail_path = f"{self.root_path}/{task.task_path}/detail"
+ try:
+ new_task_content = json.dumps(task.to_dict(),ensure_ascii=False)
+ async with aiofiles.open(detail_path, mode='w', encoding="utf-8") as f:
+ await f.write(new_task_content)
+ except Exception as e:
+ logger.error("update_task failed:%s",e)
+ return str(e)
+
+ return None
+
+ async def update_todo(self,todo:AgentTodo):
+ todo_path = self._get_obj_path(todo.todo_id)
+ if todo_path is None:
+ return f"todo {todo.todo_id} not found"
+
+ try:
+ new_todo_content = json.dumps(todo.to_dict(),ensure_ascii=False)
+ async with aiofiles.open(todo_path, mode='w', encoding="utf-8") as f:
+ await f.write(new_todo_content)
+ except Exception as e:
+ logger.error("update_todo failed:%s",e)
+ return str(e)
+
+ return None
+
+ #async def update_task_state(self,task_id,state:str):
+ # pass
+
+ #async def update_todo_state(self,task_id,state:str):
+ # pass
+
+ #todo共享其所在task的文件夹
+ # if task_id is none, means root folder in workspace
+ def _get_taskfile_path(self,task_id:str,path:str)->str:
+ root_path = self.root_path
+ if task_id is None:
+ root_path = f"{root_path}/workspace"
+ else:
+ task_path = self._get_obj_path(task_id)
+ if task_path is None:
+ return None
+ root_path = f"{task_path}/wrorkspace"
+
+ file_path = f"{root_path}/{path}"
+ return file_path
+
+ async def read_task_file(self,task_id:str,path:str)->str:
+ file_path = self._get_taskfile_path(task_id,path)
+ if not os.path.exists(file_path):
+ return None
+
+ try:
+ async with aiofiles.open(file_path, mode='r', encoding="utf-8") as f:
+ content = await f.read()
+ return content
+ except Exception as e:
+ logger.error("read_task_file failed:%s",e)
+ return None
+
+
+ async def write_task_file(self,task_id:str,path:str,content:str):
+ file_path = self._get_taskfile_path(task_id,path)
+ # write file
+ try:
+ dir_name = os.path.dirname(file_path)
+ os.makedirs(dir_name)
+ async with aiofiles.open(file_path, mode='w', encoding="utf-8") as f:
+ await f.write(content)
+ except Exception as e:
+ logger.error("write_task_file failed:%s",e)
+ return str(e)
+
+ async def append_task_file(self,task_id:str,path:str,content:str):
+ file_path = self._get_taskfile_path(task_id,path)
+ # append file
+ try:
+ async with aiofiles.open(file_path, mode='a', encoding="utf-8") as f:
+ await f.write(content)
+ except Exception as e:
+ logger.error("append_task_file failed:%s",e)
+ return str(e)
+
+
+ async def list_task_dir(self,task_id:str,path:str) -> List[str]:
+ dir_path = self._get_taskfile_path(task_id,path)
+ if not os.path.exists(dir_path):
+ return None
+
+ try:
+ result_node = os.listdir(dir_path)
+ result = []
+ for name in result_node:
+ if name.startswith("."):
+ continue
+
+ result.append(name)
+ return result
+ except Exception as e:
+ logger.error("list_task_dir failed:%s",e)
+ return None
+
+
+ async def remove_task_file(self,task_id:str,path:str):
+ file_path = self._get_taskfile_path(task_id,path)
+ try:
+ os.remove(file_path)
+ except Exception as e:
+ logger.error("remove_task_file failed:%s",e)
+ return str(e)
+
+ return None
+
+
+
+class AgentWorkspace:
+ def __init__(self,owner_id:str) -> None:
+ self.owner_id : str = owner_id
+ self.task_mgr : AgentTaskManager = LocalAgentTaskManger(owner_id)
+
+ @staticmethod
+ def register_ai_functions():
+ async def post_message(parameters):
+ _agent_id = parameters.get("_agentid")
+ if _agent_id is None:
+ return "_agentid not found"
+
+ target = parameters.get("target")
+ if target is None:
+ return "target not found"
+ message = parameters.get("message")
+ if message is None:
+ return "message not found"
+ topic = parameters.get("topic")
+
+ msg = AgentMsg()
+ msg.sender = _agent_id
+ msg.body = message
+ msg.topic = topic
+ msg.target = target
+ msg.create_time = time.time()
+
+ is_post_ok = await AIBus.get_default_bus().post_message(msg)
+ if is_post_ok:
+ return "post message ok!"
+ else:
+ return f"post message to {target} failed!"
+
+ parameters = ParameterDefine.create_parameters({
+ "target": {"type": "string", "description": "target agent/contact fullname or telephone or email"},
+ "topic": {"type": "string", "description": "optional, message topic"},
+ "message": {"type": "string", "description": "message content"},
+ })
+ post_message_action = SimpleAIFunction(
+ "post_message",
+ "Post a message to target agent/contact",
+ post_message,
+ parameters,
+ )
+ GlobaToolsLibrary.get_instance().register_tool_function(post_message_action)
+
+ async def send_message(parameters):
+ _agent_id = parameters.get("_agentid")
+ if _agent_id is None:
+ return "_agentid not found"
+
+ target = parameters.get("target")
+ if target is None:
+ return "target not found"
+ message = parameters.get("message")
+ if message is None:
+ return "message not found"
+ topic = parameters.get("topic")
+
+ msg = AgentMsg()
+ msg.sender = _agent_id
+ msg.body = message
+ msg.topic = topic
+ msg.target = target
+ msg.create_time = time.time()
+
+ resp = await AIBus.get_default_bus().send_message(msg)
+ if resp:
+ return f"resp is : {resp.body}"
+ else:
+ return f"send message to {target} failed!"
+
+ parameters = ParameterDefine.create_parameters({
+ "target": {"type": "string", "description": "target agent/contact id"},
+ "topic": {"type": "string", "description": "optional, message topic"},
+ "message": {"type": "string", "description": "message content"},
+ })
+ send_message_action = SimpleAIFunction(
+ "send_message",
+ "send a message to target agent/contact, and wait for reply",
+ send_message,
+ parameters,
+ )
+ GlobaToolsLibrary.get_instance().register_tool_function(send_message_action)
+
+ async def create_task(params):
+ _workspace = params.get("_workspace")
+ _agent_id = params.get("_agentid")
+ if _workspace is None:
+ return "_workspace not found"
+ if params.get("creator") is None:
+ params["creator"] = _agent_id
+ taskObj = AgentTask.create_by_dict(params)
+ parent_id = params.get("parent")
+ return await _workspace.task_mgr.create_task(taskObj,parent_id)
+ parameters = ParameterDefine.create_parameters({
+ "title" : {"type": "string", "description": "task title,Simple and clear, try to include the task \ Related personnel \ place \ key conditions \ time element involved in the event"},
+ "detail" : {"type": "string", "description": "task detail(simple task can not be filled)"},
+ "priority" : {"type": "int", "description": "task priority from 1-10"},
+ #"due_date": {"type": "isoformat time string", "description": "optional,confirm task due date"},
+ #"expiration_time": {"type": "isoformat time string", "description": "optional,confirm task expiration time"},
+ "parent": {"type": "string", "description": "optional,parent task id"},
+ })
+ create_task_action = SimpleAIFunction(
+ "agent.workspace.create_task",
+ "Create a task",
+ create_task,
+ parameters,
+ )
+ GlobaToolsLibrary.get_instance().register_tool_function(create_task_action)
+
+
+ async def cancel_task(parameters):
+ _workspace = parameters.get("_workspace")
+ if _workspace is None:
+ return "_workspace not found"
+ task_id = parameters.get("task_id")
+ task : AgentTask = await _workspace.task_mgr.get_task(task_id)
+ if task is None:
+ return f"task {task_id} not found"
+ task.state = AgentTaskState.TASK_STATE_CANCEL
+ await _workspace.task_mgr.update_task(task)
+ return "canncel task ok"
+
+ parameters = ParameterDefine.create_parameters({
+ "task_id": {"type": "string", "description": "task id which want to cancel"},
+ })
+ cancel_task_action = SimpleAIFunction(
+ "agent.workspace.cancel_task",
+ "Cancel this task",
+ cancel_task,
+ parameters
+ )
+ GlobaToolsLibrary.get_instance().register_tool_function(cancel_task_action)
+
+
+ async def confirm_task(parameters):
+ _workspace = parameters.get("_workspace")
+ if _workspace is None:
+ return "_workspace not found"
+ task_id = parameters.get("task_id")
+ task : AgentTask = await _workspace.task_mgr.get_task(task_id)
+ if task is None:
+ return f"task {task_id} not found"
+ if parameters.get("priority"):
+ task.priority = parameters.get("priority")
+ if parameters.get("next_attention_time"):
+ task.next_attention_time = parameters.get("next_attention_time")
+ if parameters.get("expiration_time"):
+ task.expiration_time = parameters.get("expiration_time")
+ if parameters.get("due_date"):
+ task.due_date = parameters.get("due_date")
+ task.state = AgentTaskState.TASK_STATE_CONFIRMED
+ await _workspace.task_mgr.update_task(task)
+ return "confirm task ok"
+ parameters = ParameterDefine.create_parameters({
+ "task_id": {"type": "string", "description": "task id which want to confirm"},
+ "next_attention_time": {"type": "isoformat time string", "description": "optional,confirm task next attention time"},
+ "expiration_time": {"type": "isoformat time string", "description": "optional,confirm task expiration time"},
+ #"due_date": {"type": "isoformat time string", "description": "optional,confirm task due date"},
+ "priority": {"type": "int", "description": "optional,task priority from 1-10"},
+ })
+ confirm_task_action = SimpleAIFunction(
+ "agent.workspace.confirm_task",
+ "After understanding the content of the task, the importance of the importance of the task, the priority, the deadline, etc.",
+ confirm_task,
+ parameters
+ )
+ GlobaToolsLibrary.get_instance().register_tool_function(confirm_task_action)
+
+ async def list_task(parameters):
+ _workspace = parameters.get("_workspace")
+ if _workspace is None:
+ return "_workspace not found"
+ all_task = await _workspace.task_mgr.list_task(None)
+ if all_task:
+ return json.dumps([task.to_dict() for task in all_task],ensure_ascii=False)
+ else :
+ return "no task"
+ list_task_ai_function = SimpleAIFunction("agent.workspace.list_task",
+ "list all tasks in json format",
+ list_task,{})
+ GlobaToolsLibrary.get_instance().register_tool_function(list_task_ai_function)
+
+ async def update_task(parameters):
+ _workspace : AgentWorkspace= parameters.get("_workspace")
+ if _workspace is None:
+ return "_workspace not found"
+ task_id = parameters.get("task_id")
+ task:AgentTask = await _workspace.task_mgr.get_task(task_id)
+ if task is None:
+ return f"task {task_id} not found"
+ if parameters.get("title"):
+ task.title = parameters.get("title")
+ if parameters.get("detail"):
+ task.detail = parameters.get("detail")
+ if parameters.get("priority"):
+ task.priority = parameters.get("priority")
+ if parameters.get("new_state"):
+ task.state = AgentTaskState.from_str(parameters.get("new_state"))
+ if parameters.get("next_attention_time"):
+ task.next_attention_time = parameters.get("next_attention_time")
+ if parameters.get("due_date"):
+ task.due_date = parameters.get("due_date")
+ if parameters.get("expiration_time"):
+ task.expiration_time = parameters.get("expiration_time")
+ await _workspace.task_mgr.update_task(task)
+ return "update task ok"
+ parameters = ParameterDefine.create_parameters({
+ "task_id": {"type": "string", "description": "task id which want to update"},
+ "new_state": {"type": "string", "description": "optional,new task state: cancel or done"},
+ "next_attention_time": {"type": "isoformat time string", "description": "optional,update task next attention time"},
+ "expiration_time": {"type": "isoformat time string", "description": "optional,update task expiration time"},
+ "priority": {"type": "int", "description": "optional,task priority from 1-10"},
+ "title": {"type": "string", "description": "optional, new task title"},
+ "detail": {"type": "string", "description": "optional, new task detail(simple task can not be filled)"},
+ #"due_date": {"type": "string", "description": "optional,new task due date"},
+ })
+ update_task_ai_function = SimpleAIFunction("agent.workspace.update_task",
+ "update task to new state",
+ update_task,parameters)
+ GlobaToolsLibrary.get_instance().register_tool_function(update_task_ai_function)
+
+ async def set_todos(parameters):
+ _workspace : AgentWorkspace= parameters.get("_workspace")
+ if _workspace is None:
+ return "_workspace not found"
+ task_id = parameters.get("task_id")
+ task:AgentTask = await _workspace.task_mgr.get_task(task_id)
+ if task is None:
+ return f"task {task_id} not found"
+ todos = parameters.get("todos")
+ if todos is None:
+ return "todos not found"
+ await _workspace.task_mgr.set_todos(task_id,todos)
+ return "set todos ok"
+
+ todo_demo = """
+ [
+ {
+ "title": "todo1",
+ "detail": "todo1 detail",
+ "tags": "tag1,tag2",
+ "due_date": "2021-01-01",
+ "priority": 1
+ },
+ ]
+ """
+ parameters = ParameterDefine.create_parameters({
+ "task_id": {"type": "string", "description": "task id which want to set todos"},
+ "todos": {"type": "list", "description": f"List of todo, todo is a dict like {todo_demo}"},
+ })
+ set_todos_ai_function = SimpleAIFunction("agent.workspace.set_todos",
+ "set todos for task",
+ set_todos,parameters)
+ GlobaToolsLibrary.get_instance().register_tool_function(set_todos_ai_function)
+
+ async def update_todo(parameters):
+ _workspace : AgentWorkspace= parameters.get("_workspace")
+ if _workspace is None:
+ return "_workspace not found"
+ todo_id = parameters.get("todo_id")
+ todo : AgentTodo = await _workspace.task_mgr.get_todo(todo_id)
+ if todo is None:
+ return f"todo {todo_id} not found"
+
+ parameters = ParameterDefine.create_parameters({
+ "todo_id": {"type": "string", "description": "todo id which want to update"},
+ "new_state": {"type": "string", "description": "optional,new todo state: execute_ok , execute_failed, done or check_failed"},
+ })
+ update_todo_ai_function = SimpleAIFunction("agent.workspace.update_todo",
+ "update todo to new state",
+ update_todo,parameters)
+ GlobaToolsLibrary.get_instance().register_tool_function(update_todo_ai_function)
+
+
+ # write file
+ async def write_task_file(parameters):
+ _workspace : AgentWorkspace= parameters.get("_workspace")
+ if _workspace is None:
+ return "_workspace not found"
+ task_id = parameters.get("task_id")
+ path = parameters.get("filename")
+ content = parameters.get("content")
+ await _workspace.task_mgr.write_task_file(task_id,path,content)
+ return "write task file ok"
+ parameters = ParameterDefine.create_parameters({
+ "filename": {"type": "string", "description": "filename"},
+ "content": {"type": "string", "description": "file content"},
+ })
+ write_task_file_ai_function = SimpleAIFunction("agent.workspace.write_file",
+ "write file for task",
+ write_task_file,parameters)
+ GlobaToolsLibrary.get_instance().register_tool_function(write_task_file_ai_function)
+
+ # append file
+ async def append_task_file(parameters):
+ _workspace : AgentWorkspace= parameters.get("_workspace")
+ if _workspace is None:
+ return "_workspace not found"
+ task_id = parameters.get("task_id")
+ path = parameters.get("filename")
+ content = parameters.get("content")
+ await _workspace.task_mgr.append_task_file(task_id,path,content)
+ return "append task file ok"
+ parameters = ParameterDefine.create_parameters({
+ "filename": {"type": "string", "description": "filename"},
+ "content": {"type": "string", "description": "file content"},
+ })
+ append_task_file_ai_function = SimpleAIFunction("agent.workspace.append_file",
+ "append file for task",
+ append_task_file,parameters)
+ GlobaToolsLibrary.get_instance().register_tool_function(append_task_file_ai_function)
+
+ # read file
+ async def read_task_file(parameters):
+ _workspace : AgentWorkspace= parameters.get("_workspace")
+ if _workspace is None:
+ return "_workspace not found"
+ task_id = parameters.get("task_id")
+ path = parameters.get("filename")
+ content = await _workspace.task_mgr.read_task_file(task_id,path)
+ return content
+ parameters = ParameterDefine.create_parameters({
+ "filename": {"type": "string", "description": "filename"},
+ })
+ read_task_file_ai_function = SimpleAIFunction("agent.workspace.read_file",
+ "read file for task",
+ read_task_file,parameters)
+ GlobaToolsLibrary.get_instance().register_tool_function(read_task_file_ai_function)
+
+ # list dir
+ async def list_task_dir(parameters):
+ _workspace : AgentWorkspace= parameters.get("_workspace")
+ if _workspace is None:
+ return "_workspace not found"
+ task_id = parameters.get("task_id")
+ path = parameters.get("path")
+ content = await _workspace.task_mgr.list_task_dir(task_id,path)
+ return content
+ parameters = ParameterDefine.create_parameters({
+ "path": {"type": "string", "description": "The relative path of the dir"},
+ })
+ list_task_dir_ai_function = SimpleAIFunction("agent.workspace.list_dir",
+ "list dir in task workspace",
+ list_task_dir,parameters)
+ GlobaToolsLibrary.get_instance().register_tool_function(list_task_dir_ai_function)
+
+ # remove file
+ async def remove_task_file(parameters):
+ _workspace : AgentWorkspace= parameters.get("_workspace")
+ if _workspace is None:
+ return "_workspace not found"
+ task_id = parameters.get("task_id")
+ path = parameters.get("filename")
+ content = await _workspace.task_mgr.remove_task_file(task_id,path)
+ return content
+ parameters = ParameterDefine.create_parameters({
+ "filename": {"type": "string", "description": "filename"},
+ })
+ remove_task_file_ai_function = SimpleAIFunction("agent.workspace.remove_file",
+ "remove file for task",
+ remove_task_file,parameters)
+ GlobaToolsLibrary.get_instance().register_tool_function(remove_task_file_ai_function)
+
+
+
diff --git a/src/aios/ai_functions/asr_function.py b/src/aios/ai_functions/asr_function.py
new file mode 100644
index 0000000..7c729d0
--- /dev/null
+++ b/src/aios/ai_functions/asr_function.py
@@ -0,0 +1,57 @@
+# pylint:disable=E0402
+import logging
+from typing import Dict
+
+from ..proto.ai_function import *
+from ..agent.llm_context import GlobaToolsLibrary
+from ..frame.compute_kernel import ComputeKernel
+
+logger = logging.getLogger(__name__)
+
+class AsrFunction(AIFunction):
+ def __init__(self):
+ self.func_id = "aigc.voice_to_text"
+ self.description = "Voice recognition, convert the voice into text"
+ self.parameters = ParameterDefine.create_parameters({
+ "audio_file": {"type": "string", "description": "Audio file path"},
+ "model": {"type": "string", "description": "Recognition model", "enum": ["openai-whisper"]},
+ "prompt": {"type": "string", "description": "Prompt statement, can be None"},
+ "response_format": {"type": "string", "description": "Return format", "enum": ["text", "json", "srt", "verbose_json", "vtt"]},
+ })
+
+ def register_function(self):
+ GlobaToolsLibrary.get_instance().register_tool_function(self)
+
+ def get_id(self) -> str:
+ return self.func_id
+
+ def get_description(self) -> str:
+ return self.description
+
+ def get_parameters(self) -> Dict:
+ return self.parameters
+
+ async def execute(self, **kwargs) -> str:
+ logger.info(f"execute asr function: {kwargs}")
+
+ audio_file = kwargs.get("audio_file")
+ model = kwargs.get("model")
+ prompt = kwargs.get("prompt")
+ response_format = kwargs.get("response_format")
+ if response_format is None:
+ response_format = "text"
+
+ result = await ComputeKernel.get_instance().do_speech_to_text(audio_file, model, prompt, response_format)
+ if result is not None:
+ return f"exec speech_to_text Ok. {response_format} is\n```\n{result.result_str}\n```"
+ else:
+ return "exec speech_to_text failed"
+
+ def is_local(self) -> bool:
+ return True
+
+ def is_in_zone(self) -> bool:
+ return True
+
+ def is_ready_only(self) -> bool:
+ return False
diff --git a/src/aios/ai_functions/code_interpreter.py b/src/aios/ai_functions/code_interpreter.py
new file mode 100644
index 0000000..a9c223a
--- /dev/null
+++ b/src/aios/ai_functions/code_interpreter.py
@@ -0,0 +1,460 @@
+# pylint:disable=E0402
+import logging
+import os
+import pathlib
+import shutil
+import subprocess
+import sys
+import re
+import time
+import ast
+from concurrent.futures import ThreadPoolExecutor
+from hashlib import md5
+from typing import Optional, Union, List, Tuple
+from generic_escape import GenericEscape
+
+from ..storage.storage import AIStorage
+
+try:
+ import docker
+except ImportError:
+ docker = None
+
+CODE_BLOCK_PATTERN = r"```[ \t]*(\w+)?[ \t]*\r?\n(.*?)\r?\n[ \t]*```"
+UNKNOWN = "unknown"
+TIMEOUT_MSG = "Timeout"
+DEFAULT_TIMEOUT = 600
+WIN32 = sys.platform == "win32"
+PATH_SEPARATOR = WIN32 and "\\" or "/"
+
+logger = logging.getLogger(__name__)
+
+
+BUILT_IN_MODULES = set(
+ [
+ "sys",
+ "os",
+ "math",
+ "random",
+ "datetime",
+ "json",
+ "re",
+ "subprocess",
+ "time",
+ "threading",
+ "logging",
+ "collections",
+ "itertools",
+ "functools",
+ "operator",
+ "pathlib",
+ "shutil",
+ "tempfile",
+ "pickle",
+ "io",
+ "argparse",
+ "typing",
+ "unittest",
+ "contextlib",
+ "abc",
+ "heapq",
+ "bisect",
+ "copy",
+ "decimal",
+ "fractions",
+ "hashlib",
+ "secrets",
+ "statistics",
+ "difflib",
+ "doctest",
+ "enum",
+ "inspect",
+ "traceback",
+ "weakref",
+ "gc",
+ "mmap",
+ "msvcrt",
+ "winreg",
+ "array",
+ "audioop",
+ "binascii",
+ "cProfile",
+ "concurrent.futures",
+ "configparser",
+ "csv",
+ "ctypes",
+ "dateutil",
+ "dis",
+ "fnmatch",
+ "getopt",
+ "glob",
+ "gzip",
+ "pdb",
+ "pprint",
+ "profile",
+ "pstats",
+ "queue",
+ "socket",
+ "sqlite3",
+ "ssl",
+ "struct",
+ "tarfile",
+ "telnetlib",
+ "timeit",
+ "tokenize",
+ "uuid",
+ "xml",
+ "zipfile",
+ "zlib",
+ ]
+)
+
+
+def get_imports(code: str) -> List[str]:
+ root = ast.parse(code)
+
+ imports = []
+ for node in ast.iter_child_nodes(root):
+ if isinstance(node, ast.Import):
+ module_names = [alias.name for alias in node.names]
+ elif isinstance(node, ast.ImportFrom):
+ module_names = [node.module]
+ else:
+ continue
+
+ for name in module_names:
+ # Exclude built-in modules
+ if name not in BUILT_IN_MODULES:
+ imports.append(name)
+
+ return imports
+
+
+def write_requirements(code: str, requirements_filepath: str):
+ imports = get_imports(code)
+
+ with open(requirements_filepath, "w") as file:
+ for module in imports:
+ file.write(module + "\n")
+
+
+def _cmd(lang):
+ if lang.startswith("python") or lang in ["bash", "sh", "powershell"]:
+ return lang
+ if lang in ["shell"]:
+ return "sh"
+ if lang in ["ps1"]:
+ return "powershell"
+ raise NotImplementedError(f"{lang} not recognized in code execution")
+
+
+def create_runner(code: str, timeout: int = 30) -> str:
+ """
+ Create a Python script that runs the code and prints the output
+ """
+ code = GenericEscape().escape(code)
+ # Create a runner script
+ runner = f"""
+import os
+import subprocess
+
+my_env = os.environ.copy()
+my_env["PYTHONIOENCODING"] = "utf-8"
+
+process = subprocess.Popen(
+ f"python -i -q -u".split(),
+ stdin=subprocess.PIPE,
+ stdout=subprocess.PIPE,
+ stderr=subprocess.PIPE,
+ text=True,
+ bufsize=0,
+ universal_newlines=True,
+ env=my_env
+)
+
+process.stdin.write("{code}" + "\\n")
+process.stdin.write("exit()\\n")
+process.stdin.flush()
+
+try:
+ process.wait({timeout})
+except Exception as e:
+ process.terminate()
+
+for line in iter(process.stdout.readline, ""):
+ print(line)
+
+for line in iter(process.stderr.readline, ""):
+ if line.startswith(">>>"):
+ continue
+ print(line)
+"""
+ return runner
+
+
+def _run_cmd(cmd: [str], work_dir: str, timeout: int) -> str:
+ if WIN32:
+ logger.warning("SIGALRM is not supported on Windows. No timeout will be enforced.")
+ result = subprocess.run(
+ cmd,
+ cwd=work_dir,
+ capture_output=True,
+ text=True,
+ )
+ else:
+ with ThreadPoolExecutor(max_workers=1) as executor:
+ future = executor.submit(
+ subprocess.run,
+ cmd,
+ cwd=work_dir,
+ capture_output=True,
+ text=True,
+ )
+ result = future.result(timeout=timeout)
+ return result
+
+
+def execute_code(
+ code: Optional[str] = None,
+ timeout: Optional[int] = None,
+ filename: Optional[str] = None,
+ work_dir: Optional[str] = None,
+ use_docker: Optional[Union[List[str], str, bool]] = None,
+ lang: Optional[str] = "python",
+) -> Tuple[int, str]:
+ """Execute code in a docker container.
+ This function is not tested on MacOS.
+
+ Args:
+ code (Optional, str): The code to execute.
+ If None, the code from the file specified by filename will be executed.
+ Either code or filename must be provided.
+ timeout (Optional, int): The maximum execution time in seconds.
+ If None, a default timeout will be used. The default timeout is 600 seconds. On Windows, the timeout is not enforced when use_docker=False.
+ filename (Optional, str): The file name to save the code or where the code is stored when `code` is None.
+ If None, a file with a randomly generated name will be created.
+ The randomly generated file will be deleted after execution.
+ The file name must be a relative path. Relative paths are relative to the working directory.
+ work_dir (Optional, str): The working directory for the code execution.
+ If None, a default working directory will be used.
+ The default working directory is the "extensions" directory under
+ "path_to_autogen".
+ use_docker (Optional, list, str or bool): The docker image to use for code execution.
+ If a list or a str of image name(s) is provided, the code will be executed in a docker container
+ with the first image successfully pulled.
+ If None, False or empty, the code will be executed in the current environment.
+ Default is None, which will be converted into an empty list when docker package is available.
+ Expected behaviour:
+ - If `use_docker` is explicitly set to True and the docker package is available, the code will run in a Docker container.
+ - If `use_docker` is explicitly set to True but the Docker package is missing, an error will be raised.
+ - If `use_docker` is not set (i.e., left default to None) and the Docker package is not available, a warning will be displayed, but the code will run natively.
+ If the code is executed in the current environment,
+ the code must be trusted.
+ lang (Optional, str): The language of the code. Default is "python".
+
+ Returns:
+ int: 0 if the code executes successfully.
+ str: The error message if the code fails to execute; the stdout otherwise.
+ """
+ if all((code is None, filename is None)):
+ error_msg = f"Either {code=} or {filename=} must be provided."
+ logger.error(error_msg)
+ raise AssertionError(error_msg)
+
+ # Warn if use_docker was unspecified (or None), and cannot be provided (the default).
+ # In this case the current behavior is to fall back to run natively, but this behavior
+ # is subject to change.
+ if use_docker is None:
+ if docker is None:
+ use_docker = False
+ logger.warning(
+ "execute_code was called without specifying a value for use_docker. Since the python docker package is not available, code will be run natively. Note: this fallback behavior is subject to change"
+ )
+ else:
+ # Default to true
+ use_docker = True
+
+ timeout = timeout or DEFAULT_TIMEOUT
+ original_filename = filename
+ if WIN32 and lang in ["sh", "shell"] and (not use_docker):
+ lang = "ps1"
+ if filename is None:
+ code_hash = md5(code.encode()).hexdigest()
+ # create a file with a automatically generated name
+ filename = f"tmp_code_{code_hash}.{'py' if lang.startswith('python') else lang}"
+ if work_dir is None:
+ WORKING_DIR = os.path.join(AIStorage.get_instance().get_myai_dir(), "tmp_code")
+ pathlib.Path(WORKING_DIR).mkdir(exist_ok=True)
+ work_dir = os.path.join(WORKING_DIR, code_hash)
+ pathlib.Path(work_dir).mkdir(exist_ok=True)
+ filepath = os.path.join(work_dir, filename)
+ file_dir = os.path.dirname(filepath)
+ os.makedirs(file_dir, exist_ok=True)
+ if code is not None:
+ write_requirements(code, os.path.join(file_dir, "requirements.txt"))
+ code = create_runner(code, 30)
+ with open(filepath, "w", encoding="utf-8") as fout:
+ fout.write(code)
+
+
+ # check if already running in a docker container
+ in_docker_container = os.path.exists("/.dockerenv")
+ if not use_docker or in_docker_container:
+ try:
+ env_cmd = ["python", "-m", "venv", os.path.join(file_dir, "venv")]
+ _run_cmd(env_cmd, file_dir, timeout)
+ if WIN32:
+ venv_path = os.path.join(file_dir, "venv", "Scripts")
+ else:
+ venv_path = os.path.join(file_dir, "venv", "bin")
+ pip_cmd = [os.path.join(venv_path, "python"), "-m", "pip", "install", "-r", "requirements.txt"]
+ _run_cmd(pip_cmd, file_dir, timeout)
+ # already running in a docker container
+ cmd = [
+ os.path.join(venv_path, "python"),
+ f".\\{filename}" if WIN32 else filename,
+ ]
+ result = _run_cmd(cmd, file_dir, timeout)
+ except TimeoutError:
+ if original_filename is None:
+ shutil.rmtree(os.path.join(file_dir, "venv"))
+ os.remove(filepath)
+ os.remove(os.path.join(file_dir, "requirements.txt"))
+ try:
+ os.removedirs(file_dir)
+ except Exception:
+ pass
+ return 1, TIMEOUT_MSG
+ if original_filename is None:
+ shutil.rmtree(os.path.join(file_dir, "venv"))
+ os.remove(filepath)
+ os.remove(os.path.join(file_dir, "requirements.txt"))
+ try:
+ os.removedirs(file_dir)
+ except Exception:
+ pass
+ if result.returncode:
+ logs = result.stderr
+ if original_filename is None:
+ abs_path = str(pathlib.Path(filepath).absolute())
+ logs = logs.replace(str(abs_path), "").replace(filename, "")
+ else:
+ abs_path = str(pathlib.Path(work_dir).absolute()) + PATH_SEPARATOR
+ logs = logs.replace(str(abs_path), "")
+ else:
+ logs = result.stdout
+ return result.returncode, logs
+
+ # create a docker client
+ client = docker.from_env()
+ image_list = (
+ ["python:3-alpine", "python:3", "python:3-windowsservercore"]
+ if use_docker is True
+ else [use_docker]
+ if isinstance(use_docker, str)
+ else use_docker
+ )
+ for image in image_list:
+ # check if the image exists
+ try:
+ client.images.get(image)
+ break
+ except docker.errors.ImageNotFound:
+ # pull the image
+ logger.info("Pulling image", image)
+ try:
+ client.images.pull(image, stream=True, decode=True)
+ break
+ except docker.errors.DockerException as e:
+ logger.error("Failed to pull image", image)
+ logger.exception(e)
+ # get a randomized str based on current time to wrap the exit code
+ exit_code_str = f"exitcode{time.time()}"
+ start_str = f'start{time.time()}'
+ abs_path = pathlib.Path(work_dir).absolute()
+ cmd = [
+ "sh",
+ "-c",
+ f"pip install --quiet -r requirements.txt; echo -n {start_str}; {_cmd(lang)} {filename}; exit_code=$?; echo -n {exit_code_str}; echo -n $exit_code; echo {exit_code_str};",
+ ]
+ # create a docker container
+ container = client.containers.run(
+ image,
+ command=cmd,
+ working_dir="/workspace",
+ detach=True,
+ # get absolute path to the working directory
+ volumes={abs_path: {"bind": "/workspace", "mode": "rw"}},
+ )
+ start_time = time.time()
+ while container.status != "exited" and time.time() - start_time < timeout:
+ # Reload the container object
+ container.reload()
+ if container.status != "exited":
+ container.stop()
+ container.remove()
+ if original_filename is None:
+ os.remove(filepath)
+ return 1, TIMEOUT_MSG, image
+ # get the container logs
+ logs: str = container.logs().decode("utf-8").rstrip()
+ start_pos = logs.find(start_str)
+ if start_pos != -1:
+ logs = logs[start_pos + len(start_str):]
+ # # commit the image
+ # tag = filename.replace("/", "")
+ # container.commit(repository="python", tag=tag)
+ # remove the container
+ container.remove()
+ # check if the code executed successfully
+ exit_code = container.attrs["State"]["ExitCode"]
+ if exit_code == 0:
+ # extract the exit code from the logs
+ pattern = re.compile(f"{exit_code_str}(\\d+){exit_code_str}")
+ match = pattern.search(logs)
+ exit_code = 1 if match is None else int(match.group(1))
+ # remove the exit code from the logs
+ logs = logs if match is None else pattern.sub("", logs)
+
+ if original_filename is None:
+ os.remove(filepath)
+ os.remove(os.path.join(file_dir, "requirements.txt"))
+ os.removedirs(file_dir)
+ if exit_code:
+ logs = logs.replace(f"/workspace/{filename if original_filename is None else ''}", "")
+ # return the exit code, logs and image
+ return exit_code, logs
+
+class CodeInterpreterFunction(AIFunction):
+ def __init__(self):
+ self.func_id = "system.code_interpreter"
+ self.description = "execute python code"
+ self.parameters = ParameterDefine.create_parameters({
+ "code": {"type": "string", "description": "python code"}
+ })
+
+ def get_name(self) -> str:
+ return self.func_id
+
+ def get_description(self) -> str:
+ return self.description
+
+ def get_parameters(self) -> Dict:
+ return self.parameters
+
+ async def execute(self, **kwargs) -> str:
+ code = kwargs.get("code")
+ ret_code, result = execute_code(code=code)
+ if ret_code == 0:
+ return result.strip()
+ else:
+ return result.strip()
+
+ def is_local(self) -> bool:
+ return True
+
+ def is_in_zone(self) -> bool:
+ return True
+
+ def is_ready_only(self) -> bool:
+ return False
diff --git a/src/aios/ai_functions/duckduckgo_text_search_function.py b/src/aios/ai_functions/duckduckgo_text_search_function.py
new file mode 100644
index 0000000..019d98f
--- /dev/null
+++ b/src/aios/ai_functions/duckduckgo_text_search_function.py
@@ -0,0 +1,56 @@
+# pylint:disable=E0402
+import json
+from typing import Dict
+
+from ..proto.ai_function import *
+from ..agent.llm_context import GlobaToolsLibrary
+from duckduckgo_search import AsyncDDGS
+
+
+class DuckDuckGoTextSearchFunction(AIFunction):
+ def __init__(self):
+ self.name = "web.search.duckduckgo"
+ self.description = "Search web by text (powered by DuckDuckGo)"
+ self.region = "wt-wt"
+ self.safesearch = "moderate"
+ self.time = "y"
+ self.max_results = 5
+ self.parameters = ParameterDefine.create_parameters({
+ "query": {"type": "string", "description": "The query to search for."}
+ })
+
+ def register_function(self):
+ GlobaToolsLibrary.get_instance().register_tool_function(self)
+
+ def get_id(self) -> str:
+ return self.name
+
+ def get_description(self) -> str:
+ return self.description
+
+ def get_parameters(self) -> Dict:
+ return self.parameters
+
+ async def execute(self, **kwargs) -> str:
+ query = kwargs.get("query")
+
+ async with AsyncDDGS() as ddgs:
+ results = [r async for r in ddgs.text(
+ query,
+ region=self.region,
+ safesearch=self.safesearch,
+ timelimit=self.time,
+ backend="api",
+ max_results=self.max_results
+ )]
+
+ return json.dumps(results,,ensure_ascii=False)
+
+ def is_local(self) -> bool:
+ return True
+
+ def is_in_zone(self) -> bool:
+ return True
+
+ def is_ready_only(self) -> bool:
+ return False
diff --git a/src/aios/ai_functions/image_2_text_function.py b/src/aios/ai_functions/image_2_text_function.py
new file mode 100644
index 0000000..f0a61b4
--- /dev/null
+++ b/src/aios/ai_functions/image_2_text_function.py
@@ -0,0 +1,54 @@
+# pylint:disable=E0402
+import logging
+from typing import Dict
+
+from ..frame.compute_kernel import ComputeKernel
+from ..proto.ai_function import *
+from ..agent.llm_context import GlobaToolsLibrary
+
+logger = logging.getLogger(__name__)
+
+class Image2TextFunction(AIFunction):
+
+ def __init__(self):
+ self.func_id = "aigc.image_2_text"
+ self.description = "According to the input image file address, return the description of the image content"
+ self.parameters = ParameterDefine.create_parameters({
+ "image_path": {"type": "string", "description": "image file path"}
+ })
+ logger.info(f"init Image2TextFunction")
+
+ def register_function(self):
+ GlobaToolsLibrary.get_instance().register_tool_function(self)
+
+ def get_id(self) -> str:
+ return self.func_id
+
+ def get_description(self) -> str:
+ return self.description
+
+ def get_parameters(self) -> Dict:
+ return self.parameters
+
+ async def execute(self, **kwargs) -> str:
+ logger.info(f"execute image_2_text function: {kwargs}")
+ image_path = kwargs.get("image_path")
+ data = await ComputeKernel.get_instance().do_image_2_text(image_path, '')
+ try:
+ result = data['message']['choices'][0]['message']['content']
+ except (KeyError, TypeError, IndexError):
+ logger.error(f"image_2_text error: {data}")
+ result = ""
+ return result
+
+
+ def is_local(self) -> bool:
+ return False
+
+ def is_in_zone(self) -> bool:
+ return True
+
+ def is_ready_only(self) -> bool:
+ return False
+
+
diff --git a/src/aios_kernel/text_to_speech_function.py b/src/aios/ai_functions/script_to_speech_function.py
similarity index 60%
rename from src/aios_kernel/text_to_speech_function.py
rename to src/aios/ai_functions/script_to_speech_function.py
index 90dfd49..31a11d7 100644
--- a/src/aios_kernel/text_to_speech_function.py
+++ b/src/aios/ai_functions/script_to_speech_function.py
@@ -1,57 +1,66 @@
+# pylint:disable=E0402
+
import io
import logging
import os
import random
+from pathlib import Path
from typing import Dict
-from aios_kernel import ComputeKernel
-from aios_kernel.ai_function import AIFunction
+from ..proto.ai_function import *
+from ..frame.compute_kernel import ComputeKernel
+from ..storage.storage import AIStorage
+
from pydub import AudioSegment
logger = logging.getLogger(__name__)
-class TextToSpeechFunction(AIFunction):
- def __init__(self):
- self.func_id = "text_to_speech"
- self.description = "根据输入的剧本生成音频文件，成功时会返回音频文件路径"
- def get_name(self) -> str:
+class ScriptToSpeechFunction(AIFunction):
+ def __init__(self):
+ self.func_id = "aigc.script_to_speech"
+ self.description = "Generate audio files according to the input script, and the audio file path will be returned when successful"
+ self.speech_path = os.path.join(AIStorage.get_instance().get_myai_dir(), "tts")
+ self.parameters = ParameterDefine.create_parameters({
+ "language": {"type": "string", "description": "Actual language", "enum": ["zh", "en"]},
+ "model": {"type": "string", "description": "Studio", "enum": ["tts-1", "tts-1-hd"]},
+ "roles": {"type": "array", "items": {
+ "type": "object",
+ "properties": {
+ "name": {"type": "string", "description": "Character name"},
+ "gender": {"type": "string", "description": "Gender", "enum": ["man", "female"]},
+ "age": {"type": "string", "description": "age", "enum": ["child", "adult"]},
+ }}},
+ "lines": {"type": "array", "items": {
+ "type": "object",
+ "properties": {
+ "name": {"type": "string", "description": "Character name"},
+ "tone": {"type": "string", "description": "Sovereign emotions",
+ "enum": ["happy", "sad", "angry", "fear", "disgust", "surprise", "neutral"]},
+ "text": {"type": "string", "description": "Line"},
+ }
+ }}
+ })
+
+ Path(self.speech_path).mkdir(exist_ok=True)
+
+ def get_id(self) -> str:
return self.func_id
def get_description(self) -> str:
return self.description
- def get_parameters(self) -> Dict:
- return {
- "type": "object",
- "properties": {
- "language": {"type": "string", "description": "演播语言", "enum": ["zh", "en"]},
- "roles": {"type": "array", "items": {
- "type": "object",
- "properties": {
- "name": {"type": "string", "description": "角色名字"},
- "gender": {"type": "string", "description": "角色性别", "enum": ["man", "female"]},
- "age": {"type": "string", "description": "年龄", "enum": ["child", "adult"]},
- }}},
- "lines": {"type": "array", "items": {
- "type": "object",
- "properties": {
- "name": {"type": "string", "description": "角色名字"},
- "tone": {"type": "string", "description": "演播情感",
- "enum": ["happy", "sad", "angry", "fear", "disgust", "surprise", "neutral"]},
- "text": {"type": "string", "description": "台词"},
- }
- }}
- }
- }
+ def get_parameters(self):
+ return self.parameters
async def execute(self, **kwargs) -> str:
- logger.info(f"execute text_to_speech function: {kwargs}")
+ logger.info(f"execute aigc.script_to_speech function: {kwargs}")
language = kwargs.get("language")
if language is None:
language = "zh"
+ model = kwargs.get("model")
roles = kwargs.get("roles")
lines = kwargs.get("lines")
@@ -71,23 +80,23 @@ class TextToSpeechFunction(AIFunction):
i = 0
while i < 3:
try:
- data = await ComputeKernel.get_instance().do_text_to_speech(text, language, gender, age, name, tone)
+ data = await ComputeKernel.get_instance().do_text_to_speech(text, language, gender, age, name, tone, model_name=model)
if audio is None:
audio = AudioSegment.from_mp3(io.BytesIO(data))
else:
audio = audio + AudioSegment.from_mp3(io.BytesIO(data))
break
except Exception as e:
- logger.error(f"do_text_to_speech failed: {e}")
+ logger.error(f"script_to_speech failed: {e}")
i += 1
continue
if audio is not None:
- path = os.path.join(os.path.realpath(os.curdir), "{}.mp3".format(''.join(random.sample('zyxwvutsrqponmlkjihgfedcba', 10))))
+ path = os.path.join(self.speech_path, "{}.mp3".format(''.join(random.sample('zyxwvutsrqponmlkjihgfedcba', 10))))
audio.export(path, format="mp3")
- return "exec text_to_speech OK，speech file store at {}".format(path)
+ return "exec script_to_speech OK，speech file store at ```{}```".format(path)
else:
- return "exec text_to_speech failed"
+ return "exec script_to_speech failed"
def is_local(self) -> bool:
return True
diff --git a/src/aios/ai_functions/sql_database_function.py b/src/aios/ai_functions/sql_database_function.py
new file mode 100644
index 0000000..6fd92c3
--- /dev/null
+++ b/src/aios/ai_functions/sql_database_function.py
@@ -0,0 +1,113 @@
+# pylint:disable=E0402
+from datetime import timedelta, datetime
+from typing import Dict
+
+from cachetools import TLRUCache, cached
+
+from ..proto.ai_function import *
+from ..environment.sql_database import SQLDatabase, get_from_env
+
+
+def _my_ttu(_key, _value, now):
+ return now + timedelta(seconds=600)
+
+
+database_cache = TLRUCache(ttu=_my_ttu, maxsize=10000, timer=datetime.now)
+
+
+@cached(cache=database_cache)
+def get_database(uri: str) -> SQLDatabase:
+ return SQLDatabase.from_uri(uri)
+
+
+class GetTableInfosFunction(AIFunction):
+ def __init__(self):
+ super().__init__()
+ self.name = "get_table_infos"
+ self.description = "Get table informations in the database"
+
+ def get_id(self) -> str:
+ return self.name
+
+ def get_description(self) -> str:
+ return self.description
+
+ def get_parameters(self) -> Dict:
+ return {
+ "type": "object",
+ "properties": {
+ "database_url": {"type": "string", "description": "Database URL,Can be set to None"},
+ }
+ }
+
+ async def execute(self, **kwargs) -> str:
+ database_url: str = kwargs.get("database_url")
+ if (database_url is None
+ or database_url.strip() == ""
+ or database_url.strip().lower() == "none"
+ or database_url.strip().lower() == "null"):
+ database_url = get_from_env(key="database url", env_key="DATABASE_URL")
+ if database_url is None:
+ return "error: database_url is None"
+ database = get_database(database_url)
+ tables = database.get_usable_table_names()
+ table_infos = database.get_table_info(tables)
+ return table_infos
+
+ def is_local(self) -> bool:
+ return True
+
+ def is_in_zone(self) -> bool:
+ return True
+
+ def is_ready_only(self) -> bool:
+ return False
+
+
+class ExecuteSqlFunction(AIFunction):
+ def __init__(self):
+ super().__init__()
+ self.name = "execute_sql"
+ self.description = """
+ Input to this function is a detailed and correct SQL query, output is a result from the database.
+ If the query is not correct, an error message will be returned.
+ If an error is returned, rewrite the query, check the query, and try again.
+ """
+
+ def get_id(self) -> str:
+ return self.name
+
+ def get_description(self) -> str:
+ return self.description
+
+ def get_parameters(self) -> Dict:
+ return {
+ "type": "object",
+ "properties": {
+ "database_url": {"type": "string", "description": "Database URL,Can be set to None"},
+ "sql": {"type": "string", "description": "SQL to execute"}
+ }
+ }
+
+ async def execute(self, **kwargs) -> str:
+ database_url = kwargs.get("database_url")
+ if (database_url is None
+ or database_url.strip() == ""
+ or database_url.strip().lower() == "none"
+ or database_url.strip().lower() == "null"):
+ database_url = get_from_env(key="database url", env_key="DATABASE_URL")
+ if database_url is None:
+ return "error: database_url is None"
+ sql = kwargs.get("sql")
+
+ database = get_database(database_url)
+ return database.run_no_throw(sql)
+
+ def is_local(self) -> bool:
+ return True
+
+ def is_in_zone(self) -> bool:
+ return True
+
+ def is_ready_only(self) -> bool:
+ return False
diff --git a/src/aios/ai_functions/text_to_speech_function.py b/src/aios/ai_functions/text_to_speech_function.py
new file mode 100644
index 0000000..3a555ac
--- /dev/null
+++ b/src/aios/ai_functions/text_to_speech_function.py
@@ -0,0 +1,77 @@
+import io
+import logging
+import os
+import random
+from pathlib import Path
+from typing import Dict
+
+from ..proto.ai_function import *
+from ..frame.compute_kernel import ComputeKernel
+from ..storage.storage import AIStorage
+
+
+from pydub import AudioSegment
+
+logger = logging.getLogger(__name__)
+
+
+class TextToSpeechFunction(AIFunction):
+ def __init__(self):
+ self.func_id = "aigc.text_to_speech"
+ self.description = "To generate audio files according to the input text, the audio file path will be returned when successful"
+ self.speech_path = os.path.join(AIStorage.get_instance().get_myai_dir(), "tts")
+ self.parameters = ParameterDefine.create_parameters({
+ "language": {"type": "string", "description": "Actual language", "enum": ["zh", "en"]},
+ "model": {"type": "string", "description": "Studio", "enum": ["tts-1", "tts-1-hd"]},
+ "text": {"type": "string", "description": "text"}
+ })
+ Path(self.speech_path).mkdir(exist_ok=True)
+
+ def get_id(self) -> str:
+ return self.func_id
+
+ def get_description(self) -> str:
+ return self.description
+
+ def get_parameters(self) -> Dict:
+ return self.parameters
+
+ async def execute(self, **kwargs) -> str:
+ logger.info(f"execute aigc.text_to_speech function: {kwargs}")
+
+ language = kwargs.get("language")
+ if language is None:
+ language = "en"
+ model = kwargs.get("model")
+ text = kwargs.get("text")
+
+ i = 0
+ while i < 3:
+ try:
+ data = await ComputeKernel.get_instance().do_text_to_speech(text, language, None, None, None, None,
+ model_name=model)
+ if data is not None:
+ audio = AudioSegment.from_mp3(io.BytesIO(data))
+ break
+ except Exception as e:
+ logger.error(f"do_text_to_speech failed: {e}")
+ i += 1
+ continue
+
+ if audio is not None:
+ path = os.path.join(self.speech_path, "{}.mp3".format(''.join(random.sample('zyxwvutsrqponmlkjihgfedcba', 10))))
+ audio.export(path, format="mp3")
+ return "exec text_to_speech OK，speech file store at ```{}```".format(path)
+ else:
+ return "exec text_to_speech failed"
+
+ def is_local(self) -> bool:
+ return True
+
+ def is_in_zone(self) -> bool:
+ return True
+
+ def is_ready_only(self) -> bool:
+ return False
+
+
diff --git a/src/aios/environment/environment.py b/src/aios/environment/environment.py
new file mode 100644
index 0000000..ac79eaa
--- /dev/null
+++ b/src/aios/environment/environment.py
@@ -0,0 +1,123 @@
+# basic environment class
+# we have some built-in environment: Calender(include timer),Home(connect to IoT device in your home), ,KnwoledgeBase,FileSystem,
+# pylint:disable=E0402
+from abc import ABC, abstractmethod
+from typing import Any, Callable, Optional,Dict,Awaitable,List
+import logging
+from ..proto.ai_function import *
+
+
+logger = logging.getLogger(__name__)
+
+
+class BaseEnvironment:
+ @classmethod
+ def get_env_by_id(cls,env_id:str)->'BaseEnvironment':
+ if cls.all_env is None:
+ cls.all_env = {}
+ return cls.all_env.get(env_id)
+
+ @classmethod
+ def register_env(cls,env_id:str,env:'BaseEnvironment')->None:
+ if cls.all_env is None:
+ cls.all_env = {}
+ cls.all_env[env_id] = env
+
+
+ def __init__(self, workspace: str) -> None:
+ pass
+
+ # @abstractmethod
+ # #TODO: how to use env? different env has different prompt
+ # def get_env_prompt(self) -> str:
+ # pass
+
+ @abstractmethod
+ def get_ai_function(self,func_name:str) -> AIFunction:
+ pass
+
+ @abstractmethod
+ def get_all_ai_functions(self) -> List[AIFunction]:
+ pass
+
+
+ @abstractmethod
+ def get_ai_operation(self,op_name:str) -> AIAction:
+ pass
+
+ @abstractmethod
+ def get_all_ai_operations(self) -> List[AIAction]:
+ pass
+
+ def __getitem__(self, key):
+ return self.get_value(key)
+
+ @abstractmethod
+ def get_value(self,key:str) -> Optional[str]:
+ pass
+
+ # _all_env = {}
+ # @classmethod
+ # def get_env_by_id(cls,env_id:str):
+ # return cls._all_env.get(env_id)
+
+ # @classmethod
+ # def set_env_by_id(cls,id,env):
+ # assert id == env.get_id()
+ # cls._all_env[env.get_id()] = env
+
+class SimpleEnvironment(BaseEnvironment):
+ def __init__(self, workspace: str) -> None:
+ super().__init__(workspace)
+ self.functions: Dict[str,AIFunction] = {}
+ self.operations: Dict[str,AIAction] = {}
+
+ def add_ai_function(self,func:AIFunction) -> None:
+ self.functions[func.get_id()] = func
+
+ def get_ai_function(self,func_name:str) -> AIFunction:
+ func = self.functions.get(func_name)
+ if func is not None:
+ return func
+ return None
+
+ def get_all_ai_functions(self) -> List[AIFunction]:
+ func_list = []
+ func_list.extend(self.functions.values())
+ return func_list
+
+ def add_ai_operation(self,op:AIAction) -> None:
+ self.operations[op.get_id()] = op
+
+ def get_ai_operation(self,op_name:str) -> AIAction:
+ op = self.operations.get(op_name)
+ if op is not None:
+ return op
+ return None
+
+ def get_all_ai_operations(self) -> List[AIAction]:
+ op_list = []
+ op_list.extend(self.operations.values())
+ return op_list
+
+
+
+class CompositeEnvironment(SimpleEnvironment):
+ def __init__(self, workspace: str) -> None:
+ super().__init__(workspace)
+ self.envs: List[BaseEnvironment] = []
+
+ def add_env(self, env: BaseEnvironment) -> None:
+ self.envs.append(env)
+ functions = env.get_all_ai_functions()
+ for func in functions:
+ self.functions[func.get_id()] = func
+ operations = env.get_all_ai_operations()
+ for op in operations:
+ self.operations[op.get_id()] = op
+
+ def get_value(self,key:str) -> Optional[str]:
+ for env in self.envs:
+ val = env.get_value(key)
+ if val is not None:
+ return val
\ No newline at end of file
diff --git a/src/aios/environment/simple_kb_db.py b/src/aios/environment/simple_kb_db.py
new file mode 100644
index 0000000..1bc7617
--- /dev/null
+++ b/src/aios/environment/simple_kb_db.py
@@ -0,0 +1,228 @@
+# pylint:disable=E0402
+import sqlite3
+import json
+import threading
+import logging
+from datetime import datetime
+
+from typing import Optional, List
+
+logger = logging.getLogger(__name__)
+
+class SimpleKnowledgeDB:
+ def __init__(self,db_path:str):
+ self.db_path = db_path
+ self._get_conn()
+
+ def _get_conn(self):
+ """ get db connection """
+ local = threading.local()
+ if not hasattr(local, 'conn'):
+ local.conn = self._create_connection(self.db_path)
+ return local.conn
+
+
+ def _create_connection(self, db_file):
+ """ create a database connection to a SQLite database """
+ conn = None
+ try:
+ conn = sqlite3.connect(db_file)
+ except Exception as e:
+ logger.error("Error occurred while connecting to database: %s", e)
+ return None
+
+ if conn:
+ self._create_tables(conn)
+
+ return conn
+
+ def _create_tables(self,conn):
+ cursor = conn.cursor()
+ cursor.execute('''
+ CREATE TABLE IF NOT EXISTS documents (
+ doc_path TEXT PRIMARY KEY,
+ length INTEGER,
+ last_modify TEXT,
+ doc_hash TEXT,
+ create_time TEXT
+ )
+ ''')
+ cursor.execute('''
+ CREATE TABLE IF NOT EXISTS knowledge (
+ doc_hash TEXT PRIMARY KEY,
+ title TEXT,
+ summary TEXT,
+ content TEXT,
+ catalogs TEXT,
+ tags TEXT,
+ llm_title TEXT,
+ llm_summary TEXT,
+ create_time TEXT
+ )
+ ''')
+
+ cursor.execute('''
+ CREATE INDEX IF NOT EXISTS idx_documents_doc_hash
+ ON documents (doc_hash)
+ ''')
+
+ cursor.execute('''
+ CREATE INDEX IF NOT EXISTS idx_knowledge_tags
+ ON knowledge (tags)
+ ''')
+
+ conn.commit()
+
+ def add_doc(self, doc_path: str, length: int, last_modify: str, doc_hash: Optional[str] = None):
+ conn = self._get_conn()
+ cursor = conn.cursor()
+ create_time = datetime.now().strftime("%Y-%m-%d %H:%M:%S")
+ cursor.execute('''
+ INSERT INTO documents (doc_path, length, last_modify, doc_hash,create_time)
+ VALUES (?, ?, ?, ?,?)
+ ''', (doc_path, length, last_modify, doc_hash,create_time))
+ conn.commit()
+
+ def is_doc_exist(self, doc_path: str) -> bool:
+ conn = self._get_conn()
+ cursor = conn.cursor()
+ cursor.execute('''
+ SELECT doc_path
+ FROM documents
+ WHERE doc_path = ?
+ ''', (doc_path,))
+ return len(cursor.fetchall()) > 0
+
+ def set_doc_hash(self, doc_path: str, doc_hash: str):
+ conn = self._get_conn()
+ cursor = conn.cursor()
+ cursor.execute('''
+ UPDATE documents
+ SET doc_hash = ?
+ WHERE doc_path = ?
+ ''', (doc_hash, doc_path))
+ conn.commit()
+
+ def get_docs_without_hash(self,limit:int=1024) -> List[str]:
+ conn = self._get_conn()
+ cursor = conn.cursor()
+ cursor.execute('''
+ SELECT doc_path
+ FROM documents
+ WHERE doc_hash IS NULL OR doc_hash = ''
+ ORDER BY create_time DESC
+ LIMIT ?
+ ''',(limit,))
+ return [row[0] for row in cursor.fetchall()]
+
+ #metadata["summary"]
+ #metadata["catelogs"]
+ #metadata["tags"]
+ def add_knowledge(self, doc_hash: str, title: str, metadata: dict,content:str = None,):
+ conn = self._get_conn()
+ cursor = conn.cursor()
+
+ create_time = datetime.now().strftime("%Y-%m-%d %H:%M:%S")
+ summary = metadata.get("summary", "")
+ catalogs = metadata.get("catalogs","")
+ tags = ','.join(metadata.get("tags", []))
+
+ cursor.execute('''
+ INSERT INTO knowledge (doc_hash, title , summary , catalogs , tags,create_time)
+ VALUES (?, ?, ?, ?, ?,?)
+ ''', (doc_hash, title, summary, catalogs, tags,create_time))
+ conn.commit()
+
+ #llm_result["summary"]
+ #llm_result["tags"]
+ #llm_result["catelog"]
+ def set_knowledge_llm_result(self, doc_hash: str, llm_result: dict):
+ conn = self._get_conn()
+ cursor = conn.cursor()
+
+ title = llm_result.get("title", "")
+ summary = llm_result.get("summary", "")
+ catalogs = json.dumps(llm_result.get("catalogs", {}))
+ tags = ','.join(llm_result.get("tags", []))
+
+ cursor.execute('''
+ UPDATE knowledge
+ SET llm_title = ?,llm_summary = ?, catalogs = ?, tags = ?
+ WHERE doc_hash = ?
+ ''', (title,summary, catalogs, tags, doc_hash))
+ conn.commit()
+
+ def get_hash_by_doc_path(self, doc_path: str) -> Optional[str]:
+ conn = self._get_conn()
+ cursor = conn.cursor()
+ cursor.execute('''
+ SELECT doc_hash
+ FROM documents
+ WHERE doc_path = ?
+ ''', (doc_path,))
+ row = cursor.fetchone()
+ if row is None:
+ return None
+ return row[0]
+
+ def get_knowledge(self, doc_hash: str) -> Optional[dict]:
+ conn = self._get_conn()
+ cursor = conn.cursor()
+ cursor.execute('''
+ SELECT title, summary, catalogs, tags, llm_title, llm_summary
+ FROM knowledge
+ WHERE doc_hash = ?
+ ''', (doc_hash,))
+ row = cursor.fetchone()
+ if row is None:
+ return None
+
+ # get doc path
+ cursor.execute('''
+ SELECT doc_path
+ FROM documents
+ WHERE doc_hash = ?
+ ''', (doc_hash,))
+ row2 = cursor.fetchone()
+ if row2 is None:
+ return None
+ doc_path = row2[0]
+
+
+ return {
+ "full_path": doc_path,
+ "title": row[0],
+ "summary": row[1],
+ "catalogs": row[2],
+ "tags": row[3],
+ "llm_title" : row[4],
+ "llm_summary" : row[5],
+ }
+
+ def get_knowledge_without_llm_title(self,limit:int=16) -> List[str]:
+ conn = self._get_conn()
+ cursor = conn.cursor()
+ cursor.execute('''
+ SELECT doc_hash
+ FROM knowledge
+ WHERE llm_title IS NULL OR llm_title = ''
+ ORDER BY create_time DESC
+ LIMIT ?
+ ''',(limit,))
+ return [row[0] for row in cursor.fetchall()]
+
+ def query_docs_by_tag(self, tag: str) -> List[str]:
+ conn = self._get_conn()
+ cursor = conn.cursor()
+ tag_json = json.dumps(tag,ensure_ascii=False) # 将标签转换为 JSON 字符串
+ cursor.execute('''
+ SELECT documents.doc_path
+ FROM documents
+ JOIN knowledge ON documents.doc_hash = knowledge.doc_hash
+ WHERE json_extract(knowledge.tags, '$') LIKE ?
+ ''', (tag))
+ return [row[0] for row in cursor.fetchall()]
+
+ def query(self,sql:str):
+ pass
+ #cursor = self.conn.cursor()
diff --git a/src/aios/environment/sql_database.py b/src/aios/environment/sql_database.py
new file mode 100644
index 0000000..7c13255
--- /dev/null
+++ b/src/aios/environment/sql_database.py
@@ -0,0 +1,495 @@
+"""
+Taken from: langchain
+SQLAlchemy wrapper around a database.
+"""
+# pylint:disable=E0402
+
+from __future__ import annotations
+import os
+
+import warnings
+from typing import Any, Dict, Iterable, List, Literal, Optional, Sequence, Union
+
+import sqlalchemy
+from sqlalchemy import MetaData, Table, create_engine, inspect, select, text
+from sqlalchemy.engine import Engine
+from sqlalchemy.exc import ProgrammingError, SQLAlchemyError
+from sqlalchemy.schema import CreateTable
+from sqlalchemy.types import NullType
+
+
+def get_from_env(key: str, env_key: str, default: Optional[str] = None) -> str:
+ """Get a value from a dictionary or an environment variable."""
+ if env_key in os.environ and os.environ[env_key]:
+ return os.environ[env_key]
+ elif default is not None:
+ return default
+ else:
+ raise ValueError(
+ f"Did not find {key}, please add an environment variable"
+ f" `{env_key}` which contains it, or pass"
+ f" `{key}` as a named parameter."
+ )
+
+
+def _format_index(index: sqlalchemy.engine.interfaces.ReflectedIndex) -> str:
+ return (
+ f'Name: {index["name"]}, Unique: {index["unique"]},'
+ f' Columns: {str(index["column_names"])}'
+ )
+
+
+def truncate_word(content: Any, *, length: int, suffix: str = "...") -> str:
+ """
+ Truncate a string to a certain number of words, based on the max string
+ length.
+ """
+
+ if not isinstance(content, str) or length <= 0:
+ return content
+
+ if len(content) <= length:
+ return content
+
+ return content[: length - len(suffix)].rsplit(" ", 1)[0] + suffix
+
+
+class SQLDatabase:
+ """SQLAlchemy wrapper around a database."""
+
+ def __init__(
+ self,
+ engine: Engine,
+ schema: Optional[str] = None,
+ metadata: Optional[MetaData] = None,
+ ignore_tables: Optional[List[str]] = None,
+ include_tables: Optional[List[str]] = None,
+ sample_rows_in_table_info: int = 3,
+ indexes_in_table_info: bool = False,
+ custom_table_info: Optional[dict] = None,
+ view_support: bool = False,
+ max_string_length: int = 300,
+ ):
+ """Create engine from database URI."""
+ self._engine = engine
+ self._schema = schema
+ if include_tables and ignore_tables:
+ raise ValueError("Cannot specify both include_tables and ignore_tables")
+
+ self._inspector = inspect(self._engine)
+
+ # including view support by adding the views as well as tables to the all
+ # tables list if view_support is True
+ self._all_tables = set(
+ self._inspector.get_table_names(schema=schema)
+ + (self._inspector.get_view_names(schema=schema) if view_support else [])
+ )
+
+ self._include_tables = set(include_tables) if include_tables else set()
+ if self._include_tables:
+ missing_tables = self._include_tables - self._all_tables
+ if missing_tables:
+ raise ValueError(
+ f"include_tables {missing_tables} not found in database"
+ )
+ self._ignore_tables = set(ignore_tables) if ignore_tables else set()
+ if self._ignore_tables:
+ missing_tables = self._ignore_tables - self._all_tables
+ if missing_tables:
+ raise ValueError(
+ f"ignore_tables {missing_tables} not found in database"
+ )
+ usable_tables = self.get_usable_table_names()
+ self._usable_tables = set(usable_tables) if usable_tables else self._all_tables
+
+ if not isinstance(sample_rows_in_table_info, int):
+ raise TypeError("sample_rows_in_table_info must be an integer")
+
+ self._sample_rows_in_table_info = sample_rows_in_table_info
+ self._indexes_in_table_info = indexes_in_table_info
+
+ self._custom_table_info = custom_table_info
+ if self._custom_table_info:
+ if not isinstance(self._custom_table_info, dict):
+ raise TypeError(
+ "table_info must be a dictionary with table names as keys and the "
+ "desired table info as values"
+ )
+ # only keep the tables that are also present in the database
+ intersection = set(self._custom_table_info).intersection(self._all_tables)
+ self._custom_table_info = dict(
+ (table, self._custom_table_info[table])
+ for table in self._custom_table_info
+ if table in intersection
+ )
+
+ self._max_string_length = max_string_length
+
+ self._metadata = metadata or MetaData()
+ # including view support if view_support = true
+ self._metadata.reflect(
+ views=view_support,
+ bind=self._engine,
+ only=list(self._usable_tables),
+ schema=self._schema,
+ )
+
+ @classmethod
+ def from_uri(
+ cls, database_uri: str, engine_args: Optional[dict] = None, **kwargs: Any
+ ) -> SQLDatabase:
+ """Construct a SQLAlchemy engine from URI."""
+ _engine_args = engine_args or {}
+ return cls(create_engine(database_uri, **_engine_args), **kwargs)
+
+ @classmethod
+ def from_databricks(
+ cls,
+ catalog: str,
+ schema: str,
+ host: Optional[str] = None,
+ api_token: Optional[str] = None,
+ warehouse_id: Optional[str] = None,
+ cluster_id: Optional[str] = None,
+ engine_args: Optional[dict] = None,
+ **kwargs: Any,
+ ) -> SQLDatabase:
+ """
+ Class method to create an SQLDatabase instance from a Databricks connection.
+ This method requires the 'databricks-sql-connector' package. If not installed,
+ it can be added using `pip install databricks-sql-connector`.
+
+ Args:
+ catalog (str): The catalog name in the Databricks database.
+ schema (str): The schema name in the catalog.
+ host (Optional[str]): The Databricks workspace hostname, excluding
+ 'https://' part. If not provided, it attempts to fetch from the
+ environment variable 'DATABRICKS_HOST'. If still unavailable and if
+ running in a Databricks notebook, it defaults to the current workspace
+ hostname. Defaults to None.
+ api_token (Optional[str]): The Databricks personal access token for
+ accessing the Databricks SQL warehouse or the cluster. If not provided,
+ it attempts to fetch from 'DATABRICKS_TOKEN'. If still unavailable
+ and running in a Databricks notebook, a temporary token for the current
+ user is generated. Defaults to None.
+ warehouse_id (Optional[str]): The warehouse ID in the Databricks SQL. If
+ provided, the method configures the connection to use this warehouse.
+ Cannot be used with 'cluster_id'. Defaults to None.
+ cluster_id (Optional[str]): The cluster ID in the Databricks Runtime. If
+ provided, the method configures the connection to use this cluster.
+ Cannot be used with 'warehouse_id'. If running in a Databricks notebook
+ and both 'warehouse_id' and 'cluster_id' are None, it uses the ID of the
+ cluster the notebook is attached to. Defaults to None.
+ engine_args (Optional[dict]): The arguments to be used when connecting
+ Databricks. Defaults to None.
+ **kwargs (Any): Additional keyword arguments for the `from_uri` method.
+
+ Returns:
+ SQLDatabase: An instance of SQLDatabase configured with the provided
+ Databricks connection details.
+
+ Raises:
+ ValueError: If 'databricks-sql-connector' is not found, or if both
+ 'warehouse_id' and 'cluster_id' are provided, or if neither
+ 'warehouse_id' nor 'cluster_id' are provided and it's not executing
+ inside a Databricks notebook.
+ """
+ try:
+ from databricks import sql # noqa: F401
+ except ImportError:
+ raise ValueError(
+ "databricks-sql-connector package not found, please install with"
+ " `pip install databricks-sql-connector`"
+ )
+ context = None
+ try:
+ from dbruntime.databricks_repl_context import get_context
+
+ context = get_context()
+ except ImportError:
+ pass
+
+ default_host = context.browserHostName if context else None
+ if host is None:
+ host = get_from_env("host", "DATABRICKS_HOST", default_host)
+
+ default_api_token = context.apiToken if context else None
+ if api_token is None:
+ api_token = get_from_env("api_token", "DATABRICKS_TOKEN", default_api_token)
+
+ if warehouse_id is None and cluster_id is None:
+ if context:
+ cluster_id = context.clusterId
+ else:
+ raise ValueError(
+ "Need to provide either 'warehouse_id' or 'cluster_id'."
+ )
+
+ if warehouse_id and cluster_id:
+ raise ValueError("Can't have both 'warehouse_id' or 'cluster_id'.")
+
+ if warehouse_id:
+ http_path = f"/sql/1.0/warehouses/{warehouse_id}"
+ else:
+ http_path = f"/sql/protocolv1/o/0/{cluster_id}"
+
+ uri = (
+ f"databricks://token:{api_token}@{host}?"
+ f"http_path={http_path}&catalog={catalog}&schema={schema}"
+ )
+ return cls.from_uri(database_uri=uri, engine_args=engine_args, **kwargs)
+
+ @classmethod
+ def from_cnosdb(
+ cls,
+ url: str = "127.0.0.1:8902",
+ user: str = "root",
+ password: str = "",
+ tenant: str = "cnosdb",
+ database: str = "public",
+ ) -> SQLDatabase:
+ """
+ Class method to create an SQLDatabase instance from a CnosDB connection.
+ This method requires the 'cnos-connector' package. If not installed, it
+ can be added using `pip install cnos-connector`.
+
+ Args:
+ url (str): The HTTP connection host name and port number of the CnosDB
+ service, excluding "http://" or "https://", with a default value
+ of "127.0.0.1:8902".
+ user (str): The username used to connect to the CnosDB service, with a
+ default value of "root".
+ password (str): The password of the user connecting to the CnosDB service,
+ with a default value of "".
+ tenant (str): The name of the tenant used to connect to the CnosDB service,
+ with a default value of "cnosdb".
+ database (str): The name of the database in the CnosDB tenant.
+
+ Returns:
+ SQLDatabase: An instance of SQLDatabase configured with the provided
+ CnosDB connection details.
+ """
+ try:
+ from cnosdb_connector import make_cnosdb_langchain_uri
+
+ uri = make_cnosdb_langchain_uri(url, user, password, tenant, database)
+ return cls.from_uri(database_uri=uri)
+ except ImportError:
+ raise ValueError(
+ "cnos-connector package not found, please install with"
+ " `pip install cnos-connector`"
+ )
+
+ @property
+ def dialect(self) -> str:
+ """Return string representation of dialect to use."""
+ return self._engine.dialect.name
+
+ def get_usable_table_names(self) -> Iterable[str]:
+ """Get names of tables available."""
+ if self._include_tables:
+ return sorted(self._include_tables)
+ return sorted(self._all_tables - self._ignore_tables)
+
+ def get_table_names(self) -> Iterable[str]:
+ """Get names of tables available."""
+ warnings.warn(
+ "This method is deprecated - please use `get_usable_table_names`."
+ )
+ return self.get_usable_table_names()
+
+ @property
+ def table_info(self) -> str:
+ """Information about all tables in the database."""
+ return self.get_table_info()
+
+ def get_table_info(self, table_names: Optional[List[str]] = None) -> str:
+ """Get information about specified tables.
+
+ Follows best practices as specified in: Rajkumar et al, 2022
+ (https://arxiv.org/abs/2204.00498)
+
+ If `sample_rows_in_table_info`, the specified number of sample rows will be
+ appended to each table description. This can increase performance as
+ demonstrated in the paper.
+ """
+ all_table_names = self.get_usable_table_names()
+ if table_names is not None:
+ missing_tables = set(table_names).difference(all_table_names)
+ if missing_tables:
+ raise ValueError(f"table_names {missing_tables} not found in database")
+ all_table_names = table_names
+
+ meta_tables = [
+ tbl
+ for tbl in self._metadata.sorted_tables
+ if tbl.name in set(all_table_names)
+ and not (self.dialect == "sqlite" and tbl.name.startswith("sqlite_"))
+ ]
+
+ tables = []
+ for table in meta_tables:
+ if self._custom_table_info and table.name in self._custom_table_info:
+ tables.append(self._custom_table_info[table.name])
+ continue
+
+ # Ignore JSON datatyped columns
+ for k, v in table.columns.items():
+ if type(v.type) is NullType:
+ table._columns.remove(v)
+
+ # add create table command
+ create_table = str(CreateTable(table).compile(self._engine))
+ table_info = f"{create_table.rstrip()}"
+ has_extra_info = (
+ self._indexes_in_table_info or self._sample_rows_in_table_info
+ )
+ if has_extra_info:
+ table_info += "\n\n/*"
+ if self._indexes_in_table_info:
+ table_info += f"\n{self._get_table_indexes(table)}\n"
+ if self._sample_rows_in_table_info:
+ table_info += f"\n{self._get_sample_rows(table)}\n"
+ if has_extra_info:
+ table_info += "*/"
+ tables.append(table_info)
+ tables.sort()
+ final_str = "\n\n".join(tables)
+ return final_str
+
+ def _get_table_indexes(self, table: Table) -> str:
+ indexes = self._inspector.get_indexes(table.name)
+ indexes_formatted = "\n".join(map(_format_index, indexes))
+ return f"Table Indexes:\n{indexes_formatted}"
+
+ def _get_sample_rows(self, table: Table) -> str:
+ # build the select command
+ command = select(table).limit(self._sample_rows_in_table_info)
+
+ # save the columns in string format
+ columns_str = "\t".join([col.name for col in table.columns])
+
+ try:
+ # get the sample rows
+ with self._engine.connect() as connection:
+ sample_rows_result = connection.execute(command) # type: ignore
+ # shorten values in the sample rows
+ sample_rows = list(
+ map(lambda ls: [str(i)[:100] for i in ls], sample_rows_result)
+ )
+
+ # save the sample rows in string format
+ sample_rows_str = "\n".join(["\t".join(row) for row in sample_rows])
+
+ # in some dialects when there are no rows in the table a
+ # 'ProgrammingError' is returned
+ except ProgrammingError:
+ sample_rows_str = ""
+
+ return (
+ f"{self._sample_rows_in_table_info} rows from {table.name} table:\n"
+ f"{columns_str}\n"
+ f"{sample_rows_str}"
+ )
+
+ def _execute(
+ self,
+ command: str,
+ fetch: Union[Literal["all"], Literal["one"]] = "all",
+ ) -> Sequence[Dict[str, Any]]:
+ """
+ Executes SQL command through underlying engine.
+
+ If the statement returns no rows, an empty list is returned.
+ """
+ with self._engine.begin() as connection:
+ if self._schema is not None:
+ if self.dialect == "snowflake":
+ connection.exec_driver_sql(
+ "ALTER SESSION SET search_path = %s", (self._schema,)
+ )
+ elif self.dialect == "bigquery":
+ connection.exec_driver_sql("SET @@dataset_id=?", (self._schema,))
+ elif self.dialect == "mssql":
+ pass
+ elif self.dialect == "trino":
+ connection.exec_driver_sql("USE ?", (self._schema,))
+ elif self.dialect == "duckdb":
+ # Unclear which parameterized argument syntax duckdb supports.
+ # The docs for the duckdb client say they support multiple,
+ # but `duckdb_engine` seemed to struggle with all of them:
+ # https://github.com/Mause/duckdb_engine/issues/796
+ connection.exec_driver_sql(f"SET search_path TO {self._schema}")
+ elif self.dialect == "oracle":
+ connection.exec_driver_sql(
+ f"ALTER SESSION SET CURRENT_SCHEMA = {self._schema}"
+ )
+ else: # postgresql and other compatible dialects
+ connection.exec_driver_sql("SET search_path TO %s", (self._schema,))
+ cursor = connection.execute(text(command))
+ if cursor.returns_rows:
+ if fetch == "all":
+ result = [x._asdict() for x in cursor.fetchall()]
+ elif fetch == "one":
+ first_result = cursor.fetchone()
+ result = [] if first_result is None else [first_result._asdict()]
+ else:
+ raise ValueError("Fetch parameter must be either 'one' or 'all'")
+ return result
+ return []
+
+ def run(
+ self,
+ command: str,
+ fetch: Union[Literal["all"], Literal["one"]] = "all",
+ ) -> str:
+ """Execute a SQL command and return a string representing the results.
+
+ If the statement returns rows, a string of the results is returned.
+ If the statement returns no rows, an empty string is returned.
+ """
+ result = self._execute(command, fetch)
+ # Convert columns values to string to avoid issues with sqlalchemy
+ # truncating text
+ res = [
+ tuple(truncate_word(c, length=self._max_string_length) for c in r.values())
+ for r in result
+ ]
+ if not res:
+ return ""
+ else:
+ return str(res)
+
+ def get_table_info_no_throw(self, table_names: Optional[List[str]] = None) -> str:
+ """Get information about specified tables.
+
+ Follows best practices as specified in: Rajkumar et al, 2022
+ (https://arxiv.org/abs/2204.00498)
+
+ If `sample_rows_in_table_info`, the specified number of sample rows will be
+ appended to each table description. This can increase performance as
+ demonstrated in the paper.
+ """
+ try:
+ return self.get_table_info(table_names)
+ except ValueError as e:
+ """Format the error message"""
+ return f"Error: {e}"
+
+ def run_no_throw(
+ self,
+ command: str,
+ fetch: Union[Literal["all"], Literal["one"]] = "all",
+ ) -> str:
+ """Execute a SQL command and return a string representing the results.
+
+ If the statement returns rows, a string of the results is returned.
+ If the statement returns no rows, an empty string is returned.
+
+ If the statement throws an error, the error message is returned.
+ """
+ try:
+ return self.run(command, fetch)
+ except SQLAlchemyError as e:
+ """Format the error message"""
+ return f"Error: {e}"
diff --git a/src/aios_kernel/workflow_env.py b/src/aios/environment/workflow_env.py
similarity index 80%
rename from src/aios_kernel/workflow_env.py
rename to src/aios/environment/workflow_env.py
index c62fc45..47153be 100644
--- a/src/aios_kernel/workflow_env.py
+++ b/src/aios/environment/workflow_env.py
@@ -1,4 +1,4 @@
-
+# pylint:disable=E0402
from datetime import datetime
import asyncio
import json
@@ -7,20 +7,22 @@ from sqlite3 import Error
import threading
import logging
from typing import Optional
-
-from .text_to_speech_function import TextToSpeechFunction
-from .compute_kernel import ComputeKernel, ComputeTaskResultCode
-from .environment import Environment,EnvironmentEvent
-from .ai_function import SimpleAIFunction
-from .storage import AIStorage
-from .contact_manager import ContactManager,Contact,FamilyMember
-
import aiosqlite
+from ..agent.llm_context import GlobaToolsLibrary
+from ..proto.compute_task import *
+from ..proto.ai_function import *
+from ..frame.compute_kernel import ComputeKernel
+from ..frame.contact_manager import ContactManager,Contact
+from ..storage.storage import AIStorage
+
+from .environment import SimpleEnvironment, CompositeEnvironment
+from ..ai_functions.script_to_speech_function import ScriptToSpeechFunction
+from ..ai_functions.image_2_text_function import Image2TextFunction
+
logger = logging.getLogger(__name__)
-
-class CalenderEvent(EnvironmentEvent):
+class CalenderEvent(SimpleEnvironment):
def __init__(self,data) -> None:
super().__init__()
self.event_name = "timer"
@@ -30,48 +32,45 @@ class CalenderEvent(EnvironmentEvent):
return f"#event timer:{self.data}"
# AI Calender GOAL: Let user use "create notify after 2 days" to create a timer event
-class CalenderEnvironment(Environment):
+class CalenderEnvironment(SimpleEnvironment):
def __init__(self, env_id: str) -> None:
super().__init__(env_id)
self.db_file = AIStorage.get_instance().get_myai_dir() / "calender.db"
self.is_run = False
+ gl = GlobaToolsLibrary.get_instance()
- self.add_ai_function(SimpleAIFunction("get_time",
+ gl.register_tool_function(SimpleAIFunction("system.now",
"get current time",
self._get_now))
-
- #self.add_ai_function(SimpleAIFunction("serach_events",
- # "search events in calender",
- # self._search_events))
-
- get_param = {
+
+ get_param = ParameterDefine.create_parameters({
"start_time": "start time (UTC) of event",
"end_time": "end time (UTC) of event"
- }
- self.add_ai_function(SimpleAIFunction("get_events",
+ })
+ gl.register_tool_function(SimpleAIFunction("system.calender.get_events",
"get events in calender by time range",
self._get_events_by_time_range,get_param))
- add_param = {
+ add_param = ParameterDefine.create_parameters({
"title": "title of event",
"start_time": "start time (UTC) of event",
"end_time": "end time (UTC) of event",
"participants": "participants of event",
"location": "location of event",
"details": "details of event"
- }
- self.add_ai_function(SimpleAIFunction("add_event",
+ })
+ gl.register_tool_function(SimpleAIFunction("system.calender.add_event",
"add event to calender",
self._add_event,add_param))
- delete_param = {
+ delete_param = ParameterDefine.create_parameters({
"event_id": "id of event"
- }
- self.add_ai_function(SimpleAIFunction("delete_event",
+ })
+ gl.register_tool_function(SimpleAIFunction("system.calender.delete_event",
"delete event from calender",
self._delete_event,delete_param))
- update_param = {
+ update_param = ParameterDefine.create_parameters({
"event_id": "id of event",
"new_title": "new title of event",
"new_participants": "new participants of event",
@@ -79,35 +78,11 @@ class CalenderEnvironment(Environment):
"new_details": "new details of event",
"start_time": "new start time (UTC) of event",
"end_time": "new end time (UTC) of event"
- }
- self.add_ai_function(SimpleAIFunction("update_event",
+ })
+ gl.register_tool_function(SimpleAIFunction("system.calender.update_event",
"update event in calender",
self._update_event,update_param))
-
- #maybe this function should be in other env?
- paint_param = {
- "prompt": "A description of the content of the painting",
- "model_name": "Which model to use to draw the picture, can be None"
- }
- self.add_ai_function(SimpleAIFunction("paint",
- "Draw a picture according to the description",
- self._paint,paint_param))
-
- self.add_ai_function(SimpleAIFunction("get_contact",
- "get contact info",
- self._get_contact,{"name":"name of contact"}))
-
- self.add_ai_function(SimpleAIFunction("set_contact",
- "set contact info",
- self._set_contact,{"name":"name of contact","contact_info":"A json to descrpit contact"}))
-
-
-
-
- #self.add_ai_function(SimpleAIFunction("user_confirm",
- # "user confirm",
- # self._user_confirm))
async def init_db(self):
async with aiosqlite.connect(self.db_file) as db:
@@ -151,7 +126,7 @@ class CalenderEnvironment(Environment):
_event["location"] = row[5]
_event["details"] = row[6]
result[row[0]] = _event
- return json.dumps(result, indent=4, sort_keys=True)
+ return json.dumps(result, indent=4, sort_keys=True,ensure_ascii=False)
async def _get_events_by_time_range(self,start_time, end_time):
async with aiosqlite.connect(self.db_file) as db:
@@ -173,11 +148,11 @@ class CalenderEnvironment(Environment):
_event["location"] = row[5]
_event["details"] = row[6]
result[row[0]] = _event
-
+
if not have_result:
return "No event."
-
- return json.dumps(result, indent=4, sort_keys=True)
+
+ return json.dumps(result, indent=4, sort_keys=True,ensure_ascii=False)
async def _update_event(self,event_id, new_title=None, new_participants=None, new_location=None, new_details=None ,start_time=None, end_time=None):
fields_to_update = []
@@ -234,12 +209,12 @@ class CalenderEnvironment(Environment):
def _do_get_value(self,key:str) -> Optional[str]:
return None
-
+
async def _get_contact(self,name:str) -> str:
cm = ContactManager.get_instance()
contact : Contact = cm.find_contact_by_name(name)
if contact:
- s = json.dumps(contact.to_dict())
+ s = json.dumps(contact.to_dict(),ensure_ascii=False)
return f"Execute get_contact OK , contact {name} is {s}"
else:
return f"Execute get_contact OK , contact {name} not found!"
@@ -271,9 +246,6 @@ class CalenderEnvironment(Environment):
return f"Execute set_contact OK , contact {name} updated!"
-
-
-
async def start(self) -> None:
if self.is_run:
return
@@ -309,7 +281,7 @@ class CalenderEnvironment(Environment):
formatted_time = now.strftime('%Y-%m-%d %H:%M:%S')
return formatted_time
-
+
async def _paint(self, prompt, model_name = None) -> str:
result = await ComputeKernel.get_instance().do_text_2_image(prompt, model_name)
if result.result_code == ComputeTaskResultCode.ERROR:
@@ -318,18 +290,19 @@ class CalenderEnvironment(Environment):
return f'exec paint OK, saved as a local file, path is: {result.result["file"]}'
-class PaintEnvironment(Environment):
+class PaintEnvironment(SimpleEnvironment):
def __init__(self, env_id: str) -> None:
super().__init__(env_id)
- self.is_run = False
- paint_param = {
- "prompt": "Keywords of the content of the painting",
- "model_name": "Which model to use to draw the picture, can be None",
- "negative_prompt": "Keywords that describe what is not to be drawn, can be None"
- }
- self.add_ai_function(SimpleAIFunction("paint",
- "Draw a picture according to the keywords",
+
+
+
+ def register_functions(self):
+ paint_param = ParameterDefine.create_parameters({
+ "prompt": "Description of the content of the painting",
+ })
+ GlobaToolsLibrary.get_instance().register_tool_function(SimpleAIFunction("aigc.text_2_image",
+ "Draw a picture according to the description",
self._paint,paint_param))
def _do_get_value(self,key:str) -> Optional[str]:
@@ -337,21 +310,22 @@ class PaintEnvironment(Environment):
async def _paint(self, prompt, model_name = None, negative_prompt = None) -> str:
- err, result = await ComputeKernel.get_instance().do_text_2_image(prompt, model_name, negative_prompt)
- if err is not None:
- return f"exec paint failed. err:{err}"
+ result = await ComputeKernel.get_instance().do_text_2_image(prompt, model_name, negative_prompt)
+ if result.result_code == ComputeTaskResultCode.ERROR:
+ return f"exec paint failed. err:{result.error_str}"
else:
return f'exec paint OK, saved as a local file, path is: {result.result["file"]}'
# Default Workflow Environment(Context)
-class WorkflowEnvironment(Environment):
+class WorkflowEnvironment(CompositeEnvironment):
def __init__(self, env_id: str,db_file:str) -> None:
super().__init__(env_id)
self.db_file = db_file
self.local = threading.local()
self.table_name = "WorkflowEnv_" + env_id
- self.add_ai_function(TextToSpeechFunction())
+ # self.add_ai_function(ScriptToSpeechFunction())
+ # self.add_ai_function(Image2TextFunction())
def _get_conn(self):
@@ -424,4 +398,4 @@ class WorkflowEnvironment(Environment):
logging.error(f"Error occurred while update env{self.env_id}.{key} ,error:{e}")
def get_functions(self):
- pass
\ No newline at end of file
+ pass
diff --git a/src/aios/environment/workspace_env.py b/src/aios/environment/workspace_env.py
new file mode 100644
index 0000000..2e332ba
--- /dev/null
+++ b/src/aios/environment/workspace_env.py
@@ -0,0 +1,328 @@
+# pylint:disable=E0402
+import json
+import logging
+import os
+import aiofiles
+import sqlite3
+import asyncio
+from typing import Any,List,Dict
+import chardet
+
+from ..proto.agent_task import *
+from ..proto.ai_function import *
+from ..proto.compute_task import *
+from ..agent.agent_base import *
+
+from ..storage.storage import AIStorage,ResourceLocation
+from .environment import SimpleEnvironment, CompositeEnvironment
+
+
+logger = logging.getLogger(__name__)
+
+class TodoListType:
+ TO_WORK = "work"
+ TO_LEARN = "learn"
+
+class TodoListEnvironment(SimpleEnvironment):
+ def __init__(self, workspace, list_type) -> None:
+ super().__init__(workspace)
+ self.root_path = os.path.join(workspace, list_type)
+ if not os.path.exists(self.root_path):
+ os.makedirs(self.root_path)
+
+ self.db_path = os.path.join(self.root_path, "todo.db")
+ self.conn = None
+ try:
+ self.conn = sqlite3.connect(self.db_path)
+ except Exception as e:
+ logger.error("Error occurred while connecting to database: %s", e)
+ return None
+
+ cursor = self.conn.cursor()
+ cursor.execute('''
+ CREATE TABLE IF NOT EXISTS todo_list (
+ id TEXT,
+ path TEXT
+ )
+ ''')
+ self.conn.commit()
+
+ async def create_todo(params):
+ todoObj = AgentTodo.from_dict(params["todo"])
+ parent_id = params.get("parent")
+ return await self.create_todo(parent_id,todoObj)
+
+ self.add_ai_operation(SimpleAIAction(
+ op="create_todo",
+ description="create todo",
+ func_handler=create_todo,
+ ))
+
+
+ async def update_todo(params):
+ todo_id = params["id"]
+ new_stat = params["state"]
+ return await self.update_todo(todo_id,new_stat)
+
+ self.add_ai_operation(SimpleAIAction(
+ op="update_todo",
+ description="update todo",
+ func_handler=update_todo,
+ ))
+
+ def _get_todo_path(self,todo_id:str) -> str:
+ cursor = self.conn.cursor()
+ cursor.execute('''
+ SELECT path FROM todo_list WHERE id = ?
+ ''',(todo_id,))
+ row = cursor.fetchone()
+ if row:
+ return row[0]
+ else:
+ return None
+
+ def _save_todo_path(self,todo_id:str,path:str):
+ cursor = self.conn.cursor()
+ cursor.execute('''
+ INSERT INTO todo_list (id,path) VALUES (?,?)
+ ''',(todo_id,path))
+ self.conn.commit()
+
+ # Task/todo system , create,update,delete,query
+ async def get_todo_tree(self,path:str = None,deep:int = 4):
+ if path:
+ directory_path = os.path.join(self.root_path, path)
+ else:
+ directory_path = self.root_path
+
+
+ str_result:str = "/todos\n"
+ todo_count:int = 0
+
+ async def scan_dir(directory_path:str,deep:int):
+ nonlocal str_result
+ nonlocal todo_count
+ if deep <= 0:
+ return
+
+ if os.path.exists(directory_path) is False:
+ return
+
+ for entry in os.scandir(directory_path):
+ is_dir = entry.is_dir()
+ if not is_dir:
+ continue
+
+ if entry.name.startswith("."):
+ continue
+
+ todo_count = todo_count + 1
+ str_result = str_result + f"{' '*(4-deep)}{entry.name}\n"
+ await scan_dir(entry.path,deep-1)
+
+ await scan_dir(directory_path,deep)
+ return str_result,todo_count
+
+ async def get_todo_list(self,agent_id:str,path:str = None)->List[AgentTodo]:
+ logger.info("get_todo_list:%s,%s",agent_id,path)
+ if path:
+ directory_path = os.path.join(self.root_path, path)
+ else:
+ directory_path = self.root_path
+
+ result_list:List[AgentTodo] = []
+
+ async def scan_dir(directory_path:str,deep:int,parent:AgentTodo=None):
+ nonlocal result_list
+ if os.path.exists(directory_path) is False:
+ return
+
+ for entry in os.scandir(directory_path):
+ is_dir = entry.is_dir()
+ if not is_dir:
+ continue
+
+ if entry.name.startswith("."):
+ continue
+
+ todo = await self.get_todo_by_fullpath(entry.path)
+ if todo:
+ if todo.worker:
+ if todo.worker != agent_id:
+ continue
+
+ if parent:
+ parent.sub_todos[todo.todo_id] = todo
+
+ result_list.append(todo)
+ todo.rank = int(todo.create_time)>>deep
+ await scan_dir(entry.path,deep + 1,todo)
+
+ return
+
+ await scan_dir(directory_path,0)
+ #sort by rank
+ result_list.sort(key=lambda x:(x.rank,x.title))
+ logger.info("get_todo_list return,todolist.length() is %d",len(result_list))
+ return result_list
+
+ async def get_todo_by_fullpath(self,path:str) -> AgentTodo:
+ logger.info("get_todo_by_fullpath:%s",path)
+
+ detail_path = path + "/detail"
+ try:
+ with open(detail_path, mode='r', encoding="utf-8") as f:
+ todo_dict = json.load(f)
+ result_todo = AgentTodo.from_dict(todo_dict)
+ if result_todo:
+ relative_path = os.path.relpath(path, self.root_path)
+ if not relative_path.startswith('/'):
+ relative_path = '/' + relative_path
+ result_todo.todo_path = relative_path
+ else:
+ logger.error("get_todo_by_path:%s,parse failed!",path)
+
+ return result_todo
+ except Exception as e:
+ logger.error("get_todo_by_path:%s,failed:%s",path,e)
+ return None
+
+
+ async def create_todo(self,parent_id:str,todo:AgentTodo) -> str:
+ try:
+ if parent_id:
+ parent_path = self._get_todo_path(parent_id)
+ todo_path = f"{parent_path}/{todo.todo_id}-{todo.title}"
+ else:
+ todo_path = f"{todo.todo_id}-{todo.title}"
+
+ dir_path = f"{self.root_path}/{todo_path}"
+
+ os.makedirs(dir_path)
+ detail_path = f"{dir_path}/detail"
+ if todo.todo_path is None:
+ todo.todo_path = todo_path
+ self._save_todo_path(todo.todo_id,todo_path)
+ logger.info("create_todo %s",detail_path)
+ async with aiofiles.open(detail_path, mode='w', encoding="utf-8") as f:
+ await f.write(json.dumps(todo.to_dict(),ensure_ascii=False))
+ except Exception as e:
+ logger.error("create_todo failed:%s",e)
+ return str(e)
+
+ return None
+
+ async def update_todo(self,todo_id:str,new_stat:str)->str:
+ try:
+ todo_path = self._get_todo_path(todo_id)
+ full_path = f"{self.root_path}/{todo_path}"
+ todo : AgentTodo = await self.get_todo_by_fullpath(full_path)
+ if todo:
+ todo.state = new_stat
+ detail_path = f"{full_path}/detail"
+ async with aiofiles.open(detail_path, mode='w', encoding="utf-8") as f:
+ await f.write(json.dumps(todo.to_dict()),ensure_ascii=False)
+ return None
+ else:
+ return "todo not found."
+ except Exception as e:
+ return str(e)
+
+ async def wait_todo_done(self,todo_id:str,state=AgentTodoState.TODO_STATE_EXEC_OK) -> AgentTodo:
+ todo_path = self._get_todo_path(todo_id)
+ full_path = f"{self.root_path}/{todo_path}"
+ async def check_done():
+ while True:
+ todo : AgentTodo = await self.get_todo_by_fullpath(full_path)
+ if todo is None:
+ continue
+ if todo.state == AgentTodo.TODO_STATE_CANCEL:
+ break
+ elif todo.state == AgentTodo.TODO_STATE_EXPIRED:
+ break
+ elif todo.state == AgentTodo.TODO_STATE_WAITING_CHECK:
+ if state == AgentTodo.TODO_STATE_WAITING_CHECK:
+ break
+ elif todo.state == AgentTodo.TODO_STATE_DONE:
+ if state == AgentTodo.TODO_STATE_WAITING_CHECK:
+ break
+ elif todo.state == AgentTodo.TODO_STATE_DONE:
+ break
+ elif todo.state == AgentTodo.TODO_STATE_REVIEWED:
+ break
+ await asyncio.sleep(1)
+
+ await check_done()
+ return await self.get_todo_by_fullpath(full_path)
+
+
+ # async def append_worklog(self, todo:AgentTodo, result:AgentTodoResult):
+ # worklog = f"{self.root_path}/{todo.todo_path}/.worklog"
+
+ # async with aiofiles.open(worklog, mode='w+', encoding="utf-8") as f:
+ # content = await f.read()
+ # if len(content) > 0:
+ # json_obj = json.loads(content)
+ # else:
+ # json_obj = {}
+ # logs = json_obj.get("logs")
+ # if logs is None:
+ # logs = []
+ # logs.append(result.to_dict())
+ # json_obj["logs"] = logs
+ # await f.write(json.dumps(json_obj),ensure_ascii=False)
+
+
+class WorkspaceEnvironment(CompositeEnvironment):
+ def __init__(self, env_id: str) -> None:
+ myai_path = AIStorage.get_instance().get_myai_dir()
+ root_path = f"{myai_path}/workspace/{env_id}"
+ super().__init__(root_path)
+
+ self.root_path = root_path
+ if not os.path.exists(self.root_path):
+ os.makedirs()
+
+ self.todo_list: Dict[str, TodoListEnvironment] = {}
+ self.todo_list[TodoListType.TO_WORK] = TodoListEnvironment(self.root_path,TodoListType.TO_WORK)
+ self.todo_list[TodoListType.TO_LEARN] = TodoListEnvironment(self.root_path,TodoListType.TO_LEARN)
+
+ # default environments in workspace
+ self.add_env(self.todo_list[TodoListType.TO_WORK])
+
+ def set_root_path(self,path:str):
+ self.root_path = path
+
+ def get_prompt(self) -> AgentMsg:
+ return None
+
+ def get_role_prompt(self,role_id:str) -> LLMPrompt:
+ return None
+
+ def get_do_prompt(self,todo:AgentTodo=None)->LLMPrompt:
+ return None
+
+ # result mean: list[op_error_str],have_error
+ async def exec_op_list(self,oplist:List,agent_id:str)->tuple[List[str],bool]:
+ result_str = "op list is none"
+ if oplist is None or len(oplist) == 0:
+ return None,False
+
+ result_str = []
+ have_error = False
+ for op in oplist:
+ operation = self.get_ai_operation(op["op"])
+ if operation:
+ error_str = await operation.execute(op)
+ else:
+ logger.error(f"execute op list failed: unknown op:{op['op']}")
+ error_str = f"execute op list failed: unknown op:{op['op']}"
+
+ if error_str:
+ have_error = True
+ result_str.append(error_str)
+ else:
+ result_str.append(f"execute success!")
+
+ return result_str,have_error
+
diff --git a/src/aios_kernel/bus.py b/src/aios/frame/bus.py
similarity index 95%
rename from src/aios_kernel/bus.py
rename to src/aios/frame/bus.py
index 2731702..090865c 100644
--- a/src/aios_kernel/bus.py
+++ b/src/aios/frame/bus.py
@@ -1,10 +1,11 @@
from typing import Coroutine,Dict,Any
-from .agent_message import AgentMsg,AgentMsgStatus,AgentMsgType
import asyncio
from asyncio import Queue
-
import logging
+from ..proto.agent_msg import *
+from ..agent.agent_base import *
+
logger = logging.getLogger(__name__)
class AIBusHandler:
@@ -109,6 +110,9 @@ class AIBus:
# means sub
def register_message_handler(self,handler_name:str,handler:Any) -> Queue:
handler_node = AIBusHandler(handler,self)
+ if self.handlers.get(handler_name) is not None:
+ logger.warn(f"handler {handler_name} already register on AI_BUS!")
+
self.handlers[handler_name] = handler_node
return handler_node.queue
diff --git a/src/aios_kernel/compute_kernel.py b/src/aios/frame/compute_kernel.py
similarity index 70%
rename from src/aios_kernel/compute_kernel.py
rename to src/aios/frame/compute_kernel.py
index 3f97708..9458e18 100644
--- a/src/aios_kernel/compute_kernel.py
+++ b/src/aios/frame/compute_kernel.py
@@ -3,12 +3,13 @@ import random
from typing import Optional
import logging
import asyncio
+import tiktoken
from asyncio import Queue
-from knowledge import ObjectID
-from .agent import AgentPrompt
+from ..proto.compute_task import *
+from ..knowledge import ObjectID
+
from .compute_node import ComputeNode
-from .compute_task import ComputeTask, ComputeTaskState, ComputeTaskResult, ComputeTaskType,ComputeTaskResultCode
logger = logging.getLogger(__name__)
@@ -16,7 +17,6 @@ logger = logging.getLogger(__name__)
# to suitable computing nodes, achieving a balance of speed, cost, and power consumption,
# is the CORE GOAL of the entire computing task schedule system (aios_kernel).
-
class ComputeKernel:
_instance = None
@classmethod
@@ -75,7 +75,7 @@ class ComputeKernel:
if len(support_nodes) < 1:
logger.warning(f"task {task.display()} is not support by any compute node")
return None
-
+
# hit a random node with weight
hit_pos = random.randint(0, total_weights - 1)
for i in range(min(len(support_nodes) - 1, hit_pos), -1, -1):
@@ -104,16 +104,34 @@ class ComputeKernel:
def is_task_support(self, task: ComputeTask) -> bool:
return True
+ @staticmethod
+ def llm_num_tokens_from_text(text:str,model:str) -> int:
+ if model is None:
+ model = "gpt-4-turbo-preview"
+
+ try:
+ encoding = tiktoken.encoding_for_model(model)
+ except KeyError:
+ logger.debug("Warning: model not found. Using cl100k_base encoding.")
+ encoding = tiktoken.get_encoding("cl100k_base")
+
+ token_count = len(encoding.encode(text))
+ return token_count
+
+ @staticmethod
+ def llm_num_tokens(prompt: LLMPrompt, model_name: str = None) -> int:
+ return ComputeKernel.llm_num_tokens_from_text(prompt.as_str(), model_name)
+
# friendly interface for use:
- def llm_completion(self, prompt: AgentPrompt, mode_name: Optional[str] = None, max_token: int = 0,inner_functions = None):
+ def llm_completion(self, prompt: LLMPrompt, resp_mode:str="text",mode_name: Optional[str] = None, max_token: int = 0,inner_functions = None):
# craete a llm_work_task ,push on queue's end
# then task_schedule would run this task.(might schedule some work_task to another host)
task_req = ComputeTask()
- task_req.set_llm_params(prompt, mode_name, max_token,inner_functions)
+ task_req.set_llm_params(prompt,resp_mode,mode_name, max_token,inner_functions)
self.run(task_req)
return task_req
-
- async def _wait_task(self,task_req:ComputeTask)->ComputeTaskResult:
+
+ async def _wait_task(self,task_req:ComputeTask, timeout=60)->ComputeTaskResult:
async def check_timer():
check_times = 0
while True:
@@ -123,7 +141,7 @@ class ComputeKernel:
if task_req.state == ComputeTaskState.ERROR:
break
- if check_times >= 120:
+ if timeout is not None and check_times >= timeout*2:
task_req.state = ComputeTaskState.ERROR
break
@@ -137,11 +155,13 @@ class ComputeKernel:
time_out_result = ComputeTaskResult()
time_out_result.result_code = ComputeTaskResultCode.TIMEOUT
time_out_result.set_from_task(task_req)
- ## craete timeout task_result
+ task_req.result = time_out_result
+ return time_out_result
- async def do_llm_completion(self, prompt: AgentPrompt, mode_name: Optional[str] = None, max_token: int = 0, inner_functions = None) -> str:
- task_req = self.llm_completion(prompt, mode_name, max_token,inner_functions)
- return await self._wait_task(task_req)
+
+ async def do_llm_completion(self, prompt: LLMPrompt,resp_mode:str="text", mode_name: Optional[str]=None, max_token:int=0, inner_functions=None, timeout=60) -> str:
+ task_req = self.llm_completion(prompt, resp_mode,mode_name, max_token,inner_functions)
+ return await self._wait_task(task_req, timeout)
def text_embedding(self,input:str,model_name:Optional[str] = None):
@@ -165,7 +185,7 @@ class ComputeKernel:
task_req.set_image_embedding_params(input,model_name)
self.run(task_req)
return task_req
-
+
async def do_image_embedding(self,input:ObjectID,model_name:Optional[str] = None) -> [float]:
task_req = self.image_embedding(input,model_name)
task_result = await self._wait_task(task_req)
@@ -181,7 +201,8 @@ class ComputeKernel:
gender: Optional[str] = None,
age: Optional[str] = None,
voice_name: Optional[str] = None,
- tone: Optional[str] = None):
+ tone: Optional[str] = None,
+ model_name: Optional[str] = None):
task_req = ComputeTask()
task_req.params["text"] = input
task_req.params["language_code"] = language_code
@@ -189,6 +210,7 @@ class ComputeKernel:
task_req.params["age"] = age
task_req.params["voice_name"] = voice_name
task_req.params["tone"] = tone
+ task_req.params["model_name"] = model_name
task_req.task_type = ComputeTaskType.TEXT_2_VOICE
self.run(task_req)
@@ -197,6 +219,24 @@ class ComputeKernel:
if task_req.state == ComputeTaskState.DONE:
return task_result.result
+ async def do_speech_to_text(self,
+ audio: str,
+ model: str,
+ prompt: Optional[str],
+ response_format: Optional[str]):
+ task_req = ComputeTask()
+ task_req.params["file"] = audio
+ task_req.params["model_name"] = model
+ task_req.params["prompt"] = prompt
+ task_req.params["response_format"] = response_format
+ task_req.task_type = ComputeTaskType.VOICE_2_TEXT
+
+ self.run(task_req)
+
+ task_result = await self._wait_task(task_req)
+
+ if task_req.state == ComputeTaskState.DONE:
+ return task_result
def text_2_image(self, prompt:str, model_name:Optional[str] = None, negative_prompt = None):
task = ComputeTask()
@@ -206,11 +246,19 @@ class ComputeKernel:
async def do_text_2_image(self, prompt:str, model_name:Optional[str] = None, negative_prompt = None) -> ComputeTaskResult:
task = self.text_2_image(prompt,model_name, negative_prompt)
- task = await self._wait_task(task)
+ task_result = await self._wait_task(task)
- return task.result
+ return task_result
# if task_req.state == ComputeTaskState.DONE:
# return None, task_result
- # return task_req.error_str, None
+ def image_2_text(self, image_path: str, prompt:str, model_name:Optional[str] = None, negative_prompt = None):
+ task = ComputeTask()
+ task.set_image_2_text_params(image_path,prompt,model_name, negative_prompt)
+ self.run(task)
+ return task
+ async def do_image_2_text(self, image_path: str, prompt:str, model_name:Optional[str] = None, negative_prompt = None) -> ComputeTaskResult:
+ task = self.image_2_text(image_path,prompt, model_name, negative_prompt)
+ task = await self._wait_task(task)
+ return task.result
diff --git a/src/aios_kernel/compute_node.py b/src/aios/frame/compute_node.py
similarity index 92%
rename from src/aios_kernel/compute_node.py
rename to src/aios/frame/compute_node.py
index 7ce7292..6b8a43d 100644
--- a/src/aios_kernel/compute_node.py
+++ b/src/aios/frame/compute_node.py
@@ -1,6 +1,6 @@
from abc import ABC, abstractmethod
-from .compute_task import ComputeTask, ComputeTaskType
+from ..proto.compute_task import ComputeTask, ComputeTaskType
class ComputeNode(ABC):
def __init__(self) -> None:
diff --git a/src/aios/frame/contact.py b/src/aios/frame/contact.py
new file mode 100644
index 0000000..a3fb309
--- /dev/null
+++ b/src/aios/frame/contact.py
@@ -0,0 +1,76 @@
+from typing import List
+import logging
+from datetime import datetime
+
+from ..proto.agent_msg import AgentMsg
+from .tunnel import AgentTunnel
+
+
+logger = logging.getLogger(__name__)
+class Contact:
+ def __init__(self, name, phone=None, email=None, telegram=None,added_by=None, tags=[], notes=""):
+ self.name = name
+ self.phone = phone
+ self.email = email
+ self.telegram = telegram
+ self.added_by = added_by
+ self.tags = tags
+ self.notes = notes
+ self.is_family_member = False
+ self.active_tunnels = {}
+ self.relationship = "friends"
+
+ def to_dict(self):
+ return {
+ "name": self.name,
+ "phone": self.phone,
+ "email": self.email,
+ "telegram" : self.telegram,
+ "is_family_member": self.is_family_member,
+
+ "added_by": self.added_by,
+ "tags": self.tags,
+ "notes": self.notes,
+ "now" : datetime.now().strftime('%Y-%m-%d %H:%M:%S'),
+ "relationship" : self.relationship
+ }
+
+ async def _process_msg(self,msg:AgentMsg):
+ tunnel : AgentTunnel = self.get_active_tunnel(msg.sender)
+ if tunnel is not None:
+ await tunnel.post_message(msg)
+ return None
+ else:
+ tunnel = await self.create_default_tunnel(msg.sender)
+ if tunnel is not None:
+ self.active_tunnels[msg.sender] = tunnel
+ await tunnel.post_message(msg)
+ return None
+
+
+ logger.warn(f"contact {self.name} cann't get tunnel,post message failed!")
+
+ def get_active_tunnel(self,agent_id) -> AgentTunnel:
+ tunnel = self.active_tunnels.get(agent_id)
+ return tunnel
+
+ def set_active_tunnel(self,agent_id,tunnel:AgentTunnel):
+ self.active_tunnels[agent_id] = tunnel
+
+ async def create_default_tunnel(self,agent_id:str) -> AgentTunnel:
+ #TODO:fix this
+ from .email_tunnel import EmailTunnel
+
+ result_tunnels = AgentTunnel.get_tunnel_by_agentid(agent_id)
+ for tunnel in result_tunnels:
+ if isinstance(tunnel,EmailTunnel):
+ return tunnel
+
+ return None
+
+ @classmethod
+ def from_dict(cls, data):
+ result = Contact(data.get("name"), data.get("phone"), data.get("email"), data.get("telegram"),data.get("added_by"), data.get("tags"), data.get("notes"))
+ if data.get("relationship") is not None:
+ result.relationship = data.get("relationship")
+ return result
diff --git a/src/aios_kernel/contact_manager.py b/src/aios/frame/contact_manager.py
similarity index 53%
rename from src/aios_kernel/contact_manager.py
rename to src/aios/frame/contact_manager.py
index 39c5711..5c619c6 100644
--- a/src/aios_kernel/contact_manager.py
+++ b/src/aios/frame/contact_manager.py
@@ -1,48 +1,18 @@
from typing import List
import toml
+import time
+import logging
-class Contact:
- def __init__(self, name, phone=None, email=None, telegram=None,added_by=None, tags=[], notes=""):
- self.name = name
- self.phone = phone
- self.email = email
- self.telegram = telegram
- self.added_by = added_by
- self.tags = tags
- self.notes = notes
- self.is_family_member = False
+from datetime import datetime
+from ..proto.agent_msg import AgentMsg
+from ..proto.ai_function import ParameterDefine, SimpleAIFunction
+from ..agent.llm_context import GlobaToolsLibrary
+from .tunnel import AgentTunnel
+from .contact import Contact
- def to_dict(self):
- return {
- "name": self.name,
- "phone": self.phone,
- "email": self.email,
- "telegram" : self.telegram,
- "added_by": self.added_by,
- "tags": self.tags,
- "notes": self.notes
- }
+logger = logging.getLogger(__name__)
- @classmethod
- def from_dict(cls, data):
- return Contact(data.get("name"), data.get("phone"), data.get("email"), data.get("telegram"),data.get("added_by"), data.get("tags"), data.get("notes"))
-
-class FamilyMember(Contact):
- def __init__(self, name, relationship,phone=None, email=None,telegram=None):
- super().__init__(name, phone, email, telegram)
- self.name = name
- self.relationship = relationship
- self.is_family_member = True
-
- def to_dict(self):
- result = super().to_dict()
- result["relationship"] = self.relationship
- return result
-
- @classmethod
- def from_dict(cls, data):
- return FamilyMember(data.get("name"),data.get("relationship"),data.get("phone"), data.get("email"),data.get("telegram"))
class ContactManager:
_instance = None
@@ -51,11 +21,28 @@ class ContactManager:
if cls._instance is None:
cls._instance = ContactManager(str(filename))
return cls._instance
+
+ def register_global_functions(self):
+ gl = GlobaToolsLibrary.get_instance()
+
+ get_parameters = ParameterDefine.create_parameters({"name":"contact name name"})
+ gl.register_tool_function(SimpleAIFunction("system.contacts.get",
+ "get contact info",
+ self._get_contact,get_parameters))
+
+ # todo: use json to save contact info
+ update_parameters = ParameterDefine.create_parameters({"name":"name","contact_info":"A json to descrpit contact"})
+ gl.register_tool_function(SimpleAIFunction("system.contacts.set",
+ "set contact info",
+ self._set_contact,update_parameters))
+
+ return
+
+
def __init__(self, filename="contacts.toml"):
self.filename = filename
self.contacts = []
- self.family_members = []
self.is_auto_create_contact_from_telegram = True
@@ -69,12 +56,10 @@ class ContactManager:
def load_from_config(self,config_data:dict):
self.contacts = [Contact.from_dict(item) for item in config_data.get("contacts", [])]
- self.family_members = [FamilyMember.from_dict(item) for item in config_data.get("family_members", [])]
-
+
def save_data(self):
data = {
"contacts": [contact.to_dict() for contact in self.contacts],
- "family_members": [member.to_dict() for member in self.family_members]
}
with open(self.filename, "w") as f:
toml.dump(data, f)
@@ -108,46 +93,28 @@ class ContactManager:
if contact.name == name:
return contact
- for member in self.family_members:
- if member.name == name:
- return member
return None
def find_contact_by_telegram(self, telegram:str):
for contact in self.contacts:
if contact.telegram == telegram:
return contact
- for member in self.family_members:
- if member.telegram == telegram:
- return member
+
return None
def find_contact_by_email(self, email:str):
for contact in self.contacts:
if contact.email == email:
return contact
- for member in self.family_members:
- if member.email == email:
- return member
+
return None
def find_contact_by_phone(self, phone:str):
for contact in self.contacts:
if contact.phone == phone:
return contact
- for member in self.family_members:
- if member.phone == phone:
- return member
+
return None
-
- def add_family_member(self, name, new_member:FamilyMember):
- assert name == new_member.name
- self.family_members.append(new_member)
- self.save_data()
-
def list_contacts(self):
return self.contacts
-
- def list_family_members(self):
- return self.family_members
diff --git a/src/aios_kernel/queue_compute_node.py b/src/aios/frame/queue_compute_node.py
similarity index 91%
rename from src/aios_kernel/queue_compute_node.py
rename to src/aios/frame/queue_compute_node.py
index 4eb22f4..be8a48e 100644
--- a/src/aios_kernel/queue_compute_node.py
+++ b/src/aios/frame/queue_compute_node.py
@@ -4,8 +4,7 @@ from asyncio import Queue
import logging
from abc import abstractmethod
-from .compute_task import ComputeTask, ComputeTaskResult, ComputeTaskResultCode, ComputeTaskState, ComputeTaskType
-from .compute_node import ComputeNode
+from aios import ComputeTask, ComputeNode,ComputeTaskResult, ComputeTaskResultCode, ComputeTaskState, ComputeTaskType
logger = logging.getLogger(__name__)
diff --git a/src/aios_kernel/tunnel.py b/src/aios/frame/tunnel.py
similarity index 86%
rename from src/aios_kernel/tunnel.py
rename to src/aios/frame/tunnel.py
index a3691cd..ce11f1e 100644
--- a/src/aios_kernel/tunnel.py
+++ b/src/aios/frame/tunnel.py
@@ -1,7 +1,8 @@
from abc import ABC, abstractmethod
import logging
from typing import Coroutine
-from .agent_message import AgentMsg
+
+from ..proto.agent_msg import AgentMsg
from .bus import AIBus
logger = logging.getLogger(__name__)
@@ -33,7 +34,7 @@ class AgentTunnel(ABC):
tunnel.connect_to(AIBus.get_default_bus(),tunnel.target_id)
await tunnel.start()
else:
- logger.error(f"load tunnel {tunnel_id} failed")
+ logger.error(f"load tunnel {tunnel_id} failed")
else:
logger.error(f"load tunnel {tunnel_id} failed,loader not found")
@@ -48,19 +49,27 @@ class AgentTunnel(ABC):
await tunnel.start()
return True
else:
- logger.error(f"load tunnel {tunnel_config['tunnel_id']} failed")
+ logger.error(f"load tunnel {tunnel_config['tunnel_id']} failed")
else:
logger.error(f"load tunnel {tunnel_config['type']} failed,loader not found")
return False
-
+
+ @classmethod
+ async def get_tunnel_by_agentid(cls,agent_id:str):
+ result = []
+ for tunnel in cls._all_tunnels.values():
+ if tunnel.target_id == agent_id:
+ result.append(tunnel)
+ return result
+
def __init__(self) -> None:
super().__init__()
self.tunnel_id = None
self.target_id = None
self.target_type = None
- self.ai_bus = None
+ self.ai_bus: AIBus = None
self.is_connected = False
def connect_to(self, ai_bus:AIBus,target_id: str) -> None:
@@ -75,7 +84,10 @@ class AgentTunnel(ABC):
self.ai_bus = ai_bus
self.is_connected = True
-
+ @abstractmethod
+ def post_message(self, msg: AgentMsg) -> None:
+ pass
+
@abstractmethod
async def start(self) -> bool:
diff --git a/src/knowledge/__init__.py b/src/aios/knowledge/__init__.py
similarity index 66%
rename from src/knowledge/__init__.py
rename to src/aios/knowledge/__init__.py
index 680c632..361a67d 100644
--- a/src/knowledge/__init__.py
+++ b/src/aios/knowledge/__init__.py
@@ -2,4 +2,5 @@ from .object import *
from .vector import *
from .data import *
from .store import KnowledgeStore
-from .core_object import *
\ No newline at end of file
+from .core_object import *
+from .pipeline import *
\ No newline at end of file
diff --git a/src/knowledge/core_object/__init__.py b/src/aios/knowledge/core_object/__init__.py
similarity index 100%
rename from src/knowledge/core_object/__init__.py
rename to src/aios/knowledge/core_object/__init__.py
diff --git a/src/knowledge/core_object/document_object.py b/src/aios/knowledge/core_object/document_object.py
similarity index 84%
rename from src/knowledge/core_object/document_object.py
rename to src/aios/knowledge/core_object/document_object.py
index b69f59f..43e8b9e 100644
--- a/src/knowledge/core_object/document_object.py
+++ b/src/aios/knowledge/core_object/document_object.py
@@ -1,7 +1,6 @@
from ..object import KnowledgeObject, ObjectRelationStore
from ..data import ChunkList, ChunkListWriter
from ..object import ObjectType
-from .. import KnowledgeStore
# desc
# meta
@@ -49,13 +48,13 @@ class DocumentObjectBuilder:
self.text = text
return self
- def build(self) -> DocumentObject:
- chunk_list = KnowledgeStore().get_chunk_list_writer().create_chunk_list_from_text(self.text)
+ def build(self, store) -> DocumentObject:
+ chunk_list = store.get_chunk_list_writer().create_chunk_list_from_text(self.text)
doc = DocumentObject(self.meta, self.tags, chunk_list)
doc_id = doc.calculate_id()
# Add relation to store
for chunk_id in chunk_list.chunk_list:
- KnowledgeStore().get_relation_store().add_relation(chunk_id, doc_id)
+ store.get_relation_store().add_relation(chunk_id, doc_id)
return doc
diff --git a/src/knowledge/core_object/email_object.py b/src/aios/knowledge/core_object/email_object.py
similarity index 96%
rename from src/knowledge/core_object/email_object.py
rename to src/aios/knowledge/core_object/email_object.py
index 108c9cc..64d78b3 100644
--- a/src/knowledge/core_object/email_object.py
+++ b/src/aios/knowledge/core_object/email_object.py
@@ -1,4 +1,3 @@
-from .. import KnowledgeStore
from .rich_text_object import RichTextObject, RichTextObjectBuilder
from ..object import ObjectID, ObjectType, KnowledgeObject
from .document_object import DocumentObjectBuilder
@@ -68,11 +67,11 @@ class EmailObjectBuilder:
self.folder = folder
return self
- def build(self) -> EmailObject:
+ def build(self, store) -> EmailObject:
# Just get the object store and relation store from global KnowledgeStore
- store = KnowledgeStore().get_object_store()
- relation = KnowledgeStore().get_relation_store()
+ store = store.get_object_store()
+ relation = store.get_relation_store()
# Read meta.json
meta = {}
diff --git a/src/knowledge/core_object/image_object.py b/src/aios/knowledge/core_object/image_object.py
similarity index 93%
rename from src/knowledge/core_object/image_object.py
rename to src/aios/knowledge/core_object/image_object.py
index 340bcc2..1033bf7 100644
--- a/src/knowledge/core_object/image_object.py
+++ b/src/aios/knowledge/core_object/image_object.py
@@ -1,7 +1,6 @@
from ..object import KnowledgeObject
from ..data import ChunkList, ChunkListWriter
from ..object import ObjectType
-from .. import KnowledgeStore
import os
# desc
@@ -86,10 +85,10 @@ class ImageObjectBuilder:
self.restore_file = restore_file
return self
- def build(self) -> ImageObject:
+ def build(self, store) -> ImageObject:
file_size = os.path.getsize(self.image_file)
- chunk_list = KnowledgeStore().get_chunk_list_writer().create_chunk_list_from_file(
+ chunk_list = store.get_chunk_list_writer().create_chunk_list_from_file(
self.image_file, 1024 * 1024 * 4, self.restore_file
)
exif = get_exif_data(self.image_file)
diff --git a/src/knowledge/core_object/rich_text_object.py b/src/aios/knowledge/core_object/rich_text_object.py
similarity index 97%
rename from src/knowledge/core_object/rich_text_object.py
rename to src/aios/knowledge/core_object/rich_text_object.py
index 7791689..8d9437e 100644
--- a/src/knowledge/core_object/rich_text_object.py
+++ b/src/aios/knowledge/core_object/rich_text_object.py
@@ -1,4 +1,4 @@
-from knowledge.object.object_id import ObjectType
+from ..object.object_id import ObjectType
from ..object import KnowledgeObject
from ..data import ChunkList, ChunkListWriter
from ..object import ObjectType
diff --git a/src/knowledge/core_object/video_object.py b/src/aios/knowledge/core_object/video_object.py
similarity index 93%
rename from src/knowledge/core_object/video_object.py
rename to src/aios/knowledge/core_object/video_object.py
index 56e9829..1ac9338 100644
--- a/src/knowledge/core_object/video_object.py
+++ b/src/aios/knowledge/core_object/video_object.py
@@ -1,7 +1,6 @@
from ..object import KnowledgeObject
from ..data import ChunkList, ChunkListWriter
from ..object import ObjectType
-from .. import KnowledgeStore
# desc
# meta
@@ -76,8 +75,8 @@ class VideoObjectBuilder:
self.restore_file = restore_file
return self
- def build(self) -> VideoObject:
- chunk_list = KnowledgeStore().get_chunk_list_writer().create_chunk_list_from_file(
+ def build(self, store) -> VideoObject:
+ chunk_list = store.get_chunk_list_writer().create_chunk_list_from_file(
self.video_file, 1024 * 1024 * 4, self.restore_file
)
info = get_video_info(self.video_file)
diff --git a/src/knowledge/data/__init__.py b/src/aios/knowledge/data/__init__.py
similarity index 100%
rename from src/knowledge/data/__init__.py
rename to src/aios/knowledge/data/__init__.py
diff --git a/src/knowledge/data/chunk.py b/src/aios/knowledge/data/chunk.py
similarity index 100%
rename from src/knowledge/data/chunk.py
rename to src/aios/knowledge/data/chunk.py
diff --git a/src/knowledge/data/chunk_list.py b/src/aios/knowledge/data/chunk_list.py
similarity index 100%
rename from src/knowledge/data/chunk_list.py
rename to src/aios/knowledge/data/chunk_list.py
diff --git a/src/knowledge/data/chunk_store.py b/src/aios/knowledge/data/chunk_store.py
similarity index 100%
rename from src/knowledge/data/chunk_store.py
rename to src/aios/knowledge/data/chunk_store.py
diff --git a/src/knowledge/data/reader.py b/src/aios/knowledge/data/reader.py
similarity index 100%
rename from src/knowledge/data/reader.py
rename to src/aios/knowledge/data/reader.py
diff --git a/src/knowledge/data/tracker.py b/src/aios/knowledge/data/tracker.py
similarity index 100%
rename from src/knowledge/data/tracker.py
rename to src/aios/knowledge/data/tracker.py
diff --git a/src/knowledge/data/writer.py b/src/aios/knowledge/data/writer.py
similarity index 91%
rename from src/knowledge/data/writer.py
rename to src/aios/knowledge/data/writer.py
index 0381bc6..6ebf3ac 100644
--- a/src/knowledge/data/writer.py
+++ b/src/aios/knowledge/data/writer.py
@@ -19,10 +19,10 @@ def _join_docs(docs: List[str], separator: str) -> Optional[str]:
return text
def _merge_splits(
- splits: Iterable[str],
- separator: str,
- chunk_size: int,
- chunk_overlap: int,
+ splits: Iterable[str],
+ separator: str,
+ chunk_size: int,
+ chunk_overlap: int,
length_function: Callable[[str], int]
) -> List[str]:
# We now want to combine these smaller pieces into medium size
@@ -86,11 +86,11 @@ def _split_text_with_regex(
return [s for s in splits if s != ""]
-def _split_text(
- text: str,
- separators: List[str],
- chunk_size: int,
- chunk_overlap: int,
+def split_text(
+ text: str,
+ separators: List[str],
+ chunk_size: int,
+ chunk_overlap: int,
length_function: Callable[[str], int]
) -> List[str]:
@@ -127,7 +127,7 @@ def _split_text(
if not new_separators:
final_chunks.append(s)
else:
- other_info = _split_text(s, new_separators, chunk_size, chunk_overlap, length_function)
+ other_info = split_text(s, new_separators, chunk_size, chunk_overlap, length_function)
final_chunks.extend(other_info)
if _good_splits:
merged_text = _merge_splits(_good_splits, _separator, chunk_size, chunk_overlap, length_function)
@@ -153,7 +153,7 @@ class ChunkListWriter:
chunk = file.read(chunk_size)
if not chunk:
break
-
+
chunk_len = len(chunk)
chunk_id = ChunkID.hash_data(chunk)
chunk_list.append(chunk_id)
@@ -176,14 +176,14 @@ class ChunkListWriter:
file_hash = HashValue(hash_obj.digest())
# print(f"calc file hash: {file_path}, {file_hash}")
-
+
return ChunkList(chunk_list, file_hash)
def create_chunk_list_from_text(
- self,
- text: str,
- chunk_size: int = 4000,
- chunk_overlap: int = 200,
+ self,
+ text: str,
+ chunk_size: int = 4000,
+ chunk_overlap: int = 200,
separators: str = ["\n\n", "\n", " ", ""]
) -> ChunkList:
enc = tiktoken.encoding_for_model("gpt-3.5-turbo")
@@ -196,8 +196,8 @@ class ChunkListWriter:
disallowed_special="all",
)
)
-
- text_list = _split_text(text, separators, chunk_size, chunk_overlap, length_function)
+
+ text_list = split_text(text, separators, chunk_size, chunk_overlap, length_function)
chunk_list = []
hash_obj = hashlib.sha256()
@@ -211,4 +211,4 @@ class ChunkListWriter:
self.chunk_store.put_chunk(chunk_id, chunk_bytes)
hash = HashValue(hash_obj.digest())
- return ChunkList(chunk_list, hash)
\ No newline at end of file
+ return ChunkList(chunk_list, hash)
diff --git a/src/knowledge/object/__init__.py b/src/aios/knowledge/object/__init__.py
similarity index 100%
rename from src/knowledge/object/__init__.py
rename to src/aios/knowledge/object/__init__.py
diff --git a/src/knowledge/object/blob.py b/src/aios/knowledge/object/blob.py
similarity index 100%
rename from src/knowledge/object/blob.py
rename to src/aios/knowledge/object/blob.py
diff --git a/src/knowledge/object/hash.py b/src/aios/knowledge/object/hash.py
similarity index 100%
rename from src/knowledge/object/hash.py
rename to src/aios/knowledge/object/hash.py
diff --git a/src/knowledge/object/object.py b/src/aios/knowledge/object/object.py
similarity index 79%
rename from src/knowledge/object/object.py
rename to src/aios/knowledge/object/object.py
index ac6f2af..1872acd 100644
--- a/src/knowledge/object/object.py
+++ b/src/aios/knowledge/object/object.py
@@ -47,11 +47,23 @@ class KnowledgeObject(ABC):
def get_body(self) -> dict:
return self.body
+
+ def get_summary(self) -> str:
+ return self.desc.get("summary")
+
+ # def get_articl_catelog(self) -> str:
+ # assert self.object_type == ObjectType.Document
+ # return self.desc.get("catelog")
+
+ # def get_article_full_content(self) -> str:
+ # assert self.object_type == ObjectType.Document
+ # return self.body
def calculate_id(self):
# Convert the object_type and desc to string and compute the SHA256 hash
data = json.dumps(
{"object_type": self.object_type, "desc": self.desc},
+ ensure_ascii=False,
cls=ObjectEnhancedJSONEncoder,
)
sha256 = hashlib.sha256()
@@ -63,5 +75,5 @@ class KnowledgeObject(ABC):
return pickle.dumps(self)
@staticmethod
- def decode(data: bytes) -> "ImageObject":
+ def decode(data: bytes) -> "KnowledgeObject":
return pickle.loads(data)
diff --git a/src/knowledge/object/object_id.py b/src/aios/knowledge/object/object_id.py
similarity index 80%
rename from src/knowledge/object/object_id.py
rename to src/aios/knowledge/object/object_id.py
index 46955f4..1aca474 100644
--- a/src/knowledge/object/object_id.py
+++ b/src/aios/knowledge/object/object_id.py
@@ -13,6 +13,17 @@ class ObjectType(IntEnum):
Document = 103
RichText = 104
Email = 105
+ UserDef = 200
+
+ def is_user_def(self) -> bool:
+ return self.value >= 200
+
+ def get_user_def_type_code(self):
+ return (self.value - 200) if self.is_user_def() else None
+
+ @classmethod
+ def from_user_def_type_code(cls, value):
+ return value + 200
# define a object ID class to identify a object
@@ -23,7 +34,7 @@ class ObjectID: # pylint: disable=too-few-public-methods
def __str__(self):
return self.to_base58()
-
+
def to_base58(self):
return base58.b58encode(self.value).decode()
@@ -46,13 +57,16 @@ class ObjectID: # pylint: disable=too-few-public-methods
def new_chunk_id(chunk_hash: HashValue):
assert len(chunk_hash.value) == 32, "ObjectID must be 32 bytes long"
return ObjectID(bytes([ObjectType.Chunk]) + chunk_hash.value[1:])
-
+
def get_object_type(self) -> ObjectType:
return ObjectType(self.value[0])
-
+
@staticmethod
def hash_data(data: bytes):
return ObjectID.new_chunk_id(HashValue.hash_data(data))
-
+
def __eq__(self, other) -> bool:
- return self.value == other.value
\ No newline at end of file
+ return self.value == other.value
+
+ def __hash__(self):
+ return hash(self.value)
diff --git a/src/knowledge/object/object_store.py b/src/aios/knowledge/object/object_store.py
similarity index 100%
rename from src/knowledge/object/object_store.py
rename to src/aios/knowledge/object/object_store.py
diff --git a/src/knowledge/object/relation.py b/src/aios/knowledge/object/relation.py
similarity index 100%
rename from src/knowledge/object/relation.py
rename to src/aios/knowledge/object/relation.py
diff --git a/src/aios/knowledge/pipeline.py b/src/aios/knowledge/pipeline.py
new file mode 100644
index 0000000..a9a2b58
--- /dev/null
+++ b/src/aios/knowledge/pipeline.py
@@ -0,0 +1,133 @@
+import datetime
+import sqlite3
+import os
+import logging
+from . import ObjectID, KnowledgeStore
+from enum import Enum
+
+class KnowledgePipelineJournal:
+ def __init__(self, time: datetime.datetime, input: str, parser: str):
+ self.time = time
+ self.input = input
+ self.parser = parser
+
+ def is_finish(self) -> bool:
+ return self.input is None
+
+ def get_input(self) -> str:
+ return self.input
+
+ def get_parser(self) -> str:
+ return self.parser
+
+ def __str__(self) -> str:
+ if self.is_finish():
+ return f"{self.time}: finished)"
+ else:
+ return f"{self.time}: input:{self.input}, parser:{self.parser})"
+
+# init sqlite3 client
+class KnowledgePipelineJournalClient:
+ def __init__(self, pipeline_path: str = None):
+ if not os.path.exists(pipeline_path):
+ os.makedirs(pipeline_path)
+ self.journal_path = os.path.join(pipeline_path, "journal.db")
+
+ conn = sqlite3.connect(self.journal_path)
+ conn.execute(
+ '''CREATE TABLE IF NOT EXISTS journal (
+ id INTEGER PRIMARY KEY AUTOINCREMENT,
+ time DATETIME DEFAULT CURRENT_TIMESTAMP,
+ input TEXT,
+ parser TEXT)'''
+ )
+ conn.commit()
+
+ def insert(self, input: str, parser: str, timestamp: datetime.datetime = None):
+ timestamp = datetime.datetime.now() if timestamp is None else timestamp
+ conn = sqlite3.connect(self.journal_path)
+ conn.execute(
+ "INSERT INTO journal (time, input, parser) VALUES (?, ?, ?)",
+ (timestamp, input, parser),
+ )
+ conn.commit()
+
+ def latest_journals(self, topn) -> [KnowledgePipelineJournal]:
+ conn = sqlite3.connect(self.journal_path)
+ cursor = conn.cursor()
+ cursor.execute("SELECT * FROM journal ORDER BY id DESC LIMIT ?", (topn,))
+ return [KnowledgePipelineJournal(time, input, parser) for (_, time, input, parser) in cursor.fetchall()]
+
+class KnowledgePipelineEnvironment:
+ def __init__(self, pipeline_path: str):
+ self.knowledge_store = KnowledgeStore()
+ if not os.path.exists(pipeline_path):
+ os.makedirs(pipeline_path)
+ self.pipeline_path = pipeline_path
+ self.journal = KnowledgePipelineJournalClient(pipeline_path)
+ self.logger = logging.getLogger()
+
+ def get_journal(self) -> KnowledgePipelineJournalClient:
+ return self.journal
+
+ def get_knowledge_store(self) -> KnowledgeStore:
+ return self.knowledge_store
+
+ def get_logger(self) -> logging.Logger:
+ return self.logger
+
+class KnowledgePipelineState(Enum):
+ INIT = 0
+ RUNNING = 1
+ STOPPED = 2
+ FINISHED = 3
+
+class NullParser:
+ async def parse(self, object_id):
+ return ""
+
+class KnowledgePipeline:
+ def __init__(self, name: str, env: KnowledgePipelineEnvironment, input_init, input_params=None, parser_init=None, parser_params=None):
+ self.name = name
+ self.state = KnowledgePipelineState.INIT
+ self.input_init = input_init
+ self.input_params = input_params
+ self.parser_init = parser_init
+ self.parser_params = parser_params
+ self.env = env
+ self.input = None
+ self.parser = None
+
+ def get_name(self):
+ return self.name
+
+ def get_journal(self) -> KnowledgePipelineJournalClient:
+ return self.env.journal
+
+ async def run(self):
+ if self.state == KnowledgePipelineState.INIT:
+ self.input = self.input_init(self.env, self.input_params)
+ if self.parser_init is None:
+ self.parser = NullParser()
+ else:
+ self.parser = self.parser_init(self.env, self.parser_params)
+ self.state = KnowledgePipelineState.RUNNING
+ if self.state == KnowledgePipelineState.RUNNING:
+ async for input in self.input.next():
+ if input is None:
+ self.state = KnowledgePipelineState.FINISHED
+ self.env.journal.insert(None, None)
+ return
+ (object_id, input_journal) = input
+ if object_id is not None:
+ parser_journal = await self.parser.parse(object_id)
+ self.env.journal.insert(input_journal, parser_journal)
+ else:
+ return
+ if self.state == KnowledgePipelineState.STOPPED:
+ return
+ if self.state == KnowledgePipelineState.FINISHED:
+ return
+
+
+
\ No newline at end of file
diff --git a/src/aios/knowledge/store.py b/src/aios/knowledge/store.py
new file mode 100644
index 0000000..be6a7f6
--- /dev/null
+++ b/src/aios/knowledge/store.py
@@ -0,0 +1,102 @@
+import os
+import json
+import logging
+
+from .object import ObjectStore, ObjectRelationStore, ObjectID, ObjectType, KnowledgeObject
+from .core_object import DocumentObject, ImageObject, VideoObject, RichTextObject, EmailObject
+from .data import ChunkStore, ChunkTracker, ChunkListWriter, ChunkReader
+from ..storage.storage import AIStorage
+
+# KnowledgeStore class, which aggregates ChunkStore, ChunkTracker, and ObjectStore, and is a global singleton that makes it easy to use these three built-in store examples
+class KnowledgeStore:
+ _instance = None
+
+ def __new__(cls):
+ if cls._instance is None:
+ cls._instance = super().__new__(cls)
+
+ knowledge_dir = f"{AIStorage.get_instance().get_myai_dir()}/knowledge/objects"
+
+ if not os.path.exists(knowledge_dir):
+ os.makedirs(knowledge_dir)
+
+ cls._instance.__singleton_init__(knowledge_dir)
+
+ return cls._instance
+
+ def __singleton_init__(self, root_dir: str):
+ logging.info(f"will init knowledge store, root_dir={root_dir}")
+
+ self.root = root_dir
+
+ relation_store_dir = os.path.join(root_dir, "relation")
+ self.relation_store = ObjectRelationStore(relation_store_dir)
+
+ object_store_dir = os.path.join(root_dir, "object")
+ self.object_store = ObjectStore(object_store_dir)
+
+ chunk_store_dir = os.path.join(root_dir, "chunk")
+ self.chunk_store = ChunkStore(chunk_store_dir)
+ self.chunk_tracker = ChunkTracker(chunk_store_dir)
+ self.chunk_list_writer = ChunkListWriter(self.chunk_store, self.chunk_tracker)
+ self.chunk_reader = ChunkReader(self.chunk_store, self.chunk_tracker)
+
+
+ def get_relation_store(self) -> ObjectRelationStore:
+ return self.relation_store
+
+ def get_object_store(self) -> ObjectStore:
+ return self.object_store
+
+ def get_chunk_store(self) -> ChunkStore:
+ return self.chunk_store
+
+ def get_chunk_tracker(self) -> ChunkTracker:
+ return self.chunk_tracker
+
+ def get_chunk_list_writer(self) -> ChunkListWriter:
+ return self.chunk_list_writer
+
+ def get_chunk_reader(self) -> ChunkReader:
+ return self.chunk_reader
+
+ async def insert_object(self, object: KnowledgeObject):
+ self.object_store.put_object(object.calculate_id(), object.encode())
+
+ def load_object(self, object_id: ObjectID) -> KnowledgeObject:
+ if object_id.get_object_type() == ObjectType.Document:
+ return DocumentObject.decode(self.object_store.get_object(object_id))
+ if object_id.get_object_type() == ObjectType.Image:
+ return ImageObject.decode(self.object_store.get_object(object_id))
+ if object_id.get_object_type() == ObjectType.Video:
+ return VideoObject.decode(self.object_store.get_object(object_id))
+ if object_id.get_object_type() == ObjectType.RichText:
+ return RichTextObject.decode(self.object_store.get_object(object_id))
+ if object_id.get_object_type() == ObjectType.Email:
+ return EmailObject.decode(self.object_store.get_object(object_id))
+ else:
+ pass
+
+ def parse_object_in_message(self, message: str) -> KnowledgeObject:
+ # get message's first line
+ logging.info(f"tg parse resp message: {message}")
+ lines = message.split("\n")
+ if len(lines) > 0:
+ message = lines[0]
+ try:
+ desc = json.loads(message)
+ if isinstance(desc, dict):
+ object_id = desc["id"]
+ else:
+ object_id = desc[0]["id"]
+ except Exception as e:
+ return None
+
+ if object_id is not None:
+ return self.load_object(ObjectID.from_base58(object_id))
+
+
+ def bytes_from_object(self, object: KnowledgeObject) -> bytes:
+ if object.get_object_type() == ObjectType.Image:
+ image_object = object
+ return self.get_chunk_reader().read_chunk_list_to_single_bytes(image_object.get_chunk_list())
\ No newline at end of file
diff --git a/src/knowledge/vector/__init__.py b/src/aios/knowledge/vector/__init__.py
similarity index 100%
rename from src/knowledge/vector/__init__.py
rename to src/aios/knowledge/vector/__init__.py
diff --git a/src/knowledge/vector/chroma_store.py b/src/aios/knowledge/vector/chroma_store.py
similarity index 100%
rename from src/knowledge/vector/chroma_store.py
rename to src/aios/knowledge/vector/chroma_store.py
diff --git a/src/knowledge/vector/vector_base.py b/src/aios/knowledge/vector/vector_base.py
similarity index 100%
rename from src/knowledge/vector/vector_base.py
rename to src/aios/knowledge/vector/vector_base.py
diff --git a/src/component/ndn_client/__init__.py b/src/aios/net/__init__.py
similarity index 100%
rename from src/component/ndn_client/__init__.py
rename to src/aios/net/__init__.py
diff --git a/src/component/ndn_client/cid.py b/src/aios/net/cid.py
similarity index 100%
rename from src/component/ndn_client/cid.py
rename to src/aios/net/cid.py
diff --git a/src/component/ndn_client/ndn_client.py b/src/aios/net/ndn_client.py
similarity index 100%
rename from src/component/ndn_client/ndn_client.py
rename to src/aios/net/ndn_client.py
diff --git a/src/component/package_manager/README b/src/aios/package_manager/README
similarity index 100%
rename from src/component/package_manager/README
rename to src/aios/package_manager/README
diff --git a/src/component/package_manager/__init__.py b/src/aios/package_manager/__init__.py
similarity index 100%
rename from src/component/package_manager/__init__.py
rename to src/aios/package_manager/__init__.py
diff --git a/src/component/package_manager/env.py b/src/aios/package_manager/env.py
similarity index 100%
rename from src/component/package_manager/env.py
rename to src/aios/package_manager/env.py
diff --git a/src/component/package_manager/index_db.py b/src/aios/package_manager/index_db.py
similarity index 100%
rename from src/component/package_manager/index_db.py
rename to src/aios/package_manager/index_db.py
diff --git a/src/component/package_manager/index_syncer.py b/src/aios/package_manager/index_syncer.py
similarity index 100%
rename from src/component/package_manager/index_syncer.py
rename to src/aios/package_manager/index_syncer.py
diff --git a/src/component/package_manager/installer.py b/src/aios/package_manager/installer.py
similarity index 98%
rename from src/component/package_manager/installer.py
rename to src/aios/package_manager/installer.py
index 8ab8f5f..98a5f7b 100644
--- a/src/component/package_manager/installer.py
+++ b/src/aios/package_manager/installer.py
@@ -6,7 +6,8 @@ import aiofiles
import os
from typing import Tuple
-from ndn_client import ContentId,NDN_Client
+#from ndn_client import ContentId,NDN_Client
+from ..net import ContentId,NDN_Client
from .pkg import PackageInfo,PackageMediaInfo
from .env import PackageEnv
diff --git a/src/component/package_manager/media_reader.py b/src/aios/package_manager/media_reader.py
similarity index 100%
rename from src/component/package_manager/media_reader.py
rename to src/aios/package_manager/media_reader.py
diff --git a/src/component/package_manager/pkg.py b/src/aios/package_manager/pkg.py
similarity index 100%
rename from src/component/package_manager/pkg.py
rename to src/aios/package_manager/pkg.py
diff --git a/src/aios_kernel/agent_message.py b/src/aios/proto/agent_msg.py
similarity index 50%
rename from src/aios_kernel/agent_message.py
rename to src/aios/proto/agent_msg.py
index c5dc26d..b9665c4 100644
--- a/src/aios_kernel/agent_message.py
+++ b/src/aios/proto/agent_msg.py
@@ -1,10 +1,13 @@
-from enum import Enum
-import uuid
-import time
-import re
+# pylint:disable=E0402
+import json
+import logging
import shlex
-from typing import List
-from .ai_function import FunctionItem
+import uuid
+from enum import Enum
+import time
+from typing import Tuple, List
+
+logger = logging.getLogger(__name__)
class AgentMsgType(Enum):
TYPE_MSG = 0
@@ -40,9 +43,9 @@ class AgentMsg:
def __init__(self,msg_type=AgentMsgType.TYPE_MSG) -> None:
self.msg_id = "msg#" + uuid.uuid4().hex
self.msg_type:AgentMsgType = msg_type
-
+
self.prev_msg_id:str = None
- self.quote_msg_id:str = None
+ self.quote_msg_id:str = None
self.rely_msg_id:str = None # if not none means this is a respone msg
self.session_id:str = None
@@ -61,7 +64,7 @@ class AgentMsg:
self.body_mime:str = None #//default is "text/plain",encode is utf8
#type is call / action
- self.func_name = None
+ self.func_name = None
self.args = None
self.result_str = None
@@ -73,6 +76,17 @@ class AgentMsg:
self.inner_call_chain = []
self.resp_msg = None
+ self.action_list = []
+
+ #context info
+ self.context_info:dict= {}
+
+ @classmethod
+ def from_json(cls,json_obj:dict) -> 'AgentMsg':
+ msg = AgentMsg()
+
+ return msg
+
@classmethod
def create_internal_call_msg(self,func_name:str,args:dict,prev_msg_id:str,caller:str):
msg = AgentMsg(AgentMsgType.TYPE_INTERNAL_CALL)
@@ -82,7 +96,7 @@ class AgentMsg:
msg.prev_msg_id = prev_msg_id
msg.sender = caller
return msg
-
+
def create_action_msg(self,action_name:str,args:dict,caller:str):
msg = AgentMsg(AgentMsgType.TYPE_ACTION)
msg.create_time = time.time()
@@ -92,11 +106,11 @@ class AgentMsg:
msg.topic = self.topic
msg.sender = caller
return msg
-
+
def create_error_resp(self,error_msg:str):
resp_msg = AgentMsg(AgentMsgType.TYPE_SYSTEM)
resp_msg.create_time = time.time()
-
+
resp_msg.rely_msg_id = self.msg_id
resp_msg.body = error_msg
resp_msg.topic = self.topic
@@ -116,27 +130,124 @@ class AgentMsg:
resp_msg.topic = self.topic
return resp_msg
-
+
def create_group_resp_msg(self,sender_id,resp_body):
resp_msg = AgentMsg(AgentMsgType.TYPE_GROUPMSG)
resp_msg.create_time = time.time()
resp_msg.rely_msg_id = self.msg_id
- resp_msg.target = self.target
+ resp_msg.target = self.sender
resp_msg.sender = sender_id
resp_msg.body = resp_body
resp_msg.topic = self.topic
return resp_msg
- def set(self,sender:str,target:str,body:str,topic:str=None) -> None:
+ def set(self,sender:str,target:str,body:str,topic:str=None,body_mime:str=None) -> None:
self.sender = sender
self.target = target
self.body = body
+ self.body_mime = body_mime
self.create_time = time.time()
if topic:
self.topic = topic
+ @staticmethod
+ def create_image_body(images: [str], prompt: str = None):
+ return json.dumps({"images": images, "prompt": prompt}, ensure_ascii=False)
+
+ @staticmethod
+ def parse_image_body(image_body: str) -> Tuple[str, List[str]]:
+ body = json.loads(image_body)
+ return body.get("prompt"), body.get("images")
+
+ @staticmethod
+ def create_video_body(video: str, prompt: str = None):
+ return json.dumps({"video": video, "prompt": prompt}, ensure_ascii=False)
+
+ @staticmethod
+ def parse_video_body(video_body: str) -> Tuple[str, str]:
+ body = json.loads(video_body)
+ return body.get("prompt"), body.get("video")
+
+ @staticmethod
+ def create_audio_body(audio: str, prompt: str = None):
+ return json.dumps({"audio": audio, "prompt": prompt}, ensure_ascii=False)
+
+ @staticmethod
+ def parse_audio_body(audio_body: str) -> Tuple[str, str]:
+ body = json.loads(audio_body)
+ return body.get("prompt"), body.get("audio")
+
+ def set_image(self, sender: str, target: str, image_format: str, images: [str], prompt: str = None, topic: str = None):
+ self.sender = sender
+ self.target = target
+ self.create_time = time.time()
+ self.body_mime = f"image/{image_format}"
+ self.body = self.create_image_body(images, prompt)
+ if topic:
+ self.topic = topic
+
+ def is_image_msg(self) -> bool:
+ if self.body_mime is None:
+ return False
+ if self.body_mime.startswith("image/"):
+ return True
+ return False
+
+ def get_image_body(self) -> Tuple[str, List[str]]:
+ if self.body_mime is None:
+ return None
+ if self.body_mime.startswith("image/"):
+ return self.parse_image_body(self.body)
+ return None
+
+ def set_video(self, sender: str, target: str, video_format: str, video: str, prompt: str = None, topic: str = None):
+ self.sender = sender
+ self.target = target
+ self.create_time = time.time()
+ self.body_mime = f"video/{video_format}"
+ self.body = self.create_video_body(video, prompt)
+ if topic:
+ self.topic = topic
+
+ def get_video_body(self) -> Tuple[str, str]:
+ if self.body_mime is None:
+ return None
+ if self.body_mime.startswith("video/"):
+ return self.parse_video_body(self.body)
+ return None
+
+ def is_video_msg(self) -> bool:
+ if self.body_mime is None:
+ return False
+ if self.body_mime.startswith("video/"):
+ return True
+ return False
+
+ def set_audio(self, sender: str, target: str, audio_format: str, audio: str, prompt: str = None, topic: str = None):
+ self.sender = sender
+ self.target = target
+ self.create_time = time.time()
+ self.body_mime = f"audio/{audio_format}"
+ self.body = self.create_audio_body(audio, prompt)
+ if topic:
+ self.topic = topic
+
+ def get_audio_body(self) -> Tuple[str, str]:
+ if self.body_mime is None:
+ return None
+ if self.body_mime.startswith("audio/"):
+ return self.parse_audio_body(self.body)
+ return None
+
+ def is_audio_msg(self) -> bool:
+ if self.body_mime is None:
+ return False
+ if self.body_mime.startswith("audio/"):
+ return True
+ return False
+
def get_msg_id(self) -> str:
return self.msg_id
@@ -145,25 +256,11 @@ class AgentMsg:
def get_target(self) -> str:
return self.target
-
+
def get_prev_msg_id(self) -> str:
return self.prev_msg_id
-
+
def get_quote_msg_id(self) -> str:
return self.quote_msg_id
-
- @classmethod
- def parse_function_call(cls,func_string:str):
- str_list = shlex.split(func_string)
- func_name = str_list[0]
- params = str_list[1:]
- return func_name, params
-
-class LLMResult:
- def __init__(self) -> None:
- self.state : str = "ignore"
- self.resp : str = ""
- self.post_msgs : List[AgentMsg] = []
- self.send_msgs : List[AgentMsg] = []
- self.calls : List[FunctionItem] = []
- self.post_calls : List[FunctionItem] = []
\ No newline at end of file
+
+
diff --git a/src/aios/proto/agent_task.py b/src/aios/proto/agent_task.py
new file mode 100644
index 0000000..7ddee87
--- /dev/null
+++ b/src/aios/proto/agent_task.py
@@ -0,0 +1,558 @@
+# pylint:disable=E0402
+from abc import ABC, abstractmethod
+import json
+from typing import List, Optional
+import datetime
+import time
+import uuid
+from anyio import Path
+import logging
+from enum import Enum
+from datetime import datetime
+
+logger = logging.getLogger(__name__)
+
+# class AgentTodoResult:
+# TODO_RESULT_CODE_OK = 0,
+# TODO_RESULT_CODE_LLM_ERROR = 1,
+# TODO_RESULT_CODE_EXEC_OP_ERROR = 2
+
+
+# def __init__(self) -> None:
+# self.result_code = AgentTodoResult.TODO_RESULT_CODE_OK
+# self.result_str = None
+# self.error_str = None
+# self.op_list = None
+
+# def to_dict(self) -> dict:
+# result = {}
+# result["result_code"] = self.result_code
+# result["result_str"] = self.result_str
+# result["error_str"] = self.error_str
+# result["op_list"] = self.op_list
+# return result
+
+# class AgentTodo:
+# TODO_STATE_WAIT_ASSIGN = "wait_assign"
+# TODO_STATE_INIT = "init"
+
+# TODO_STATE_PENDING = "pending"
+# TODO_STATE_WAITING_CHECK = "wait_check"
+# TODO_STATE_EXEC_FAILED = "exec_failed"
+# TDDO_STATE_CHECKFAILED = "check_failed"
+
+# TODO_STATE_CASNCEL = "cancel"
+# TODO_STATE_DONE = "done"
+# TODO_STATE_EXPIRED = "expired"
+
+# def __init__(self):
+# self.todo_id = "todo#" + uuid.uuid4().hex
+# self.title = None
+# self.detail = None
+# self.todo_path = None # get parent todo,sub todo by path
+# #self.parent = None
+# self.create_time = time.time()
+
+# self.state = "wait_assign"
+# self.worker = None
+# self.checker = None
+# self.createor = None
+
+# self.need_check = True
+# self.due_date = time.time() + 3600 * 24 * 2
+# self.last_do_time = None
+# self.last_check_time = None
+# self.last_review_time = None
+
+# self.depend_todo_ids = []
+# self.sub_todos = {}
+
+# self.result : AgentTodoResult = None
+# self.last_check_result = None
+# self.retry_count = 0
+# self.raw_obj = None
+
+
+# @classmethod
+# def from_dict(cls,json_obj:dict) -> 'AgentTodo':
+# todo = AgentTodo()
+# if json_obj.get("id") is not None:
+# todo.todo_id = json_obj.get("id")
+
+# todo.title = json_obj.get("title")
+# todo.state = json_obj.get("state")
+# create_time = json_obj.get("create_time")
+# if create_time:
+# todo.create_time = datetime.fromisoformat(create_time).timestamp()
+
+# todo.detail = json_obj.get("detail")
+# due_date = json_obj.get("due_date")
+# if due_date:
+# todo.due_date = datetime.fromisoformat(due_date).timestamp()
+
+# last_do_time = json_obj.get("last_do_time")
+# if last_do_time:
+# todo.last_do_time = datetime.fromisoformat(last_do_time).timestamp()
+# last_check_time = json_obj.get("last_check_time")
+# if last_check_time:
+# todo.last_check_time = datetime.fromisoformat(last_check_time).timestamp()
+# last_review_time = json_obj.get("last_review_time")
+# if last_review_time:
+# todo.last_review_time = datetime.fromisoformat(last_review_time).timestamp()
+
+# todo.depend_todo_ids = json_obj.get("depend_todo_ids")
+# todo.need_check = json_obj.get("need_check")
+# #todo.result = json_obj.get("result")
+# #todo.last_check_result = json_obj.get("last_check_result")
+# todo.worker = json_obj.get("worker")
+# todo.checker = json_obj.get("checker")
+# todo.createor = json_obj.get("createor")
+# if json_obj.get("retry_count"):
+# todo.retry_count = json_obj.get("retry_count")
+
+# todo.raw_obj = json_obj
+
+# return todo
+
+# def to_dict(self) -> dict:
+# if self.raw_obj:
+# result = self.raw_obj
+# else:
+# result = {}
+
+# result["id"] = self.todo_id
+# #result["parent_id"] = self.parent_id
+# result["title"] = self.title
+# result["state"] = self.state
+# result["create_time"] = datetime.fromtimestamp(self.create_time).isoformat()
+# result["detail"] = self.detail
+# result["due_date"] = datetime.fromtimestamp(self.due_date).isoformat()
+# result["last_do_time"] = datetime.fromtimestamp(self.last_do_time).isoformat() if self.last_do_time else None
+# result["last_check_time"] = datetime.fromtimestamp(self.last_check_time).isoformat() if self.last_check_time else None
+# result["last_review_time"] = datetime.fromtimestamp(self.last_review_time).isoformat() if self.last_review_time else None
+# result["depend_todo_ids"] = self.depend_todo_ids
+# result["need_check"] = self.need_check
+# result["worker"] = self.worker
+# result["checker"] = self.checker
+# result["createor"] = self.createor
+# result["retry_count"] = self.retry_count
+
+# return result
+
+# def can_check(self)->bool:
+# if self.state != AgentTodo.TODO_STATE_WAITING_CHECK:
+# return False
+
+# now = datetime.now().timestamp()
+# if self.last_check_time:
+# time_diff = now - self.last_check_time
+# if time_diff < 60*15:
+# logger.info(f"todo {self.title} is already checked, ignore")
+# return False
+
+# return True
+
+# def can_do(self) -> bool:
+# match self.state:
+# case AgentTodo.TODO_STATE_DONE:
+# logger.info(f"todo {self.title} is done, ignore")
+# return False
+# case AgentTodo.TODO_STATE_CASNCEL:
+# logger.info(f"todo {self.title} is cancel, ignore")
+# return False
+# case AgentTodo.TODO_STATE_EXPIRED:
+# logger.info(f"todo {self.title} is expired, ignore")
+# return False
+# case AgentTodo.TODO_STATE_EXEC_FAILED:
+# if self.retry_count > 3:
+# logger.info(f"todo {self.title} retry count ({self.retry_count}) is too many, ignore")
+# return False
+
+# now = datetime.now().timestamp()
+# time_diff = self.due_date - now
+# if time_diff < 0:
+# logger.info(f"todo {self.title} is expired, ignore")
+# self.state = AgentTodo.TODO_STATE_EXPIRED
+# return False
+
+# if time_diff > 7*24*3600:
+# logger.info(f"todo {self.title} is far before due date, ignore")
+# return False
+
+# if self.last_do_time:
+# time_diff = now - self.last_do_time
+# if time_diff < 60*15:
+# logger.info(f"todo {self.title} is already do ignore")
+# return False
+
+# logger.info(f"todo {self.title} can do.")
+# return True
+
+############################################################################################
+class AgentTaskState(Enum):
+ TASK_STATE_WAIT= "wait_assign"
+ TASK_STATE_ASSIGNED = "assigned"
+ TASK_STATE_CONFIRMED = "confirmed"
+
+ TASK_STATE_CANCEL = "cancel"
+ TASK_STATE_EXPIRED = "expired"
+
+ TASK_STATE_DOING = "doing"
+ TASK_STATE_WAITING_REVIEW = "wait_review"
+ TASK_STATE_CHECKFAILED = "check_failed"
+ TASK_STATE_DONE = "done"
+ TASK_STATE_FAILED = "failed"
+ @staticmethod
+ def from_str(value):
+ return next((s for s in AgentTaskState.__members__.values() if s.value == value), None)
+
+class AgentTodoState(Enum):
+ TODO_STATE_WAITING = "waiting"
+ #TODO_STATE_WORKING = "working"
+
+ TODO_STATE_EXEC_OK = "execute_ok"
+ TODO_STATE_EXEC_FAILED = "execute_failed"
+
+ TODO_STATE_CHECK_FAILED = "check_failed"
+ TODO_STATE_DONE = "done"
+ TODO_STATE_FAILED = "failed"
+
+ @staticmethod
+ def from_str(value):
+ return next((s for s in AgentTodoState.__members__.values() if s.value == value), None)
+
+class AgentTodo:
+ def __init__(self) -> None:
+ self.todo_id = "todo#" + uuid.uuid4().hex
+ self.todo_path : str = None
+ self.owner_taskid = None
+ self.title:str = None
+ self.detail:str = None
+ self.state = AgentTodoState.TODO_STATE_WAITING
+ self.category = None
+ self.step_order:int = 0
+
+ def to_dict(self) -> dict:
+ result = {}
+ result["todo_id"] = self.todo_id
+ result["owner_taskid"] = self.owner_taskid
+ result["title"] = self.title
+ result["detail"] = self.detail
+ result["state"] = self.state.value
+ result["category"] = self.category
+ result["step_order"] = self.step_order
+
+ return result
+
+ @classmethod
+ def from_dict(cls,json_obj:dict) -> 'AgentTodo':
+ result_obj = AgentTodo()
+ #todo_id
+ if json_obj.get("todo_id") is not None:
+ result_obj.todo_id = json_obj.get("todo_id")
+ #owner_taskid
+ if json_obj.get("owner_taskid") is not None:
+ result_obj.owner_taskid = json_obj.get("owner_taskid")
+ #name
+ if json_obj.get("title") is not None:
+ result_obj.title = json_obj.get("title")
+ #detail
+ if json_obj.get("detail") is not None:
+ result_obj.detail = json_obj.get("detail")
+ #state
+ if json_obj.get("state") is not None:
+ result_obj.state = AgentTodoState.from_str(json_obj.get("state"))
+ #category
+ if json_obj.get("category") is not None:
+ result_obj.category = json_obj.get("category")
+ #step_order
+ if json_obj.get("step_order") is not None:
+ result_obj.step_order = json_obj.get("step_order")
+
+ return result_obj
+
+
+class AgentTask:
+ def __init__(self) -> None:
+ self.task_id : str = "task#" + uuid.uuid4().hex
+ self.task_path : str = None # get parent todo,sub todo by path
+ self.title = None
+ self.detail = None
+ self.state = AgentTaskState.TASK_STATE_WAIT
+ self.priority:int = 5 # 1-10
+ self.tags:List[str] = []
+ self.worker = None
+ self.createor = None
+
+ now = time.time()
+ self.create_time = datetime.fromtimestamp(now).isoformat()
+ # dead line,set by llm
+ self.due_date = None
+ # set by llm
+ self.next_attention_time = None
+ # set by llm
+ self.expiration_time = None
+
+ self.depend_task_ids = []
+ #self.step_todo_ids = []
+ self.done_time = None
+
+ self.last_plan_time = None
+ self.last_review_time = None
+
+ def is_finish(self) -> bool:
+ if self.state == AgentTaskState.TASK_STATE_DONE:
+ return True
+
+ if self.state == AgentTaskState.TASK_STATE_CANCEL:
+ return True
+
+ if self.state == AgentTaskState.TASK_STATE_EXPIRED:
+ return True
+
+ if self.state == AgentTaskState.TASK_STATE_FAILED:
+ return True
+
+ if self.expiration_time:
+ try:
+ expiration_time = datetime.fromisoformat(self.expiration_time).timestamp()
+ if expiration_time < time.time():
+ self.state = AgentTaskState.TASK_STATE_EXPIRED
+ return True
+ except Exception as e:
+ logger.warning(f"invalid expiration_time {self.expiration_time}")
+
+ return False
+
+ def can_plan(self) -> bool:
+ if self.state == AgentTaskState.TASK_STATE_CONFIRMED or self.state == AgentTaskState.TASK_STATE_CHECKFAILED:
+ if self.next_attention_time:
+ try:
+ next_attention_time = datetime.fromisoformat(self.next_attention_time).timestamp()
+ if time.time() >= next_attention_time:
+ return True
+ except Exception as e:
+ logger.warning(f"invalid next_attention_time {self.next_attention_time}")
+ else:
+ return True
+
+ return False
+
+ def to_simple_dict(self) -> dict:
+ result = {}
+ result["task_id"] = self.task_id
+ result["title"] = self.title
+ result["priority"] = self.priority
+ result["detail"] = self.detail
+ result["create_time"] = self.create_time
+ if self.due_date:
+ result["due_date"] = self.due_date
+ if self.expiration_time:
+ result["expiration_time"] = self.expiration_time
+ if self.next_attention_time:
+ result["next_attention_time"] = self.next_attention_time
+ return result
+
+
+ def to_dict(self) -> dict:
+ result = {}
+ result["task_id"] = self.task_id
+ result["title"] = self.title
+ result["detail"] = self.detail
+ result["state"] = self.state.value
+ result["priority"] = self.priority
+
+ if self.tags:
+ result["tags"] = self.tags
+
+ if self.worker:
+ result["worker"] = self.worker
+
+ if self.createor:
+ result["createor"] = self.createor
+ result["create_time"] = self.create_time
+ if self.due_date:
+ result["due_date"] = self.due_date
+ if self.expiration_time:
+ result["expiration_time"] = self.expiration_time
+
+ if self.next_attention_time:
+ result["next_attention_time"] = self.next_attention_time
+ #if self.next_check_time:
+ # result["next_check_time"] = datetime.fromtimestamp(self.next_check_time).isoformat()
+ if self.depend_task_ids:
+ if len(self.depend_task_ids) > 0:
+ result["depend_task_ids"] = self.depend_task_ids
+
+ if self.done_time:
+ result["done_time"] = self.done_time
+ if self.last_plan_time:
+ result["last_plan_time"] = self.last_plan_time
+ if self.last_review_time:
+ result["last_review_time"] = self.last_review_time
+
+ return result
+ @classmethod
+ def from_dict(cls,json_obj:dict) -> 'AgentTask':
+ result = AgentTask()
+ result.task_id = json_obj.get("task_id")
+ result.title = json_obj.get("title")
+ result.detail = json_obj.get("detail")
+ result.state = AgentTaskState.from_str(json_obj.get("state"))
+ result.priority = json_obj.get("priority")
+ result.tags = json_obj.get("tags")
+ result.worker = json_obj.get("worker")
+ result.createor = json_obj.get("createor")
+ result.due_date = json_obj.get("due_date")
+ result.next_attention_time = json_obj.get("next_attention_time")
+ result.depend_task_ids = json_obj.get("depend_task_ids")
+ #result.step_todo_ids = json_obj.get("step_todo_ids")
+ result.expiration_time = json_obj.get("expiration_time")
+ result.create_time = json_obj.get("create_time")
+ result.done_time = json_obj.get("done_time")
+ result.last_plan_time = json_obj.get("last_plan_time")
+ result.last_review_time = json_obj.get("last_review_time")
+
+ if result.task_id is None or result.title is None or result.create_time is None or result.create_time is None:
+ logger.error(f"invalid task {json_obj}")
+ return None
+
+ return result
+ @classmethod
+ def create_by_dict(cls,json_obj:dict) -> 'AgentTask':
+ creator = json_obj.get("creator")
+ if creator is None:
+ logger.error(f"invalid create task, creator is None")
+ return None
+
+ result = AgentTask()
+
+ result.title = json_obj.get("title")
+ result.detail = json_obj.get("detail")
+ if result.detail is None:
+ result.detail = result.title
+ result.priority = json_obj.get("priority")
+ if result.priority is None:
+ result.priority = 5
+ if json_obj.get("due_date"):
+ result.due_date = json_obj.get("due_date")
+ result.tags = json_obj.get("tags")
+ result.worker = json_obj.get("worker")
+ result.createor = creator
+
+ return result
+
+# 谁在什么时间做了什么
+class AgentWorkLog:
+ # work type : [triage,plan,do,check]
+ def __init__(self) -> None:
+ self.logid = "worklog#" + uuid.uuid4().hex
+ self.owner_id:str = None # taskid or todoid
+ self.work_type:str = "" # 默认为普通类型的log,特殊类型的Log一般伴随着重要的状态改变
+ self.timestamp = time.time()
+ self.content:str = None
+ self.result:str = None
+ self.meta : dict = None
+ self.operator = None
+
+ @classmethod
+ def create_by_content(cls,owner_id:str,work_type:str,content:str,operator:str) -> 'AgentWorkLog':
+ log = AgentWorkLog()
+ log.owner_id = owner_id
+ log.work_type = work_type
+ log.content = content
+ log.operator = operator
+ log.result = "OK"
+ return log
+
+
+
+class AgentTaskManager(ABC):
+ def __init__(self) -> None:
+ pass
+
+ @abstractmethod
+ async def create_task(self,task:AgentTask,parent_id:str = None) -> str:
+ pass
+
+ @abstractmethod
+ async def set_todos(self,owner_task_id:str,todos:List[AgentTodo]):
+ # return todo_id
+ pass
+
+ @abstractmethod
+ async def append_worklog(self,log:AgentWorkLog):
+ pass
+
+ @abstractmethod
+ async def get_worklog(self,obj_id:str)->List[AgentWorkLog]:
+ pass
+
+ @abstractmethod
+ async def get_task(self,task_id:str) -> AgentTask:
+ pass
+
+ #@abstractmethod
+ #async def get_task_by_fullpath(self,task_path:str) -> AgentTask:
+ # pass
+
+ @abstractmethod
+ async def get_todo(self,todo_id:str) -> AgentTodo:
+ pass
+
+ @abstractmethod
+ async def get_sub_tasks(self,task_id:str) -> List[AgentTask]:
+ pass
+
+ @abstractmethod
+ async def get_sub_todos(self,task_id:str) -> List[AgentTodo]:
+ pass
+
+ #@abstractmethod
+ #async def get_task_depends(self,task_id:str) -> List[AgentTask]:
+ # pass
+
+ @abstractmethod
+ async def list_task(self,filter:Optional[dict] = None) -> List[AgentTask]:
+ pass
+
+ @abstractmethod
+ async def update_task(self,task:AgentTask):
+ pass
+
+ @abstractmethod
+ async def update_todo(self,todo:AgentTodo):
+ pass
+
+ #@abstractmethod
+ #async def update_task_state(self,task_id,state:str):
+ # pass
+
+ #@abstractmethod
+ #async def update_todo_state(self,task_id,state:str):
+ # pass
+
+ #subtask,todo共享其所在task的文件夹
+ @abstractmethod
+ async def read_task_file(self,task_id:str,path:str)->str:
+ pass
+
+ @abstractmethod
+ async def write_task_file(self,task_id:str,path:str,content:str):
+ pass
+
+ @abstractmethod
+ async def append_task_file(self,task_id:str,path:str,content:str):
+ pass
+
+ @abstractmethod
+ async def list_task_dir(self,task_id:str,path:str):
+ pass
+
+ @abstractmethod
+ async def remove_task_file(self,task_id:str,path:str):
+ pass
+
+
+
+
diff --git a/src/aios/proto/ai_function.py b/src/aios/proto/ai_function.py
new file mode 100644
index 0000000..fb2ef18
--- /dev/null
+++ b/src/aios/proto/ai_function.py
@@ -0,0 +1,306 @@
+# pylint:disable=E0402
+from abc import ABC, abstractmethod
+from typing import Dict,Coroutine,Callable,List
+
+class ParameterDefine:
+ def __init__(self,name:str,desc:str) -> None:
+ self.name:str = name
+ self.type:str = "string"
+ self.enum:List[str] = None
+ self.description = desc
+ self.is_required = True
+
+ @classmethod
+ def create_parameters(cls,json_obj:dict) -> Dict[str,'ParameterDefine']:
+ result = {}
+ for k,v in json_obj.items():
+ param = ParameterDefine(k,v)
+ result[k] = param
+
+ return result
+
+
+class AIFunction:
+ @abstractmethod
+ def get_id(self) -> str:
+ """
+ return the id of the function (should be snake case)
+ """
+ pass
+
+ @abstractmethod
+ def get_name(self) -> str:
+ """
+ return the name of the function (should be snake case)
+ """
+ pass
+
+ @abstractmethod
+ def get_description(self) -> str:
+ """
+ return a detailed description of what the function does
+ """
+ pass
+
+ def get_detail_description(self) -> str:
+ """
+ return a detailed description of what the function does
+ """
+ parameters = self.get_parameters()
+ parameters_str = ""
+ for k,v in parameters.items():
+ if len(v.description) <= 0:
+ parameters_str +=f"{k},"
+ else:
+ if v.description == k:
+ parameters_str += f"{k},"
+ else:
+ if v.is_required:
+ parameters_str += f"{k}: {v.description},"
+ else:
+ parameters_str += f"{k} (Optional): {v.description},"
+ if len(parameters_str) > 0:
+ return f"{self.get_description()} Parameters: {parameters_str}"
+ return f"f{self.get_description()}, no parameters"
+
+ @abstractmethod
+ def get_parameters(self) -> Dict[str,ParameterDefine]:
+ pass
+
+ def get_openai_parameters(self) -> Dict:
+ """
+ Return the list of parameters to execute this function in the form of
+ JSON schema as specified in the OpenAI documentation:
+ https://platform.openai.com/docs/api-reference/chat/create#chat/create-parameters
+
+ tools = [
+ {
+ "type": "function",
+ "function": {
+ "name": "get_current_weather",
+ "description": "Get the current weather",
+ "parameters": {
+ "type": "object",
+ "properties": {
+ "location": {
+ "type": "string",
+ "description": "The city and state, e.g. San Francisco, CA",
+ },
+ "format": {
+ "type": "string",
+ "enum": ["celsius", "fahrenheit"],
+ "description": "The temperature unit to use. Infer this from the users location.",
+ },
+ },
+ "required": ["location", "format"],
+ },
+ }
+ },
+ {
+ "type": "function",
+ "function": {
+ "name": "get_n_day_weather_forecast",
+ "description": "Get an N-day weather forecast",
+ "parameters": {
+ "type": "object",
+ "properties": {
+ "location": {
+ "type": "string",
+ "description": "The city and state, e.g. San Francisco, CA",
+ },
+ "format": {
+ "type": "string",
+ "enum": ["celsius", "fahrenheit"],
+ "description": "The temperature unit to use. Infer this from the users location.",
+ },
+ "num_days": {
+ "type": "integer",
+ "description": "The number of days to forecast",
+ }
+ },
+ "required": ["location", "format", "num_days"]
+ },
+ }
+ },
+ ]
+
+ """
+ parameters = self.get_parameters()
+ if parameters is not None:
+ result = {}
+ result["type"] = "object"
+ required = []
+ parm_defines = {}
+ for parm_name,parm in parameters.items():
+ parm_item = {}
+ parm_item["type"] = parm.type
+ parm_item["description"] = parm.description
+ if parm.enum is not None:
+ parm_item["enum"] = parm.enum
+ parm_defines[parm_name] = parm_item
+ if parm.is_required:
+ required.append(parm_name)
+ result["properties"] = parm_defines
+ result["required"] = required
+ return result
+
+ return {"type": "object", "properties": {}}
+
+ @abstractmethod
+ async def execute(self, arguments:Dict) -> str:
+ """
+ Execute the function and return a JSON serializable dict by LLM
+ The parameters are passed in the form of kwargs
+
+ [{'id': 'call_fLsKR5vGllhbWxvpqsDT3jBj',
+ 'type': 'function',
+ 'function': {'name': 'get_n_day_weather_forecast',
+ 'arguments': '{"location": "San Francisco, CA", "format": "celsius", "num_days": 4}'}},
+ {'id': 'call_CchlsGE8OE03QmeyFbg7pkDz',
+ 'type': 'function',
+ 'function': {'name': 'get_n_day_weather_forecast',
+ 'arguments': '{"location": "Glasgow", "format": "celsius", "num_days": 4}'}}
+ ]
+ """
+ pass
+
+ @abstractmethod
+ def is_local(self) -> bool:
+ """
+ is this function call need network?
+ """
+ pass
+
+ @abstractmethod
+ def is_in_zone(self) -> bool:
+ """
+ is this function call in Lan?
+ """
+ pass
+
+ @abstractmethod
+ def is_ready_only(self) -> bool:
+ pass
+
+#TODO need to be upgrade
+class ActionNode:
+ def __init__(self,name:str,args:List[str]) -> None:
+ self.name:str= name
+ self.args:List[str]= args
+ self.body:str = None
+ self.parms : Dict = None
+
+ def append_body(self,body:str) -> None:
+ if self.body is None:
+ self.body = body
+ else:
+ self.body += body
+
+ def dumps(self) -> str:
+ pass
+
+ @classmethod
+ def from_json(cls,json_obj:dict) -> 'ActionNode':
+ args = json_obj.get("args",[])
+ r = ActionNode(json_obj["name"],args)
+ if json_obj.get("body"):
+ r.body = json_obj["body"]
+ r.parms = json_obj
+
+ return r
+
+
+class SimpleAIFunction(AIFunction):
+ def __init__(self,func_id:str,description:str,func_handler:Coroutine,parameters:Dict[str,ParameterDefine] = None) -> None:
+ self.func_id = func_id
+ self.description = description
+ self.func_handler = func_handler
+ self.parameters:Dict[str,ParameterDefine] = parameters
+
+ def get_id(self) -> str:
+ return self.func_id
+
+ def get_name(self) -> str:
+ return self.func_id.split('.')[-1].strip()
+
+ def get_description(self) -> str:
+ return self.description
+
+ def get_parameters(self) -> Dict[str,ParameterDefine]:
+ return self.parameters
+
+ async def execute(self,parameters:Dict) -> str:
+ if self.func_handler is None:
+ return f"error: function {self.func_id} not implemented"
+
+ return await self.func_handler(parameters)
+
+ def is_local(self) -> bool:
+ return True
+
+ def is_in_zone(self) -> bool:
+ return True
+
+ def is_ready_only(self) -> bool:
+ return False
+
+class AIAction:
+ @abstractmethod
+ def get_id(self) -> str:
+ """
+ return the name of the operation (should be snake case)
+ """
+ pass
+
+ def get_name(self)->str:
+ return self.get_id().split('.')[-1].strip()
+
+
+ @abstractmethod
+ def get_description(self) -> str:
+ """
+ return a detailed description of what the operation does
+ """
+ pass
+
+
+ @abstractmethod
+ async def execute(self, params: dict) -> str:
+ """
+ Execute the function and return a JSON serializable dict.
+ The parameters are passed in the form of kwargs
+ """
+ pass
+
+class SimpleAIAction(AIAction):
+ def __init__(self,op:str,description:str,func_handler:Coroutine) -> None:
+ self.op = op
+ self.description = description
+ self.func_handler = func_handler
+
+ def get_id(self) -> str:
+ return self.op
+
+ def get_description(self) -> str:
+ return self.description
+
+ async def execute(self, params: Dict) -> str:
+ if self.func_handler is None:
+ return "error: function not implemented"
+
+ return await self.func_handler(params)
+
+
+class AIFunction2Action(AIAction):
+ def __init__(self, func: AIFunction) -> None:
+ super().__init__()
+ self.ai_func = func
+
+ def get_id(self) -> str:
+ return self.ai_func.get_id()
+
+ def get_description(self) -> str:
+ return self.ai_func.get_detail_description()
+
+ async def execute(self, params: dict) -> str:
+ return await self.ai_func.execute(params)
\ No newline at end of file
diff --git a/src/aios/proto/compute_task.py b/src/aios/proto/compute_task.py
new file mode 100644
index 0000000..f7f4371
--- /dev/null
+++ b/src/aios/proto/compute_task.py
@@ -0,0 +1,377 @@
+# pylint:disable=E0402
+import copy
+from enum import Enum
+import json
+import shlex
+import uuid
+import time
+from typing import List, Union,Dict
+from .ai_function import AIFunction,ActionNode
+from .agent_msg import AgentMsg
+from ..knowledge import ObjectID
+from ..storage.storage import AIStorage
+
+
+import logging
+
+logger = logging.getLogger(__name__)
+
+class ComputeTaskResultCode(Enum):
+ OK = 0
+ TIMEOUT = 1
+ NO_WORKER = 2
+ ERROR = 3
+
+
+class ComputeTaskState(Enum):
+ DONE = 0
+ INIT = 1
+ RUNNING = 2
+ ERROR = 3
+ PENDING = 4
+
+class ComputeTaskType(Enum):
+ NONE = "None"
+ LLM_COMPLETION = "llm_completion"
+ TEXT_EMBEDDING ="text_embedding"
+ IMAGE_EMBEDDING ="image_embedding"
+
+ TEXT_2_IMAGE = "text_2_image"
+ IMAGE_2_TEXT = "image_2_text"
+ IMAGE_2_IMAGE = "image_2_image"
+ VOICE_2_TEXT = "voice_2_text"
+ TEXT_2_VOICE = "text_2_voice"
+
+
+# class Function(TypedDict, total=False):
+# name: Required[str]
+# """The name of the function to be called.
+
+# Must be a-z, A-Z, 0-9, or contain underscores and dashes, with a maximum length
+# of 64.
+# """
+
+# parameters: Required[shared_params.FunctionParameters]
+# """The parameters the functions accepts, described as a JSON Schema object.
+
+# See the [guide](https://platform.openai.com/docs/guides/gpt/function-calling)
+# for examples, and the
+# [JSON Schema reference](https://json-schema.org/understanding-json-schema/) for
+# documentation about the format.
+
+# To describe a function that accepts no parameters, provide the value
+# `{"type": "object", "properties": {}}`.
+# """
+
+# description: str
+# """
+# A description of what the function does, used by the model to choose when and
+# how to call the function.
+# """
+
+class LLMPrompt:
+ def __init__(self,prompt_str = None) -> None:
+ self.messages : List[Dict] = []
+ if prompt_str:
+ self.messages.append({"role":"user","content":prompt_str})
+ self.system_message : Dict = None
+ self.inner_functions : List[Dict] = []
+
+ def append_system_message(self,content:str):
+ if content is None:
+ return
+
+ if self.system_message is None:
+ self.system_message = {"role":"system","content":content}
+ else:
+ self.system_message["content"] += content
+
+ def append_user_message(self,content:str):
+ if content is None:
+ return
+
+ self.messages.append({"role":"user","content":content})
+
+ def as_str(self)->str:
+ result_str = ""
+ if self.system_message:
+ result_str = json.dumps(self.system_message,ensure_ascii=False)
+ if self.messages:
+ result_str += json.dumps(self.messages,ensure_ascii=False)
+ if self.inner_functions:
+ result_str += json.dumps(self.inner_functions,ensure_ascii=False)
+
+ return result_str
+
+ def to_message_list(self):
+ result = []
+ if self.system_message:
+ result.append(self.system_message)
+ result.extend(self.messages)
+ return result
+
+
+
+ def append(self,prompt:'LLMPrompt'):
+ if prompt is None:
+ return
+
+ if prompt.inner_functions:
+ if self.inner_functions is None:
+ self.inner_functions = copy.deepcopy(prompt.inner_functions)
+ else:
+ self.inner_functions.extend(prompt.inner_functions)
+
+ if prompt.system_message is not None:
+ if self.system_message is None:
+ self.system_message = copy.deepcopy(prompt.system_message)
+ else:
+ self.system_message["content"] += prompt.system_message.get("content")
+
+ self.messages.extend(prompt.messages)
+
+ def load_from_config(self,config:List[Dict]) -> bool:
+ if isinstance(config,list) is not True:
+ logger.error("prompt is not list!")
+ return False
+ self.messages : List[Dict] = []
+ for msg in config:
+ if msg.get("content"):
+ if msg.get("role") == "system":
+ self.system_message = msg
+ else:
+ self.messages.append(msg)
+ else:
+ logger.error("prompt message has no content!")
+ return True
+
+
+class LLMResultStates(Enum):
+ IGNORE = "ignore"
+ OK = "ok" # process done
+ ERROR = "error"
+
+class LLMResult:
+ def __init__(self) -> None:
+ self.state : str = LLMResultStates.IGNORE
+ self.compute_error_str = None
+ self.resp : str = "" # llm say:
+ self.raw_result = None # raw result from compute kernel
+ #self.inner_functions : List[AIFunction] = []
+ self.action_list : List[ActionNode] = [] # op_list is a optimize design for saving token
+
+
+ @classmethod
+ def from_error_str(self,error_str:str) -> 'LLMResult':
+ r = LLMResult()
+ r.state = LLMResultStates.ERROR
+ r.error_str = error_str
+ return r
+
+ @classmethod
+ def from_json_str(self,llm_json_str:str) -> 'LLMResult':
+ r = LLMResult()
+ if llm_json_str is None:
+ r.state = LLMResultStates.IGNORE
+ return r
+ if llm_json_str == "**IGNORE**":
+ r.state = LLMResultStates.IGNORE
+ return r
+
+ r.state = LLMResultStates.OK
+
+ llm_json = json.loads(llm_json_str)
+ r.resp = llm_json.get("resp")
+ r.raw_result = llm_json
+ action_list = llm_json.get("actions")
+ if action_list:
+ for action in action_list:
+ action_item = ActionNode.from_json(action)
+ if action_item:
+ r.action_list.append(action_item)
+
+ return r
+
+ @classmethod
+ def parse_action(cls,func_string:str):
+ str_list = shlex.split(func_string)
+ func_name = str_list[0]
+ params = str_list[1:]
+ return func_name, params
+
+ @classmethod
+ def from_str(self,llm_result_str:str,valid_func:List[str]=None) -> 'LLMResult':
+ r = LLMResult()
+
+ if llm_result_str is None:
+ r.state = LLMResultStates.IGNORE
+ return r
+ if llm_result_str == "**IGNORE**":
+ r.state = LLMResultStates.IGNORE
+ return r
+
+ try:
+ if llm_result_str[0] == "{":
+ return LLMResult.from_json_str(llm_result_str)
+
+ if llm_result_str.lstrip().rstrip().startswith("```json"):
+ return LLMResult.from_json_str(llm_result_str[7:-3])
+ except:
+ pass
+
+ lines = llm_result_str.splitlines()
+ is_need_wait = False
+
+ def check_args(action_item:ActionNode):
+ match action_item.name:
+ case "post_msg":# /post_msg $target_id
+ if len(action_item.args) != 1:
+ return False
+
+ new_msg = AgentMsg()
+ target_id = action_item.args[0]
+ msg_content = action_item.body
+ new_msg.set("",target_id,msg_content)
+
+ return True
+
+
+ return False
+
+
+ current_action : ActionNode = None
+ for line in lines:
+ if line.startswith("##/"):
+ if current_action:
+ if check_args(current_action) is False:
+ r.resp += current_action.dumps()
+ else:
+ r.action_list.append(current_action)
+
+ action_name,action_args = LLMResult.parse_action(line[3:])
+ current_action = ActionNode(action_name,action_args)
+ else:
+ if current_action:
+ current_action.append_body(line + "\n")
+ else:
+ r.resp += line + "\n"
+
+ if current_action:
+ if check_args(current_action) is False:
+ r.resp += current_action.dumps()
+ else:
+ r.action_list.append(current_action)
+
+ r.state = LLMResultStates.OK
+ return r
+
+class ComputeTask:
+ def __init__(self) -> None:
+ self.task_type = ComputeTaskType.NONE
+ self.create_time = None
+
+ self.task_id: str = None
+ self.callchain_id: str = None
+ self.params: dict = {}
+ self.refers: dict = None
+ self.pading_data: bytearray = None
+
+ self.state = ComputeTaskState.INIT
+ self.result = None
+ self.error_str = None
+
+ def set_llm_params(self, prompts, resp_mode,model_name, max_token_size, inner_functions = None, callchain_id=None):
+ self.task_type = ComputeTaskType.LLM_COMPLETION
+ self.create_time = time.time()
+ self.task_id = uuid.uuid4().hex
+ self.callchain_id = callchain_id
+ self.params["prompts"] = prompts.to_message_list()
+ self.params["resp_mode"] = resp_mode
+ if model_name is None:
+ model_name = AIStorage.get_instance().get_user_config().get_value("llm_model_name")
+ self.params["model_name"] = model_name
+ if max_token_size is None:
+ self.params["max_token_size"] = 4000
+ else:
+ self.params["max_token_size"] = max_token_size
+
+ if inner_functions is not None:
+ self.params["inner_functions"] = inner_functions
+
+ def set_text_embedding_params(self, input: str, model_name=None, callchain_id = None):
+ self.task_type = ComputeTaskType.TEXT_EMBEDDING
+ self.create_time = time.time()
+ self.task_id = uuid.uuid4().hex
+ self.callchain_id = callchain_id
+ if model_name is not None:
+ self.params["model_name"] = model_name
+ else:
+ self.params["model_name"] = "text-embedding-ada-002"
+ self.params["input"] = input
+
+ def set_image_embedding_params(self, input = Union[ObjectID, bytes], model_name=None, callchain_id = None):
+ self.task_type = ComputeTaskType.IMAGE_EMBEDDING
+ self.create_time = time.time()
+ self.task_id = uuid.uuid4().hex
+ self.callchain_id = callchain_id
+ if model_name is not None:
+ self.params["model_name"] = model_name
+ else:
+ self.params["model_name"] = None
+ self.params["input"] = input
+
+ def set_text_2_image_params(self, prompt: str, model_name, negative_prompt="", callchain_id=None):
+ self.task_type = ComputeTaskType.TEXT_2_IMAGE
+ self.create_time = time.time()
+ self.task_id = uuid.uuid4().hex
+ self.callchain_id = callchain_id
+ self.params["prompt"] = prompt
+ self.params["negative_prompt"] = negative_prompt
+ if model_name is not None:
+ self.params["model_name"] = model_name
+ else:
+ self.params["model_name"] = "v1-5-pruned-emaonly"
+
+ def set_image_2_text_params(self, image_path: str, prompt: str, model_name, negative_prompt="", callchain_id=None):
+ self.task_type = ComputeTaskType.IMAGE_2_TEXT
+ self.create_time = time.time()
+ self.task_id = uuid.uuid4().hex
+ self.callchain_id = callchain_id
+ self.params["image_path"] = image_path
+ if prompt == '':
+ self.params["prompt"] = "What's in this image?"
+ else:
+ self.params["prompt"] = prompt
+ self.params["negative_prompt"] = negative_prompt
+ if model_name is not None:
+ self.params["model_name"] = model_name
+ else:
+ self.params["model_name"] = "gpt-4-vision-preview"
+
+
+ def display(self) -> str:
+ return f"ComputeTask: {self.task_id} {self.task_type} {self.state}"
+
+
+class ComputeTaskResult:
+ def __init__(self) -> None:
+ self.create_time = None
+ self.task_id: str = None
+ self.callchain_id: str = None
+ self.worker_id: str = None
+ self.error_str : str = None
+ self.result_code: int = ComputeTaskResultCode.OK
+ self.result_str: str = None # easy to use,can read from result
+
+ self.result : dict = {}
+
+ self.result_refers: dict = {}
+ self.pading_data: bytearray = None
+
+
+ def set_from_task(self, task: ComputeTask):
+ self.task_id = task.task_id
+ self.callchain_id = task.callchain_id
+ task.result = self
+
+
diff --git a/src/aios_kernel/storage.py b/src/aios/storage/storage.py
similarity index 86%
rename from src/aios_kernel/storage.py
rename to src/aios/storage/storage.py
index 2d83af6..9d098ad 100644
--- a/src/aios_kernel/storage.py
+++ b/src/aios/storage/storage.py
@@ -40,6 +40,17 @@ class UserConfig:
self.config_table = {}
self.user_config_path:str = None
+ self._init_default_value("llm_model_name","gpt-4-turbo-preview")
+
+ def _init_default_value(self,key:str,value:Any) -> None:
+ if self.config_table.get(key) is not None:
+ logger.warning("user config key %s already exist, will be overrided",key)
+
+ new_config_item = UserConfigItem()
+ new_config_item.default_value = value
+ new_config_item.is_optional = True
+ self.config_table[key] = new_config_item
+
def add_user_config(self,key:str,desc:str,is_optional:bool,default_value:Any=None,item_type="str") -> None:
if self.config_table.get(key) is not None:
@@ -145,7 +156,7 @@ class AIStorage:
self.feature_init_results = {}
async def initial(self)->bool:
- self.user_config.user_config_path = str(self.get_myai_dir() / "etc/system.cfg.toml")
+ self.user_config.user_config_path = str(self.get_myai_dir() + "/etc/system.cfg.toml")
await self.user_config.load_value_from_file(self.get_system_dir() + "/system.cfg.toml")
await self.user_config.load_value_from_file(self.user_config.user_config_path,True)
@@ -184,7 +195,7 @@ class AIStorage:
/opt/aios
"""
if self.is_dev_mode:
- return os.path.abspath(_file_dir + "/../")
+ return os.path.abspath(_file_dir + "/../../")
else:
return "/opt/aios/"
@@ -195,7 +206,7 @@ class AIStorage:
/opt/aios/app
"""
if self.is_dev_mode:
- return os.path.abspath(_file_dir + "/../../rootfs/")
+ return os.path.abspath(_file_dir + "/../../../rootfs/")
else:
return "/opt/aios/app/"
@@ -204,7 +215,14 @@ class AIStorage:
my ai dir is the dir for user to store their ai app and data
~/myai/
"""
- return Path.home() / "myai"
+ return f"{Path.home()}/myai"
+
+ def get_download_dir(self) -> str:
+ """
+ download dir is the dir for user to store the files downloaded with the system.
+ ~/myai/download
+ """
+ return f"{self.get_myai_dir()}/download"
def get_db(self,app_name:str)->ResourceLocation:
pass
@@ -232,4 +250,9 @@ class AIStorage:
except Exception as e:
logger.error(f"open or create file {path} failed! {str(e)}")
+ @staticmethod
+ def ensure_directory_exists(directory_path):
+ if not os.path.exists(directory_path):
+ os.makedirs(directory_path)
+ logger.info(f"Directory created: {directory_path}")
diff --git a/src/aios/utils/__init__.py b/src/aios/utils/__init__.py
new file mode 100644
index 0000000..dbaab83
--- /dev/null
+++ b/src/aios/utils/__init__.py
@@ -0,0 +1,2 @@
+from . import image_utils
+from . import video_utils
diff --git a/src/aios/utils/image_utils.py b/src/aios/utils/image_utils.py
new file mode 100644
index 0000000..3d07b13
--- /dev/null
+++ b/src/aios/utils/image_utils.py
@@ -0,0 +1,40 @@
+import base64
+import os.path
+from typing import Tuple
+
+import cv2
+
+
+def to_base64(image_path: str, resize: Tuple[int, int] = None) -> str:
+ """Convert image to base64."""
+ ext = os.path.splitext(image_path)[1][1:]
+ if resize is None:
+ with open(image_path, "rb") as image_file:
+ base64_image = base64.b64encode(image_file.read()).decode("utf-8")
+ return f"data:image/{ext};base64,{base64_image}"
+ else:
+ dest_width, dest_height = resize
+ img = cv2.imread(image_path)
+ width, height = img.shape[:2]
+ if width > dest_width or height > dest_height:
+ width_rate = dest_width / width
+ height_rate = dest_height / height
+ rate = min(width_rate, height_rate)
+ dest_width = int(width * rate)
+ dest_height = int(height * rate)
+ img = cv2.resize(img, (dest_width, dest_height), interpolation=cv2.INTER_AREA)
+ _, buf = cv2.imencode(f".{ext}", img)
+ base64_image = base64.b64encode(buf).decode("utf-8")
+ return f"data:image/{ext};base64,{base64_image}"
+
+
+def is_file(image_path: str) -> bool:
+ return os.path.isfile(image_path)
+
+
+def is_base64(image_path: str) -> bool:
+ return image_path.startswith("data:image/")
+
+
+def is_url(image_path: str) -> bool:
+ return image_path.startswith("http://") or image_path.startswith("https://")
diff --git a/src/aios/utils/video_utils.py b/src/aios/utils/video_utils.py
new file mode 100644
index 0000000..15cbb4e
--- /dev/null
+++ b/src/aios/utils/video_utils.py
@@ -0,0 +1,128 @@
+import base64
+from typing import List, Tuple
+
+import cv2
+import numpy as np
+import moviepy.editor as mp
+
+
+def precess_image(image):
+ '''
+ Graying and GaussianBlur
+ :param image: The image matrix,np.array
+ :return: The processed image matrix,np.array
+ '''
+ gray_image = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
+ gray_image = cv2.GaussianBlur(gray_image, (3, 3), 0)
+ return gray_image
+
+
+def abs_diff(pre_image, curr_image):
+ '''
+ Calculate absolute difference between pre_image and curr_image
+ :param pre_image:The image in past frame,np.array
+ :param curr_image:The image in current frame,np.array
+ :return:
+ '''
+ gray_pre_image = precess_image(pre_image)
+ gray_curr_image = precess_image(curr_image)
+ diff = cv2.absdiff(gray_pre_image, gray_curr_image)
+ res, diff = cv2.threshold(diff, 0, 255, cv2.THRESH_BINARY+cv2.THRESH_OTSU)
+ cnt_diff = np.sum(np.sum(diff))
+ return cnt_diff
+
+
+def exponential_smoothing(alpha, s):
+ '''
+ Primary exponential smoothing
+ :param alpha: Smoothing factor,num
+ :param s: List of data,list
+ :return: List of data after smoothing,list
+ '''
+ s_temp = [s[0]]
+ print(s_temp)
+ for i in range(1, len(s), 1):
+ s_temp.append(alpha * s[i - 1] + (1 - alpha) * s_temp[i - 1])
+ return s_temp
+
+
+def extract_frames(video_path: str, resize: Tuple[int, int] = None, smooth=False, alpha=0.07, window=25) -> List[str]:
+ """Extract frames from video."""
+ frames = []
+ vidcap = cv2.VideoCapture(video_path)
+ diff = []
+ frm = 0
+ pre_image = np.array([])
+ cur_image = np.array([])
+
+ while True:
+ frm = frm + 1
+ success, image = vidcap.read()
+ if not success:
+ break
+
+ if frm == 1:
+ pre_image = image
+ cur_image = image
+ else:
+ pre_image = cur_image
+ cur_image = image
+
+ diff.append(abs_diff(pre_image, cur_image))
+
+ if smooth:
+ diff = exponential_smoothing(alpha, diff)
+
+ diff = np.array(diff)
+ mean = np.mean(diff)
+ dev = np.std(diff)
+ diff = (diff - mean) / dev
+
+ idx = []
+ for i, d in enumerate(diff):
+ ub = len(diff) - 1
+ lb = 0
+ if not i - window // 2 < lb:
+ lb = i - window // 2
+ if not i + window // 2 > ub:
+ ub = i + window // 2
+
+ comp_window = diff[lb: ub]
+ if d >= max(comp_window):
+ idx.append(i)
+
+ tmp = np.array(idx)
+ tmp = tmp + 1
+ idx = set(tmp.tolist())
+ vidcap.release()
+
+ vidcap = cv2.VideoCapture(video_path)
+ i = 0
+ frm = 0
+ while vidcap.isOpened() and i < 10:
+ frm = frm + 1
+ success, image = vidcap.read()
+ if not success:
+ break
+ if frm not in idx:
+ continue
+ if resize is not None:
+ dest_width, dest_height = resize
+ width, height = image.shape[:2]
+ if width > dest_width or height > dest_height:
+ width_rate = dest_width / width
+ height_rate = dest_height / height
+ rate = min(width_rate, height_rate)
+ dest_width = int(width * rate)
+ dest_height = int(height * rate)
+ image = cv2.resize(image, (dest_width, dest_height), interpolation=cv2.INTER_AREA)
+ _, buffer = cv2.imencode(".jpg", image)
+ frames.append(f"data:image/jpg;base64,{base64.b64encode(buffer).decode('utf-8')}")
+ i += 1
+ vidcap.release()
+ return frames
+
+
+def extract_audio(video_path: str, audio_path: str):
+ my_clip = mp.VideoFileClip(video_path)
+ my_clip.audio.write_audiofile(audio_path)
diff --git a/src/aios_kernel/__init__.py b/src/aios_kernel/__init__.py
deleted file mode 100644
index 42d1b65..0000000
--- a/src/aios_kernel/__init__.py
+++ /dev/null
@@ -1,29 +0,0 @@
-from .environment import Environment,EnvironmentEvent
-from .agent_message import AgentMsg,AgentMsgStatus,AgentMsgType
-from .chatsession import AIChatSession
-from .agent import AIAgent,AIAgentTemplete,AgentPrompt
-from .compute_kernel import ComputeKernel,ComputeTask,ComputeTaskResult,ComputeTaskState,ComputeTaskType
-from .compute_node import ComputeNode,LocalComputeNode
-from .open_ai_node import OpenAI_ComputeNode
-from .knowledge_base import KnowledgeBase, KnowledgeEnvironment
-from .knowledge_pipeline import KnowledgeEmailSource, KnowledgeDirSource, KnowledgePipline
-from .role import AIRole,AIRoleGroup
-from .workflow import Workflow
-from .bus import AIBus
-from .workflow_env import WorkflowEnvironment,CalenderEnvironment,CalenderEvent,PaintEnvironment
-from .local_llama_compute_node import LocalLlama_ComputeNode
-from .whisper_node import WhisperComputeNode
-from .google_text_to_speech_node import GoogleTextToSpeechNode
-from .tunnel import AgentTunnel
-from .tg_tunnel import TelegramTunnel
-from .email_tunnel import EmailTunnel
-from .storage import ResourceLocation,AIStorage,UserConfig,UserConfigItem
-from .contact_manager import ContactManager,Contact,FamilyMember
-from .text_to_speech_function import TextToSpeechFunction
-from .workspace_env import WorkspaceEnvironment
-from .local_stability_node import Local_Stability_ComputeNode
-from .stability_node import Stability_ComputeNode
-from .local_st_compute_node import LocalSentenceTransformer_Text_ComputeNode,LocalSentenceTransformer_Image_ComputeNode
-from .compute_node_config import ComputeNodeConfig
-AIOS_Version = "0.5.1, build 2023-9-28"
-
diff --git a/src/aios_kernel/agent.py b/src/aios_kernel/agent.py
deleted file mode 100644
index 563d49e..0000000
--- a/src/aios_kernel/agent.py
+++ /dev/null
@@ -1,702 +0,0 @@
-from typing import Optional
-
-from asyncio import Queue
-import asyncio
-import logging
-import uuid
-import time
-import json
-import shlex
-import datetime
-import copy
-
-from .agent_message import AgentMsg, AgentMsgStatus, AgentMsgType,FunctionItem,LLMResult
-from .chatsession import AIChatSession
-from .compute_task import ComputeTaskResult,ComputeTaskResultCode
-from .ai_function import AIFunction
-from .environment import Environment
-from .contact_manager import ContactManager,Contact,FamilyMember
-
-logger = logging.getLogger(__name__)
-
-class AgentPrompt:
- def __init__(self,prompt_str = None) -> None:
- self.messages = []
- if prompt_str:
- self.messages.append({"role":"user","content":prompt_str})
- self.system_message = None
-
- def as_str(self)->str:
- result_str = ""
- if self.system_message:
- result_str += self.system_message.get("role") + ":" + self.system_message.get("content") + "\n"
- if self.messages:
- for msg in self.messages:
- result_str += msg.get("role") + ":" + msg.get("content") + "\n"
-
- return result_str
-
- def to_message_list(self):
- result = []
- if self.system_message:
- result.append(self.system_message)
- result.extend(self.messages)
- return result
-
- def append(self,prompt):
- if prompt is None:
- return
-
- if prompt.system_message is not None:
- if self.system_message is None:
- self.system_message = copy.deepcopy(prompt.system_message)
- else:
- self.system_message["content"] += prompt.system_message.get("content")
-
- self.messages.extend(prompt.messages)
-
- def get_prompt_token_len(self):
- result = 0
-
- if self.system_message:
- result += len(self.system_message.get("content"))
- for msg in self.messages:
- result += len(msg.get("content"))
-
- return result
-
- def load_from_config(self,config:list) -> bool:
- if isinstance(config,list) is not True:
- logger.error("prompt is not list!")
- return False
- self.messages = []
- for msg in config:
- if msg.get("role") == "system":
- self.system_message = msg
- else:
- self.messages.append(msg)
- return True
-
-
-class AIAgentTemplete:
- def __init__(self) -> None:
- self.llm_model_name:str = "gpt-4-0613"
- self.max_token_size:int = 0
- self.template_id:str = None
- self.introduce:str = None
- self.author:str = None
- self.prompt:AgentPrompt = None
-
- def load_from_config(self,config:dict) -> bool:
- if config.get("llm_model_name") is not None:
- self.llm_model_name = config["llm_model_name"]
- if config.get("max_token_size") is not None:
- self.max_token_size = config["max_token_size"]
- if config.get("template_id") is not None:
- self.template_id = config["template_id"]
- if config.get("prompt") is not None:
- self.prompt = AgentPrompt()
- if self.prompt.load_from_config(config["prompt"]) is False:
- logger.error("load prompt from config failed!")
- return False
-
-
- return True
-
-
-class AIAgent:
- def __init__(self) -> None:
- self.agent_prompt:AgentPrompt = None
- self.agent_think_prompt:AgentPrompt = None
- self.llm_model_name:str = None
- self.max_token_size:int = 3600
- self.agent_id:str = None
- self.template_id:str = None
- self.fullname:str = None
- self.powerby = None
- self.enable = True
- self.enable_kb = False
- self.enable_timestamp = False
- self.guest_prompt_str = None
- self.owner_promp_str = None
- self.contact_prompt_str = None
- self.history_len = 10
-
- self.chat_db = None
- self.unread_msg = Queue() # msg from other agent
- self.owner_env : Environment = None
- self.owenr_bus = None
- self.enable_function_list = None
-
- @classmethod
- def create_from_templete(cls,templete:AIAgentTemplete, fullname:str):
- # Agent just inherit from templete on craete,if template changed,agent will not change
- result_agent = AIAgent()
- result_agent.llm_model_name = templete.llm_model_name
- result_agent.max_token_size = templete.max_token_size
- result_agent.template_id = templete.template_id
- result_agent.agent_id = "agent#" + uuid.uuid4().hex
- result_agent.fullname = fullname
- result_agent.powerby = templete.author
- result_agent.agent_prompt = templete.prompt
- return result_agent
-
- def load_from_config(self,config:dict) -> bool:
- if config.get("instance_id") is None:
- logger.error("agent instance_id is None!")
- return False
- self.agent_id = config["instance_id"]
-
- if config.get("fullname") is None:
- logger.error(f"agent {self.agent_id} fullname is None!")
- return False
- self.fullname = config["fullname"]
-
- if config.get("prompt") is not None:
- self.agent_prompt = AgentPrompt()
- self.agent_prompt.load_from_config(config["prompt"])
-
- if config.get("think_prompt") is not None:
- self.agent_think_prompt = AgentPrompt()
- self.agent_think_prompt.load_from_config(config["think_prompt"])
-
- if config.get("guest_prompt") is not None:
- self.guest_prompt_str = config["guest_prompt"]
-
- if config.get("owner_prompt") is not None:
- self.owner_promp_str = config["owner_prompt"]
-
- if config.get("contact_prompt") is not None:
- self.contact_prompt_str = config["contact_prompt"]
-
- if config.get("owner_env") is not None:
- self.owner_env = Environment.get_env_by_id(config["owner_env"])
-
- if config.get("powerby") is not None:
- self.powerby = config["powerby"]
- if config.get("template_id") is not None:
- self.template_id = config["template_id"]
- if config.get("llm_model_name") is not None:
- self.llm_model_name = config["llm_model_name"]
- if config.get("max_token_size") is not None:
- self.max_token_size = config["max_token_size"]
- if config.get("enable_function") is not None:
- self.enable_function_list = config["enable_function"]
- if config.get("enable_kb") is not None:
- self.enable_kb = bool(config["enable_kb"])
- if config.get("enable_timestamp") is not None:
- self.enable_timestamp = bool(config["enable_timestamp"])
- if config.get("history_len"):
- self.history_len = int(config.get("history_len"))
- return True
-
-
- def _get_llm_result_type(self,llm_result_str:str) -> LLMResult:
- r = LLMResult()
- if llm_result_str is None:
- r.state = "ignore"
- return r
- if llm_result_str == "ignore":
- r.state = "ignore"
- return r
-
- lines = llm_result_str.splitlines()
- is_need_wait = False
-
- def check_args(func_item:FunctionItem):
- match func_name:
- case "send_msg":# sendmsg($target_id,$msg_content)
- if len(func_args) != 1:
- logger.error(f"parse sendmsg failed! {func_name}")
- return False
- new_msg = AgentMsg()
- target_id = func_item.args[0]
- msg_content = func_item.body
- new_msg.set(self.agent_id,target_id,msg_content)
-
- r.send_msgs.append(new_msg)
- is_need_wait = True
-
- case "post_msg":# postmsg($target_id,$msg_content)
- if len(func_args) != 1:
- logger.error(f"parse postmsg failed! {func_name}")
- return False
- new_msg = AgentMsg()
- target_id = func_item.args[0]
- msg_content = func_item.body
- new_msg.set(self.agent_id,target_id,msg_content)
- r.post_msgs.append(new_msg)
-
- case "call":# call($func_name,$args_str)
- r.calls.append(func_item)
- is_need_wait = True
- return True
- case "post_call": # post_call($func_name,$args_str)
- r.post_calls.append(func_item)
- return True
-
- current_func : FunctionItem = None
- for line in lines:
- if line.startswith("##/"):
- if current_func:
- if check_args(current_func) is False:
- r.resp += current_func.dumps()
-
- func_name,func_args = AgentMsg.parse_function_call(line[3:])
- current_func = FunctionItem(func_name,func_args)
- else:
- if current_func:
- current_func.append_body(line + "\n")
- else:
- r.resp += line + "\n"
-
- if current_func:
- if check_args(current_func) is False:
- r.resp += current_func.dumps()
-
- if len(r.send_msgs) > 0 or len(r.calls) > 0:
- r.state = "waiting"
- else:
- r.state = "reponsed"
-
- return r
-
- def _get_remote_user_prompt(self,remote_user:str) -> AgentPrompt:
- cm = ContactManager.get_instance()
- contact = cm.find_contact_by_name(remote_user)
- if contact is None:
- #create guest prompt
- if self.guest_prompt_str is not None:
- prompt = AgentPrompt()
- prompt.system_message = {"role":"system","content":self.guest_prompt_str}
- return prompt
- return None
- else:
- if contact.is_family_member:
- if self.owner_promp_str is not None:
- real_str = self.owner_promp_str.format_map(contact.to_dict())
- prompt = AgentPrompt()
- prompt.system_message = {"role":"system","content":real_str}
- return prompt
- else:
- if self.contact_prompt_str is not None:
- real_str = self.contact_prompt_str.format_map(contact.to_dict())
- prompt = AgentPrompt()
- prompt.system_message = {"role":"system","content":real_str}
- return prompt
-
- return None
-
- def _get_inner_functions(self) -> dict:
- if self.owner_env is None:
- return None,0
-
- all_inner_function = self.owner_env.get_all_ai_functions()
- if all_inner_function is None:
- return None,0
-
- result_func = []
- result_len = 0
- for inner_func in all_inner_function:
- func_name = inner_func.get_name()
- if self.enable_function_list is not None:
- if len(self.enable_function_list) > 0:
- if func_name not in self.enable_function_list:
- logger.debug(f"ageint {self.agent_id} ignore inner func:{func_name}")
- continue
-
- this_func = {}
- this_func["name"] = func_name
- this_func["description"] = inner_func.get_description()
- this_func["parameters"] = inner_func.get_parameters()
- result_len += len(json.dumps(this_func)) / 4
- result_func.append(this_func)
-
- return result_func,result_len
-
- async def _execute_func(self,inenr_func_call_node:dict,prompt:AgentPrompt,org_msg:AgentMsg,stack_limit = 5) -> [str,int]:
- from .compute_kernel import ComputeKernel
-
- func_name = inenr_func_call_node.get("name")
- arguments = json.loads(inenr_func_call_node.get("arguments"))
- logger.info(f"llm execute inner func:{func_name} ({json.dumps(arguments)})")
-
- func_node : AIFunction = self.owner_env.get_ai_function(func_name)
- if func_node is None:
- result_str = f"execute {func_name} error,function not found"
- else:
- ineternal_call_record = AgentMsg.create_internal_call_msg(func_name,arguments,org_msg.get_msg_id(),org_msg.target)
- try:
- result_str:str = await func_node.execute(**arguments)
- except Exception as e:
- result_str = f"execute {func_name} error:{str(e)}"
- logger.error(f"llm execute inner func:{func_name} error:{e}")
-
-
- logger.info("llm execute inner func result:" + result_str)
- inner_functions,inner_function_len = self._get_inner_functions()
- prompt.messages.append({"role":"function","content":result_str,"name":func_name})
- task_result:ComputeTaskResult = await ComputeKernel.get_instance().do_llm_completion(prompt,self.llm_model_name,self.max_token_size,inner_functions)
- if task_result.result_code != ComputeTaskResultCode.OK:
- logger.error(f"llm compute error:{task_result.error_str}")
- return task_result.error_str,1
-
- ineternal_call_record.result_str = task_result.result_str
- ineternal_call_record.done_time = time.time()
- org_msg.inner_call_chain.append(ineternal_call_record)
-
- if stack_limit > 0:
- result_message = task_result.result.get("message")
- if result_message:
- inner_func_call_node = result_message.get("function_call")
-
- if inner_func_call_node:
- return await self._execute_func(inner_func_call_node,prompt,org_msg,stack_limit-1)
- else:
- return task_result.result_str,0
-
- async def _get_agent_prompt(self) -> AgentPrompt:
- return self.agent_prompt
-
- async def _get_agent_think_prompt(self) -> AgentPrompt:
- return self.agent_think_prompt
-
- def _format_msg_by_env_value(self,prompt:AgentPrompt):
- if self.owner_env is None:
- return
-
- for msg in prompt.messages:
- old_content = msg.get("content")
- msg["content"] = old_content.format_map(self.owner_env)
-
- async def _handle_event(self,event):
- if event.type == "AgentThink":
- return await self._do_think()
-
-
- async def _do_think(self):
- #1) load all sessions
- session_id_list = AIChatSession.list_session(self.agent_id,self.chat_db)
- #2) get history from session in token limit
- for session_id in session_id_list:
- await self.think_chatsession(session_id)
-
- #4) advanced: reload all chatrecord,and think the topic of message.
- #5) some topic could be end(not be thinked in futured )
- return
-
- async def think_chatsession(self,session_id):
- if self.agent_think_prompt is None:
- return
- logger.info(f"agent {self.agent_id} think session {session_id}")
- from .compute_kernel import ComputeKernel
- chatsession = AIChatSession.get_session_by_id(session_id,self.chat_db)
-
- while True:
- cur_pos = chatsession.summarize_pos
- summary = chatsession.summary
- prompt:AgentPrompt = AgentPrompt()
- #prompt.append(self._get_agent_prompt())
- prompt.append(await self._get_agent_think_prompt())
- system_prompt_len = prompt.get_prompt_token_len()
- #think env?
- history_prompt,next_pos = await self._get_history_prompt_for_think(chatsession,summary,system_prompt_len,cur_pos)
- prompt.append(history_prompt)
- is_finish = next_pos - cur_pos < 2
- if is_finish:
- logger.info(f"agent {self.agent_id} think session {session_id} is finished!,no more history")
- break
- #3) llm summarize chat history
- task_result:ComputeTaskResult = await ComputeKernel.get_instance().do_llm_completion(prompt,self.llm_model_name,self.max_token_size,None)
- if task_result.result_code != ComputeTaskResultCode.OK:
- logger.error(f"llm compute error:{task_result.error_str}")
- break
- else:
- new_summary= task_result.result_str
- logger.info(f"agent {self.agent_id} think session {session_id} from {cur_pos} to {next_pos} summary:{new_summary}")
- chatsession.update_think_progress(next_pos,new_summary)
-
-
-
- return
-
- async def _process_group_chat_msg(self,msg:AgentMsg) -> AgentMsg:
- from .compute_kernel import ComputeKernel
- from .bus import AIBus
-
- session_topic = msg.target + "#" + msg.topic
- chatsession = AIChatSession.get_session(self.agent_id,session_topic,self.chat_db)
- need_process = False
- if msg.mentions is not None:
- if self.agent_id in msg.mentions:
- need_process = True
- logger.info(f"agent {self.agent_id} recv a group chat message from {msg.sender},but is not mentioned,ignore!")
-
- if need_process is not True:
- chatsession.append(msg)
- resp_msg = msg.create_group_resp_msg(self.agent_id,"")
- return resp_msg
- else:
- msg_prompt = AgentPrompt()
- msg_prompt.messages = [{"role":"user","content":f"{msg.sender}:{msg.body}"}]
-
- prompt = AgentPrompt()
- prompt.append(await self._get_agent_prompt())
- self._format_msg_by_env_value(prompt)
- inner_functions,function_token_len = self._get_inner_functions()
-
- system_prompt_len = prompt.get_prompt_token_len()
- input_len = len(msg.body)
-
- history_prmpt,history_token_len = await self._get_prompt_from_session_for_groupchat(chatsession,system_prompt_len + function_token_len,input_len)
- prompt.append(history_prmpt) # chat context
- prompt.append(msg_prompt)
-
- logger.debug(f"Agent {self.agent_id} do llm token static system:{system_prompt_len},function:{function_token_len},history:{history_token_len},input:{input_len}, totoal prompt:{system_prompt_len + function_token_len + history_token_len} ")
- task_result:ComputeTaskResult = await ComputeKernel.get_instance().do_llm_completion(prompt,self.llm_model_name,self.max_token_size,inner_functions)
- if task_result.result_code != ComputeTaskResultCode.OK:
- logger.error(f"llm compute error:{task_result.error_str}")
- error_resp = msg.create_error_resp(task_result.error_str)
- return error_resp
-
- final_result = task_result.result_str
-
- result_message = task_result.result.get("message")
- if result_message:
- inner_func_call_node = result_message.get("function_call")
- if inner_func_call_node:
- #TODO to save more token ,can i use msg_prompt?
- call_prompt : AgentPrompt = copy.deepcopy(prompt)
- final_result,error_code = await self._execute_func(inner_func_call_node,call_prompt,msg)
- if error_code != 0:
- error_resp = msg.create_error_resp(final_result)
- return error_resp
-
- llm_result : LLMResult = self._get_llm_result_type(final_result)
- is_ignore = False
- result_prompt_str = ""
- match llm_result.state:
- case "ignore":
- is_ignore = True
- case "waiting":
- for sendmsg in llm_result.send_msgs:
- target = sendmsg.target
- sendmsg.topic = msg.topic
- sendmsg.prev_msg_id = msg.get_msg_id()
- send_resp = await AIBus.get_default_bus().send_message(sendmsg)
- if send_resp is not None:
- result_prompt_str += f"\n{target} response is :{send_resp.body}"
- agent_sesion = AIChatSession.get_session(self.agent_id,f"{sendmsg.target}#{sendmsg.topic}",self.chat_db)
- agent_sesion.append(sendmsg)
- agent_sesion.append(send_resp)
-
- final_result = llm_result.resp + result_prompt_str
-
- if is_ignore is not True:
- resp_msg = msg.create_group_resp_msg(self.agent_id,final_result)
- chatsession.append(msg)
- chatsession.append(resp_msg)
-
- return resp_msg
-
- return None
-
- async def _process_msg(self,msg:AgentMsg) -> AgentMsg:
- from .compute_kernel import ComputeKernel
- from .bus import AIBus
-
- if msg.msg_type == AgentMsgType.TYPE_GROUPMSG:
- return await self._process_group_chat_msg(msg)
-
- session_topic = msg.get_sender() + "#" + msg.topic
- chatsession = AIChatSession.get_session(self.agent_id,session_topic,self.chat_db)
-
-
- msg_prompt = AgentPrompt()
- msg_prompt.messages = [{"role":"user","content":msg.body}]
-
- prompt = AgentPrompt()
- prompt.append(await self._get_agent_prompt())
- self._format_msg_by_env_value(prompt)
- prompt.append(self._get_remote_user_prompt(msg.sender))
-
- inner_functions,function_token_len = self._get_inner_functions()
-
- system_prompt_len = prompt.get_prompt_token_len()
- input_len = len(msg.body)
-
- history_prmpt,history_token_len = await self._get_prompt_from_session(chatsession,system_prompt_len + function_token_len,input_len)
- prompt.append(history_prmpt) # chat context
- prompt.append(msg_prompt)
-
- logger.debug(f"Agent {self.agent_id} do llm token static system:{system_prompt_len},function:{function_token_len},history:{history_token_len},input:{input_len}, totoal prompt:{system_prompt_len + function_token_len + history_token_len} ")
- task_result:ComputeTaskResult = await ComputeKernel.get_instance().do_llm_completion(prompt,self.llm_model_name,self.max_token_size,inner_functions)
- if task_result.result_code != ComputeTaskResultCode.OK:
- logger.error(f"llm compute error:{task_result.error_str}")
- error_resp = msg.create_error_resp(task_result.error_str)
- return error_resp
-
- final_result = task_result.result_str
-
- result_message = task_result.result.get("message")
- if result_message:
- inner_func_call_node = result_message.get("function_call")
- if inner_func_call_node:
- #TODO to save more token ,can i use msg_prompt?
- call_prompt : AgentPrompt = copy.deepcopy(prompt)
- final_result,error_code = await self._execute_func(inner_func_call_node,call_prompt,msg)
- if error_code != 0:
- error_resp = msg.create_error_resp(final_result)
- return error_resp
-
- llm_result : LLMResult = self._get_llm_result_type(final_result)
- is_ignore = False
- result_prompt_str = ""
- match llm_result.state:
- case "ignore":
- is_ignore = True
- case "waiting":
- for sendmsg in llm_result.send_msgs:
- target = sendmsg.target
- sendmsg.topic = msg.topic
- sendmsg.prev_msg_id = msg.get_msg_id()
- send_resp = await AIBus.get_default_bus().send_message(sendmsg)
- if send_resp is not None:
- result_prompt_str += f"\n{target} response is :{send_resp.body}"
- agent_sesion = AIChatSession.get_session(self.agent_id,f"{sendmsg.target}#{sendmsg.topic}",self.chat_db)
- agent_sesion.append(sendmsg)
- agent_sesion.append(send_resp)
-
- final_result = llm_result.resp + result_prompt_str
-
- if is_ignore is not True:
- resp_msg = msg.create_resp_msg(final_result)
- chatsession.append(msg)
- chatsession.append(resp_msg)
-
- return resp_msg
-
- return None
-
- def get_id(self) -> str:
- return self.agent_id
-
- def get_fullname(self) -> str:
- return self.fullname
-
- def get_template_id(self) -> str:
- return self.template_id
-
- def get_llm_model_name(self) -> str:
- return self.llm_model_name
-
- def get_max_token_size(self) -> int:
- return self.max_token_size
-
- async def _get_history_prompt_for_think(self,chatsession:AIChatSession,summary:str,system_token_len:int,pos:int)->(AgentPrompt,int):
- history_len = (self.max_token_size * 0.7) - system_token_len
-
- messages = chatsession.read_history(self.history_len,pos,"natural") # read
- result_token_len = 0
- result_prompt = AgentPrompt()
- have_summary = False
- if summary is not None:
- if len(summary) > 1:
- have_summary = True
-
- if have_summary:
- result_prompt.messages.append({"role":"user","content":summary})
- result_token_len -= len(summary)
- else:
- result_prompt.messages.append({"role":"user","content":"There is no summary yet."})
- result_token_len -= 6
-
- read_history_msg = 0
- history_str : str = ""
- for msg in messages:
- read_history_msg += 1
- dt = datetime.datetime.fromtimestamp(float(msg.create_time))
- formatted_time = dt.strftime('%y-%m-%d %H:%M:%S')
- record_str = f"{msg.sender},[{formatted_time}]\n{msg.body}\n"
- history_str = history_str + record_str
-
- history_len -= len(msg.body)
- result_token_len += len(msg.body)
- if history_len < 0:
- logger.warning(f"_get_prompt_from_session reach limit of token,just read {read_history_msg} history message.")
- break
-
- result_prompt.messages.append({"role":"user","content":history_str})
- return result_prompt,pos+read_history_msg
-
- async def _get_prompt_from_session_for_groupchat(self,chatsession:AIChatSession,system_token_len,input_token_len,is_groupchat=False):
- history_len = (self.max_token_size * 0.7) - system_token_len - input_token_len
- messages = chatsession.read_history(self.history_len) # read
- result_token_len = 0
- result_prompt = AgentPrompt()
- read_history_msg = 0
- for msg in reversed(messages):
- read_history_msg += 1
- dt = datetime.datetime.fromtimestamp(float(msg.create_time))
- formatted_time = dt.strftime('%y-%m-%d %H:%M:%S')
-
- if msg.sender == self.agent_id:
- if self.enable_timestamp:
- result_prompt.messages.append({"role":"assistant","content":f"(create on {formatted_time}) {msg.body} "})
- else:
- result_prompt.messages.append({"role":"assistant","content":msg.body})
-
- else:
- if self.enable_timestamp:
- result_prompt.messages.append({"role":"user","content":f"(create on {formatted_time}) {msg.body} "})
- else:
- result_prompt.messages.append({"role":"user","content":f"{msg.sender}:{msg.body}"})
-
- history_len -= len(msg.body)
- result_token_len += len(msg.body)
- if history_len < 0:
- logger.warning(f"_get_prompt_from_session reach limit of token,just read {read_history_msg} history message.")
- break
-
- return result_prompt,result_token_len
-
- async def _get_prompt_from_session(self,chatsession:AIChatSession,system_token_len,input_token_len) -> AgentPrompt:
- # TODO: get prompt from group chat is different from single chat
-
- history_len = (self.max_token_size * 0.7) - system_token_len - input_token_len
- messages = chatsession.read_history(self.history_len) # read
- result_token_len = 0
- result_prompt = AgentPrompt()
- read_history_msg = 0
-
- if chatsession.summary is not None:
- if len(chatsession.summary) > 1:
- result_prompt.messages.append({"role":"user","content":chatsession.summary})
- result_token_len -= len(chatsession.summary)
-
- for msg in reversed(messages):
- read_history_msg += 1
- dt = datetime.datetime.fromtimestamp(float(msg.create_time))
- formatted_time = dt.strftime('%y-%m-%d %H:%M:%S')
-
- if msg.sender == self.agent_id:
-
- if self.enable_timestamp:
- result_prompt.messages.append({"role":"assistant","content":f"(create on {formatted_time}) {msg.body} "})
- else:
- result_prompt.messages.append({"role":"assistant","content":msg.body})
-
- else:
- if self.enable_timestamp:
- result_prompt.messages.append({"role":"user","content":f"(create on {formatted_time}) {msg.body} "})
- else:
- result_prompt.messages.append({"role":"user","content":msg.body})
-
- history_len -= len(msg.body)
- result_token_len += len(msg.body)
- if history_len < 0:
- logger.warning(f"_get_prompt_from_session reach limit of token,just read {read_history_msg} history message.")
- break
-
- return result_prompt,result_token_len
-
diff --git a/src/aios_kernel/ai_function.py b/src/aios_kernel/ai_function.py
deleted file mode 100644
index bca41d2..0000000
--- a/src/aios_kernel/ai_function.py
+++ /dev/null
@@ -1,144 +0,0 @@
-from abc import ABC, abstractmethod
-from typing import Dict,Coroutine,Callable
-
-class ParameterDefine:
- def __init__(self) -> None:
- self.name = None
- self.type = None
- self.description = None
-
-
-class AIFunction:
- def __init__(self) -> None:
- self.description : str = None
-
- @abstractmethod
- def get_name(self) -> str:
- """
- return the name of the function (should be snake case)
- """
- pass
-
- @abstractmethod
- def get_description(self) -> str:
- """
- return a detailed description of what the function does
- """
- return self.description
-
- @abstractmethod
- def get_parameters(self) -> Dict:
- """
- Return the list of parameters to execute this function in the form of
- JSON schema as specified in the OpenAI documentation:
- https://platform.openai.com/docs/api-reference/chat/create#chat/create-parameters
-
- str = run_code(code:str)
- parameters = {
- "type": "object",
- "properties": {
- "code": {
- "type": "string",
- "description": "Python code which needs to be executed"
- }
- }
- }
-
- """
- pass
-
- @abstractmethod
- async def execute(self, **kwargs) -> str:
- """
- Execute the function and return a JSON serializable dict.
- The parameters are passed in the form of kwargs
- """
- pass
-
- @abstractmethod
- def is_local(self) -> bool:
- """
- is this function call need network?
- """
- pass
-
- @abstractmethod
- def is_in_zone(self) -> bool:
- """
- is this function call in Lan?
- """
- pass
-
- @abstractmethod
- def is_ready_only(self) -> bool:
- pass
-
- #def load_from_config(self,config:dict) -> bool:
- # pass
-
-class FunctionItem:
- def __init__(self,name,args) -> None:
- self.name = name
- self.args = args
- self.body = None
-
- def append_body(self,body:str) -> None:
- if self.body is None:
- self.body = body
- else:
- self.body += body
-
- def dumps(self) -> str:
- pass
-
-# call chain is a combination of ai_function,group of ai_function.
-class CallChain:
- def __init__(self) -> None:
- pass
-
- def load_from_config(self,config:dict) -> bool:
- pass
-
- async def execute(self):
- pass
-
-class SimpleAIFunction(AIFunction):
- def __init__(self,func_id:str,description:str,func_handler:Coroutine,parameters:Dict = None) -> None:
- self.func_id = func_id
- self.description = description
- self.func_handler = func_handler
- self.parameters = parameters
-
- def get_name(self) -> str:
- return self.func_id
-
- def get_parameters(self) -> Dict:
- if self.parameters is not None:
- result = {}
- result["type"] = "object"
- parm_defines = {}
- for parm,desc in self.parameters.items():
- parm_item = {}
- parm_item["type"] = "string"
- parm_item["description"] = desc
- parm_defines[parm] = parm_item
- result["properties"] = parm_defines
- return result
- return {"type": "object", "properties": {}}
-
-
- async def execute(self,**kwargs) -> str:
- if self.func_handler is None:
- return "error: function not implemented"
-
- return await self.func_handler(**kwargs)
-
- def is_local(self) -> bool:
- return True
-
- def is_in_zone(self) -> bool:
- return True
-
- def is_ready_only(self) -> bool:
- return False
-
diff --git a/src/aios_kernel/compute_task.py b/src/aios_kernel/compute_task.py
deleted file mode 100644
index ea7c431..0000000
--- a/src/aios_kernel/compute_task.py
+++ /dev/null
@@ -1,122 +0,0 @@
-
-from enum import Enum
-import uuid
-import time
-from typing import Union
-from knowledge import ObjectID
-
-class ComputeTaskResultCode(Enum):
- OK = 0
- TIMEOUT = 1
- NO_WORKER = 2
- ERROR = 3
-
-
-class ComputeTaskState(Enum):
- DONE = 0
- INIT = 1
- RUNNING = 2
- ERROR = 3
- PENDING = 4
-
-class ComputeTaskType(Enum):
- NONE = "None"
- LLM_COMPLETION = "llm_completion"
- TEXT_2_IMAGE = "text_2_image"
- IMAGE_2_IMAGE = "image_2_image"
- VOICE_2_TEXT = "voice_2_text"
- TEXT_2_VOICE = "text_2_voice"
- TEXT_EMBEDDING ="text_embedding"
- IMAGE_EMBEDDING ="image_embedding"
-
-
-class ComputeTask:
- def __init__(self) -> None:
- self.task_type = ComputeTaskType.NONE
- self.create_time = None
-
- self.task_id: str = None
- self.callchain_id: str = None
- self.params: dict = {}
- self.refers: dict = None
- self.pading_data: bytearray = None
-
- self.state = ComputeTaskState.INIT
- self.result = None
- self.error_str = None
-
- def set_llm_params(self, prompts, model_name, max_token_size, inner_functions = None, callchain_id=None):
- self.task_type = ComputeTaskType.LLM_COMPLETION
- self.create_time = time.time()
- self.task_id = uuid.uuid4().hex
- self.callchain_id = callchain_id
- self.params["prompts"] = prompts.to_message_list()
- if model_name is not None:
- self.params["model_name"] = model_name
- else:
- self.params["model_name"] = "gpt-4-0613"
- if max_token_size is None:
- self.params["max_token_size"] = 4000
- else:
- self.params["max_token_size"] = max_token_size
-
- if inner_functions is not None:
- self.params["inner_functions"] = inner_functions
-
- def set_text_embedding_params(self, input: str, model_name=None, callchain_id = None):
- self.task_type = ComputeTaskType.TEXT_EMBEDDING
- self.create_time = time.time()
- self.task_id = uuid.uuid4().hex
- self.callchain_id = callchain_id
- if model_name is not None:
- self.params["model_name"] = model_name
- else:
- self.params["model_name"] = "text-embedding-ada-002"
- self.params["input"] = input
-
- def set_image_embedding_params(self, input = Union[ObjectID, bytes], model_name=None, callchain_id = None):
- self.task_type = ComputeTaskType.IMAGE_EMBEDDING
- self.create_time = time.time()
- self.task_id = uuid.uuid4().hex
- self.callchain_id = callchain_id
- if model_name is not None:
- self.params["model_name"] = model_name
- else:
- self.params["model_name"] = None
- self.params["input"] = input
-
- def set_text_2_image_params(self, prompt: str, model_name, negative_prompt="", callchain_id=None):
- self.task_type = ComputeTaskType.TEXT_2_IMAGE
- self.create_time = time.time()
- self.task_id = uuid.uuid4().hex
- self.callchain_id = callchain_id
- self.params["prompt"] = prompt
- self.params["negative_prompt"] = negative_prompt
- if model_name is not None:
- self.params["model_name"] = model_name
- else:
- self.params["model_name"] = "v1-5-pruned-emaonly"
-
- def display(self) -> str:
- return f"ComputeTask: {self.task_id} {self.task_type} {self.state}"
-
-
-class ComputeTaskResult:
- def __init__(self) -> None:
- self.create_time = None
- self.task_id: str = None
- self.callchain_id: str = None
- self.worker_id: str = None
- self.error_str : str = None
- self.result_code: int = 0
- self.result_str: str = None # easy to use,can read from result
-
- self.result : dict = {}
-
- self.result_refers: dict = {}
- self.pading_data: bytearray = None
-
- def set_from_task(self, task: ComputeTask):
- self.task_id = task.task_id
- self.callchain_id = task.callchain_id
- task.result = self
diff --git a/src/aios_kernel/environment.py b/src/aios_kernel/environment.py
deleted file mode 100644
index 1e8cdab..0000000
--- a/src/aios_kernel/environment.py
+++ /dev/null
@@ -1,135 +0,0 @@
-# basic environment class
-# we have some built-in environment: Calender(include timer),Home(connect to IoT device in your home), ,KnwoledgeBase,FileSystem,
-
-from abc import ABC, abstractmethod
-from typing import Any, Callable, Optional,Dict,Awaitable,List
-import logging
-
-from .ai_function import AIFunction
-
-logger = logging.getLogger(__name__)
-
-class EnvironmentEvent(ABC):
- @abstractmethod
- def display(self) -> str:
- pass
-
-EnvironmentEventHandler = Callable[[str,EnvironmentEvent],Awaitable[Any]]
-
-class Environment:
- _all_env = {}
- @classmethod
- def get_env_by_id(cls,env_id:str):
- return cls._all_env.get(env_id)
-
- @classmethod
- def set_env_by_id(cls,id,env):
- assert id == env.get_id()
- cls._all_env[env.get_id()] = env
-
- def __init__(self,env_id:str) -> None:
- self.env_id = env_id
- self.values:Dict[str,str] = {}
- self.get_handlers:Dict[str,Callable] = {}
- self.owner_env:Dict[str,Environment] = {}
- # self.valid_keys:Dict[str,bool] = None
- self.event_handlers:Dict[str,List[EnvironmentEventHandler]]= {}
-
- self.functions : Dict[str,AIFunction] = {}
-
- def get_id(self) -> str:
- return self.env_id
-
- def add_owner_env(self,env) -> None:
- self.owner_env[env.get_id()] = env
-
- #@abstractmethod
- #TODO: how to use env? different env has different prompt
- #def get_env_prompt(self) -> str:
- # pass
-
- def add_ai_function(self,func:AIFunction) -> None:
- if self.functions.get(func.get_name()) is not None:
- logger.warn(f"add ai_function {func.get_name()} in env {self.env_id}:function already exist")
-
- self.functions[func.get_name()] = func
-
- def get_ai_function(self,func_name:str) -> AIFunction:
- return self.functions.get(func_name)
-
- #def enable_ai_function(self,func_name:str) -> None:
- # pass
-
- #def disable_ai_function(self,func_name:str) -> None:
- # pass
-
- def get_all_ai_functions(self) -> List[AIFunction]:
- return self.functions.values()
-
- @abstractmethod
- def _do_get_value(self,key:str) -> Optional[str]:
- pass
-
- def register_get_handler(self,key:str,handler:Callable) -> None:
- h = self.get_handlers.get(key)
- if h is not None:
- logger.warn(f"register get_handler {key} in env {self.env_id}:handler already exist")
-
- self.get_handlers[key] = handler
-
-
- def attach_event_handler(self,event_id:str,handler:Callable) -> None:
- handler_list = self.event_handlers.get(event_id)
- if handler_list is None:
- handler_list = []
- self.event_handlers[event_id] = handler_list
-
- handler_list.append(handler)
-
- def remove_event_handler(self,event_id:str,handler:Callable) -> None:
- handler_list = self.event_handlers.get(event_id)
- if handler is not None:
- handler_list.remove(handler)
- return
-
- logger.warn(f"remove event_handler {event_id} in env {self.env_id}:handler not found")
-
- async def fire_event(self,event_id:str,event:EnvironmentEvent) -> None:
- handler_list = self.event_handlers.get(event_id)
- if handler_list is not None:
- for handler in handler_list:
- await handler(self.env_id,event)
- else:
- logger.debug(f"fire event {event_id} in env {self.env_id}:handler not found")
- return
-
- def __getitem__(self, key):
- return self.get_value(key)
-
- def get_value(self,key:str) -> Optional[str]:
- handler = self.get_handlers.get(key)
- if handler is not None:
- return handler()
-
- s = self.values.get(key)
- if isinstance(s,str):
- return s
- else:
- logger.warn(f"get value {key} in env {self.env_id} failed!,type is not str")
-
- s = self._do_get_value(key)
- if s is not None:
- return s
- if self.owner_env is not None:
- for env in self.owner_env.values():
- s = env.get_value(key)
- if s is not None:
- return s
-
- logger.warn(f"get value {key} in env {self.env_id} failed!,not found")
- return None
-
- def set_value(self, key: str, str_value: str,is_storage:bool = True):
- logger.info(f"set value {key} in env {self.env_id} to {str_value}")
- self.values[key] = str_value
-
diff --git a/src/aios_kernel/knowledge_base.py b/src/aios_kernel/knowledge_base.py
deleted file mode 100644
index 63c7d04..0000000
--- a/src/aios_kernel/knowledge_base.py
+++ /dev/null
@@ -1,295 +0,0 @@
-# define a knowledge base class
-import json
-import logging
-from .agent import AgentPrompt
-from .compute_kernel import ComputeKernel
-from .storage import AIStorage
-from .environment import Environment
-from .ai_function import SimpleAIFunction
-from knowledge import *
-
-
-class KnowledgeBase:
- _instance = None
-
- def __new__(cls):
- if cls._instance is None:
- cls._instance = super().__new__(cls)
- cls._instance.__singleton_init__()
-
- return cls._instance
-
- def __singleton_init__(self) -> None:
- self.store = KnowledgeStore()
- self.compute_kernel = ComputeKernel.get_instance()
- self._default_text_model = "all-MiniLM-L6-v2"
- self._default_image_model = "clip-ViT-B-32"
-
- async def __embedding_document(self, document: DocumentObject):
- for chunk_id in document.get_chunk_list():
- chunk = self.store.get_chunk_reader().get_chunk(chunk_id)
- if chunk is None:
- raise ValueError(f"text chunk not found: {chunk_id}")
-
- text = chunk.read().decode("utf-8")
- vector = await self.compute_kernel.do_text_embedding(text, self._default_text_model)
- if vector:
- await self.store.get_vector_store(self._default_text_model).insert(vector, chunk_id)
-
- async def __embedding_image(self, image: ImageObject):
- # desc = {}
- # if not not image.get_meta():
- # desc["meta"] = image.get_meta()
- # if not not image.get_exif():
- # desc["exif"] = image.get_exif()
- # if not not image.get_tags():
- # desc["tags"] = image.get_tags()
- # vector = await self.compute_kernel.do_text_embedding(json.dumps(desc), self._default_text_model)
- vector = await self.compute_kernel.do_image_embedding(image.calculate_id(), self._default_image_model)
- if vector:
- await self.store.get_vector_store(self._default_image_model).insert(vector, image.calculate_id())
-
- async def __embedding_video(self, vedio: VideoObject):
- desc = {}
- if not not vedio.get_meta():
- desc["meta"] = vedio.get_meta()
- if not not vedio.get_info():
- desc["info"] = vedio.get_info()
- if not not vedio.get_tags():
- desc["tags"] = vedio.get_tags()
- vector = await self.compute_kernel.do_text_embedding(json.dumps(desc), self._default_text_model)
- await self.store.get_vector_store(self._default_text_model).insert(vector, vedio.calculate_id())
-
- async def __embedding_rich_text(self, rich_text: RichTextObject):
- for document_id in rich_text.get_documents().values():
- document = DocumentObject.decode(self.store.get_object_store().get_object(document_id))
- await self.__embedding_document(document)
- for image_id in rich_text.get_images().values():
- image = ImageObject.decode(self.store.get_object_store().get_object(image_id))
- await self.__embedding_image(image)
- for video_id in rich_text.get_videos().values():
- video = VideoObject.decode(self.store.get_object_store().get_object(video_id))
- await self.__embedding_video(video)
- for rich_text_id in rich_text.get_rich_texts().values():
- rich_text = RichTextObject.decode(self.store.get_object_store().get_object(rich_text_id))
- await self.__embedding_rich_text(rich_text)
-
- async def __embedding_email(self, email: EmailObject):
- vector = await self.compute_kernel.do_text_embedding(json.dumps(email.get_desc()), self._default_text_model)
- await self.store.get_vector_store(self._default_text_model).insert(vector, email.calculate_id())
- await self.__embedding_rich_text(email.get_rich_text())
-
-
- async def __do_embedding(self, object: KnowledgeObject):
- if object.get_object_type() == ObjectType.Document:
- await self.__embedding_document(object)
- if object.get_object_type() == ObjectType.Image:
- await self.__embedding_image(object)
- if object.get_object_type() == ObjectType.Video:
- await self.__embedding_video(object)
- if object.get_object_type() == ObjectType.RichText:
- await self.__embedding_rich_text(object)
- if object.get_object_type() == ObjectType.Email:
- await self.__embedding_email(object)
- else:
- pass
-
- # def __save_document(self, document: DocumentObject):
- # doc_id = document.calculate_id()
- # self.store.get_object_store().put_object(doc_id, document.encode())
- # for chunk_id in document.get_chunk_list():
- # self.store.get_relation_store().add_relation(chunk_id, doc_id)
-
- # def __save_image(self, image: ImageObject):
- # image_id = image.calculate_id()
- # self.store.get_object_store().put_object(image_id, image.encode())
-
- # def __save_video(self, video: VideoObject):
- # video_id = video.calculate_id()
- # self.store.get_object_store().put_object(video_id, video.encode())
-
- # def __save_rich_text(self, rich_text: RichTextObject):
- # rich_text_id = rich_text.calculate_id()
- # # rich_text_enc = dict()
- # # rich_text_enc["desc"] = rich_text.desc
- # # rich_text_enc["body"] = {"documents": {}, "images": {}, "videos": {}, "rich_texts": {}}
- # for key, document in rich_text.get_documents().items():
- # self.__save_document(document)
- # doc_id = document.calculate_id()
- # self.store.get_relation_store().add_relation(doc_id, rich_text_id)
- # # rich_text_enc["body"]["documents"][key] = doc_id
- # for key, image in rich_text.get_images().items():
- # self.__save_image(image)
- # image_id = image.calculate_id()
- # self.store.get_relation_store().add_relation(image_id, rich_text_id)
- # # rich_text_enc["body"]["images"][key] = image_id
- # for key, video in rich_text.get_videos().items():
- # self.__save_video(video)
- # video_id = video.calculate_id()
- # self.store.get_relation_store().add_relation(video_id, rich_text_id)
- # # rich_text_enc["body"]["videos"][key] = video_id
- # for key, rich_text in rich_text.get_rich_texts().items():
- # self.__save_rich_text(rich_text)
- # rich_text_id = rich_text.calculate_id()
- # self.store.get_relation_store().add_relation(rich_text_id, rich_text_id)
- # # rich_text_enc["body"]["rich_texts"][key] = rich_text_id
-
-
- # self.store.get_object_store().put_object(rich_text_id, rich_text.encode())
-
- # def __save_email(self, email: EmailObject):
- # email_id = email.calculate_id()
- # # email_enc = dict()
- # # email_enc["desc"] = email.desc
- # # email_enc["body"] = {"content": None}
- # self.__save_rich_text(email.get_rich_text())
- # rich_text_id = email.get_rich_text().calculate_id()
- # self.store.get_relation_store().add_relation(rich_text_id, email_id)
- # # email_enc["body"]["content"] = rich_text_id
- # self.store.get_object_store().put_object(email_id, email.encode())
-
-
- # def __save_object(self, object: KnowledgeObject):
- # if object.get_object_type() == ObjectType.Document:
- # self.__save_document(object)
- # if object.get_object_type() == ObjectType.Image:
- # self.__save_image(object)
- # if object.get_object_type() == ObjectType.Video:
- # self.__save_video(object)
- # if object.get_object_type() == ObjectType.RichText:
- # self.__save_rich_text(object)
- # if object.get_object_type() == ObjectType.Email:
- # self.__save_email(object)
- # else:
- # pass
-
- async def insert_object(self, object: KnowledgeObject):
- self.store.get_object_store().put_object(object.calculate_id(), object.encode())
- await self.__do_embedding(object)
-
- async def query_objects(self, tokens: str, types: list[str], topk: int) -> [ObjectID]:
- texts = []
- if "text" in types:
- vector = await self.compute_kernel.do_text_embedding(tokens, self._default_text_model)
- texts = await self.store.get_vector_store(self._default_text_model).query(vector, topk)
- images = []
- if "image" in types:
- vector = await self.compute_kernel.do_text_embedding(tokens, self._default_image_model)
- images = await self.store.get_vector_store(self._default_image_model).query(vector, topk)
- return texts + images
-
- def load_object(self, object_id: ObjectID) -> KnowledgeObject:
- if object_id.get_object_type() == ObjectType.Document:
- return DocumentObject.decode(self.store.get_object_store().get_object(object_id))
- if object_id.get_object_type() == ObjectType.Image:
- return ImageObject.decode(self.store.get_object_store().get_object(object_id))
- if object_id.get_object_type() == ObjectType.Video:
- return VideoObject.decode(self.store.get_object_store().get_object(object_id))
- if object_id.get_object_type() == ObjectType.RichText:
- return RichTextObject.decode(self.store.get_object_store().get_object(object_id))
- if object_id.get_object_type() == ObjectType.Email:
- return EmailObject.decode(self.store.get_object_store().get_object(object_id))
- else:
- pass
-
-
- def tokens_from_objects(self, object_ids: [ObjectID]) -> list[str]:
- results = dict()
- for object_id in object_ids:
- parents = self.store.get_relation_store().get_related_root_objects(object_id)
- # last parent is the root object
- root_object_id = parents[0] if parents else object_id
- logging.info(f"object_id: {str(object_id)} root_object_id: {str(root_object_id)}")
- if str(root_object_id) in results:
- results[str(root_object_id)].append(object_id)
- else:
- results[str(root_object_id)] = [root_object_id, object_id]
- content = ""
- result_desc = []
- for result in results.values():
- # first element in result is the root object
- root_object_id = result[0]
- if root_object_id.get_object_type() == ObjectType.Email:
- email = self.load_object(root_object_id)
- desc = email.get_desc()
- desc["type"] = "email"
- desc["contents"] = []
- result_desc.append(desc)
- upper_list = desc["contents"]
- result = result[1:]
- else:
- upper_list = result_desc
-
- for object_id in result:
- if object_id.get_object_type() == ObjectType.Chunk:
- upper_list.append({"type": "text", "content": self.store.get_chunk_reader().get_chunk(object_id).read().decode("utf-8")})
- if object_id.get_object_type() == ObjectType.Image:
- # image = self.load_object(object_id)
- desc = dict()
- desc["id"] = str(object_id)
- desc["type"] = "image"
- upper_list.append(desc)
- if object_id.get_object_type() == ObjectType.Video:
- video = self.load_object(object_id)
- desc = video.get_desc()
- desc["type"] = "video"
- upper_list.append(desc)
- else:
- pass
- content += json.dumps(result_desc)
- content += ".\n"
-
- return content
-
- def parse_object_in_message(self, message: str) -> KnowledgeObject:
- # get message's first line
- logging.info(f"tg parse resp message: {message}")
- lines = message.split("\n")
- if len(lines) > 0:
- message = lines[0]
- try:
- desc = json.loads(message)
- if isinstance(desc, dict):
- object_id = desc["id"]
- else:
- object_id = desc[0]["id"]
- except Exception as e:
- return None
-
- if object_id is not None:
- return self.load_object(ObjectID.from_base58(object_id))
-
-
- def bytes_from_object(self, object: KnowledgeObject) -> bytes:
- if object.get_object_type() == ObjectType.Image:
- image_object = object
- return self.store.get_chunk_reader().read_chunk_list_to_single_bytes(image_object.get_chunk_list())
-
-
-
-
-
-
-
-class KnowledgeEnvironment(Environment):
- def __init__(self, env_id: str) -> None:
- super().__init__(env_id)
-
- query_param = {
- "tokens": "key words to query",
- "types": "prefered knowledge types, one or more of [text, image]",
- "index": "index of query result"
- }
- self.add_ai_function(SimpleAIFunction("query_knowledge",
- "vector query content from local knowledge base",
- self._query,
- query_param))
-
- async def _query(self, tokens: str, types: list[str] = ["text"], index: str=0):
- index = int(index)
- object_ids = await KnowledgeBase().query_objects(tokens, types, 4)
- if len(object_ids) <= index:
- return "*** I have no more information for your reference.\n"
- else:
- content = "*** I have provided the following known information for your reference with json format:\n"
- return content + KnowledgeBase().tokens_from_objects(object_ids[index:index+1])
\ No newline at end of file
diff --git a/src/aios_kernel/knowledge_pipeline.py b/src/aios_kernel/knowledge_pipeline.py
deleted file mode 100644
index 759793a..0000000
--- a/src/aios_kernel/knowledge_pipeline.py
+++ /dev/null
@@ -1,411 +0,0 @@
-"""
-Capture your email locally, and parse out the pictures in the email body and the pictures, videos and other files in the attachment. Subsequently, it supports vectorized analysis of your personal data and serves as a knowledge base to enable large language model answers. Better results.
-
-An example of a local file is as follows:
-├── data
-│ └── alex0072@gmail.com
-│ └── 5de3e52f3a6b90cabe6cbdd4ae3a5c5b
-│ ├── email.txt
-│ ├── meta.json
-│ ├── image
-│ │ ├── 0648B869@99C03070.DB94B354.jpg
-│ └── body_image
-│ ├── 11044884873.jpg
-│ ├── 282985198265470.gif
-│ └── dd-login-service-min.png
-
-"""
-import asyncio
-import datetime
-import sqlite3
-import imaplib
-import logging
-import mailparser
-import hashlib
-import json
-import base64
-import chardet
-import aiofiles
-
-from bs4 import BeautifulSoup
-import requests
-import os
-import toml
-from .storage import AIStorage, UserConfigItem
-from .knowledge_base import KnowledgeBase, ImageObjectBuilder, ObjectID, ObjectType, DocumentObjectBuilder, EmailObjectBuilder, EmailObject
-
-class KnowledgeJournal:
- def __init__(self, source_type: str, source_id: str, item_id: str, object_id: str, timestamp=None):
- # define a timestamp variable
- self.timestamp = datetime.datetime.now() if timestamp is None else timestamp
- self.object_id = object_id
- self.source_type = source_type
- self.source_id = source_id
- self.item_id = item_id
-
- def __str__(self) -> str:
- if self.source_type == "dir":
- object_id = ObjectID.from_base58(self.object_id)
- object_type = None
- if object_id.get_object_type() == ObjectType.Image:
- object_type = "image"
- else:
- pass
- return f"Add {object_type} from {os.path.join(self.source_id, self.item_id)}"
- if self.source_type == "email":
- object_id = ObjectID.from_base58(self.object_id)
- email = EmailObject.decode(KnowledgeBase().store.get_object_store().get_object(object_id))
- meta = email.get_meta()
- return f'Add email from {os.path.join(self.source_id)} subject {meta["subject"]}'
-
-
-# init sqlite3 client
-class KnowledgeJournalClient:
- def __init__(self):
- knowledge_dir = os.path.join(AIStorage.get_instance().get_myai_dir(), "knowledge")
- if not os.path.exists(knowledge_dir):
- os.makedirs(knowledge_dir)
- self.journal_path = os.path.join(knowledge_dir, "journal.db")
-
- conn = sqlite3.connect(self.journal_path)
- conn.execute(
- '''CREATE TABLE IF NOT EXISTS journal (
- id INTEGER PRIMARY KEY AUTOINCREMENT,
- time DATETIME DEFAULT CURRENT_TIMESTAMP,
- source_type TEXT,
- source_id TEXT,
- item_id TEXT,
- object_id TEXT)'''
- )
- conn.commit()
-
- def insert(self, journal: KnowledgeJournal):
- conn = sqlite3.connect(self.journal_path)
- conn.execute(
- "INSERT INTO journal (time, source_type, source_id, item_id, object_id) VALUES (?, ?, ?, ?, ?)",
- (journal.timestamp, journal.source_type, journal.source_id, journal.item_id, journal.object_id),
- )
- conn.commit()
-
- def latest_journal(self, source_id: str) -> KnowledgeJournal:
- conn = sqlite3.connect(self.journal_path)
- cursor = conn.cursor()
- cursor.execute("SELECT * FROM journal WHERE source_id = ? ORDER BY id DESC LIMIT 1", (source_id,))
- result = cursor.fetchone()
- if result is None:
- return None
- else:
- (_, timestamp, source_type, sorce_id, item_id, object_id) = result
- return KnowledgeJournal(source_type, sorce_id, item_id, object_id, timestamp)
-
- def latest_journals(self, topn) -> [KnowledgeJournal]:
- conn = sqlite3.connect(self.journal_path)
- cursor = conn.cursor()
- cursor.execute("SELECT * FROM journal ORDER BY id DESC LIMIT ?", (topn,))
- return [KnowledgeJournal(source_type, sorce_id, item_id, object_id, timestamp) for (_, timestamp, source_type, sorce_id, item_id, object_id) in cursor.fetchall()]
-
-
-class KnowledgeEmailSource:
- def __init__(self, config:dict):
- self.config = config
- self.config["type"] = "email"
-
- def id(self):
- return self.config["address"]
-
- @classmethod
- def user_config_items(cls):
- return [("address", "email address"),
- ("password", "email password"),
- ("imap_server", "imap server"),
- ("imap_port", "imap port")
- ]
-
- @classmethod
- def local_root(cls):
- user_data_dir = AIStorage.get_instance().get_myai_dir()
- return os.path.abspath(f"{user_data_dir}/knowledge/email")
-
- async def run_once(self):
- # read config from toml file
- # and read from config config.local.toml if exists (config.local.toml is ignored by git)
- logging.debug(f"knowledge email source {self.id()} run once")
- filter = "ALL"
- self.client = self.email_client()
- await self.read_emails(imap_keyword=filter)
-
- def email_client(self) -> imaplib.IMAP4_SSL:
- logging.info(f"read email config from {self.config.get('imap_server')}")
- client = imaplib.IMAP4_SSL(
- host=self.config.get('imap_server'),
- port=self.config.get('imap_port')
- )
- client.login(self.config.get('address'), self.config.get('password'))
- return client
-
- async def read_emails(self, folder: str = 'INBOX', imap_keyword: str = "UNSEEN"):
- journal_client = KnowledgeJournalClient()
- latest_journal = journal_client.latest_journal(self.id())
- latest_uid = 0 if latest_journal is None else int(latest_journal.item_id)
- self.client.select(folder)
- _, data = self.client.uid('search', None, imap_keyword)
-
- # get email uid list
- email_list = data[0].split()
- logging.info(f"got {len(email_list)} emails")
- journal_client = KnowledgeJournalClient()
- for uid in email_list:
- _uid = int.from_bytes(uid)
- if _uid > latest_uid:
- email_dir = self.check_email_saved(uid)
- if email_dir is not None:
- logging.info(f"email uid {uid} already saved")
- else:
- email_dir = self.read_and_save_email(uid)
- logging.info(f"email uid {uid} saved")
- email_object = EmailObjectBuilder({}, email_dir).build()
- await KnowledgeBase().insert_object(email_object)
- journal_client.insert(KnowledgeJournal("email", self.id(), str(int.from_bytes(uid)), str(email_object.calculate_id())))
-
-
- def read_and_save_email(self, uid: str) -> str:
- message_parts = "(BODY.PEEK[])"
- _, email_data = self.client.uid('fetch', uid, message_parts)
- mail = mailparser.parse_from_bytes(email_data[0][1])
- logging.info(f"got email subject [{mail.subject}]")
- self.save_email(mail)
- return self.get_local_dir_name(mail)
-
- def get_local_dir_name(self, mail: mailparser.MailParser) -> str:
- dir = f"{self.local_root()}/{self.config.get('address')}"
- name = f"{mail.subject}__{mail.date}"
- name = hashlib.md5(name.encode('utf-8')).hexdigest()
- return f"{dir}/{name}"
-
- def check_email_saved(self, uid: str) -> str:
- message_parts = "(BODY[HEADER])"
- _, email_data = self.client.uid('fetch', uid, message_parts)
- mail = mailparser.parse_from_bytes(email_data[0][1])
- logging.info(f"[{uid}]check email subject [{mail.subject}]")
- dir = self.get_local_dir_name(mail)
- logging.info(f"check email saved {dir}")
- file = f"{dir}/email.txt"
- if os.path.exists(file):
- return dir
- return None
-
- # save email attachment(images)
- def save_email_attachment(self, mail: mailparser.MailParser, email_dir: str):
- for attachment in mail.attachments:
- if attachment['mail_content_type'] in ['image/png', 'image/jpeg', 'image/gif']:
- print('current mail have image attachment')
- img_dir = f"{email_dir}/image"
- if not os.path.exists(img_dir):
- os.makedirs(img_dir)
- filename = attachment['filename']
- filefullname = f"{img_dir}/{filename}"
- image_data = attachment['payload']
- try:
- image_data = base64.b64decode(image_data)
- except base64.binascii.Error:
- image_data = image_data.encode()
- with open(filefullname, 'wb') as f:
- f.write(image_data)
- logging.info(f"save email image {filename} success")
-
- # save email body images(html content)
- def save_body_images(self, html_content: str, email_dir: str):
- # get all image urls
- soup = BeautifulSoup(html_content, 'html.parser')
- img_tags = soup.find_all('img')
- img_urls = [img['src'] for img in img_tags if 'src' in img.attrs]
- logging.info(f'Found {len(img_urls)} images in email body')
-
- name_count = 0
-
- if not os.path.exists(email_dir):
- os.makedirs(email_dir)
-
- for img_url in img_urls:
- # keep the original image filename(last of url)
- ext = img_url.split('/')[-1].split('.')[-1]
- img_filename = os.path.join(email_dir, f"{name_count}.{ext}")
- name_count += 1
- # download image
- response = requests.get(img_url, stream=True)
- if response.status_code == 200:
- with open(img_filename, 'wb') as img_file:
- for chunk in response.iter_content(1024):
- img_file.write(chunk)
- logging.info(f'Downloaded {img_url} to {img_filename}')
- else:
- logging.info(f'Failed to download {img_url}')
-
- # save email content to local dir
- def save_email(self, mail: mailparser.MailParser):
- dir = f"{self.local_root()}/{self.config.get('address')}"
- if not os.path.exists(dir):
- os.makedirs(dir)
- email_dir = self.get_local_dir_name(mail)
- logging.info(f"save email to {email_dir}")
- if not os.path.exists(email_dir):
- os.makedirs(email_dir)
- with open(f"{email_dir}/email.txt", "w", encoding='utf-8') as f:
- # soup = BeautifulSoup(mail.body, 'html.parser')
- f.write(mail.body)
- with open(f"{email_dir}/meta.json", "w", encoding='utf-8') as f:
- mail_dict = json.loads(mail.mail_json)
- if 'body' in mail_dict:
- del mail_dict['body']
- json.dump(mail_dict, f, ensure_ascii=False, indent=4)
- logging.info(f"save email meta info {f.name}")
-
- self.save_email_attachment(mail, email_dir)
- self.save_body_images(mail.body, f"{email_dir}/body_image")
-
-
-class KnowledgeDirSource:
- def __init__(self, config):
- self.config = config
- config["path"] = os.path.abspath(config["path"])
- self.config["type"] = "dir"
-
- @classmethod
- def user_config_items(cls):
- return [("path", "local dir path")]
-
- def id(self):
- return self.config["path"]
-
- def path(self):
- return self.config["path"]
-
- @staticmethod
- async def read_txt_file(file_path:str)->str:
- cur_encode = "utf-8"
- async with aiofiles.open(file_path,'rb') as f:
- cur_encode = chardet.detect(await f.read())['encoding']
-
- async with aiofiles.open(file_path,'r',encoding=cur_encode) as f:
- return await f.read()
-
- async def run_once(self):
- logging.debug(f"knowledge dir source {self.id()} run once")
- journal_client = KnowledgeJournalClient()
- latest_journal = journal_client.latest_journal(self.id())
- if latest_journal is not None:
- if os.path.getmtime(self.path()) <= latest_journal.timestamp:
- logging.debug(f"knowledge dir source {self.id()} ingnored for no update")
- return
- file_pathes = sorted(os.listdir(self.path()), key=lambda x: os.path.getctime(os.path.join(self.path(), x)))
- for rel_path in file_pathes:
- file_path = os.path.join(self.path(), rel_path)
- timestamp = os.path.getctime(file_path)
- if latest_journal is not None:
- if timestamp <= latest_journal.timestamp:
- continue
- ext = os.path.splitext(file_path)[1].lower()
- if ext in ['.jpg', '.jpeg', '.png', '.gif', '.bmp']:
- logging.info(f"knowledge dir source {self.id()} found image file {file_path}")
- image = ImageObjectBuilder({}, {}, file_path).build()
- await KnowledgeBase().insert_object(image)
- journal_client.insert(KnowledgeJournal("dir", self.id(), rel_path, str(image.calculate_id()), timestamp))
- if ext in ['.txt']:
- logging.info(f"knowledge dir source {self.id()} found text file {file_path}")
- text = await self.read_txt_file(file_path)
-
- document = DocumentObjectBuilder({}, {}, text).build()
- await KnowledgeBase().insert_object(document)
- journal_client.insert(KnowledgeJournal("dir", self.id(), rel_path, str(document.calculate_id()), timestamp))
-
-
-
-
-# define singleton class knowledge pipline
-class KnowledgePipline:
- _instance = None
-
- @classmethod
- def get_instance(cls):
- if cls._instance is None:
- cls._instance = KnowledgePipline()
- cls._instance.__singleton_init__()
-
- return cls._instance
-
- def initial(self):
- config_path = self.__config_path()
- logging.info(f"initial knowledge pipline from {config_path}")
- if os.path.exists(config_path):
- config = toml.load(self.__config_path())
- for source_config in config["sources"]:
- if source_config['type'] == 'email':
- self.add_email_source(KnowledgeEmailSource(source_config))
- if source_config['type'] == 'dir':
- self.add_dir_source(KnowledgeDirSource(source_config))
- user_data_dir = AIStorage.get_instance().get_myai_dir()
- default_dir = os.path.abspath(f"{user_data_dir}/data")
- if not os.path.exists(default_dir):
- os.makedirs(default_dir)
- self.add_dir_source(KnowledgeDirSource({"path": default_dir}))
-
- return True
-
- def __singleton_init__(self):
- self.knowledge_base = KnowledgeBase()
- self.email_sources = dict()
- self.dir_sources = dict()
- self.source_queue = list()
- self.run_lock = asyncio.Lock()
- asyncio.create_task(self.run_loop())
-
-
- def save_config(self):
- config = dict()
- config["sources"] = [source.config for source in self.source_queue]
- with open(self.__config_path(), "w") as f:
- toml.dump(config, f)
-
-
- @classmethod
- def __config_path(cls) -> str:
- user_data_dir = AIStorage.get_instance().get_myai_dir()
- return os.path.abspath(f"{user_data_dir}/etc/knowledge.cfg.toml")
-
-
- def add_email_source(self, source: KnowledgeEmailSource):
- if self.email_sources.get(source.id()) is not None:
- return "already exists"
- self.email_sources[source.id()] = source
- self.source_queue.append(source)
- return None
-
- def add_dir_source(self, source: KnowledgeDirSource):
- if self.dir_sources.get(source.id()) is not None:
- logging.info(f"knowledge add source {source.id()} failed for already exists")
- return "already exists"
- logging.info(f"knowledge added source {source.id()}")
- self.dir_sources[source.id()] = source
- self.source_queue.append(source)
- return None
-
- def get_latest_journals(self, topn) -> [KnowledgeJournal]:
- return KnowledgeJournalClient().latest_journals(topn)
-
- async def run_loop(self):
- while True:
- await self.run_once()
- await asyncio.sleep(5)
-
- async def run_once(self):
- logging.info(f"knowledge pipeline started")
- # sources = list()
- # async with self.run_lock:
- # for source in self.source_queue:
- # sources.append(source)
- # for source in sources:
- # await source.run_once()
- for source in self.source_queue:
- await source.run_once()
-
- logging.info(f"knowledge pipeline finished")
\ No newline at end of file
diff --git a/src/aios_kernel/whisper_node.py b/src/aios_kernel/whisper_node.py
deleted file mode 100644
index b036ad5..0000000
--- a/src/aios_kernel/whisper_node.py
+++ /dev/null
@@ -1,111 +0,0 @@
-from asyncio import Queue
-import asyncio
-import openai
-import os
-import logging
-
-from .compute_node import ComputeNode
-from .compute_task import ComputeTask, ComputeTaskResult, ComputeTaskState, ComputeTaskType
-
-logger = logging.getLogger(__name__)
-
-
-class WhisperComputeNode(ComputeNode):
- _instance = None
-
- def __new__(cls):
- if cls._instance is None:
- cls._instance = super().__new__(cls)
- cls._instance.is_start = False
- return cls._instance
-
- def __init__(self) -> None:
- super().__init__()
- if self.is_start is True:
- logger.warn("WhisperComputeNode is already start")
- return
-
- self.is_start = True
- self.node_id = "whisper_node"
- self.enable = True
- self.task_queue = Queue()
- self.open_api_key = None
-
- if self.open_api_key is None and os.getenv("OPENAI_API_KEY") is not None:
- self.open_api_key = os.getenv("OPENAI_API_KEY")
-
- if self.open_api_key is None:
- raise Exception("WhisperComputeNode open_api_key is None")
-
- self.start()
-
- def start(self):
- async def _run_task_loop():
- while True:
- task = await self.task_queue.get()
- try:
- result = self._run_task(task)
- if result is not None:
- task.state = ComputeTaskState.DONE
- task.result = result
- except Exception as e:
- logger.error(f"whisper_node run task error: {e}")
- task.state = ComputeTaskState.ERROR
- task.result = ComputeTaskResult()
- task.result.set_from_task(task)
- task.result.worker_id = self.node_id
- task.result.result_str = str(e)
-
- asyncio.create_task(_run_task_loop())
-
- def _run_task(self, task: ComputeTask):
- task.state = ComputeTaskState.RUNNING
- prompt = task.params["prompt"]
- response_format = None
- if "response_format" in task.params:
- response_format = task.params["response_format"]
- temperature = None
- if "temperature" in task.params:
- temperature = task.params["temperature"]
- language = None
- if "language" in task.params:
- language = task.params["language"]
- file = task.params["file"]
-
- resp = openai.Audio.transcribe("whisper-1",
- file,
- self.open_api_key,
- prompt=prompt,
- response_format=response_format,
- temperature=temperature,
- language=language)
- result = ComputeTaskResult()
- result.set_from_task(task)
- result.worker_id = self.node_id
- result.result_str = resp["text"]
- result.result = resp
- return result
-
- async def push_task(self, task: ComputeTask, proiority: int = 0):
- logger.info(f"whisper_node push task: {task.display()}")
- self.task_queue.put_nowait(task)
-
- async def remove_task(self, task_id: str):
- pass
-
- def get_task_state(self, task_id: str):
- pass
-
- def display(self) -> str:
- return f"WhisperComputeNode: {self.node_id}"
-
- def get_capacity(self):
- return 0
-
- def is_support(self, task_type: ComputeTaskType) -> bool:
- if task_type == ComputeTaskType.VOICE_2_TEXT:
- return True
- return False
-
- def is_local(self) -> bool:
- return False
diff --git a/src/aios_kernel/workspace_env.py b/src/aios_kernel/workspace_env.py
deleted file mode 100644
index 8bfe00e..0000000
--- a/src/aios_kernel/workspace_env.py
+++ /dev/null
@@ -1,173 +0,0 @@
-# this env is designed for workflow owner filesystem, support file/directory operations
-
-import subprocess
-import tempfile
-import threading
-import traceback
-import time
-import ast
-import sys
-import os
-import re
-
-from .environment import Environment,EnvironmentEvent
-from .ai_function import AIFunction,SimpleAIFunction
-
-
-class CodeInterpreter:
- def __init__(self, language, debug_mode):
- self.language = language
- self.proc = None
- self.active_line = None
- self.debug_mode = debug_mode
-
- def start_process(self):
- start_cmd = sys.executable + " -i -q -u"
- self.proc = subprocess.Popen(start_cmd.split(),
- stdin=subprocess.PIPE,
- stdout=subprocess.PIPE,
- stderr=subprocess.PIPE,
- text=True,
- bufsize=0)
-
- # Start watching ^ its `stdout` and `stderr` streams
- threading.Thread(target=self.save_and_display_stream,
- args=(self.proc.stdout, False), # Passes False to is_error_stream
- daemon=True).start()
- threading.Thread(target=self.save_and_display_stream,
- args=(self.proc.stderr, True), # Passes True to is_error_stream
- daemon=True).start()
-
- def warp_code(self,pycode:str)->str:
- # Add import traceback
- code = "import traceback\n" + pycode
- # Parse the input code into an AST
- parsed_code = ast.parse(code)
- # Wrap the entire code's AST in a single try-except block
- try_except = ast.Try(
- body=parsed_code.body,
- handlers=[
- ast.ExceptHandler(
- type=ast.Name(id="Exception", ctx=ast.Load()),
- name=None,
- body=[
- ast.Expr(
- value=ast.Call(
- func=ast.Attribute(value=ast.Name(id="traceback", ctx=ast.Load()), attr="print_exc", ctx=ast.Load()),
- args=[],
- keywords=[]
- )
- ),
- ]
- )
- ],
- orelse=[],
- finalbody=[]
- )
-
- parsed_code.body = [try_except]
- return ast.unparse(parsed_code)
-
- def run(self,py_code:str):
- """
- Executes code.
- """
- # Get code to execute
- self.code = py_code
-
- # Start the subprocess if it hasn't been started
- if not self.proc:
- try:
- self.start_process()
- except Exception as e:
- # Sometimes start_process will fail!
- # Like if they don't have `node` installed or something.
-
- traceback_string = traceback.format_exc()
- self.output = traceback_string
- # Before you return, wait for the display to catch up?
- # (I'm not sure why this works)
- time.sleep(0.1)
-
- return self.output
-
- self.output = ""
-
- self.print_cmd = 'print("{}")'
- code = self.warp_code(py_code)
-
- if self.debug_mode:
- print("Running code:")
- print(code)
- print("---")
-
- self.done = threading.Event()
- self.done.clear()
-
- # Write code to stdin of the process
- try:
- self.proc.stdin.write(code + "\n")
- self.proc.stdin.flush()
- except BrokenPipeError:
- return
- self.done.wait()
- time.sleep(0.1)
- return self.output
-
- def save_and_display_stream(self, stream, is_error_stream):
-
- for line in iter(stream.readline, ''):
- if self.debug_mode:
- print("Recieved output line:")
- print(line)
- print("---")
-
- line = line.strip()
- if is_error_stream and "KeyboardInterrupt" in line:
- raise KeyboardInterrupt
- elif "END_OF_EXECUTION" in line:
- self.done.set()
- self.active_line = None
- else:
- self.output += "\n" + line
- self.output = self.output.strip()
-
-
-
-class WorkspaceEnvironment(Environment):
- def __init__(self, env_id: str) -> None:
- super().__init__(env_id)
-
- operator_param = {
- "command": "command will execute",
- }
- self.add_ai_function(SimpleAIFunction("shell_exec",
- "execute shell command in linux bash",
- self.shell_exec,operator_param))
-
- #run_code_param = {
- # "pycode": "python code will execute",
- #}
- #self.add_ai_function(SimpleAIFunction("run_code",
- # "execute python code",
- # self.run_code,run_code_param))
-
-
- async def shell_exec(self,command:str) -> str:
- import asyncio.subprocess
- process = await asyncio.create_subprocess_shell(
- command,
- stdout=asyncio.subprocess.PIPE,
- stderr=asyncio.subprocess.PIPE
- )
- stdout, stderr = await process.communicate()
- returncode = process.returncode
- if returncode == 0:
- return f"Execute success! stdout is:\n{stdout}\n"
- else:
- return f"Execute failed! stderr is:\n{stderr}\n"
-
- async def run_code(self,pycode:str) -> str:
- interpreter = CodeInterpreter("python",True)
- return interpreter.run(pycode)
-
diff --git a/src/component/agent_manager/agent_manager.py b/src/component/agent_manager/agent_manager.py
index 083b680..36d799b 100644
--- a/src/component/agent_manager/agent_manager.py
+++ b/src/component/agent_manager/agent_manager.py
@@ -1,10 +1,12 @@
-
+import importlib
import logging
import toml
+import os
+import sys
+import runpy
from typing import Any, Callable, Dict, List, Optional, Union
-from aios_kernel import AIAgent,AIAgentTemplete,AIStorage
-from package_manager import PackageEnv,PackageEnvManager,PackageMediaInfo,PackageInstallTask
+from aios import AIAgent,AIAgentTemplete,AIStorage,BaseAIAgent,PackageEnv,PackageEnvManager,PackageMediaInfo,PackageInstallTask,WorkspaceEnvironment
logger = logging.getLogger(__name__)
@@ -15,19 +17,19 @@ cache = "./.agents"
class AgentManager:
_instance = None
-
+
@classmethod
def get_instance(cls)->'AgentManager':
if cls._instance is None:
cls._instance = AgentManager()
return cls._instance
-
+
def __init__(self) -> None:
self.agent_templete_env : PackageEnv = None
self.agent_env : PackageEnv = None
- self.db_path : str = None
- self.loaded_agent_instance : Dict[str,AIAgent] = None
-
+ self.db_path : str = None
+ self.loaded_agent_instance : Dict[str,BaseAIAgent] = None
+
async def initial(self) -> None:
system_app_dir = AIStorage.get_instance().get_system_app_dir()
user_data_dir = AIStorage.get_instance().get_myai_dir()
@@ -40,30 +42,42 @@ class AgentManager:
self.agent_env.parent_envs.append(sys_agent_env)
self.db_path = f"{user_data_dir}/messages.db"
+ self.agent_memory_base_dir = f"{user_data_dir}/memory"
+ self.workspace_base_dir = f"{user_data_dir}/workspace"
self.loaded_agent_instance = {}
-
+
return True
-
+
async def scan_all_agent(self)->None:
pass
-
-
+
+ async def is_exist(self,agent_id:str) -> bool:
+ the_aget = await self.get(agent_id)
+ if the_aget:
+ return True
+ return False
+
async def get(self,agent_id:str) -> AIAgent:
the_agent = self.loaded_agent_instance.get(agent_id)
if the_agent:
return the_agent
-
+
# try load from disk
agent_media_info = self.agent_env.load(agent_id)
if agent_media_info is None:
return None
-
+
the_agent : AIAgent = await self._load_agent_from_media(agent_media_info)
if the_agent is None:
logger.warn(f"load agent {agent_id} from media failed!")
-
- the_agent.chat_db = self.db_path
- return the_agent
+ return None
+
+ if await the_agent.initial():
+ return the_agent
+ else:
+ logger.warn(f"initial agent {agent_id} failed!")
+ return None
+
def remove(self,agent_id:str)->int:
pass
@@ -77,19 +91,19 @@ class AgentManager:
def install(self,templete_id) -> PackageInstallTask:
installer = self.agent_templete_env.get_installer()
return installer.install(templete_id)
-
+
def uninstall(self,templete_id) -> int:
- pass
-
+ pass
+
async def _load_templete_from_media(self,templete_media:PackageMediaInfo) -> AIAgentTemplete:
pass
- async def _load_agent_from_media(self,agent_media:PackageMediaInfo) -> AIAgent:
+ async def _load_agent_from_media(self,agent_media:PackageMediaInfo) -> BaseAIAgent:
reader = self.agent_env._create_media_loader(agent_media)
if reader is None:
logger.error(f"create media loader for {agent_media} failed!")
return None
-
+
try:
config_file = await reader.read("agent.toml","r")
if config_file is None:
@@ -99,16 +113,25 @@ class AgentManager:
config_data = await config_file.read()
config = toml.loads(config_data)
result_agent = AIAgent()
- if result_agent.load_from_config(config) is False:
+
+ if await result_agent.load_from_config(config) is False:
logger.error(f"load agent from {agent_media} failed!")
return None
return result_agent
except Exception as e:
- logger.error(f"read agent.toml cfg from {agent_media} failed! unexpected error occurred: {str(e)}")
- return None
-
+ custom_agent = os.path.join(agent_media.full_path,"agent.py")
+ if not os.path.exists(custom_agent):
+ logger.error(f"read agent.toml cfg from {agent_media} failed! unexpected error occurred: {str(e)}")
+ return None
+
+ agent = runpy.run_path(custom_agent)
+ if "init" not in agent:
+ logger.error(f"read agent.toml cfg from {agent_media} failed! unexpected error occurred: {str(e)}")
+ return None
+ return agent["init"]()
+
+
-
def create(self,template,agent_name,agent_last_name,agent_introduce) -> AIAgent:
pass
diff --git a/src/component/common_environment/__init__.py b/src/component/common_environment/__init__.py
new file mode 100644
index 0000000..5587238
--- /dev/null
+++ b/src/component/common_environment/__init__.py
@@ -0,0 +1,3 @@
+from .local_document import LocalKnowledgeBase, ScanLocalDocument, ParseLocalDocument
+from .local_file_system import FilesystemEnvironment
+from .shell import ShellEnvironment
\ No newline at end of file
diff --git a/src/component/common_environment/local_document.py b/src/component/common_environment/local_document.py
new file mode 100644
index 0000000..8929d94
--- /dev/null
+++ b/src/component/common_environment/local_document.py
@@ -0,0 +1,618 @@
+import os
+import aiofiles
+import chardet
+import string
+import sqlite3
+import json
+import re
+import threading
+import logging
+import hashlib
+from markdown import Markdown
+import PyPDF2
+import datetime
+from typing import Optional, List
+from aios import *
+from aios.environment.workspace_env import TodoListEnvironment, TodoListType
+from .local_file_system import FilesystemEnvironment
+
+logger = logging.getLogger(__name__)
+
+class MetaDatabase:
+ def __init__(self,db_path:str):
+ self.db_path = db_path
+ self._get_conn()
+
+ def _get_conn(self):
+ """ get db connection """
+ local = threading.local()
+ if not hasattr(local, 'conn'):
+ local.conn = self._create_connection(self.db_path)
+ return local.conn
+
+
+ def _create_connection(self, db_file):
+ """ create a database connection to a SQLite database """
+ conn = None
+ try:
+ conn = sqlite3.connect(db_file)
+ except Exception as e:
+ logger.error("Error occurred while connecting to database: %s", e)
+ return None
+
+ if conn:
+ self._create_tables(conn)
+
+ return conn
+
+ def _create_tables(self,conn):
+ cursor = conn.cursor()
+ cursor.execute('''
+ CREATE TABLE IF NOT EXISTS documents (
+ doc_path TEXT PRIMARY KEY,
+ length INTEGER,
+ last_modify TEXT,
+ doc_hash TEXT,
+ create_time TEXT
+ )
+ ''')
+ cursor.execute('''
+ CREATE TABLE IF NOT EXISTS knowledge (
+ doc_hash TEXT PRIMARY KEY,
+ title TEXT,
+ summary TEXT,
+ content TEXT,
+ catalogs TEXT,
+ tags TEXT,
+ llm_title TEXT,
+ llm_summary TEXT,
+ create_time TEXT
+ )
+ ''')
+
+ cursor.execute('''
+ CREATE INDEX IF NOT EXISTS idx_documents_doc_hash
+ ON documents (doc_hash)
+ ''')
+
+ cursor.execute('''
+ CREATE INDEX IF NOT EXISTS idx_knowledge_tags
+ ON knowledge (tags)
+ ''')
+
+ conn.commit()
+
+ def add_doc(self, doc_path: str, length: int, last_modify: str, doc_hash: Optional[str] = None):
+ conn = self._get_conn()
+ cursor = conn.cursor()
+ create_time = datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S")
+ cursor.execute('''
+ INSERT INTO documents (doc_path, length, last_modify, doc_hash,create_time)
+ VALUES (?, ?, ?, ?,?)
+ ''', (doc_path, length, last_modify, doc_hash,create_time))
+ conn.commit()
+
+ def is_doc_exist(self, doc_path: str) -> bool:
+ conn = self._get_conn()
+ cursor = conn.cursor()
+ cursor.execute('''
+ SELECT doc_path
+ FROM documents
+ WHERE doc_path = ?
+ ''', (doc_path,))
+ return len(cursor.fetchall()) > 0
+
+ def set_doc_hash(self, doc_path: str, doc_hash: str):
+ conn = self._get_conn()
+ cursor = conn.cursor()
+ cursor.execute('''
+ UPDATE documents
+ SET doc_hash = ?
+ WHERE doc_path = ?
+ ''', (doc_hash, doc_path))
+ conn.commit()
+
+ def get_docs_without_hash(self,limit:int=1024) -> List[str]:
+ conn = self._get_conn()
+ cursor = conn.cursor()
+ cursor.execute('''
+ SELECT doc_path
+ FROM documents
+ WHERE doc_hash IS NULL OR doc_hash = ''
+ ORDER BY create_time DESC
+ LIMIT ?
+ ''',(limit,))
+ return [row[0] for row in cursor.fetchall()]
+
+ #metadata["summary"]
+ #metadata["catalogs"]
+ #metadata["tags"]
+ def add_knowledge(self, doc_hash: str, metadata: dict,content:str = None,):
+ conn = self._get_conn()
+ cursor = conn.cursor()
+
+ create_time = datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S")
+ summary = metadata.get("summary", "")
+ catalogs = json.dumps(metadata.get("catalogs", {}),ensure_ascii=False)
+ title = metadata.get("title","")
+ tags = ','.join(metadata.get("tags", []))
+
+ cursor.execute('''
+ INSERT INTO knowledge (doc_hash, title , summary , catalogs , tags,create_time)
+ VALUES (?, ?, ?, ?, ?,?)
+ ''', (doc_hash, title, summary, catalogs, tags,create_time))
+ conn.commit()
+
+ #llm_result["summary"]
+ #llm_result["tags"]
+ #llm_result["catalog"]
+ def set_knowledge_llm_result(self, doc_hash: str, meta: dict):
+ conn = self._get_conn()
+ cursor = conn.cursor()
+
+ title = meta.get("title", "")
+ summary = meta.get("summary", "")
+ catalogs = json.dumps(meta.get("catalogs", {}),ensure_ascii=False)
+ tags = ','.join(meta.get("tags", []))
+
+ cursor.execute('''
+ UPDATE knowledge
+ SET llm_title = ?,llm_summary = ?, catalogs = ?, tags = ?
+ WHERE doc_hash = ?
+ ''', (title,summary, catalogs, tags, doc_hash))
+ conn.commit()
+
+
+ def get_hash_by_doc_path(self, doc_path: str) -> Optional[str]:
+ conn = self._get_conn()
+ cursor = conn.cursor()
+ cursor.execute('''
+ SELECT doc_hash
+ FROM documents
+ WHERE doc_path = ?
+ ''', (doc_path,))
+ row = cursor.fetchone()
+ if row is None:
+ return None
+ return row[0]
+
+ def get_knowledge(self, doc_hash: str) -> Optional[dict]:
+ conn = self._get_conn()
+ cursor = conn.cursor()
+ cursor.execute('''
+ SELECT title, summary, catalogs, tags, llm_title, llm_summary
+ FROM knowledge
+ WHERE doc_hash = ?
+ ''', (doc_hash,))
+ row = cursor.fetchone()
+ if row is None:
+ return None
+
+ # get doc path
+ cursor.execute('''
+ SELECT doc_path
+ FROM documents
+ WHERE doc_hash = ?
+ ''', (doc_hash,))
+ row2 = cursor.fetchone()
+ if row2 is None:
+ return None
+ doc_path = row2[0]
+
+
+ return {
+ "full_path": doc_path,
+ "title": row[0],
+ "summary": row[1],
+ "catalogs": row[2],
+ "tags": row[3],
+ "llm_title" : row[4],
+ "llm_summary" : row[5],
+ }
+
+ def get_knowledge_without_llm_title(self,limit:int=16) -> List[str]:
+ conn = self._get_conn()
+ cursor = conn.cursor()
+ cursor.execute('''
+ SELECT doc_hash
+ FROM knowledge
+ WHERE llm_title IS NULL OR llm_title = ''
+ ORDER BY create_time DESC
+ LIMIT ?
+ ''',(limit,))
+ return [row[0] for row in cursor.fetchall()]
+
+ def query_docs_by_tag(self, tag: str) -> List[str]:
+ conn = self._get_conn()
+ cursor = conn.cursor()
+ tag_json = json.dumps(tag,ensure_ascii=False) # 将标签转换为 JSON 字符串
+ cursor.execute('''
+ SELECT documents.doc_path
+ FROM documents
+ JOIN knowledge ON documents.doc_hash = knowledge.doc_hash
+ WHERE json_extract(knowledge.tags, '$') LIKE ?
+ ''', (tag))
+ return [row[0] for row in cursor.fetchall()]
+
+# singleton
+class LearningCache:
+ _instance_lock = threading.Lock()
+ _instance = None
+
+ def __instance_init__(self):
+ self.cache = {}
+ self.cache_lock = threading.Lock()
+
+ def __new__(cls, *args, **kwargs):
+ if cls._instance is None:
+ with LearningCache._instance_lock:
+ if cls._instance is None:
+ cls._instance = super().__new__(cls)
+ cls._instance.__instance_init__()
+ return cls._instance
+
+ def add(self, key, value):
+ with self.cache_lock:
+ self.cache[key] = value
+
+ def get(self, key):
+ with self.cache_lock:
+ return self.cache.get(key)
+
+ def remove(self, key):
+ with self.cache_lock:
+ return self.cache.pop(key, None)
+
+
+class LocalKnowledgeBase(CompositeEnvironment):
+ def __init__(self, workspace: str) -> None:
+ super().__init__(workspace)
+ self.root_path = f"{workspace}/knowledge"
+ if os.path.exists(self.root_path) is False:
+ os.makedirs(self.root_path)
+ self.meta_db = MetaDatabase(f"{self.root_path}/kb.db")
+ self.learning_cache = LearningCache()
+
+ async def learn(op:dict):
+ full_path = op.get("original_path")
+ if not full_path:
+ return
+ meta = self.learning_cache.get(full_path)
+ meta.update(op)
+
+ self.add_ai_operation(SimpleAIAction(
+ op="learn",
+ description="update knowledge llm summary",
+ func_handler=learn,
+ ))
+
+ self.fs = FilesystemEnvironment(self.root_path)
+ self.add_env(self.fs)
+
+ async def get_knowledege_catalog(self,path:str=None,only_dir =True,max_depth:int=5)->str:
+ if path:
+ full_path = f"{self.root_path}/{path}"
+ else:
+ full_path = self.root_path
+
+ catlogs,file_count = await self.get_directory_structure(full_path,max_depth,only_dir)
+ return catlogs
+
+ async def get_directory_structure(self,root_dir, max_depth:int=4, only_dir=True, indent=1):
+ file_count = 0
+ structure_str = ''
+ if os.path.isdir(root_dir):
+ sub_files = []
+ with os.scandir(root_dir) as it:
+ for entry in it:
+ if entry.is_dir():
+ sub_structure, sub_count = await self.get_directory_structure(entry.path, max_depth, only_dir, indent + 1)
+ if sub_structure:
+ structure_str += sub_structure
+ file_count += sub_count
+ else:
+ file_count += 1
+ sub_files.append(entry.name)
+
+ if only_dir is False:
+ for file_name in sub_files:
+ structure_str = structure_str + ' ' * (indent+1) + file_name + '\n'
+
+ dir_name = os.path.basename(root_dir)
+ dir_info = f"{dir_name} "
+
+
+ structure_str = ' ' * indent + dir_info + '\n' + structure_str
+
+ if indent - 1 >= max_depth:
+ return None, file_count
+ else:
+ return structure_str, file_count
+
+ # inner_function
+ async def get_knowledge_meta(self,path:str) -> str:
+ full_path = f"{self.root_path}/{path}"
+ if os.islink(full_path):
+ org_path = os.readlink(full_path)
+ hash = self.meta_db.get_hash_by_doc_path(org_path)
+ if hash:
+ return self.meta_db.get_knowledge(org_path)
+
+ return "not found"
+
+ async def load_knowledge_content(self,path:str,pos:int=0,length:int=None) -> str:
+ if path.endswith("pdf"):
+ logger.info("load_knowledge_content:pdf")
+ dir_path = os.path.dirname(path)
+ base_name = os.path.basename(path)
+ text_content_path = f"{dir_path}/.{base_name}.txt"
+ if os.path.exists(text_content_path) is False:
+ return None
+ async with aiofiles.open(path, mode='r', encoding=cur_encode) as f:
+ await f.seek(pos)
+ content = await f.read(length)
+ return content
+ else:
+ async with aiofiles.open(path,'rb') as f:
+ cur_encode = chardet.detect(await f.read())['encoding']
+
+ async with aiofiles.open(path, mode='r', encoding=cur_encode) as f:
+ await f.seek(pos)
+ content = await f.read(length)
+ return content
+
+
+class ScanLocalDocument:
+ def __init__(self, env: KnowledgePipelineEnvironment, config):
+ self.env = env
+ workspace = string.Template(config["workspace"]).substitute(myai_dir=AIStorage.get_instance().get_myai_dir())
+ path = string.Template(config["path"]).substitute(myai_dir=AIStorage.get_instance().get_myai_dir())
+ self.knowledge_base = LocalKnowledgeBase(workspace)
+ self.path = path
+
+ def _support_file(self,file_name:str) -> bool:
+ if file_name.startswith("."):
+ return False
+
+ if file_name.endswith(".pdf"):
+ return True
+ if file_name.endswith(".md"):
+ return True
+ if file_name.endswith(".txt"):
+ return True
+ return False
+
+ async def next(self):
+ while True:
+ for root, dirs, files in os.walk(self.path):
+ for file in files:
+ if self._support_file(file):
+ full_path = os.path.join(root, file)
+ full_path = os.path.normpath(full_path)
+ if self.knowledge_base.meta_db.is_doc_exist(full_path):
+ continue
+ yield(full_path, full_path)
+ else:
+ continue
+ yield(None, None)
+
+
+
+class ParseLocalDocument:
+ def __init__(self, env: KnowledgePipelineEnvironment, config: dict):
+ self.env = env
+ workspace = string.Template(config["workspace"]).substitute(myai_dir=AIStorage.get_instance().get_myai_dir())
+ self.todo_list = TodoListEnvironment(workspace, TodoListType.TO_LEARN)
+ self.knowledge_base = LocalKnowledgeBase(workspace)
+ self.token_limit = config.get("token_limit", 4000)
+ self.assign_to = config.get("assign_to")
+
+
+ async def parse(self, full_path: str) -> str:
+ file_stat = os.stat(full_path)
+ if file_stat.st_size < 1:
+ return full_path
+ hash, parse_meta = self._parse_document(full_path)
+ parse_meta["original_path"] = full_path
+ llm_meta = await self._learn_by_agent(parse_meta)
+ self.knowledge_base.meta_db.add_doc(full_path,file_stat.st_size,file_stat.st_mtime,hash)
+ self.knowledge_base.meta_db.add_knowledge(hash,parse_meta)
+ self.knowledge_base.meta_db.set_knowledge_llm_result(hash,llm_meta)
+ path_list = llm_meta.get("path")
+ new_title = llm_meta.get("title")
+ if path_list:
+ for new_path in path_list:
+ new_path = f"{new_path}/{new_title}"
+ await self.knowledge_base.fs.symlink(full_path, new_path)
+ logger.info(f"create soft link {full_path} -> {new_path}")
+ return full_path
+
+ async def _get_meta_prompt(self,meta: dict,temp_meta = None,need_catalogs = False) -> str:
+ kb_tree = await self.knowledge_base.get_knowledege_catalog()
+
+ known_obj = {}
+ title = meta.get("title")
+ if title:
+ known_obj["title"] = title
+ summary = meta.get("summary")
+ if summary:
+ known_obj["summary"] = summary
+ tags = meta.get("tags")
+ if tags:
+ known_obj["tags"] = tags
+ if need_catalogs:
+ catalogs = meta.get("catalogs")
+ if catalogs:
+ known_obj["catalogs"] = catalogs
+
+ if temp_meta:
+ for key in temp_meta.keys():
+ known_obj[key] = temp_meta[key]
+
+ org_path = meta.get("original_path")
+ known_obj["original_path"] = org_path
+ return f"# Known information:\n## Current directory structure:\n{kb_tree}\n## Knowlege Metadata:\n{json.dumps(known_obj,ensure_ascii=False)}\n"
+
+ def _token_len(self, text: str) -> int:
+ return CustomAIAgent("", "gpt-4-turbo-preview", self.token_limit).token_len(text=text)
+
+
+ async def _learn_by_agent(self, meta:dict) -> dict:
+ # Objectives:
+ # Obtain better titles, abstracts, table of contents (if necessary), tags
+ # Determine the appropriate place to put it (in line with the organization's goals)
+ # Known information:
+ # The reason why the target service's learn_prompt is being sorted
+ # Summary of the organization's work (if any)
+ # The current structure of the knowledge base (note the size control) gen_kb_tree_prompt (when empty, LLM should generate an appropriate initial directory structure)
+ # Original path, current title, abstract, table of contents
+
+ # Sorting long files (general tricks)
+ # Indicate that the input is part of the content, let LLM generate intermediate results for the task
+ # Enter the content in sequence, when the last content block is input, LLM gets the result
+ full_content = await self.knowledge_base.load_knowledge_content(meta["original_path"])
+ full_content_len = self._token_len(full_content)
+ full_path = meta["original_path"]
+ self.knowledge_base.learning_cache.add(full_path, meta)
+
+
+ if full_content_len < self.token_limit:
+ # 短文章不用总结catalog
+ todo = AgentTodo()
+ todo.worker = self.assign_to
+ todo.title = meta["title"]
+ meta_prompt = await self._get_meta_prompt(meta,None)
+ todo.detail = meta_prompt + full_content
+ await self.todo_list.create_todo(None, todo)
+ await self.todo_list.wait_todo_done(todo.todo_id)
+ else:
+ logger.warning(f"llm_read_article: article {full_path} use LLM loop learn!")
+ pos = 0
+ read_len = int(self.token_limit * 1.2)
+
+ is_final = False
+ while pos < full_content_len:
+ _content = full_content[pos:pos+read_len]
+ part_cotent_len = len(_content)
+ if part_cotent_len < read_len:
+ # last chunk
+ is_final = True
+ part_content = f"<>\n{_content}"
+ else:
+ part_content = f"<>\n{_content}"
+
+ pos = pos + read_len
+ temp_meta = self.knowledge_base.learning_cache.get(full_path)
+ todo = AgentTodo()
+ todo.worker = self.assign_to
+ todo.title = meta["title"]
+ meta_prompt = await self._get_meta_prompt(meta,temp_meta)
+ todo.detail = meta_prompt + part_content
+ self.todo_list.create_todo(None, todo)
+ todo = await self.todo_list.wait_todo_done(todo.todo_id)
+ if is_final:
+ break
+ return self.knowledge_base.learning_cache.remove(full_path)
+
+ def _parse_pdf_bookmarks(self,bookmarks, parent:list):
+ for item in bookmarks:
+ if isinstance(item,list):
+ self._parse_pdf_bookmarks(item,parent)
+ else:
+ if item.title:
+ new_item = {}
+ new_item["page"] = item.page.idnum
+ new_item["title"] = item.title
+ my_childs = []
+ if item.childs:
+ if len(item.childs) > 0:
+ self._parse_pdf_bookmarks(item.childs, my_childs)
+ new_item["childs"] = my_childs
+ parent.append(new_item)
+ else:
+ logger.warning("parse pdf bookmarks failed: item.title is None!")
+
+ return
+
+ def _parse_pdf(self,doc_path:str):
+ metadata = {}
+ with open(doc_path, 'rb') as file:
+ reader = PyPDF2.PdfReader(file)
+ try:
+ doc_info = reader.metadata
+ if doc_info:
+ if doc_info.title:
+ metadata["title"] = doc_info.title
+ if doc_info.author:
+ metadata["authors"] = doc_info.author
+ except Exception as e:
+ logger.warn("parse pdf metadata failed:%s",e)
+
+ dir_path = os.path.dirname(doc_path)
+ base_name = os.path.basename(doc_path)
+ text_content_path = f"{dir_path}/.{base_name}.txt"
+ full_text = ""
+
+ for page in reader.pages:
+ text = page.extract_text()
+ full_text += text
+ with open(text_content_path, 'w', encoding='utf-8') as f:
+ f.write(full_text)
+
+ try:
+ bookmarks = reader.outline
+ if bookmarks:
+ catalogs = []
+ self._parse_pdf_bookmarks(bookmarks,catalogs)
+ metadata["catalogs"] = json.dumps(catalogs,ensure_ascii=False)
+ except Exception as e:
+ logger.warn("parse pdf bookmarks failed:%s",e)
+
+ return metadata
+
+ def _parse_txt(self,doc_path:str):
+ return {}
+
+ def _parse_md(self,doc_path:str):
+ metadata = {}
+ cur_encode = "utf-8"
+ with open(doc_path,'rb') as f:
+ cur_encode = chardet.detect(f.read(1024))['encoding']
+
+ with open(doc_path, mode='r', encoding=cur_encode) as f:
+ content = f.read()
+ match = re.search(r'^# (.*)', content, re.MULTILINE)
+ if match:
+ metadata['title'] = match.group(1).strip()
+ md = Markdown(extensions=['toc'])
+ html_str = md.convert(content)
+ toc = md.toc
+ if toc:
+ metadata['catalogs'] = toc
+
+ return metadata
+
+ def _parse_document(self,doc_path:str):
+ hash_result = None
+ title = os.path.basename(doc_path)
+ meta_data = {}
+
+ with open(doc_path, "rb") as f:
+ hash_md5 = hashlib.md5()
+ for chunk in iter(lambda: f.read(1024*1024), b""):
+ hash_md5.update(chunk)
+ hash_result = hash_md5.hexdigest()
+ try:
+ if doc_path.endswith(".md"):
+ meta_data = self._parse_md(doc_path)
+ elif doc_path.endswith(".pdf"):
+ meta_data = self._parse_pdf(doc_path)
+ except Exception as e:
+ logger.error("parse document %s failed:%s",doc_path,e)
+ # traceback.print_exc()
+
+ if not "title" in meta_data:
+ meta_data["title"] = title
+ logger.info("parse document %s!",doc_path)
+ return hash_result, meta_data
+
diff --git a/src/component/common_environment/local_file_system.py b/src/component/common_environment/local_file_system.py
new file mode 100644
index 0000000..f65b99e
--- /dev/null
+++ b/src/component/common_environment/local_file_system.py
@@ -0,0 +1,139 @@
+import json
+import os
+import aiofiles
+from typing import Any,List,Dict
+import chardet
+from aios import SimpleAIAction
+from aios import SimpleEnvironment
+
+class FilesystemEnvironment(SimpleEnvironment):
+ def __init__(self, workspace: str) -> None:
+ super().__init__(workspace)
+ self.root_path = workspace
+
+ # if op["op"] == "create":
+ # await self.create(op["path"],op["content"])
+
+ async def write(op):
+ is_append = op.get("is_append")
+ if is_append is None:
+ is_append = False
+ return await self.write(op["path"],op["content"],is_append)
+ self.add_ai_operation(SimpleAIAction(
+ op="write",
+ description="write file",
+ func_handler=write,
+ ))
+
+ async def delete(op):
+ return await self.delete(op["path"])
+ self.add_ai_operation(SimpleAIAction(
+ op="delete",
+ description="delete path",
+ func_handler=delete,
+ ))
+
+ async def rename(op):
+ return await self.move(op["path"],op["new_name"])
+ self.add_ai_operation(SimpleAIAction(
+ op="rename",
+ description="rename path",
+ func_handler=rename,
+ ))
+
+ # file system operation: list,read,write,delete,move,stat
+ # inner_function
+ async def list(self,path:str,only_dir:bool=False) -> str:
+ directory_path = self.root_path + path
+ items = []
+
+ with await aiofiles.os.scandir(directory_path) as entries:
+ async for entry in entries:
+ is_dir = entry.is_dir()
+ if only_dir and not is_dir:
+ continue
+ item_type = "directory" if is_dir else "file"
+ items.append({"name": entry.name, "type": item_type})
+
+ return json.dumps(items,ensure_ascii=False)
+
+ # inner_function
+ async def read(self,path:str) -> str:
+ file_path = self.root_path + path
+ cur_encode = "utf-8"
+ async with aiofiles.open(file_path,'rb') as f:
+ cur_encode = chardet.detect(await f.read())['encoding']
+
+ async with aiofiles.open(file_path, mode='r', encoding=cur_encode) as f:
+ content = await f.read(2048)
+ return content
+
+
+ # operation or inner_function (MOST IMPORTANT FUNCTION)
+ async def write(self,path:str,content:str,is_append:bool=False) -> str:
+ file_path = self.root_path + path
+ try:
+ if is_append:
+ async with aiofiles.open(file_path, mode='a', encoding="utf-8") as f:
+ await f.write(content)
+ else:
+ if content is None:
+ # create dir
+ dir_path = self.root_path + path
+ os.makedirs(dir_path)
+ return True
+ else:
+ file_path = self.root_path + path
+ os.makedirs(os.path.dirname(file_path),exist_ok=True)
+ async with aiofiles.open(file_path, mode='w', encoding="utf-8") as f:
+ await f.write(content)
+ return True
+
+ except Exception as e:
+ return str(e)
+ return None
+
+
+ # operation or inner_function
+ async def delete(self,path:str) -> str:
+ try:
+ file_path = self.root_path + path
+ os.remove(file_path)
+ except Exception as e:
+ return str(e)
+
+ return None
+
+ # operation or inner_function
+ async def move(self,path:str,new_path:str) -> str:
+ try:
+ file_path = self.root_path + path
+ new_path = self.root_path + new_path
+ os.rename(file_path,new_path)
+ except Exception as e:
+ return str(e)
+
+ return None
+
+ # inner_function
+ async def stat(self,path:str) -> str:
+ try:
+ file_path = self.root_path + path
+ stat = os.stat(file_path)
+ return json.dumps(stat,ensure_ascii=False)
+ except Exception as e:
+ return str(e)
+
+ # operation or inner_function
+ async def symlink(self,path:str,target:str) -> str:
+ try:
+ #file_path = self.root_path + path
+ target_path = self.root_path + target
+ dir_path = os.path.dirname(target_path)
+ os.makedirs(dir_path,exist_ok=True)
+ os.symlink(path,target_path)
+ except Exception as e:
+ logger.error("symlink failed:%s",e)
+ return str(e)
+
+ return None
\ No newline at end of file
diff --git a/src/component/common_environment/shell.py b/src/component/common_environment/shell.py
new file mode 100644
index 0000000..0026240
--- /dev/null
+++ b/src/component/common_environment/shell.py
@@ -0,0 +1,30 @@
+import os
+from typing import Any,List,Dict
+from aios import SimpleAIFunction
+from aios import SimpleEnvironment
+from aios import GlobaToolsLibrary,ParameterDefine
+
+class ShellEnvironment(SimpleEnvironment):
+ def __init__(self) -> None:
+ super().__init__("shell")
+
+ @classmethod
+ def register_ai_functions(cls):
+ operator_param = ParameterDefine.create_parameters({"command":"command will execute"})
+ GlobaToolsLibrary.get_instance().register_tool_function(SimpleAIFunction("system.shell.exec",
+ "execute shell command in linux bash",
+ ShellEnvironment.shell_exec,operator_param))
+ @staticmethod
+ async def shell_exec(command:str) -> str:
+ import asyncio.subprocess
+ process = await asyncio.create_subprocess_shell(
+ command,
+ stdout=asyncio.subprocess.PIPE,
+ stderr=asyncio.subprocess.PIPE
+ )
+ stdout, stderr = await process.communicate()
+ returncode = process.returncode
+ if returncode == 0:
+ return f"Execute success! stdout is:\n{stdout}\n"
+ else:
+ return f"Execute failed! stderr is:\n{stderr}\n"
diff --git a/src/component/discord_tunnel.py b/src/component/discord_tunnel.py
new file mode 100644
index 0000000..1332e58
--- /dev/null
+++ b/src/component/discord_tunnel.py
@@ -0,0 +1,214 @@
+import asyncio
+import datetime
+import time
+import logging
+import os
+import re
+import uuid
+import aiofiles
+from urllib.parse import urlparse
+from typing import Optional
+
+from aios import KnowledgeStore, ObjectType
+from aios.frame.tunnel import AgentTunnel
+from aios.proto.agent_msg import AgentMsg, AgentMsgType
+import discord
+
+from aios.storage.storage import AIStorage
+
+logger = logging.getLogger(__name__)
+
+
+IMAGE_FORMATS = ["jpg", "jpeg", "png", "gif", "bmp", "webp", "tiff", "tif"]
+VIDEO_FORMATS = ["mp4", "avi", "mov", "wmv", "flv", "mkv", "webm"]
+AUDIO_FORMATS = ["mp3", "wav", "ogg", "flac", "aac", "m4a", "wma", "ape", "alac", "opus", "oga"]
+
+
+class DiscordTunnel(AgentTunnel):
+ target_id: str
+ type: str
+ tunnel_id: str
+
+ def __init__(self, token: Optional[str] = None):
+ super().__init__()
+ self.token = token
+ self.client: discord.Client = None
+
+ self.discord_cache = os.path.join(AIStorage.get_instance().get_myai_dir(), "discord")
+ if not os.path.exists(self.discord_cache):
+ os.makedirs(self.discord_cache)
+
+ @classmethod
+ def register_to_loader(cls):
+ async def load_discord_tunnel(config: dict) -> AgentTunnel:
+ result_tunnel = DiscordTunnel()
+ if await result_tunnel.load_from_config(config):
+ return result_tunnel
+ else:
+ return None
+
+ AgentTunnel.register_loader("DiscordTunnel", load_discord_tunnel)
+
+ async def load_from_config(self, config: dict) -> bool:
+ self.type = "DiscordTunnel"
+ self.token = config["token"]
+ self.target_id = config["target"]
+ self.tunnel_id = config["tunnel_id"]
+ return True
+
+ def get_cache_path(self) -> str:
+ today = datetime.datetime.today()
+ path = os.path.join(self.discord_cache, str(today.year), str(today.month))
+ if not os.path.exists(path):
+ os.makedirs(path)
+ return path
+
+
+ def post_message(self, msg: AgentMsg) -> None:
+ if self.client is None:
+ logger.error("DiscordTunnel is not started")
+ return
+
+ logger.warn("post_message not implemented")
+
+ async def start(self) -> bool:
+ if self.client is not None:
+ logger.warn("DiscordTunnel is already started")
+ return False
+
+ if self.token is None:
+ self.token = os.environ.get("DISCORD_TOKEN")
+ if self.token is None:
+ raise ValueError("Discord token must be provided")
+
+ intents = discord.Intents.default()
+ intents.message_content = True
+ intents.members = True
+ client = discord.Client(intents=intents)
+
+ @client.event
+ async def on_ready():
+ print(f"Logged in as {self.client.user}")
+
+ @client.event
+ async def on_message(message: discord.Message):
+ logger.info(f"Message from {message.author}: {message.content}")
+ if message.author == self.client.user:
+ return
+
+ content = re.sub("<@.+>", "", message.content).strip()
+
+ attach_type = None
+ images = []
+ ext = None
+ video_file = None
+ audio_file = None
+ if message.attachments is not None and len(message.attachments) > 0:
+ for attachment in message.attachments:
+ logger.info(f"Attachment: {attachment.url}")
+ url = urlparse(attachment.url)
+ ext = url.path.rsplit(".")[-1]
+
+ file_path = os.path.join(self.get_cache_path(), attachment.filename)
+ with open(file_path, "wb") as f:
+ await attachment.save(f)
+
+ if ext in IMAGE_FORMATS:
+ if attach_type is None:
+ attach_type = "image"
+ elif attach_type != "image":
+ break
+ images.append(file_path)
+ elif ext in VIDEO_FORMATS:
+ if attach_type is None:
+ attach_type = "video"
+ video_file = file_path
+ break
+ elif ext in AUDIO_FORMATS:
+ if attach_type is None:
+ attach_type = "audio"
+ audio_file = file_path
+ break
+
+ agent_msg = AgentMsg()
+ agent_msg.topic = "_discord"
+ agent_msg.msg_id = "discord_msg#" + str(message.id) + "#" + uuid.uuid4().hex
+ agent_msg.target = self.target_id
+ agent_msg.create_time = time.time()
+ agent_msg.sender = message.author.name
+ self.ai_bus.register_message_handler(agent_msg.sender, self._process_message)
+
+ if len(message.channel.members) > 2:
+ if self.client.user not in message.mentions:
+ agent_msg.msg_type = AgentMsgType.TYPE_GROUP_MSG
+
+ if attach_type is None:
+ agent_msg.body = content
+ elif attach_type == "image":
+ agent_msg.body = agent_msg.create_image_body(images, content)
+ agent_msg.body_mime = f"image/{ext}"
+ elif attach_type == "video":
+ agent_msg.body = agent_msg.create_video_body(video_file, content)
+ agent_msg.body_mime = f"video/{ext}"
+ elif attach_type == "audio":
+ agent_msg.body = agent_msg.create_audio_body(audio_file, content)
+ agent_msg.body_mime = f"audio/{ext}"
+
+ resp_msg: AgentMsg = await self.ai_bus.send_message(agent_msg)
+ if resp_msg is None:
+ await message.channel.send(f"System Error: Timeout,{self.target_id} no resopnse! Please check logs/aios.log for more details!")
+ else:
+ if resp_msg.body_mime is None:
+ if resp_msg.body is None:
+ return
+
+ if len(resp_msg.body) < 1:
+ return
+
+ knownledge_object = KnowledgeStore().parse_object_in_message(resp_msg.body)
+ if knownledge_object is not None:
+ if knownledge_object.get_object_type() == ObjectType.Image:
+ image = KnowledgeStore().bytes_from_object(knownledge_object)
+ try:
+ async with aiofiles.open("image.jpg", "wb") as f:
+ await f.write(image)
+ await message.channel.send(file=discord.File("image.jpg"))
+ except Exception as e:
+ logger.error(f"save image error:{e}")
+ logger.exception(e)
+ return
+ else:
+ pos = resp_msg.body.find("audio file")
+ if pos != -1:
+ audio_file = resp_msg.body[pos+11:].strip()
+ if audio_file.startswith("\""):
+ audio_file = audio_file[1:-1]
+ await message.channel.send(file=discord.File(audio_file))
+ return
+ await message.channel.send(resp_msg.body)
+ else:
+ if resp_msg.is_image_msg():
+ text, images = resp_msg.get_image_body()
+ files = []
+ for image in images:
+ files.append(discord.File(image))
+ await message.channel.send(content=text, files=files)
+ elif resp_msg.is_video_msg():
+ text, video_file = resp_msg.get_video_body()
+ await message.channel.send(text, file=discord.File(video_file))
+ elif resp_msg.is_audio_msg():
+ text, audio_file = resp_msg.get_audio_body()
+ await message.channel.send(text, file=discord.File(audio_file))
+ else:
+ await message.channel.send(resp_msg.body)
+
+ asyncio.create_task(client.start(self.token))
+ self.client = client
+ print("start finish")
+
+ async def close(self) -> None:
+ await self.client.close()
+ self.client = None
+
+ async def _process_message(self, msg: AgentMsg) -> None:
+ logger.warn(f"process message {msg.msg_id} from {msg.sender} to {msg.target}")
diff --git a/src/aios_kernel/email_tunnel.py b/src/component/email_tunnel.py
similarity index 78%
rename from src/aios_kernel/email_tunnel.py
rename to src/component/email_tunnel.py
index c4f75b4..c5e906a 100644
--- a/src/aios_kernel/email_tunnel.py
+++ b/src/component/email_tunnel.py
@@ -7,8 +7,8 @@ import mailparser
import logging
import time
import datetime
-from .tunnel import AgentTunnel
-from .agent_message import AgentMsg
+
+from aios import AgentTunnel,AgentMsg,ContactManager
from email.message import EmailMessage
@@ -30,7 +30,7 @@ class EmailTunnel(AgentTunnel):
self.target_id = config["target"]
self.tunnel_id = config["tunnel_id"]
- self.type = "TelegramTunnel"
+ self.type = "EmailTunnel"
self.email = config["email"]
self.imap_server = config["imap"]
s = self.imap_server.split(":")
@@ -46,8 +46,10 @@ class EmailTunnel(AgentTunnel):
self.login_user = config["user"]
self.login_password = config["password"]
- self.folder = config["folder"]
- self.check_interval = config["interval"]
+ if config.get("folder") is not None:
+ self.folder = config["folder"]
+ if config.get("interval") is not None:
+ self.check_interval = config["interval"]
return True
@@ -55,6 +57,8 @@ class EmailTunnel(AgentTunnel):
super().__init__()
self.is_start = False
self.read_email = {}
+ self.folder = "INBOX"
+ self.check_interval = 60
async def on_new_email(self,mail:mailparser.MailParser) -> None:
remote_email_addr = mail.from_[0][1]
@@ -86,7 +90,32 @@ class EmailTunnel(AgentTunnel):
password=self.login_password,
)
+ async def post_message(self, msg: AgentMsg) -> None:
+ cm = ContactManager.get_instance()
+ contact = cm.find_contact_by_name(msg.target)
+ if contact is None:
+ logger.error(f"can't find contact {msg.target} , post message through email_tunnel failed!")
+ return
+
+ target_email = contact.email
+ if target_email is None:
+ logger.error(f"contact {msg.target} has no email, post message through email_tunnel failed!")
+ return
+
+ email_msg = EmailMessage()
+ email_msg['Subject'] = f"{msg.topic},From AIAgent {msg.sender}"
+ email_msg['From'] = self.email
+ email_msg['To'] = target_email
+ email_msg.set_content(msg)
+ await aiosmtplib.send(
+ email_msg,
+ hostname = self.smtp_server,
+ port=self.smtp_port,
+ username=self.login_user,
+ password=self.login_password,
+ )
+
def conver_mail_to_agent_msg(self,mail:mailparser.MailParser) -> AgentMsg:
msg = AgentMsg()
@@ -141,3 +170,5 @@ class EmailTunnel(AgentTunnel):
async def _process_message(self, msg: AgentMsg) -> None:
logger.warn(f"process message {msg.msg_id} from {msg.sender} to {msg.target}")
+
+
diff --git a/src/component/google_node/__init__.py b/src/component/google_node/__init__.py
new file mode 100644
index 0000000..22ac89b
--- /dev/null
+++ b/src/component/google_node/__init__.py
@@ -0,0 +1 @@
+from .google_text_to_speech_node import *
\ No newline at end of file
diff --git a/src/aios_kernel/google_text_to_speech_node.py b/src/component/google_node/google_text_to_speech_node.py
similarity index 95%
rename from src/aios_kernel/google_text_to_speech_node.py
rename to src/component/google_node/google_text_to_speech_node.py
index 0fdba47..ecdb874 100644
--- a/src/aios_kernel/google_text_to_speech_node.py
+++ b/src/component/google_node/google_text_to_speech_node.py
@@ -7,9 +7,7 @@ from typing import Optional
from google.cloud import texttospeech
-from .storage import AIStorage
-from .compute_task import ComputeTask, ComputeTaskResult, ComputeTaskState, ComputeTaskType
-from .compute_node import ComputeNode
+from aios import AIStorage,ComputeTask, ComputeTaskResult, ComputeTaskState, ComputeTaskType,ComputeNode
logger = logging.getLogger(__name__)
@@ -117,9 +115,18 @@ class GoogleTextToSpeechNode(ComputeNode):
def _run_task(self, task: ComputeTask):
task.state = ComputeTaskState.RUNNING
language_code = task.params["language_code"]
+ if language_code is None:
+ language_code = "en"
+
text = task.params["text"]
voice_name = task.params["voice_name"]
+ if voice_name is None:
+ voice_name = "default"
+
gender = task.params["gender"]
+ if gender is None:
+ gender = "female"
+
age = task.params["age"]
if language_code == "zh":
diff --git a/src/component/knowledge_manager/__init__.py b/src/component/knowledge_manager/__init__.py
new file mode 100644
index 0000000..8f6fd1f
--- /dev/null
+++ b/src/component/knowledge_manager/__init__.py
@@ -0,0 +1 @@
+from .pipeline import KnowledgePipelineManager
\ No newline at end of file
diff --git a/src/component/knowledge_manager/pipeline.py b/src/component/knowledge_manager/pipeline.py
new file mode 100644
index 0000000..c8a1ded
--- /dev/null
+++ b/src/component/knowledge_manager/pipeline.py
@@ -0,0 +1,90 @@
+import os
+import runpy
+import toml
+import asyncio
+from aios import KnowledgePipelineEnvironment, KnowledgePipeline
+
+
+class KnowledgePipelineManager:
+ @classmethod
+ def initial(cls, root_dir: str):
+ cls._instance = KnowledgePipelineManager(root_dir)
+ return cls._instance
+
+ @classmethod
+ def get_instance(cls):
+ return cls._instance
+
+ def __init__(self, root_dir: str):
+ self.root_dir = root_dir
+ self.input_modules = {}
+ self.parser_modules = {}
+ self.pipelines = {
+ "names": {},
+ "running": []
+ }
+
+ def register_input(self, name: str, init_method):
+ self.input_modules[name] = init_method
+
+ def register_parser(self, name: str, parser_method):
+ self.parser_modules[name] = parser_method
+
+ def add_pipeline(self, config: dict, path: str):
+ name = config["name"]
+ if name in self.pipelines["names"]:
+ return
+
+ input_module = config["input"]["module"]
+ _, ext = os.path.splitext(input_module)
+ if ext == ".py":
+ input_module = os.path.join(path, input_module)
+ input_init = runpy.run_path(input_module)["init"]
+ else:
+ input_init = self.input_modules.get(input_module)
+ input_params = config["input"].get("params")
+
+ parser_config = config.get("parser")
+ if parser_config is None:
+ parser_init = None
+ parser_params = None
+ else:
+ parser_module = parser_config["module"]
+ _, ext = os.path.splitext(parser_module)
+ if ext == ".py":
+ parser_module = os.path.join(path, parser_module)
+ parser_init = runpy.run_path(parser_module)["init"]
+ else:
+ parser_init = self.parser_modules.get(parser_module)
+ parser_params = parser_config.get("params")
+
+
+ data_path = os.path.join(self.root_dir, name)
+ env = KnowledgePipelineEnvironment(data_path)
+ pipeline = KnowledgePipeline(name, env, input_init, input_params, parser_init, parser_params)
+ self.pipelines["names"][name] = pipeline
+ self.pipelines["running"].append(pipeline)
+
+ def get_pipelines(self) -> [KnowledgePipeline]:
+ return self.pipelines["running"]
+
+ def get_pipeline(self, name: str) -> KnowledgePipeline:
+ return self.pipelines["names"].get(name)
+
+ async def run(self):
+ while True:
+ for pipeline in self.pipelines["running"]:
+ await pipeline.run()
+ await asyncio.sleep(5)
+
+ def load_dir(self, root: str):
+ config_path = os.path.join(root, "pipelines.toml")
+ if not os.path.exists(config_path):
+ return
+ with open(config_path, "r") as f:
+ config = toml.load(f)
+ for path in config["pipelines"]:
+ pipeline_path = os.path.join(root, path)
+ with open(os.path.join(pipeline_path, "pipeline.toml"), 'r', encoding='utf-8') as f:
+ pipeline_config = toml.load(f)
+ self.add_pipeline(pipeline_config, pipeline_path)
diff --git a/src/component/llama_node/__init__.py b/src/component/llama_node/__init__.py
new file mode 100644
index 0000000..e6623f9
--- /dev/null
+++ b/src/component/llama_node/__init__.py
@@ -0,0 +1 @@
+from .local_llama_compute_node import LocalLlama_ComputeNode
\ No newline at end of file
diff --git a/src/aios_kernel/local_llama_compute_node.py b/src/component/llama_node/local_llama_compute_node.py
similarity index 85%
rename from src/aios_kernel/local_llama_compute_node.py
rename to src/component/llama_node/local_llama_compute_node.py
index 7f8ad0a..efb9712 100644
--- a/src/aios_kernel/local_llama_compute_node.py
+++ b/src/component/llama_node/local_llama_compute_node.py
@@ -1,13 +1,8 @@
-import json
import logging
import requests
-from typing import Optional, List
-from pydantic import BaseModel
-from .compute_task import ComputeTask, ComputeTaskResult, ComputeTaskResultCode, ComputeTaskState, ComputeTaskType
-from .queue_compute_node import Queue_ComputeNode
-from .storage import AIStorage,UserConfig
+from aios import ComputeTask,Queue_ComputeNode, ComputeTaskResult, ComputeTaskResultCode, ComputeTaskState, ComputeTaskType,AIStorage,UserConfig
logger = logging.getLogger(__name__)
@@ -113,11 +108,26 @@ class LocalLlama_ComputeNode(Queue_ComputeNode):
llm_inner_functions = task.params.get("inner_functions")
if max_token_size is None:
max_token_size = max_token_size
-
+
body = {
"messages": [],
+ "functions": llm_inner_functions,
+ "tools": [],
+ "tool_choices": [],
"max_tokens": 4000
}
+ if llm_inner_functions is not None:
+ for fun in llm_inner_functions:
+ body["tools"].append({
+ "type": "function",
+ "function": fun,
+ })
+ body["tool_choices"].append({
+ "type": "function",
+ "function": {
+ "name": fun["name"]
+ }
+ })
for prompt in prompts:
body["messages"].append({
@@ -126,6 +136,8 @@ class LocalLlama_ComputeNode(Queue_ComputeNode):
})
try:
+ logger.info(f"will post http request to {self.url}/v1/chat/completions, body: {body}")
+
response = requests.post(self.url + "/v1/chat/completions", json = body, verify=False, headers={"Content-Type": "application/json"})
response.close()
@@ -138,8 +150,12 @@ class LocalLlama_ComputeNode(Queue_ComputeNode):
token_usage = resp["usage"]
match status_code:
- case "function_call":
+ case "tool_calls":
task.state = ComputeTaskState.DONE
+ # rebuild the function name
+ fun_name = resp["choices"][0]["message"]["function_call"]["name"]
+ if len(llm_inner_functions) == 1 and (fun_name is None or fun_name == ""):
+ resp["choices"][0]["message"]["function_call"]["name"] = llm_inner_functions[0]["name"]
case "stop":
task.state = ComputeTaskState.DONE
case _:
diff --git a/src/component/mail_environment/__init__.py b/src/component/mail_environment/__init__.py
new file mode 100644
index 0000000..402cf90
--- /dev/null
+++ b/src/component/mail_environment/__init__.py
@@ -0,0 +1,3 @@
+from .issue import IssueParser
+from .local import LocalEmail
+from .spider import EmailSpider
\ No newline at end of file
diff --git a/src/component/mail_environment/issue.py b/src/component/mail_environment/issue.py
new file mode 100644
index 0000000..7bddcb5
--- /dev/null
+++ b/src/component/mail_environment/issue.py
@@ -0,0 +1,321 @@
+# define a knowledge base class
+import json
+import string
+from aios import *
+from .mail import MailStorage, Mail
+
+class IssueState(Enum):
+ Open = 1
+ InProgress = 2
+ Closed = 3
+
+class IssueUpdateHistory:
+ def __init__(self, source: str, changes: dict) -> None:
+ self.source = source
+ self.changes = changes
+
+ def to_json_dict(self) -> dict:
+ return {
+ "source": self.source,
+ "changes": self.changes,
+ }
+
+ @classmethod
+ def from_json_dict(cls, json_dict: dict) -> "IssueUpdateHistory":
+ return IssueUpdateHistory(json_dict["source"], json_dict["changes"])
+
+class Issue:
+ def __init__(self) -> None:
+ self.id = None
+ self.summary = ""
+ self.state = IssueState.Open
+ self.source: str = None
+ self.create_time: datetime = None
+ self.deadline: datetime = None
+ self.update_history = []
+ self.children = []
+ self.parent: str = None
+
+ def to_json_dict(self) -> dict:
+ json_dict = {
+ "id": self.id,
+ "summary": self.summary,
+ "state": self.state.name,
+ "create_time": self.create_time,
+ "deadline": self.deadline,
+ "source": self.source,
+ "parent": self.parent,
+ }
+ if self.children is not None and len(self.children) > 0:
+ json_dict["children"] = []
+ for child in self.children:
+ json_dict["children"].append(child.to_json_dict())
+ if self.update_history is not None and len(self.update_history) > 0:
+ json_dict["update_history"] = []
+ for history in self.update_history:
+ json_dict["update_history"].append(history.to_json_dict())
+
+ return json_dict
+
+ @classmethod
+ def from_json_dict(cls, json_dict: dict) -> "Issue":
+ issue = Issue()
+ issue.id = json_dict["id"]
+ issue.summary = json_dict["summary"]
+ issue.state = IssueState[json_dict["state"]]
+ issue.create_time = json_dict["create_time"]
+ issue.deadline = json_dict["deadline"]
+ issue.source = json_dict["source"]
+ issue.parent = json_dict["parent"]
+ if "children" in json_dict:
+ issue.children = []
+ for child_json_dict in json_dict["children"]:
+ child = Issue.from_json_dict(child_json_dict)
+ issue.children.append(child)
+ if "update_history" in json_dict:
+ issue.update_history = []
+ for history_json_dict in json_dict["update_history"]:
+ history = IssueUpdateHistory.from_json_dict(history_json_dict)
+ issue.update_history.append(history)
+ return issue
+
+
+ @classmethod
+ def object_type(cls) -> ObjectType:
+ return ObjectType.from_user_def_type_code(0)
+
+ def __to_desc(self, desc_list:[], recursion=None):
+ desc = {
+ "id": self.id,
+ "summary": self.summary,
+ "state": self.state.name,
+ "deadline": self.deadline,
+ }
+ desc_list.append(desc)
+ if not recursion or not self.parent:
+ return
+ else:
+ parent = recursion.get_issue_by_id(self.parent)
+ parent.__to_desc(desc_list, recursion)
+
+ def to_prompt(self, recursion=None) -> str:
+ desc_list = []
+ self.__to_desc(desc_list, recursion)
+ root = desc_list.pop()
+ while len(desc_list) > 0:
+ child = desc_list.pop()
+ root["child"] = child
+ root = child
+ return json.dumps(root,ensure_ascii=False)
+
+
+ @classmethod
+ def prompt_desc(cls) -> str:
+ return '''a issue contains following fileds: {
+ id: a guid string to identify a issue
+ summary: summary of this issue
+ state: state of this issue, will be one of [Open, InProgress, Closed],
+ deadline: if issue is not closed, deadline is the time to close this issue,
+ children: child issues of this issue
+ }
+ '''
+
+ def calculate_id(self) -> str:
+ desc = {
+ "summary": self.summary,
+ "source": self.source,
+ "create_time": self.create_time,
+ "deadline": self.deadline,
+ "parent": self.parent,
+ }
+ id = str(KnowledgeObject(Issue.object_type(), desc).calculate_id())
+ self.id = id
+ return id
+
+
+class IssueStorage:
+ def __init__(self, path: str, root: Issue=None) -> None:
+ self.path = path
+ if not os.path.exists(path):
+ self.root = root
+ self.__flush()
+ else:
+ root_dict = json.load(open(path, "r", encoding="utf-8"))
+ self.root = Issue.from_json_dict(root_dict)
+
+ def __flush(self):
+ json.dump(self.root.to_json_dict(), open(self.path, "w", encoding="utf-8"), ensure_ascii=False, indent=4)
+
+ def __get_issue_by_id_in_subtree(self, root_issue: Issue, id: str):
+ if root_issue.id == id:
+ return root_issue
+ if root_issue.children is None or len(root_issue.children) == 0:
+ return None
+ for child_issue in root_issue.children:
+ this_issue = self.__get_issue_by_id_in_subtree(child_issue, id)
+ if this_issue is not None:
+ return this_issue
+ return None
+
+ def get_issue_by_id(self, id: str) -> Issue:
+ return self.__get_issue_by_id_in_subtree(self.root, id)
+
+ def __get_issue_by_mail_in_subtree(self, root_issue: Issue, mail_id: str):
+ if root_issue.source == mail_id:
+ return root_issue
+ if root_issue.children is None or len(root_issue.children) == 0:
+ return None
+ for child_issue in root_issue.children:
+ this_issue = self.__get_issue_by_mail_in_subtree(child_issue, mail_id)
+ if this_issue is not None:
+ return this_issue
+ return None
+
+ def get_issue_by_mail(self, mail_storage: MailStorage, mail: Mail) -> Issue:
+ if mail.reply_to is None:
+ return self.root
+ this_mail = mail_storage.get_mail_by_id(mail.reply_to)
+ while True:
+ issue = self.__get_issue_by_mail_in_subtree(self.root, this_mail.id)
+ if issue is not None:
+ return issue
+ if this_mail.replay_to is None:
+ return self.root
+ this_mail = mail_storage.get_mail_by_id(this_mail.reply_to)
+
+
+ def add_issue(self, source_id: str, parent_id: str, summary: str):
+ parent_issue = self.get_issue_by_id(parent_id)
+ issue = Issue()
+ issue.summary = summary
+ issue.source = source_id
+ issue.parent = parent_id
+ issue.calculate_id()
+ parent_issue.children.append(issue)
+ self.__flush()
+ return issue
+
+ def update_issue(self, source_id: str, issue_id: str, update: dict):
+ issue = self.get_issue_by_id(issue_id)
+ changes = {}
+ for key, value in update.items():
+ changes[key] = {
+ "old": issue[key],
+ "new": value,
+ }
+ issue.__dict__[key] = value
+ issue.update_history.append(IssueUpdateHistory(source_id, changes))
+
+ self.__flush()
+ return issue
+
+
+class IssueAgent(CustomAIAgent):
+ async def _process_msg(self, msg: AgentMsg, workspace=None) -> AgentMsg:
+ pass
+
+ def __init__(self, agent_id: str, llm_model_name: str, max_token_size: int) -> None:
+ super().__init__(agent_id, llm_model_name, max_token_size)
+
+
+class IssueParserEnvironment(SimpleEnvironment):
+ def __init__(self, env_id: str, storage: IssueStorage) -> None:
+ super().__init__(env_id)
+ self.storage = storage
+
+ create_description = '''create a new issue'''
+ create_param = {
+ "mail_id": "new issue with which email object id",
+ "issue_id": '''new issue's parent issue id''',
+ "summary": '''new issue's summary''',
+ }
+ self.add_ai_function(SimpleAIFunction("create_issue",
+ create_description,
+ self._create,
+ create_param))
+
+ update_description = '''update an existing issue'''
+ update_param = {
+ "mail_id": "update issue with which email object id",
+ "issue_id": '''update issue's id''',
+ "summary": '''issue's new summary''',
+ }
+ self.add_ai_function(SimpleAIFunction("update_issue",
+ update_description,
+ self._update,
+ update_param))
+
+ async def _create(self, mail_id: str, issue_id: str, summary: str):
+ issue = self.storage.add_issue(mail_id, issue_id, summary)
+ return issue.id
+
+ async def _update(self, mail_id: str, issue_id: str, summary: str):
+ update = {}
+ update["summary"] = summary
+ issue = self.storage.update_issue(mail_id, issue_id, update)
+ return issue.id
+
+
+class IssueParser:
+ def __init__(self, env: KnowledgePipelineEnvironment, config: dict):
+ mail_path = string.Template(config["mail_path"]).substitute(myai_dir=AIStorage.get_instance().get_myai_dir())
+ issue_path = string.Template(config["issue_path"]).substitute(myai_dir=AIStorage.get_instance().get_myai_dir())
+ config["path"] = issue_path
+
+ self.env = env
+ self.config = config
+ self.mail_storage = MailStorage(mail_path)
+
+ root_issue = None
+ if "root_issue" in config:
+ root_config = config["root_issue"]
+ root_issue = IssueParser.__load_issue_config(root_config)
+ IssueParser.__calac_issue_id(root_issue)
+
+ self.issue_storage = IssueStorage(issue_path, root_issue)
+ self.llm_env = IssueParserEnvironment("issue_parser", self.issue_storage)
+
+ @classmethod
+ def __load_issue_config(cls, issue_config: dict) -> Issue:
+ issue = Issue()
+ issue.summary = issue_config["summary"]
+ if "children" in issue_config:
+ for child_config in issue_config["children"]:
+ child_issue = cls.__load_issue_config(child_config)
+ issue.children.append(child_issue)
+ return issue
+
+ @classmethod
+ def __calac_issue_id(cls, issue: Issue):
+ issue_id = issue.calculate_id()
+ for child in issue.children:
+ child.parent = issue_id
+ cls.__calac_issue_id(child)
+
+
+ def get_path(self) -> str:
+ return self.config["path"]
+
+ async def parse(self, mail_id: ObjectID) -> str:
+ mail_id = str(mail_id)
+ mail = self.mail_storage.get_mail_by_id(mail_id)
+ issue = self.issue_storage.get_issue_by_mail(self.mail_storage, mail)
+ mail_str = mail.to_prompt()
+ issue_str = issue.to_prompt(recursion=self.issue_storage)
+
+ mail_desc = Mail.prompt_desc()
+ issue_desc = Issue.prompt_desc()
+ prompt = LLMPrompt()
+ prompt.system_message = {"role": "system", "content": f'''
+ I'm a CEO of a company named 巴克云; You'ar my assistant, and you should help me to manage my issues. Issues is a concept in software development of this company, but I use it to manage my work.
+ I'll give you a mail in json format, {mail_desc};
+ and a issue in json format, {issue_desc}. Read mail's fileds and issue's fileds, and decide if you should update the issue or create a new issue with this mail.
+ Then call the function create_issue or update_issue.
+ if this mail is not associated with issue, you should ignore this mail.'''}
+
+ prompt.append(IssueAgent(f'''Mail is {mail_str}, issue is {issue_str}. Answer me the function's return value or None if igonred.
+ '''))
+
+ llm_result = await CustomAIAgent("issue parser", "gpt-4-1106-preview", 4000).do_llm_complection(prompt, env=self.llm_env)
+ return "update issue"
+
diff --git a/src/component/mail_environment/local.py b/src/component/mail_environment/local.py
new file mode 100644
index 0000000..af9726c
--- /dev/null
+++ b/src/component/mail_environment/local.py
@@ -0,0 +1,36 @@
+import os
+import logging
+import json
+import string
+from aios import *
+from .mail import Mail, MailStorage
+
+
+class LocalEmail:
+ def __init__(self, env: KnowledgePipelineEnvironment, config:dict):
+ self.config = config
+ self.env = env
+ path = string.Template(config["path"]).substitute(myai_dir=AIStorage.get_instance().get_myai_dir())
+ self.mail_storage = MailStorage(path, config.get("watch"))
+
+ async def next(self):
+ while True:
+ parsed = None
+ journals = self.env.journal.latest_journals(1)
+ if len(journals) == 1:
+ latest_journal = journals[0]
+ if latest_journal.is_finish():
+ yield None
+ continue
+ parsed = latest_journal.get_input()
+
+ mail_id = self.mail_storage.next_mail_id(parsed)
+ if mail_id is None:
+ yield (None, None)
+ else:
+ yield (mail_id, str(mail_id))
+
+
+class LocalEmailWithFilter:
+ def __init__(self, env: KnowledgePipelineEnvironment, config:dict):
+ pass
\ No newline at end of file
diff --git a/src/component/mail_environment/mail.py b/src/component/mail_environment/mail.py
new file mode 100644
index 0000000..47cde8b
--- /dev/null
+++ b/src/component/mail_environment/mail.py
@@ -0,0 +1,297 @@
+import asyncio
+import json
+import mailparser
+import base64
+import requests
+import datetime
+from bs4 import BeautifulSoup
+import sqlite3
+import html2text
+from urllib.parse import urlparse
+from aios import *
+
+
+
+class Mail:
+ def __init__(self, **kwargs) -> None:
+ self.from_addr = kwargs.get("from")
+ self.to_addr = kwargs.get("to")
+ self.subject = kwargs.get("subject")
+ self.date = kwargs.get("date")
+ self.bcc = kwargs.get("bcc")
+ self.cc = kwargs.get("cc")
+ self.reply_to = kwargs.get("reply_to")
+ self.id: str = None
+ self.content: str = None
+
+ def to_prompt(self) -> str:
+ prompt = {
+ "id": self.id,
+ "subject": self.subject,
+ "from": self.from_addr,
+ "date": self.date,
+ "content": self.content
+ }
+ return json.dumps(prompt,ensure_ascii=False)
+
+ @classmethod
+ def prompt_desc(cls) -> dict:
+ return '''a mail contains following fileds: {
+ id: a guid string to identify a mail
+ subject: subject of this mail
+ from: sender address of this mail
+ date: date of this mail
+ content: content of this mail
+ }
+ '''
+
+ def get_date(self) -> datetime.datetime:
+ datetime.datetime.strptime(self.date, "%Y-%m-%d %H:%M")
+
+ def calculate_id(self) -> str:
+ desc = {
+ "from_addr": self.from_addr,
+ "to_addr": self.to_addr,
+ "subject": self.subject,
+ "date": self.date,
+ "content": self.content,
+ "reply_to": self.reply_to
+ }
+ id = str(KnowledgeObject(ObjectType.Email, desc).calculate_id())
+ self.id = id
+ return id
+
+class MailStorage:
+ def __init__(self, root, watch=False):
+ self.root = root
+ if not os.path.exists(root):
+ os.makedirs(root)
+ db_file = os.path.join(root, "mail.db")
+
+ self.conn = sqlite3.connect(db_file)
+ cursor = self.conn.cursor()
+ cursor.execute(
+ """
+ CREATE TABLE IF NOT EXISTS mails (
+ uid INTEGER PRIMARY KEY,
+ object_id TEXT,
+ date DATETIME,
+ from_addr TEXT
+ )
+ """
+ )
+
+ if watch:
+ asyncio.create_task(self.watch_root())
+
+ def object_id_to_uid(self, object_id):
+ cursor = self.conn.cursor()
+ cursor.execute(
+ """
+ SELECT uid FROM mails WHERE object_id = ?
+ """,
+ (object_id,),
+ )
+ row = cursor.fetchone()
+ if row:
+ return row[0]
+ return None
+
+ def uid_to_object_id(self, uid):
+ cursor = self.conn.cursor()
+ cursor.execute(
+ """
+ SELECT object_id FROM mails WHERE uid = ?
+ """,
+ (uid,),
+ )
+ row = cursor.fetchone()
+ if row:
+ return row[0]
+ return None
+
+ def lastest_uid(self):
+ cursor = self.conn.cursor()
+ cursor.execute(
+ """
+ SELECT uid FROM mails ORDER BY uid DESC LIMIT 1
+ """
+ )
+ row = cursor.fetchone()
+ if row:
+ return row[0]
+ return None
+
+ def lastest_mail_id(self):
+ cursor = self.conn.cursor()
+ cursor.execute(
+ """
+ SELECT object_id FROM mails ORDER BY uid DESC LIMIT 1
+ """
+ )
+ row = cursor.fetchone()
+ if row:
+ return row[0]
+ return None
+
+ def next_mail_id(self, id):
+ uid = 0 if id is None else self.object_id_to_uid(id)
+ cursor = self.conn.cursor()
+ cursor.execute(
+ """
+ SELECT object_id FROM mails WHERE uid > ? ORDER BY uid ASC LIMIT 1
+ """,
+ (uid,),
+ )
+ row = cursor.fetchone()
+ if row:
+ return row[0]
+ return None
+
+
+
+ def get_mail_by_id(self, id):
+ uid = self.object_id_to_uid(id)
+ mail = Mail()
+ mail.id = id
+ mail_dir = self.mail_dir(uid)
+ mail_json = json.load(open(f"{mail_dir}/mail.json", "r", encoding='utf-8'))
+ mail.__dict__.update(mail_json)
+ with open(f"{mail_dir}/mail.txt", "r", encoding='utf-8') as f:
+ mail_content = f.read()
+ mail.content = mail_content
+ return mail
+
+ def mail_dir(self, uid):
+ return os.path.join(self.root, str(uid))
+
+ # for debug
+ async def watch_root(self):
+ while True:
+ latest_uid = self.lastest_uid()
+ for uid in os.listdir(self.root):
+ mail_dir = os.path.join(self.root, uid)
+ if uid.isdigit() and os.path.isdir(mail_dir):
+ uid = int(uid)
+ if uid <= latest_uid:
+ continue
+ mail = Mail()
+ mail_json = json.load(open(f"{mail_dir}/mail.json", "r", encoding='utf-8'))
+
+ mail.__dict__.update(mail_json)
+ # mail content
+ with open(f"{mail_dir}/mail.txt", "r", encoding='utf-8') as f:
+ mail_content = f.read()
+ mail.content = mail_content
+ mail.calculate_id()
+ cursor = self.conn.cursor()
+ cursor.execute(
+ """
+ INSERT INTO mails (uid, object_id, date, from_addr)
+ VALUES (?, ?, ?, ?)
+ """,
+ (uid, mail.id, mail.get_date(), mail.from_addr),
+ )
+ self.conn.commit()
+ await asyncio.sleep(10)
+
+ def download(self, uid, parser: mailparser.MailParser,
+ save_image=True,
+ from_field="From",
+ to_field="To",
+ subject_field="Subject",
+ date_field="Date",
+ reply_to_field="In-Reply-To",
+ cc_field="CC",
+ bcc_field="BCC"):
+ mail_dir = self.mail_dir(uid)
+ if not os.path.exists(mail_dir):
+ os.makedirs(mail_dir)
+
+ src_meta = json.loads(parser.mail_json)
+ meta = {}
+ meta["from"] = src_meta.get(from_field)
+ meta["to"] = src_meta.get(to_field)
+ meta["subject"] = src_meta.get(subject_field)
+ meta["date"] = src_meta.get(date_field)
+ meta["bcc"] = src_meta.get(bcc_field)
+ meta["cc"] = src_meta.get(cc_field)
+ reply_to = src_meta.get(reply_to_field)
+ if reply_to:
+ meta["reply_to"] = self.uid_to_object_id(reply_to)
+ mail = Mail(**meta)
+
+ h = html2text.HTML2Text()
+ h.ignore_links = True
+ h.ignore_images = True
+ mail_content = h.handle(parser.body)
+ mail.content = mail_content
+
+ mail.calculate_id()
+ del mail.content
+ json.dump(mail.__dict__, open(f"{mail_dir}/mail.json", "w", encoding='utf-8'))
+
+ # save mail content
+ with open(f"{mail_dir}/mail.txt", "w", encoding='utf-8') as f:
+ f.write(mail_content)
+
+ if save_image:
+ for attachment in parser.attachments:
+ if attachment['mail_content_type'] in ['image/png', 'image/jpg', 'image/jpeg', 'image/gif', 'image/svg']:
+ filename = attachment['filename']
+ filefullname = f"{mail_dir}/{filename}"
+ image_data = attachment['payload']
+ try:
+ image_data = base64.b64decode(image_data)
+ except base64.binascii.Error:
+ image_data = image_data.encode()
+ with open(filefullname, 'wb') as f:
+ f.write(image_data)
+ logging.info(f"save email image {filename} success")
+
+ # get all image urls
+ soup = BeautifulSoup(parser.body, 'html.parser')
+ img_tags = soup.find_all('img')
+ img_urls = [img['src'] for img in img_tags if 'src' in img.attrs]
+ logging.info(f'Found {len(img_urls)} images in email body')
+
+ name_count = 0
+
+ for img_url in img_urls:
+ # keep the original image filename(last of url)
+ url_result = urlparse(img_url)
+ if url_result.scheme not in ['http', 'https']:
+ continue
+ ext = url_result.path.split('/')[-1].split('.')[-1]
+ if ext in ['png', 'jpg', 'jpeg', 'gif', 'svg']:
+ img_filename = os.path.join(mail_dir, f"{name_count}.{ext}")
+ else :
+ img_filename = os.path.join(mail_dir, f"{name_count}")
+ name_count += 1
+ # download image
+ try:
+ response = requests.get(img_url, stream=True)
+ except requests.exceptions.RequestException as e:
+ logging.error(f'Failed to download {img_url}: {e}')
+ continue
+ if response.status_code == 200:
+ with open(img_filename, 'wb') as img_file:
+ for chunk in response.iter_content(1024):
+ img_file.write(chunk)
+ logging.info(f'Downloaded {img_url} to {img_filename}')
+ else:
+ logging.error(f'Failed to download {img_url}')
+
+ cursor = self.conn.cursor()
+ cursor.execute(
+ """
+ INSERT INTO mails (uid, object_id, date, from_addr)
+ VALUES (?, ?, ?, ?)
+ """,
+ (uid, mail.id, mail.date, mail.from_addr),
+ )
+ self.conn.commit()
+
+ return mail.id
+
+
\ No newline at end of file
diff --git a/src/component/mail_environment/spider.py b/src/component/mail_environment/spider.py
new file mode 100644
index 0000000..0119919
--- /dev/null
+++ b/src/component/mail_environment/spider.py
@@ -0,0 +1,71 @@
+import os
+import logging
+import json
+import string
+import imaplib
+import mailparser
+
+from knowledge import *
+from aios_kernel.storage import AIStorage
+from .mail import Mail, MailStorage
+
+
+class EmailSpider:
+ def __init__(self, env: KnowledgePipelineEnvironment, config:dict):
+ self.config = config
+ self.env = env
+ self.env.get_logger().info(f"read email config from {self.config.get('imap_server')}")
+ self.client = imaplib.IMAP4_SSL(
+ host=self.config.get('imap_server'),
+ port=self.config.get('imap_port')
+ )
+ self.client.login(self.config.get('address'), self.config.get('password'))
+ self.client.select("INBOX")
+ local_path = string.Template(config["path"]).substitute(myai_dir=AIStorage.get_instance().get_myai_dir())
+ local_path = os.path.join(local_path, self.config.get('address'))
+ self.mail_storage = MailStorage(local_path)
+
+
+ async def next(self):
+ while True:
+ try:
+ _, data = self.client.uid('search', None, "ALL")
+ except Exception as e:
+ self.env.get_logger().error(f"email spider error: {e}")
+ yield (None, None)
+ continue
+ uid_list = data[0].split()
+ if len(uid_list) == 0:
+ yield (None, None)
+ continue
+
+ journals = self.env.journal.latest_journals(1)
+ from_uid = 0
+ if len(journals) == 1:
+ latest_journal = journals[0]
+ if latest_journal.is_finish():
+ yield None
+ continue
+ from_uid = int(latest_journal.get_input())
+ if int.from_bytes(uid_list[-1]) <= from_uid:
+ yield (None, None)
+ continue
+
+ for uid in uid_list:
+ _uid = int.from_bytes(uid)
+ if _uid > from_uid:
+ message_parts = "(BODY.PEEK[])"
+ try:
+ _, email_data = self.client.uid('fetch', uid, message_parts)
+ mail = mailparser.parse_from_bytes(email_data[0][1])
+ id = self.mail_storage.download(_uid, mail)
+ except Exception as e:
+ self.env.get_logger().error(f"email spider error: {e}")
+ yield (None, None)
+ break
+ yield (ObjectID.from_base58(id), str(_uid))
+
+
+ yield (None, None)
+
+
\ No newline at end of file
diff --git a/src/component/openai_node/__init__.py b/src/component/openai_node/__init__.py
new file mode 100644
index 0000000..40c265a
--- /dev/null
+++ b/src/component/openai_node/__init__.py
@@ -0,0 +1,4 @@
+from .open_ai_node import *
+from .openai_tts_node import *
+from .whisper_node import *
+from .dall_e_compute_node import *
\ No newline at end of file
diff --git a/src/component/openai_node/dall_e_compute_node.py b/src/component/openai_node/dall_e_compute_node.py
new file mode 100644
index 0000000..241934c
--- /dev/null
+++ b/src/component/openai_node/dall_e_compute_node.py
@@ -0,0 +1,144 @@
+import os
+import io
+import asyncio
+from asyncio import Queue
+import logging
+from pathlib import Path
+from openai import OpenAI
+import base64
+
+from PIL import Image
+
+from aios import ComputeTask, ComputeTaskResult, ComputeTaskState, ComputeTaskType, ComputeTaskResultCode,ComputeNode, AIStorage, UserConfig
+
+logger = logging.getLogger(__name__)
+
+
+class DallEComputeNode(ComputeNode):
+ _instance = None
+
+ @classmethod
+ def get_instance(cls):
+ if cls._instance is None:
+ cls._instance = DallEComputeNode()
+ return cls._instance
+
+ @classmethod
+ def declare_user_config(cls):
+ user_config = AIStorage.get_instance().get_user_config()
+
+ if os.getenv("TEXT2IMG_OUTPUT_DIR") is None:
+ home_dir = Path.home()
+ output_dir = Path.joinpath(home_dir, "text2img_output")
+ Path.mkdir(output_dir, exist_ok=True)
+ user_config.add_user_config(
+ "text2img_output_dir", "text2image output dir", True, output_dir)
+
+ def __init__(self):
+ super().__init__()
+
+ self.is_start = False
+ self.node_id = "dall_e_node"
+ self.openai_api_key = ""
+ self.default_model = "dall-e-3"
+
+ self.task_queue = Queue()
+
+ async def initial(self):
+ if os.getenv("OPENAI_API_KEY") is not None:
+ self.openai_api_key = os.getenv("OPENAI_API_KEY")
+ else:
+ self.openai_api_key = AIStorage.get_instance().get_user_config().get_value("openai_api_key")
+
+ if self.openai_api_key is None:
+ logger.error("openai_api_key is None!")
+ return False
+
+ if os.getenv("TEXT2IMG_OUTPUT_DIR") is not None:
+ self.output_dir = os.getenv("TEXT2IMG_OUTPUT_DIR")
+ else:
+ self.output_dir = AIStorage.get_instance(
+ ).get_user_config().get_value("text2img_output_dir")
+
+ if self.output_dir is None:
+ self.output_dir = "./"
+ self.output_dir = os.path.abspath(self.output_dir)
+
+ await self.start()
+
+ return True
+
+ async def push_task(self, task: ComputeTask, proiority: int = 0):
+ logger.info(f"DallE_node push task: {task.display()}")
+ self.task_queue.put_nowait(task)
+
+ async def remove_task(self, task_id: str):
+ pass
+
+ def _run_task(self, task: ComputeTask):
+ task.state = ComputeTaskState.RUNNING
+ result = ComputeTaskResult()
+ result.result_code = ComputeTaskResultCode.ERROR
+ result.set_from_task(task)
+
+ try:
+ prompt = task.params["prompt"]
+ logging.info(f"Call DallE {self.default_model} prompts: {prompt}")
+ client = OpenAI(api_key=self.openai_api_key)
+
+ response = client.images.generate(
+ model=self.default_model,
+ prompt=prompt,
+ size="1024x1024",
+ quality="standard",
+ n=1,
+ response_format="b64_json",
+ )
+
+ binary_data = base64.b64decode(response.data[0].b64_json)
+ image = Image.open(io.BytesIO(binary_data))
+ file_name = os.path.join(self.output_dir, task.task_id + ".png")
+ image.save(file_name)
+
+ task.state = ComputeTaskState.DONE
+ result.result_code = ComputeTaskResultCode.OK
+ result.worker_id = self.node_id
+ result.result = {"file": file_name}
+
+ return result
+
+ except Exception as e:
+ logging.error(f"Call DallE failed. err: {e}")
+ task.error_str = str(e)
+ result.error_str = str(e)
+ task.state = ComputeTaskState.ERROR
+ return result
+
+ async def start(self):
+ if self.is_start:
+ return
+ self.is_start = True
+
+ async def _run_task_loop():
+ while True:
+ logger.info("Dall E node is waiting for task...")
+ task = await self.task_queue.get()
+ logger.info(f"Dall E node get task: {task.display()}")
+ result = self._run_task(task)
+
+ asyncio.create_task(_run_task_loop())
+
+ def display(self) -> str:
+ return f"DallE_ComputeNode: {self.node_id}"
+
+ def get_task_state(self, task_id: str):
+ pass
+
+ def get_capacity(self):
+ pass
+
+ def is_support(self, task: ComputeTask) -> bool:
+ return task.task_type == ComputeTaskType.TEXT_2_IMAGE
+
+ def is_local(self) -> bool:
+ return False
diff --git a/src/aios_kernel/open_ai_node.py b/src/component/openai_node/open_ai_node.py
similarity index 51%
rename from src/aios_kernel/open_ai_node.py
rename to src/component/openai_node/open_ai_node.py
index de68656..345c6c4 100644
--- a/src/aios_kernel/open_ai_node.py
+++ b/src/component/openai_node/open_ai_node.py
@@ -1,13 +1,18 @@
+import asyncio
import openai
+from openai import AsyncOpenAI
import os
import asyncio
from asyncio import Queue
import logging
import json
+import aiohttp
+import base64
+import requests
+from openai._types import NOT_GIVEN
-from .compute_task import ComputeTask, ComputeTaskResult, ComputeTaskState, ComputeTaskType,ComputeTaskResultCode
-from .compute_node import ComputeNode
-from .storage import AIStorage,UserConfig
+from aios import ComputeTask, ComputeTaskResult, ComputeTaskState, ComputeTaskType,ComputeTaskResultCode,ComputeNode,AIStorage,UserConfig
+from aios import image_utils
logger = logging.getLogger(__name__)
@@ -57,9 +62,87 @@ class OpenAI_ComputeNode(ComputeNode):
async def remove_task(self, task_id: str):
pass
- def _run_task(self, task: ComputeTask):
+ def message_to_dict(self, message)->dict:
+ result = message.dict()
+ # result_msg = {}
+ # #message.json()
+ # if message.content:
+ # result_msg["content"] = message.content
+ # result_msg["role"] = message.role
+ # if message.function_call:
+ # function_call = {}
+ # function_call["arguments"] = message.function_call.arguments
+ # function_call["name"] = message.function_call.name
+ # result_msg["function_call"] = function_call
+
+ # if message.tool_calls:
+ # tool_calls = []
+ # for tool_call in message.tool_calls:
+ # tool_call_dict = {}
+ # tool_call_dict["id"] = tool_call.id
+ # tool_call_dict["type"] = tool_call.type
+ # func_call_dict = {}
+ # func_call_dict["name"] = tool_call.function.name
+ # func_call_dict["arguments"] = tool_call.function.arguments
+ # tool_call_dict["function"] = func_call_dict
+
+ # tool_calls.append(tool_call_dict)
+ # result_msg["tool_calls"] = message.tool_calls
+
+ # result["message"] = result_msg
+ return result
+
+ def _image_2_text(self, task: ComputeTask):
+ logger.info('openai image_2_text')
+ # 本地图片处理
+
+ headers = {
+ "Content-Type": "application/json",
+ "Authorization": f"Bearer {self.openai_api_key }"
+ }
+ model_name = task.params["model_name"]
+ image_path = task.params["image_path"]
+
+ if image_utils.is_file(image_path):
+ url = image_utils.to_base64(image_path, (1024, 1024))
+ else:
+ url = image_path
+
+ payload = {
+ "model": model_name,
+ "messages": [
+ {
+ "role": "user",
+ "content": [
+ {
+ "type": "text",
+ "text": task.params["prompt"]
+ },
+ {
+ "type": "image_url",
+ "image_url": {
+ "url": url
+ }
+ }
+ ]
+ }
+ ],
+ "max_tokens": 300
+ }
+ logger.info('openai send image_2_text request ')
+ # openai 的库的Vision只支持传图片的url地址。本地图片得用request
+ response = requests.post("https://api.openai.com/v1/chat/completions", headers=headers, json=payload)
+ if response.status_code == 200:
+ logger.info('openai image_2_text success')
+ return response.json()
+ else:
+ logger.error('openai image_2_text error')
+ logger.error(response.json())
+ return None
+
+ async def _run_task(self, task: ComputeTask):
task.state = ComputeTaskState.RUNNING
-
+
result = ComputeTaskResult()
result.result_code = ComputeTaskResultCode.ERROR
result.set_from_task(task)
@@ -78,7 +161,7 @@ class OpenAI_ComputeNode(ComputeNode):
task.error_str = str(e)
result.error_str = str(e)
return result
-
+
# resp = {
# "object": "list",
# "data": [
@@ -100,30 +183,55 @@ class OpenAI_ComputeNode(ComputeNode):
result.worker_id = self.node_id
result.result_str = resp["data"][0]["embedding"]
+ return result
+ case ComputeTaskType.IMAGE_2_TEXT:
+ result.result_code = ComputeTaskResultCode.OK
+ result.worker_id = self.node_id
+ # result.result_str = resp["data"][0]["image_2_text"]
+ result.result["message"] = self._image_2_text(task)
return result
case ComputeTaskType.LLM_COMPLETION:
mode_name = task.params["model_name"]
prompts = task.params["prompts"]
+ resp_mode = task.params["resp_mode"]
+ if resp_mode == "json":
+ response_format = { "type": "json_object" }
+ else:
+ response_format = None
max_token_size = task.params.get("max_token_size")
llm_inner_functions = task.params.get("inner_functions")
if max_token_size is None:
max_token_size = 4000
- result_token = max_token_size
- try:
- if llm_inner_functions is None:
- logger.info(f"call openai {mode_name} prompts: {prompts}")
- resp = openai.ChatCompletion.create(model=mode_name,
- messages=prompts,
- #max_tokens=result_token,
- temperature=0.7)
+ if mode_name == "gpt-4-vision-preview":
+ response_format = NOT_GIVEN
+ llm_inner_functions = None
+ if max_token_size > 4096 or max_token_size < 50:
+ result_token = 4096
else:
- logger.info(f"call openai {mode_name} prompts: {prompts} functions: {json.dumps(llm_inner_functions)}")
- resp = openai.ChatCompletion.create(model=mode_name,
+ result_token = -1
+ else:
+ result_token = NOT_GIVEN
+
+ client = AsyncOpenAI(api_key=self.openai_api_key)
+ try:
+ if llm_inner_functions is None or len(llm_inner_functions) == 0:
+ if mode_name != "gpt-4-vision-preview":
+ logger.info(f"call openai {mode_name} prompts: {prompts}")
+ resp = await client.chat.completions.create(model=mode_name,
+ messages=prompts,
+ response_format = response_format,
+ max_tokens=result_token,
+ )
+ else:
+ if mode_name != "gpt-4-vision-preview":
+ logger.info(f"call openai {mode_name} prompts: \n\t {prompts} \nfunctions: \n\t{json.dumps(llm_inner_functions,ensure_ascii=False)}")
+ resp = await client.chat.completions.create(model=mode_name,
messages=prompts,
+ response_format = response_format,
functions=llm_inner_functions,
- #max_tokens=result_token,
- temperature=0.7) # TODO: add temperature to task params?
+ max_tokens=result_token,
+ ) # TODO: add temperature to task params?
except Exception as e:
logger.error(f"openai run LLM_COMPLETION task error: {e}")
task.state = ComputeTaskState.ERROR
@@ -131,10 +239,16 @@ class OpenAI_ComputeNode(ComputeNode):
result.error_str = str(e)
return result
- logger.info(f"openai response: {json.dumps(resp, indent=4)}")
+ #logger.info(f"openai response: {resp}")
+ #TODO: gpt-4v api is image_2_text ?
+ if mode_name == "gpt-4-vision-preview":
+ status_code = resp.choices[0].finish_reason
+ if status_code is None:
+ status_code = resp.choices[0].finish_details['type']
+ else:
+ status_code = resp.choices[0].finish_reason
+ token_usage = resp.usage
- status_code = resp["choices"][0]["finish_reason"]
- token_usage = resp.get("usage")
match status_code:
case "function_call":
task.state = ComputeTaskState.DONE
@@ -149,11 +263,13 @@ class OpenAI_ComputeNode(ComputeNode):
result.result_code = ComputeTaskResultCode.OK
result.worker_id = self.node_id
- result.result_str = resp["choices"][0]["message"]["content"]
- result.result["message"] = resp["choices"][0]["message"]
+ result.result_str = resp.choices[0].message.content
+
+ result.result["message"] = self.message_to_dict(resp.choices[0].message)
if token_usage:
result.result_refers["token_usage"] = token_usage
+
logger.info(f"openai success response: {result.result_str}")
return result
case _:
@@ -171,10 +287,10 @@ class OpenAI_ComputeNode(ComputeNode):
while True:
task = await self.task_queue.get()
logger.info(f"openai_node get task: {task.display()}")
- result = self._run_task(task)
+ result = await self._run_task(task)
if result is not None:
- task.state = ComputeTaskState.DONE
task.result = result
+ task.state = ComputeTaskState.DONE
asyncio.create_task(_run_task_loop())
@@ -195,7 +311,11 @@ class OpenAI_ComputeNode(ComputeNode):
model_name : str = task.params["model_name"]
if model_name.startswith("gpt-"):
return True
-
+
+ if task.task_type == ComputeTaskType.IMAGE_2_TEXT:
+ model_name : str = task.params["model_name"]
+ if model_name.startswith("gpt-4"):
+ return True
#if task.task_type == ComputeTaskType.TEXT_EMBEDDING:
# if task.params["model_name"] == "text-embedding-ada-002":
# return True
diff --git a/src/component/openai_node/openai_tts_node.py b/src/component/openai_node/openai_tts_node.py
new file mode 100644
index 0000000..f71ee41
--- /dev/null
+++ b/src/component/openai_node/openai_tts_node.py
@@ -0,0 +1,118 @@
+import asyncio
+import io
+import logging
+import os
+from asyncio import Queue
+
+from aios import ComputeNode, ComputeTask, ComputeTaskState, ComputeTaskResult, ComputeTaskType, AIStorage
+
+logger = logging.getLogger(__name__)
+
+
+class OpenAITTSComputeNode(ComputeNode):
+ _instance = None
+
+ @classmethod
+ def get_instance(cls):
+ if cls._instance is None:
+ cls._instance = cls()
+ return cls._instance
+
+ def __init__(self):
+ super().__init__()
+ self.is_start = False
+ self.node_id = "openai_tts_node"
+ self.task_queue = Queue()
+ self.voice_list = {
+ "female": ["nova", "shimmer"],
+ "man": ["alloy", "echo", "fable", "onyx"]
+ }
+ if os.getenv("OPENAI_API_KEY") is not None:
+ self.openai_api_key = os.getenv("OPENAI_API_KEY")
+ else:
+ self.openai_api_key = AIStorage.get_instance().get_user_config().get_value("openai_api_key")
+
+ self.start()
+
+ def start(self):
+ if self.is_start is True:
+ logger.warn("OpenAITTSComputeNode is already start")
+ return
+ self.is_start = True
+
+ async def _run_task_loop():
+ while True:
+ task = await self.task_queue.get()
+ try:
+ result = await self._run_task(task)
+ if result is not None:
+ task.state = ComputeTaskState.DONE
+ task.result = result
+ except Exception as e:
+ logger.error(f"openai_tts_node run task error: {e}")
+ task.state = ComputeTaskState.ERROR
+ task.result = ComputeTaskResult()
+ task.result.set_from_task(task)
+ task.result.worker_id = self.node_id
+ task.result.result_str = str(e)
+
+ asyncio.create_task(_run_task_loop())
+
+ async def _run_task(self,task: ComputeTask):
+ task.state = ComputeTaskState.RUNNING
+ text = task.params["text"]
+ voice_name = task.params["voice_name"]
+ if voice_name is None:
+ voice_name = "default"
+ gender = task.params["gender"]
+ if gender is None:
+ gender = "female"
+
+ voice_list = self.voice_list[gender]
+ voice = voice_list[hash(voice_name)%len(voice_list)]
+
+ model_name = task.params['model_name']
+ if model_name is None:
+ model_name = 'tts-1'
+
+ client = AsyncOpenAI(api_key=self.openai_api_key)
+
+ response = await client.audio.speech.create(model=model_name, voice=voice, input=text)
+
+ cache = io.BytesIO()
+ async for data in await response.aiter_bytes():
+ cache.write(data)
+
+ cache.seek(0)
+
+ result = ComputeTaskResult()
+ result.set_from_task(task)
+ result.worker_id = self.node_id
+ result.result = cache.read()
+ return result
+
+ async def push_task(self, task: ComputeTask, proiority: int = 0):
+ logger.info(f"openai_tts_node push task: {task.display()}")
+ self.task_queue.put_nowait(task)
+
+ async def remove_task(self, task_id: str):
+ pass
+
+ def get_task_state(self, task_id: str):
+ pass
+
+ def display(self) -> str:
+ return f"OpenAITTSComputeNode: {self.node_id}"
+
+ def get_capacity(self):
+ return 0
+
+ def is_support(self, task: ComputeTask) -> bool:
+ if task.task_type == ComputeTaskType.TEXT_2_VOICE:
+ if task.params['model_name'] is None or task.params['model_name'] == 'tts-1' or task.params['model_name'] == 'tts-1-hd':
+ return True
+ return False
+
+
+ def is_local(self) -> bool:
+ return False
diff --git a/src/component/openai_node/whisper_node.py b/src/component/openai_node/whisper_node.py
new file mode 100644
index 0000000..b1a8e14
--- /dev/null
+++ b/src/component/openai_node/whisper_node.py
@@ -0,0 +1,226 @@
+import io
+import json
+from asyncio import Queue
+import asyncio
+import openai
+import os
+import logging
+import srt
+import webvtt
+
+from openai import AsyncOpenAI
+from openai.cli._progress import BufferReader
+from pydub import AudioSegment
+from datetime import timedelta
+
+from aios import AIStorage,ComputeNode,ComputeTask, ComputeTaskResult, ComputeTaskState, ComputeTaskType
+
+logger = logging.getLogger(__name__)
+
+SECONDS_IN_HOUR = 3600
+SECONDS_IN_MINUTE = 60
+HOURS_IN_DAY = 24
+MICROSECONDS_IN_MILLISECOND = 1000
+
+def timedelta_to_vtt_timestamp(timedelta_timestamp):
+ hrs, secs_remainder = divmod(timedelta_timestamp.seconds, SECONDS_IN_HOUR)
+ hrs += timedelta_timestamp.days * HOURS_IN_DAY
+ mins, secs = divmod(secs_remainder, SECONDS_IN_MINUTE)
+ msecs = timedelta_timestamp.microseconds // MICROSECONDS_IN_MILLISECOND
+ return "%02d:%02d:%02d.%03d" % (hrs, mins, secs, msecs)
+
+
+class WhisperComputeNode(ComputeNode):
+ _instance = None
+
+ @classmethod
+ def get_instance(cls):
+ if cls._instance is None:
+ cls._instance = cls()
+ return cls._instance
+
+ def __init__(self) -> None:
+ super().__init__()
+ self.is_start = False
+ self.node_id = "whisper_node"
+ self.enable = True
+ self.task_queue = Queue()
+
+ if os.getenv("OPENAI_API_KEY") is not None:
+ self.openai_api_key = os.getenv("OPENAI_API_KEY")
+ else:
+ self.openai_api_key = AIStorage.get_instance().get_user_config().get_value("openai_api_key")
+
+ self.start()
+
+ def start(self):
+ if self.is_start is True:
+ logger.warn("WhisperComputeNode is already start")
+ return
+ self.is_start = True
+ async def _run_task_loop():
+ while True:
+ task = await self.task_queue.get()
+ try:
+ result = await self._run_task(task)
+ if result is not None:
+ task.state = ComputeTaskState.DONE
+ task.result = result
+ except Exception as e:
+ logger.error(f"whisper_node run task error: {e}")
+ logger.exception(e)
+ task.state = ComputeTaskState.ERROR
+ task.result = ComputeTaskResult()
+ task.result.set_from_task(task)
+ task.result.worker_id = self.node_id
+ task.result.result_str = str(e)
+
+ asyncio.create_task(_run_task_loop())
+
+ async def _run_task(self, task: ComputeTask):
+ task.state = ComputeTaskState.RUNNING
+ prompt = task.params["prompt"]
+ response_format = None
+ if "response_format" in task.params:
+ response_format = task.params["response_format"]
+ temperature = None
+ if "temperature" in task.params:
+ temperature = task.params["temperature"]
+ language = None
+ if "language" in task.params:
+ language = task.params["language"]
+ file = task.params["file"]
+
+ client = AsyncOpenAI(api_key=self.openai_api_key)
+
+ if os.path.getsize(file) > 25 * 1024 * 1024:
+ audio = AudioSegment.from_file(file)
+ text = ""
+ results = []
+ latest_resp = None
+ step = 10 * 60 * 1000
+ for i in range(0, len(audio), step):
+ if i + step < len(audio):
+ chunk = audio[i:i + step]
+ else:
+ chunk = audio[i:]
+ seg_file = io.BytesIO()
+ chunk.export(seg_file, format="mp3")
+ seg_file.seek(0)
+
+ resp = await client.audio.transcriptions.create(model="whisper-1",
+ file = ("test.mp3", seg_file),
+ language=language,
+ temperature=temperature,
+ prompt=prompt,
+ response_format=response_format)
+ if response_format == "json":
+ if text == "":
+ text = resp.text
+ else:
+ text += "," + resp.text
+ elif response_format == "text":
+ if text == "":
+ text = resp
+ else:
+ text += "," + resp
+ elif response_format == "verbose_json":
+ if text == "":
+ text = resp.text
+ else:
+ text += "," + resp.text
+ results.extend(resp.segments)
+ elif response_format == "srt":
+ srt_list = list(srt.parse(resp))
+ for item in srt_list:
+ item.start += timedelta(milliseconds=i)
+ item.end += timedelta(milliseconds=i)
+ results.append(item)
+ elif response_format == "vtt":
+ vtt = webvtt.read_buffer(io.StringIO(resp))
+ for caption in vtt.captions:
+ start = timedelta_to_vtt_timestamp(
+ srt.srt_timestamp_to_timedelta(caption.start) + timedelta(milliseconds=i))
+ end = timedelta_to_vtt_timestamp(
+ srt.srt_timestamp_to_timedelta(caption.end) + timedelta(milliseconds=i))
+ results.append(webvtt.Caption(start, end, caption.text))
+ else:
+ raise Exception(f"not support response_format: {response_format}")
+
+ latest_resp = resp
+
+ result = ComputeTaskResult()
+ result.set_from_task(task)
+ result.worker_id = self.node_id
+ if response_format == "text":
+ result.result_str = text
+ result.result = text
+ elif response_format == "json":
+ result.result_str = json.dumps({"text": text},ensure_ascii=False)
+ resp.text = text
+ result.result = resp
+ elif response_format == "verbose_json":
+ result.result_str = json.dumps({"text": text, "segments": results},ensure_ascii=False)
+ latest_resp.text = text
+ latest_resp.segments = results
+ result.result = latest_resp
+ elif response_format == "srt":
+ result.result_str = srt.compose(results)
+ result.result = result.result_str
+ elif response_format == "vtt":
+ vtt = webvtt.WebVTT()
+ vtt.captions.extend(results)
+ f = io.StringIO()
+ vtt.write(f)
+ f.seek(0)
+ result.result_str = f.read()
+ result.result = result.result_str
+ return result
+ else:
+ with open(file, "rb") as file_reader:
+ buffer_reader = BufferReader(file_reader.read(), desc="Upload progress")
+
+ resp = await client.audio.transcriptions.create(model="whisper-1",
+ file = (file, buffer_reader),
+ language=language,
+ temperature=temperature,
+ prompt=prompt,
+ response_format=response_format)
+ result = ComputeTaskResult()
+ result.set_from_task(task)
+ result.worker_id = self.node_id
+ if response_format == "json":
+ result.result_str = json.dumps({"text": resp.text},ensure_ascii=False)
+ elif response_format == "verbose_json":
+ result.result_str = json.dumps({"text": resp.text, "segments": resp.segments},ensure_ascii=False)
+ elif response_format == "srt" or response_format == "vtt" or response_format == "text":
+ result.result_str = resp
+ else:
+ raise Exception(f"not support response_format: {response_format}")
+ result.result = resp
+ return result
+
+ async def push_task(self, task: ComputeTask, proiority: int = 0):
+ logger.info(f"whisper_node push task: {task.display()}")
+ self.task_queue.put_nowait(task)
+
+ async def remove_task(self, task_id: str):
+ pass
+
+ def get_task_state(self, task_id: str):
+ pass
+
+ def display(self) -> str:
+ return f"WhisperComputeNode: {self.node_id}"
+
+ def get_capacity(self):
+ return 0
+
+ def is_support(self, task: ComputeTask) -> bool:
+ if task.task_type == ComputeTaskType.VOICE_2_TEXT:
+ if task.params['model_name'] is None or task.params['model_name'] == 'openai-whisper':
+ return True
+ return False
+
+ def is_local(self) -> bool:
+ return False
diff --git a/src/component/sd_node/__init__.py b/src/component/sd_node/__init__.py
new file mode 100644
index 0000000..bd6d2b8
--- /dev/null
+++ b/src/component/sd_node/__init__.py
@@ -0,0 +1,2 @@
+from .local_stability_node import *
+from .stability_node import *
\ No newline at end of file
diff --git a/src/aios_kernel/local_stability_node.py b/src/component/sd_node/local_stability_node.py
similarity index 97%
rename from src/aios_kernel/local_stability_node.py
rename to src/component/sd_node/local_stability_node.py
index d0d7f9d..6341590 100644
--- a/src/aios_kernel/local_stability_node.py
+++ b/src/component/sd_node/local_stability_node.py
@@ -9,9 +9,7 @@ import requests
from typing import Tuple
from pathlib import Path
-from .compute_task import ComputeTask, ComputeTaskResult, ComputeTaskState, ComputeTaskType, ComputeTaskResultCode
-from .compute_node import ComputeNode
-from .storage import AIStorage, UserConfig
+from aios import ComputeTask, ComputeTaskResult, ComputeTaskState, ComputeTaskType,ComputeTaskResultCode,ComputeNode,AIStorage,UserConfig
logger = logging.getLogger(__name__)
diff --git a/src/aios_kernel/stability_node.py b/src/component/sd_node/stability_node.py
similarity index 97%
rename from src/aios_kernel/stability_node.py
rename to src/component/sd_node/stability_node.py
index 2ea1f44..8c8cc9f 100644
--- a/src/aios_kernel/stability_node.py
+++ b/src/component/sd_node/stability_node.py
@@ -9,9 +9,7 @@ from PIL import Image
from stability_sdk import client
import stability_sdk.interfaces.gooseai.generation.generation_pb2 as generation
-from .compute_task import ComputeTask, ComputeTaskResult, ComputeTaskState, ComputeTaskType, ComputeTaskResultCode
-from .compute_node import ComputeNode
-from .storage import AIStorage, UserConfig
+from aios import ComputeTask, ComputeTaskResult, ComputeTaskState, ComputeTaskType,ComputeTaskResultCode,ComputeNode,AIStorage,UserConfig
logger = logging.getLogger(__name__)
diff --git a/src/component/slack_tunnel.py b/src/component/slack_tunnel.py
new file mode 100644
index 0000000..52a5b63
--- /dev/null
+++ b/src/component/slack_tunnel.py
@@ -0,0 +1,239 @@
+import asyncio
+import datetime
+import json
+import logging
+import os.path
+import re
+import uuid
+import time
+
+import aiofiles
+import aiohttp
+from slack_bolt.adapter.socket_mode.websockets import AsyncSocketModeHandler
+from slack_bolt.app.async_app import AsyncApp
+
+from aios import KnowledgeStore, ObjectType
+from aios.frame.tunnel import AgentTunnel
+from aios.proto.agent_msg import AgentMsg, AgentMsgType
+from aios.storage.storage import AIStorage
+
+logger = logging.getLogger(__name__)
+
+
+async def download_file(url: str, file_path: str, token: str) -> None:
+ async with aiohttp.ClientSession() as session:
+ async with session.get(url, headers={"Authorization": f"Bearer {token}"}) as resp:
+ if resp.status == 200:
+ f = await aiofiles.open(file_path, mode='wb')
+ while True:
+ chunk = await resp.content.read(1024)
+ if not chunk:
+ break
+ await f.write(chunk)
+ await f.close()
+
+
+class SlackTunnel(AgentTunnel):
+ type: str
+ token: str
+ app_token: str
+ slack_cache: str
+ app: AsyncSocketModeHandler
+
+ def __init__(self):
+ super().__init__()
+ self.type = "SlackTunnel"
+ self.token = ""
+ self.slack_cache = os.path.join(AIStorage.get_instance().get_myai_dir(), "slack")
+ if not os.path.exists(self.slack_cache):
+ os.makedirs(self.slack_cache)
+
+ @classmethod
+ def register_to_loader(cls):
+ async def load_slack_tunnel(config: dict) -> AgentTunnel:
+ result_tunnel = SlackTunnel()
+ if await result_tunnel.load_from_config(config):
+ return result_tunnel
+ else:
+ return None
+
+ AgentTunnel.register_loader("SlackTunnel", load_slack_tunnel)
+
+ async def load_from_config(self, config: dict) -> bool:
+ self.target_id = config["target"]
+ self.tunnel_id = config["tunnel_id"]
+
+ self.type = "SlackTunnel"
+ self.token = config["token"]
+ self.app_token = config["app_token"]
+
+ return True
+
+ def get_cache_path(self) -> str:
+ today = datetime.datetime.today()
+ path = os.path.join(self.slack_cache, str(today.year), str(today.month))
+ if not os.path.exists(path):
+ os.makedirs(path)
+ return path
+
+ def post_message(self, msg: AgentMsg) -> None:
+ pass
+
+ async def start(self) -> bool:
+ app = AsyncApp(token=self.token)
+
+ bot_info = await app.client.auth_test()
+
+ @app.event("message")
+ async def _handle_message(event):
+ logger.info(json.dumps(event))
+ ty = event["type"]
+ if ty != "message":
+ return
+
+ user = event["user"]
+ user_info = await app.client.users_info(user=user)
+
+ if not user_info["ok"]:
+ return
+
+ user_info = user_info["user"]
+
+ mime_type = None
+ images = []
+ file_type = None
+ video_file = None
+ audio_file = None
+
+
+ if "files" in event:
+ files = event["files"]
+ if files is not None and len(files) > 0:
+ for file in files:
+ if file["mode"] == "tombstone":
+ continue
+
+ file_path = os.path.join(self.get_cache_path(), file["id"] + "." + file["filetype"])
+ file_info = await app.client.files_info(file=file["id"])
+ if not file_info["ok"]:
+ continue
+ await download_file(file_info["file"]["url_private_download"], file_path, self.token)
+
+ mime_type = file["mimetype"]
+ if file["mimetype"].startswith("image/"):
+ if file_type is None:
+ file_type = "image"
+ elif file_type != "image":
+ break
+ images.append(file_path)
+ elif file["mimetype"].startswith("video/"):
+ if file_type is None:
+ file_type = "video"
+ video_file = file_path
+ break
+ elif file["mimetype"].startswith("audio/"):
+ if file_type is None:
+ file_type = "audio"
+ audio_file = file_path
+ break
+
+ agent_msg = AgentMsg()
+ agent_msg.topic = "_slack"
+ agent_msg.msg_id = "discord_msg#" + event["client_msg_id"] + "#" + uuid.uuid4().hex
+ agent_msg.target = self.target_id
+ agent_msg.create_time = time.time()
+ agent_msg.sender = user_info["name"]
+ self.ai_bus.register_message_handler(agent_msg.sender, self._process_message)
+
+ content = re.sub("<@.+>", "", event["text"]).strip()
+
+ blocks = event["blocks"]
+ is_metion = False
+ for block in blocks:
+ if block["type"] == "rich_text":
+ elements = block["elements"]
+ for element in elements:
+ if element["type"] == "rich_text_section":
+ elements = element["elements"]
+
+ for element in elements:
+ if element["type"] == "user":
+ if element["user_id"] == bot_info.get("user_id"):
+ is_metion = True
+ break
+
+ if not is_metion:
+ agent_msg.msg_type = AgentMsgType.TYPE_GROUPMSG
+
+ if file_type is None:
+ agent_msg.body = content
+ elif file_type == "image":
+ agent_msg.body = agent_msg.create_image_body(images, content)
+ agent_msg.body_mime = mime_type
+ elif file_type == "video":
+ agent_msg.body = agent_msg.create_video_body(video_file, content)
+ agent_msg.body_mime = mime_type
+ elif file_type == "audio":
+ agent_msg.body = agent_msg.create_audio_body(audio_file, content)
+ agent_msg.body_mime = mime_type
+
+
+ resp_msg: AgentMsg = await self.ai_bus.send_message(agent_msg)
+ if resp_msg is None:
+ await app.client.chat_postMessage(channel=event["channel"], text=f"System Error: Timeout,{self.target_id} no resopnse! Please check logs/aios.log for more details!")
+ else:
+ if resp_msg.body_mime is None:
+ if resp_msg.body is None:
+ return
+
+ if len(resp_msg.body) < 1:
+ return
+
+ knownledge_object = KnowledgeStore().parse_object_in_message(resp_msg.body)
+ if knownledge_object is not None:
+ if knownledge_object.get_object_type() == ObjectType.Image:
+ image = KnowledgeStore().bytes_from_object(knownledge_object)
+ try:
+ async with aiofiles.open("image.jpg", "wb") as f:
+ await f.write(image)
+ await app.client.files_upload_v2(channel=event["channel"], file="image.jpg")
+ except Exception as e:
+ logger.error(f"save image error:{e}")
+ logger.exception(e)
+ return
+ else:
+ pos = resp_msg.body.find("audio file")
+ if pos != -1:
+ audio_file = resp_msg.body[pos+11:].strip()
+ if audio_file.startswith("\""):
+ audio_file = audio_file[1:-1]
+ await app.client.files_upload_v2(channel=event["channel"], file=audio_file)
+ return
+ await app.client.chat_postMessage(channel=event["channel"], text=resp_msg.body)
+ else:
+ if resp_msg.is_image_msg():
+ text, images = resp_msg.get_image_body()
+ file_uploads = []
+ for image in images:
+ file_uploads.append({"file": image})
+ await app.client.files_upload_v2(channel=event["channel"], file_uploads=file_uploads, initial_comment=text)
+ elif resp_msg.is_video_msg():
+ text, video_file = resp_msg.get_video_body()
+ await app.client.files_upload_v2(channel=event["channel"], file=video_file, initial_comment=text)
+ elif resp_msg.is_audio_msg():
+ text, audio_file = resp_msg.get_audio_body()
+ await app.client.files_upload_v2(channel=event["channel"], file=audio_file, initial_comment=text)
+ else:
+ await app.client.chat_postMessage(channel=event["channel"], text=resp_msg.body)
+
+ handle = AsyncSocketModeHandler(app, self.app_token)
+ asyncio.create_task(handle.start_async())
+ self.app = handle
+ return True
+
+ async def close(self) -> None:
+ await self.app.close_async()
+ self.app = None
+
+ async def _process_message(self, msg: AgentMsg) -> None:
+ logger.warn(f"process message {msg.msg_id} from {msg.sender} to {msg.target}")
diff --git a/src/component/st_node/__init__.py b/src/component/st_node/__init__.py
new file mode 100644
index 0000000..a1fd2e0
--- /dev/null
+++ b/src/component/st_node/__init__.py
@@ -0,0 +1 @@
+from .local_st_compute_node import *
\ No newline at end of file
diff --git a/src/aios_kernel/local_st_compute_node.py b/src/component/st_node/local_st_compute_node.py
similarity index 96%
rename from src/aios_kernel/local_st_compute_node.py
rename to src/component/st_node/local_st_compute_node.py
index 4142d5a..1887ae3 100644
--- a/src/aios_kernel/local_st_compute_node.py
+++ b/src/component/st_node/local_st_compute_node.py
@@ -5,9 +5,8 @@ from pydantic import BaseModel
from typing import Union
from PIL import Image
import io
-from .compute_task import ComputeTask, ComputeTaskState, ComputeTaskType,ComputeTaskResult,ComputeTaskResultCode
-from .queue_compute_node import Queue_ComputeNode
-from knowledge import ObjectID
+
+from aios import ComputeTask, ComputeTaskResult, ComputeTaskState, ComputeTaskType,ComputeTaskResultCode,ComputeNode,AIStorage,UserConfig,ObjectID,Queue_ComputeNode
logger = logging.getLogger(__name__)
@@ -117,7 +116,7 @@ class LocalSentenceTransformer_Image_ComputeNode(Queue_ComputeNode):
def _load_image(self, source: Union[ObjectID, bytes]) -> Optional[Image]:
image_data = None
if isinstance(source, ObjectID):
- from knowledge import KnowledgeStore, ImageObject
+ from aios import KnowledgeStore, ImageObject
buf = KnowledgeStore().get_object_store().get_object(source)
if buf is None:
diff --git a/src/aios_kernel/tg_tunnel.py b/src/component/tg_tunnel.py
similarity index 55%
rename from src/aios_kernel/tg_tunnel.py
rename to src/component/tg_tunnel.py
index 53aed7a..094a98e 100644
--- a/src/aios_kernel/tg_tunnel.py
+++ b/src/component/tg_tunnel.py
@@ -1,4 +1,6 @@
+import datetime
import logging
+import os.path
import threading
import asyncio
import uuid
@@ -10,20 +12,13 @@ from telegram import Bot
from telegram.ext import Updater
from telegram.error import Forbidden, NetworkError
-from knowledge.object.object_id import ObjectType
-
-from .knowledge_base import KnowledgeBase
-
-from .tunnel import AgentTunnel
-from .storage import AIStorage
-from .contact_manager import ContactManager,Contact,FamilyMember
-from .agent_message import AgentMsg,AgentMsgType
-
+from aios import ObjectType, KnowledgeStore,AgentTunnel,AIStorage,ContactManager,Contact,AgentMsg,AgentMsgType
logger = logging.getLogger(__name__)
class TelegramTunnel(AgentTunnel):
-
+ all_bots = {}
+ default_chatid = {}
@classmethod
def register_to_loader(cls):
async def load_tg_tunnel(config:dict) -> AgentTunnel:
@@ -57,6 +52,10 @@ class TelegramTunnel(AgentTunnel):
self.update_queue = None
self.allow_group = "contact"
self.in_process_tg_msg = {}
+ self.chatid_record = {}
+ self.telegram_cache = os.path.join(AIStorage.get_instance().get_myai_dir(), "telegram")
+ if not os.path.exists(self.telegram_cache):
+ os.makedirs(self.telegram_cache)
async def _do_process_raw_message(self,bot: Bot, update_id: int) -> int:
# Request updates after the last update_id
@@ -64,7 +63,7 @@ class TelegramTunnel(AgentTunnel):
for update in updates:
next_update_id = update.update_id + 1
- if update.message and update.message.text:
+ if update.message and (update.message.text or (update.message.photo and len(update.message.photo) > 0) or update.message.video or update.message.voice or update.message.audio):
await self.on_message(bot,update)
return next_update_id
@@ -83,11 +82,17 @@ class TelegramTunnel(AgentTunnel):
self.update_queue = asyncio.Queue()
self.bot_updater = Updater(self.bot,update_queue=self.update_queue)
+ TelegramTunnel.all_bots[self.target_id] = self.bot
async def _run_app():
+ update_id = 0
try:
- update_id = (await self.bot.get_updates())[0].update_id
- except IndexError:
+ update = await self.bot.get_updates()
+ if len(update) > 0:
+ update_id = update[0].update_id
+ except Exception as e:
+ logger.error(f"tg_tunnel error:{e}")
+ logger.exception(e)
update_id = None
#logger.info("listening for new messages...")
@@ -101,9 +106,10 @@ class TelegramTunnel(AgentTunnel):
update_id += 1
except Exception as e:
logger.error(f"tg_tunnel error:{e}")
+ logger.exception(e)
await asyncio.sleep(1)
-
+
asyncio.create_task(_run_app())
logger.info(f"tunnel {self.tunnel_id} started.")
@@ -112,16 +118,87 @@ class TelegramTunnel(AgentTunnel):
async def close(self) -> None:
pass
- async def _process_message(self, msg: AgentMsg) -> None:
- logger.warn(f"process message {msg.msg_id} from {msg.sender} to {msg.target}")
+ async def _process_message(self, msg: AgentMsg) -> bool:
+ logger.warn(f"tg_tunnel process message {msg.msg_id} from agent {msg.sender} to human {msg.target}")
+ # async def _process_message(self, msg: AgentMsg) -> bool:
+ # logger.info(f"tg_tunnel process message {msg.msg_id} from agent {msg.sender} to human {msg.target}")
+ # cm = ContactManager.get_instance()
+ # contact = cm.find_contact_by_name(msg.target)
+ # bot = TelegramTunnel.all_bots.get(msg.sender)
+ # chatid_index = f"{self.target_id}#{msg.target}"
+ # chatid = TelegramTunnel.default_chatid.get(chatid_index)
+ # if chatid is None:
+ # logger.warning(f"tg_tunnel process message {msg.msg_id} from agent {msg.sender} to human {msg.target} failed! chatid not found!")
+ # return None
+
+ # if bot is None:
+ # logger.warning(f"tg_tunnel process message {msg.msg_id} from agent {msg.sender} to human {msg.target} failed! bot not found!")
+ # return None
+
+ # if contact:
+ # if contact.telegram:
+ # await bot.send_message(chat_id=chatid,text=msg.body)
+ # logging.info(f"tg_tunnel send message {msg.msg_id} from agent {msg.sender} to human {msg.target} @ chatid:{chatid}success!")
+ # return None
+
+ # logger.warning(f"tg_tunnel process message {msg.msg_id} from agent {msg.sender} to human {msg.target} failed! contact not found!")
+ # return None
+
+ async def post_message(self, msg: AgentMsg) -> None:
+ chatid = self.chatid_record.get(msg.target)
+ if chatid:
+ # TODO: support image and audio
+ await self.bot.send_message(chat_id=chatid,text=msg.body)
+ logging.info(f"tg_tunnel send message {msg.msg_id} from agent {msg.sender} to human {msg.target} @ chatid:{chatid}success!")
+ else:
+ logger.warning(f"tg_tunnel process message {msg.msg_id} from agent {msg.sender} to human {msg.target} failed! chatid not found!")
+
+ def get_cache_path(self) -> str:
+ today = datetime.datetime.today()
+ path = os.path.join(self.telegram_cache, str(today.year), str(today.month))
+ if not os.path.exists(path):
+ os.makedirs(path)
+ return path
async def conver_tg_msg_to_agent_msg(self,message:Message) -> AgentMsg:
agent_msg = AgentMsg()
agent_msg.topic = "_telegram"
agent_msg.msg_id = "tg_msg#" + str(message.message_id) + "#" + uuid.uuid4().hex
agent_msg.target = self.target_id
- agent_msg.body = message.text
+ if message.text is not None:
+ agent_msg.body = message.text
+ elif message.photo is not None and len(message.photo) > 0:
+ photo_files = []
+ photo_file = await message.photo[-1].get_file()
+ ext = photo_file.file_path.rsplit(".")[-1]
+ file_path = os.path.join(self.get_cache_path(), photo_file.file_id + f".{ext}")
+ await photo_file.download_to_drive(file_path)
+ photo_files.append(file_path)
+ agent_msg.body = agent_msg.create_image_body(photo_files, message.caption)
+ agent_msg.body_mime = f"image/{ext}"
+ elif message.video is not None:
+ video_file = await message.video.get_file()
+ ext = video_file.file_path.rsplit(".")[-1]
+ file_path = os.path.join(self.get_cache_path(), video_file.file_id + f".{ext}")
+ await video_file.download_to_drive(file_path)
+ agent_msg.body = agent_msg.create_video_body(file_path, message.caption)
+ agent_msg.body_mime = f"video/{ext}"
+ elif message.audio is not None:
+ audio_file = await message.audio.get_file()
+ ext = audio_file.file_path.rsplit(".")[-1]
+ file_path = os.path.join(self.get_cache_path(), audio_file.file_id + f".{ext}")
+ await audio_file.download_to_drive(file_path)
+ agent_msg.body = agent_msg.create_audio_body(file_path, message.caption)
+ agent_msg.body_mime = f"audio/{ext}"
+ elif message.voice is not None:
+ audio_file = await message.voice.get_file()
+ ext = audio_file.file_path.rsplit(".")[-1]
+ file_path = os.path.join(self.get_cache_path(), audio_file.file_id + f".{ext}")
+ await audio_file.download_to_drive(file_path)
+ agent_msg.body = agent_msg.create_audio_body(file_path, message.caption)
+ agent_msg.body_mime = f"audio/{ext}"
+
agent_msg.create_time = time.time()
messag_type = message.chat.type
if messag_type == "supergroup" or messag_type == "group":
@@ -130,6 +207,7 @@ class TelegramTunnel(AgentTunnel):
agent_msg.mentions = []
else:
agent_msg.msg_type = AgentMsgType.TYPE_MSG
+ agent_msg.mentions = []
if message.entities:
for entity in message.entities:
@@ -139,7 +217,7 @@ class TelegramTunnel(AgentTunnel):
agent_msg.mentions.append(self.target_id)
else:
agent_msg.mentions.append(mention)
-
+
if message.caption_entities:
for entity in message.caption_entities:
if entity.type == 'mention':
@@ -174,11 +252,11 @@ class TelegramTunnel(AgentTunnel):
if update.effective_user.is_bot:
logger.warning(f"ignore message from telegram bot {update.effective_user.id}")
return None
-
+
if self.in_process_tg_msg.get(update.message.message_id) is not None:
logger.warning(f"ignore message from telegram bot {update.effective_user.id}")
return None
-
+
self.in_process_tg_msg[update.message.message_id] = True
agent_msg = await self.conver_tg_msg_to_agent_msg(message)
@@ -191,14 +269,16 @@ class TelegramTunnel(AgentTunnel):
if contact is not None:
reomte_user_name = contact.name
+
+ #TelegramTunnel.default_chatid[f"{self.target_id}#{reomte_user_name}"] = update.effective_chat.id
if not contact.is_family_member:
if self.allow_group != "contact" and self.allow_group !="guest":
await update.message.reply_text(f"You're not supposed to talk to me! Please contact my father~")
return
-
+
else:
if self.allow_group != "guest":
- await update.message.reply_text(f"You're not supposed to talk to me! Please contact my father~")
+ await update.message.reply_text(f"The current Telegram account is not in the contact list. If you want to receive a reply, you can add the configuration in the contacts.toml file or switch tunnel to guest mode.")
return
if cm.is_auto_create_contact_from_telegram:
@@ -211,16 +291,20 @@ class TelegramTunnel(AgentTunnel):
contact.added_by = self.target_id
cm.add_contact(contact.name, contact)
reomte_user_name = contact.name
-
+
+ if contact is not None:
+ contact.set_active_tunnel(self.target_id,self)
+ self.chatid_record[reomte_user_name] = update.effective_chat.id
+ self.ai_bus.register_message_handler(reomte_user_name,contact._process_msg)
agent_msg.sender = reomte_user_name
logger.info(f"process message {agent_msg.msg_id} from {agent_msg.sender} to {agent_msg.target}")
if agent_msg.msg_type == AgentMsgType.TYPE_GROUPMSG:
self.ai_bus.register_message_handler(agent_msg.target, self._process_message)
- resp_msg = await self.ai_bus.send_message(agent_msg,self.target_id,agent_msg.target)
+ resp_msg: AgentMsg = await self.ai_bus.send_message(agent_msg,self.target_id,agent_msg.target)
else:
- self.ai_bus.register_message_handler(reomte_user_name, self._process_message)
- resp_msg = await self.ai_bus.send_message(agent_msg)
+ #self.ai_bus.register_message_handler(reomte_user_name, self._process_message)
+ resp_msg: AgentMsg = await self.ai_bus.send_message(agent_msg)
#await bot.send_chat_action(chat_id=update.effective_chat.id, action="typing")
@@ -231,15 +315,15 @@ class TelegramTunnel(AgentTunnel):
if resp_msg.body_mime is None:
if resp_msg.body is None:
return
-
+
if len(resp_msg.body) < 1:
await update.message.reply_text("")
return
-
- knowledge_object = KnowledgeBase().parse_object_in_message(resp_msg.body)
+
+ knowledge_object = KnowledgeStore().parse_object_in_message(resp_msg.body)
if knowledge_object is not None:
if knowledge_object.get_object_type() == ObjectType.Image:
- image = KnowledgeBase().bytes_from_object(knowledge_object)
+ image = KnowledgeStore().bytes_from_object(knowledge_object)
try:
async with aiofiles.open("tg_send_temp.png", mode='wb') as local_file:
if local_file:
@@ -258,14 +342,32 @@ class TelegramTunnel(AgentTunnel):
return
await update.message.reply_text(resp_msg.body)
else:
- if resp_msg.body_mime.startswith("image"):
- photo_file = open(resp_msg.body,"rb")
- if photo_file:
- await update.message.reply_photo(resp_msg.body)
- photo_file.close()
+ if resp_msg.is_image_msg():
+ text, images = resp_msg.get_image_body()
+ if text is not None:
+ await update.message.reply_text(text)
+ for image in images:
+ if os.path.exists(image):
+ await update.message.reply_photo(image)
+ else:
+ await update.message.reply_text(image)
+ elif resp_msg.is_video_msg():
+ text, video_file = resp_msg.get_video_body()
+ if text is not None:
+ await update.message.reply_text(text)
+ if os.path.exists(video_file):
+ await update.message.reply_video(video_file)
else:
- await update.message.reply_text(resp_msg.body)
+ await update.message.reply_text(video_file)
+ elif resp_msg.is_audio_msg():
+ text, audio_file = resp_msg.get_audio_body()
+ if text is not None:
+ await update.message.reply_text(text)
+ if os.path.exists(audio_file):
+ await update.message.reply_voice(audio_file)
+ else:
+ await update.message.reply_text(audio_file)
else:
await update.message.reply_text(resp_msg.body)
diff --git a/src/component/workflow_manager/workflow_manager.py b/src/component/workflow_manager/workflow_manager.py
index ee5ef9e..4da3885 100644
--- a/src/component/workflow_manager/workflow_manager.py
+++ b/src/component/workflow_manager/workflow_manager.py
@@ -2,8 +2,7 @@ import logging
import toml
import os
-from aios_kernel import Workflow,AIStorage
-from package_manager import PackageEnv,PackageEnvManager,PackageMediaInfo,PackageInstallTask
+from aios import AIStorage,PackageEnv,PackageEnvManager,PackageMediaInfo,PackageInstallTask,Workflow
from agent_manager import AgentManager
logger = logging.getLogger(__name__)
@@ -57,7 +56,13 @@ class WorkflowManager:
if await self._load_workflow_agents(sub_workflow) is False:
return False
return True
-
+
+ async def is_exist(self,workflow_id:str) -> bool:
+ the_workflow = await self.get_workflow(workflow_id)
+ if the_workflow:
+ return True
+ return False
+
async def get_workflow(self,workflow_id:str) -> Workflow:
the_workflow : Workflow = self.loaded_workflow.get(workflow_id)
if the_workflow:
diff --git a/src/knowledge/store.py b/src/knowledge/store.py
deleted file mode 100644
index d8b4bbb..0000000
--- a/src/knowledge/store.py
+++ /dev/null
@@ -1,66 +0,0 @@
-import os
-
-from .object import ObjectStore, ObjectRelationStore
-from .data import ChunkStore, ChunkTracker, ChunkListWriter, ChunkReader
-from .vector import ChromaVectorStore, VectorBase
-import logging
-
-
-# KnowledgeStore class, which aggregates ChunkStore, ChunkTracker, and ObjectStore, and is a global singleton that makes it easy to use these three built-in store examples
-class KnowledgeStore:
- _instance = None
-
- def __new__(cls):
- if cls._instance is None:
- cls._instance = super().__new__(cls)
-
- import aios_kernel
- knowledge_dir = aios_kernel.storage.AIStorage().get_myai_dir() / "knowledge"
-
- if not os.path.exists(knowledge_dir):
- os.makedirs(knowledge_dir)
-
- cls._instance.__singleton_init__(knowledge_dir)
-
- return cls._instance
-
- def __singleton_init__(self, root_dir: str):
- logging.info(f"will init knowledge store, root_dir={root_dir}")
-
- self.root = root_dir
-
- relation_store_dir = os.path.join(root_dir, "relation")
- self.relation_store = ObjectRelationStore(relation_store_dir)
-
- object_store_dir = os.path.join(root_dir, "object")
- self.object_store = ObjectStore(object_store_dir)
-
- chunk_store_dir = os.path.join(root_dir, "chunk")
- self.chunk_store = ChunkStore(chunk_store_dir)
- self.chunk_tracker = ChunkTracker(chunk_store_dir)
- self.chunk_list_writer = ChunkListWriter(self.chunk_store, self.chunk_tracker)
- self.chunk_reader = ChunkReader(self.chunk_store, self.chunk_tracker)
- self.vector_store = {}
-
- def get_relation_store(self) -> ObjectRelationStore:
- return self.relation_store
-
- def get_object_store(self) -> ObjectStore:
- return self.object_store
-
- def get_chunk_store(self) -> ChunkStore:
- return self.chunk_store
-
- def get_chunk_tracker(self) -> ChunkTracker:
- return self.chunk_tracker
-
- def get_chunk_list_writer(self) -> ChunkListWriter:
- return self.chunk_list_writer
-
- def get_chunk_reader(self) -> ChunkReader:
- return self.chunk_reader
-
- def get_vector_store(self, model_name: str) -> VectorBase:
- if model_name not in self.vector_store:
- self.vector_store[model_name] = ChromaVectorStore(self.root, model_name)
- return self.vector_store[model_name]
diff --git a/src/node_daemon/__init__.py b/src/node_daemon/__init__.py
new file mode 100644
index 0000000..66f65c6
--- /dev/null
+++ b/src/node_daemon/__init__.py
@@ -0,0 +1 @@
+# might implement by Rust in the future
\ No newline at end of file
diff --git a/src/package-lock.json b/src/package-lock.json
new file mode 100644
index 0000000..48e341a
--- /dev/null
+++ b/src/package-lock.json
@@ -0,0 +1,3 @@
+{
+ "lockfileVersion": 1
+}
diff --git a/src/readme.txt b/src/readme.txt
new file mode 100644
index 0000000..a9a6729
--- /dev/null
+++ b/src/readme.txt
@@ -0,0 +1,34 @@
+#目录说明
+## aios
+系统的主要内核部分
+### agent
+智能体相关目录（包括agent和workflow），也是aios的关键抽象
+
+### frame
+aios的核心框架
+
+### proto
+非系统内部的，承诺长期兼容的协议定义
+
+### enviroment
+enviroment定义
+其它内置的基础环境实现，包含其它非LLM AI能力的function
+
+### knowledge
+知识库相关实现
+
+### storage
+传统的存储组件
+
+### net
+基础网络库（主要是NDN，NON网络）的基础组件
+
+## node_daemon
+运行在host_os上，通过传统os概念控制aios的各个基础组件的启动。可以看成是aios的bios
+aios是一个network os, 因此这个组件里还包含了最为基础的仓库实现，以支持各个组件的在线发布\安装\更新
+
+
+## component
+可以按需加载的build-in组件
+## service
+内置的service
\ No newline at end of file
diff --git a/src/requirements.txt b/src/requirements.txt
index b83e564..656343e 100644
--- a/src/requirements.txt
+++ b/src/requirements.txt
@@ -19,7 +19,7 @@ click>=8.1.7
colorama>=0.4.6
coloredlogs>=15.0.1
decorator>=4.4.2
-fastapi>=0.99.1
+fastapi
filelock>=3.12.3
flatbuffers>=23.5.26
frozenlist>=1.4.0
@@ -47,7 +47,7 @@ mpmath>=1.3.0
multidict>=6.0.4
numpy>=1.25.2
onnxruntime>=1.15.1
-openai>=0.28.0
+
overrides>=7.4.0
packaging>=23.1
pandas>=2.1.0
@@ -59,7 +59,7 @@ proto-plus>=1.22.3
pulsar-client>=3.3.0
pyasn1>=0.5.0
pyasn1-modules>=0.3.0
-pydantic>=1.10.12
+pydantic
PyPika>=0.48.9
pyreadline3>=3.4.1
python-dateutil>=2.8.2
@@ -97,7 +97,6 @@ mpmath==1.3.0
multidict==6.0.4
numpy==1.25.2
onnxruntime==1.15.1
-openai==0.28.0
overrides==7.4.0
packaging==23.1
pandas==2.1.0
@@ -137,4 +136,24 @@ python-telegram-bot
pydub
stability_sdk
sentence-transformers==2.2.2
-tiktoken
\ No newline at end of file
+tiktoken
+markdown
+PyPDF2
+srt
+webvtt-py
+openai
+docker
+generic_escape
+duckduckgo-search
+SQLAlchemy
+mysqlclient
+psycopg2-binary
+pyodbc
+oracledb
+html2text
+docx2txt
+opencv-python
+discord.py
+slack_bolt
+wget
+moviepy
diff --git a/src/service/aios_shell/aios_shell.py b/src/service/aios_shell/aios_shell.py
index f015776..2c26929 100644
--- a/src/service/aios_shell/aios_shell.py
+++ b/src/service/aios_shell/aios_shell.py
@@ -11,7 +11,6 @@ from logging.handlers import RotatingFileHandler
from typing import Any, Optional, TypeVar, Tuple, Sequence
import argparse
-
from prompt_toolkit import HTML, PromptSession, prompt,print_formatted_text
from prompt_toolkit.formatted_text import FormattedText
from prompt_toolkit.selection import SelectionState
@@ -20,18 +19,37 @@ from prompt_toolkit.auto_suggest import AutoSuggestFromHistory
from prompt_toolkit.completion import WordCompleter
from prompt_toolkit.styles import Style
+
directory = os.path.dirname(__file__)
sys.path.append(directory + '/../../')
-
+# import os
+# os.environ['HTTP_PROXY'] = '127.0.0.1:10809'
+# os.environ['HTTPS_PROXY'] = '127.0.0.1:10809'
import proxy
-from aios_kernel import *
-
+from aios import *
+import local_compute_node_builder
+from component.llama_node.local_llama_compute_node import LocalLlama_ComputeNode
sys.path.append(directory + '/../../component/')
+
+from google_node import *
+from llama_node import *
+from openai_node import *
+from sd_node import *
+from st_node import *
+
from agent_manager import AgentManager
from workflow_manager import WorkflowManager
+from knowledge_manager import KnowledgePipelineManager
+from tg_tunnel import TelegramTunnel
+from email_tunnel import EmailTunnel
+from discord_tunnel import DiscordTunnel
+from slack_tunnel import SlackTunnel
+from common_environment import LocalKnowledgeBase, FilesystemEnvironment, ShellEnvironment, ScanLocalDocument, ParseLocalDocument
+
+from compute_node_config import *
logger = logging.getLogger(__name__)
@@ -43,7 +61,6 @@ shell_style = Style.from_dict({
'error': '#8F0000 bold'
})
-
class AIOS_Shell:
def __init__(self,username:str) -> None:
self.username = username
@@ -59,8 +76,9 @@ class AIOS_Shell:
user_config = AIStorage.get_instance().get_user_config()
user_config.add_user_config("username","username is your full name when using AIOS",False,None)
- user_config.add_user_config("telegram","Your telgram username",False,None)
- user_config.add_user_config("email","Your email",False,None)
+ user_config.add_user_config("user_telegram","Your telgram username",False,None)
+ user_config.add_user_config("user_email","Your email",False,None)
+ user_config.add_user_config("user_notes","Introduce yourself to your Agent!",False,None)
user_config.add_user_config("feature.llama","enable Local-llama feature",True,"False")
user_config.add_user_config("feature.aigc","enable AIGC feature",True,"False")
@@ -71,13 +89,15 @@ class AIOS_Shell:
user_config.add_user_config("shell.current","last opened target and topic",True,"default@Jarvis")
proxy.declare_user_config()
- google_text_to_speech = GoogleTextToSpeechNode.get_instance()
- google_text_to_speech.declare_user_config()
+ # google_text_to_speech = GoogleTextToSpeechNode.get_instance()
+ # google_text_to_speech.declare_user_config()
Local_Stability_ComputeNode.declare_user_config()
#Stability_ComputeNode.declare_user_config()
+ def init_global_action_lib(self):
+ AgentMemory.register_actions()
async def _handle_no_target_msg(self,bus:AIBus,target_id:str) -> bool:
@@ -91,6 +111,11 @@ class AIOS_Shell:
bus.register_message_handler(target_id,a_workflow._process_msg)
return True
+ a_contact = ContactManager.get_instance().find_contact_by_name(target_id)
+ if a_contact is not None:
+ bus.register_message_handler(target_id,a_contact._process_msg)
+ return True
+
return False
async def is_agent(self,target_id:str) -> bool:
@@ -102,35 +127,41 @@ class AIOS_Shell:
async def initial(self) -> bool:
cm = ContactManager.get_instance()
- owenr = cm.find_contact_by_name(self.username)
- if owenr is None:
- owenr = Contact(self.username)
- owenr.added_by = self.username
- owenr.is_family_member = True
- owenr.email = AIStorage.get_instance().get_user_config().get_value("email")
- owenr.telegram = AIStorage.get_instance().get_user_config().get_value("telegram")
+ owner = cm.find_contact_by_name(self.username)
+ if owner is None:
+ owner = Contact(self.username)
+ owner.added_by = self.username
+ owner.relationship = "Principal"
+ owner.email = AIStorage.get_instance().get_user_config().get_value("user_email")
+ owner.telegram = AIStorage.get_instance().get_user_config().get_value("user_telegram")
+ owner.notes = AIStorage.get_instance().get_user_config().get_value("user_notes")
- cm.add_family_member(self.username,owenr)
+ cm.add_contact(self.username,owner)
- knowledge_env = KnowledgeEnvironment("knowledge")
- Environment.set_env_by_id("knowledge",knowledge_env)
+ # cal_env = CalenderEnvironment("calender")
+ # await cal_env.start()
+ # Environment.set_env_by_id("calender",cal_env)
- cal_env = CalenderEnvironment("calender")
- await cal_env.start()
- Environment.set_env_by_id("calender",cal_env)
+ # workspace_env = ShellEnvironment("bash")
+ # Environment.set_env_by_id("bash",workspace_env)
- workspace_env = WorkspaceEnvironment("bash")
- Environment.set_env_by_id("bash",workspace_env)
+ # paint_env = PaintEnvironment("paint")
+ # Environment.set_env_by_id("paint",paint_env)
- paint_env = PaintEnvironment("paint")
- Environment.set_env_by_id("paint",paint_env)
+ #AgentManager.get_instance().register_environment("bash", ShellEnvironment)
+ #AgentManager.get_instance().register_environment("fs", FilesystemEnvironment)
+ #AgentManager.get_instance().register_environment("knowledge", LocalKnowledgeBase)
+ AgentWorkspace.register_ai_functions()
+ AgentMemory.register_ai_functions()
+ ShellEnvironment.register_ai_functions()
+
if await AgentManager.get_instance().initial() is not True:
logger.error("agent manager initial failed!")
return False
if await WorkflowManager.get_instance().initial() is not True:
- logger.error("workflow manager initial failed!")
- return False
+ logger.error("workflow manager initial failed!")
+ return False
open_ai_node = OpenAI_ComputeNode.get_instance()
if await open_ai_node.initial() is not True:
@@ -138,6 +169,19 @@ class AIOS_Shell:
return False
ComputeKernel.get_instance().add_compute_node(open_ai_node)
+ whisper_node = WhisperComputeNode.get_instance()
+ ComputeKernel.get_instance().add_compute_node(whisper_node);
+
+ openai_tts_node = OpenAITTSComputeNode.get_instance()
+ ComputeKernel.get_instance().add_compute_node(openai_tts_node)
+
+ dall_e_node = DallEComputeNode.get_instance()
+ if await dall_e_node.initial() is not True:
+ logger.error("dall-e node initial failed!")
+ else:
+ await dall_e_node.start()
+ ComputeKernel.get_instance().add_compute_node(dall_e_node)
+
llama_nodes = ComputeNodeConfig.get_instance().initial()
for llama_node in llama_nodes:
llama_node.start()
@@ -153,43 +197,52 @@ class AIOS_Shell:
await AIStorage.get_instance().set_feature_init_result("llama",False)
- if await AIStorage.get_instance().is_feature_enable("aigc"):
- try:
- google_text_to_speech_node = GoogleTextToSpeechNode.get_instance()
- google_text_to_speech_node.init()
- ComputeKernel.get_instance().add_compute_node(google_text_to_speech_node)
- except Exception as e:
- logger.error(f"google text to speech node initial failed! {e}")
- await AIStorage.get_instance.set_feature_init_result("aigc",False)
+ # if await AIStorage.get_instance().is_feature_enable("aigc"):
+ # try:
+ # google_text_to_speech_node = GoogleTextToSpeechNode.get_instance()
+ # google_text_to_speech_node.init()
+ # ComputeKernel.get_instance().add_compute_node(google_text_to_speech_node)
+ # except Exception as e:
+ # logger.error(f"google text to speech node initial failed! {e}")
+ # await AIStorage.get_instance.set_feature_init_result("aigc",False)
# stability_api_node = Stability_ComputeNode()
# if await stability_api_node.initial() is not True:
# logger.error("stability api node initial failed!")
# ComputeKernel.get_instance().add_compute_node(stability_api_node)
-
-
+
+
local_st_text_compute_node = LocalSentenceTransformer_Text_ComputeNode()
if local_st_text_compute_node.initial() is not True:
logger.error("local sentence transformer text embedding node initial failed!")
else:
ComputeKernel.get_instance().add_compute_node(local_st_text_compute_node)
-
+
local_st_image_compute_node = LocalSentenceTransformer_Image_ComputeNode()
if local_st_image_compute_node.initial() is not True:
logger.error("local sentence transformer image embedding node initial failed!")
else:
ComputeKernel.get_instance().add_compute_node(local_st_image_compute_node)
-
+
await ComputeKernel.get_instance().start()
AIBus().get_default_bus().register_unhandle_message_handler(self._handle_no_target_msg)
- AIBus().get_default_bus().register_message_handler(self.username,self._user_process_msg)
- KnowledgePipline.get_instance().initial()
+ #AIBus().get_default_bus().register_message_handler(self.username,self._user_process_msg)
+
+
+ pipelines = KnowledgePipelineManager.initial(os.path.join(AIStorage().get_instance().get_myai_dir(), "knowledge/pipelines"))
+ pipelines.register_input("scan_local", ScanLocalDocument)
+ pipelines.register_parser("parse_local", ParseLocalDocument)
+ pipelines.load_dir(os.path.join(AIStorage().get_instance().get_system_app_dir(), "knowledge_pipelines"))
+ pipelines.load_dir(os.path.join(AIStorage().get_instance().get_myai_dir(), "knowledge_pipelines"))
+ asyncio.create_task(pipelines.run())
TelegramTunnel.register_to_loader()
EmailTunnel.register_to_loader()
+ DiscordTunnel.register_to_loader()
+ SlackTunnel.register_to_loader()
user_data_dir = str(AIStorage.get_instance().get_myai_dir())
tunnels_config_path = os.path.abspath(f"{user_data_dir}/etc/tunnels.cfg.toml")
tunnel_config = None
@@ -198,18 +251,21 @@ class AIOS_Shell:
if tunnel_config is not None:
await AgentTunnel.load_all_tunnels_from_config(tunnel_config)
except Exception as e:
- logger.warning(f"load tunnels config from {tunnels_config_path} failed!")
+ logger.warning(f"load tunnels config from {tunnels_config_path} failed! {e}")
return True
def get_version(self) -> str:
- return "0.5.1"
+ return "0.5.2"
+
+ async def send_msg(self,msg:str,target_id:str,topic:str,sender:str = None, msg_mime:str=None) -> str:
+ if sender == self.username:
+ AIBus().get_default_bus().register_message_handler(self.username,self._user_process_msg)
- async def send_msg(self,msg:str,target_id:str,topic:str,sender:str = None) -> str:
agent_msg = AgentMsg()
- agent_msg.set(sender,target_id,msg)
+ agent_msg.set(sender,target_id,msg,body_mime=msg_mime)
agent_msg.topic = topic
resp = await AIBus.get_default_bus().send_message(agent_msg)
if resp is not None:
@@ -222,7 +278,7 @@ class AIOS_Shell:
async def _user_process_msg(self,msg:AgentMsg) -> AgentMsg:
pass
-
+
async def get_tunnel_config_from_input(self,tunnel_target,tunnel_type):
tunnel_config = {}
@@ -233,10 +289,22 @@ class AIOS_Shell:
match tunnel_type:
case "telegram":
tunnel_config["type"] = "TelegramTunnel"
- input_table["token"] = UserConfigItem("telegram bot token")
+ input_table["token"] = UserConfigItem("telegram bot token\n You can get it from https://t.me/BotFather ,read https://core.telegram.org/bots#how-do-i-create-a-bot for more details")
input_table["allow"] = UserConfigItem("allow group (default is member,you can choose contact or guest)")
case "email":
tunnel_config["type"] = "EmailTunnel"
+ input_table["email"] = UserConfigItem("email address agent will use \n")
+ input_table["imap"] = UserConfigItem("imap server address,like hostname:port")
+ input_table["smtp"] = UserConfigItem("smtp server address,like hostname:port")
+ input_table["user"] = UserConfigItem("mail server login user name")
+ input_table["password"] = UserConfigItem("main server login password")
+ case "discord":
+ tunnel_config["type"] = "DiscordTunnel"
+ input_table["token"] = UserConfigItem("discord bot token\n You can get it from https://discord.com/developers/applications ,read https://discordpy.readthedocs.io/en/stable/discord.html for more details")
+ case "slack":
+ tunnel_config["type"] = "SlackTunnel"
+ input_table["token"] = UserConfigItem("slack bot token\n You can get it from https://api.slack.com/apps")
+ input_table["app_token"] = UserConfigItem("slack app token\n You can get it from https://api.slack.com/apps")
case _:
error_text = FormattedText([("class:error", f"tunnel type {tunnel_type}not support!")])
print_formatted_text(error_text,style=shell_style)
@@ -256,7 +324,7 @@ class AIOS_Shell:
async def append_tunnel_config(self,tunnel_config):
user_data_dir = AIStorage.get_instance().get_myai_dir()
tunnels_config_path = os.path.abspath(f"{user_data_dir}/etc/tunnels.cfg.toml")
- all_tunnels = None
+ all_tunnels = None
try:
all_tunnels = toml.load(tunnels_config_path)
except Exception as e:
@@ -309,68 +377,43 @@ class AIOS_Shell:
if contact_telegram is None:
return None
contact.telegram = contact_telegram
-
+
contact_email = await try_get_input(f"Input {contact_name}'s email:")
if contact_email is None:
return None
contact.email = contact_email
-
+
contact_phone = await try_get_input(f"Input {contact_name}'s phone (optional):")
if contact_phone is not None:
contact.phone = contact_phone
contact_note = await try_get_input(f"Input {contact_name}'s note (optional):")
if contact_note is not None:
- contact.note = contact_note
-
+ contact.notes = contact_note
+
contact.added_by = self.username
if is_update:
cm.set_contact(contact_name,contact)
else:
cm.add_contact(contact_name,contact)
-
+
async def handle_knowledge_commands(self, args):
- show_text = FormattedText([("class:title", "sub command not support!\n"
- "/knowledge add email | dir\n"
- "/knowledge journal [$topn]\n"
+ show_text = FormattedText([("class:title", "sub command not support!\n"
+ "/knowledge pipelines\n"
+ "/knowledge journal $pipeline [$topn]\n"
"/knowledge query $object_id\n")])
if len(args) < 1:
return show_text
sub_cmd = args[0]
- if sub_cmd == "add":
- if len(args) < 2:
- return show_text
- if args[1] == "email":
- config = dict()
- for key, item in KnowledgeEmailSource.user_config_items():
- user_input = await try_get_input(f"{key} : {item}")
- if user_input is None:
- return show_text
- config[key] = user_input
- error = KnowledgePipline.get_instance().add_email_source(KnowledgeEmailSource(config))
- if error is not None:
- return FormattedText([("class:title", f"/knowledge add email failed {error}\n")])
- else:
- KnowledgePipline.get_instance().save_cosnfig()
- if args[1] == "dir":
- config = dict()
- for key, item in KnowledgeDirSource.user_config_items():
- user_input = await try_get_input(f"{key} : {item}")
- if user_input is None:
- return show_text
- config[key] = user_input
- error = KnowledgePipline.get_instance().add_dir_source(KnowledgeDirSource(config))
- if error is not None:
- return FormattedText([("class:title", f"/knowledge add dir failed {error}\n")])
- else:
- KnowledgePipline.get_instance().save_config()
- else:
- return show_text
+ if sub_cmd == "pipelines":
+ pipelines = KnowledgePipelineManager.get_instance().get_pipelines()
+ print_formatted_text("\r\n".join(pipeline.get_name() for pipeline in pipelines))
if sub_cmd == "journal":
try:
- topn = 10 if len(args) == 1 else int(args[1])
- journals = [str(journal) for journal in KnowledgePipline.get_instance().get_latest_journals(topn)]
- print_formatted_text("\r\n".join(journals))
+ name = args[1]
+ topn = 10 if len(args) == 2 else int(args[2])
+ journals = [str(journal) for journal in KnowledgePipelineManager.get_instance().get_pipeline(name).get_journal().latest_journals(topn)]
+ print_formatted_text("\r\n".join(str(journal) for journal in journals))
except ValueError:
return FormattedText([("class:title", f"/knowledge journal failed: {args[1]} is not a valid integer.\n")])
if sub_cmd == "query":
@@ -381,48 +424,41 @@ class AIOS_Shell:
if object_id.get_object_type() == ObjectType.Image:
from PIL import Image
import io
- image = KnowledgeBase().load_object(object_id)
- image_data = KnowledgeBase().bytes_from_object(image)
+ image = KnowledgeStore().load_object(object_id)
+ image_data = KnowledgeStore().bytes_from_object(image)
image = Image.open(io.BytesIO(image_data))
image.show()
async def handle_node_commands(self, args):
show_text = FormattedText([("class:title", "sub command not support!\n"
- "/node add llama $model_name $url\n"
- "/node rm llama $model_name $url\n"
+ "/node add $model_name $url\n"
+ "/node create\n"
+ "/node rm $model_name $url\n"
"/node list\n")])
if len(args) < 1:
return show_text
sub_cmd = args[0]
- if sub_cmd == "add":
- if len(args) < 2:
- return show_text
- if args[1] == "llama":
- if len(args) < 4:
- return show_text
-
- model_name = args[2]
- url = args[3]
- ComputeNodeConfig.get_instance().add_node("llama", url, model_name)
- ComputeNodeConfig.get_instance().save()
- node = LocalLlama_ComputeNode(url, model_name)
- node.start()
- ComputeKernel.get_instance().add_compute_node(node)
- else:
+ if sub_cmd == "create":
+ await local_compute_node_builder.build(session, shell_style)
+ elif sub_cmd == "add":
+ if len(args) < 3:
return show_text
+
+ model_name = args[1]
+ url = args[2]
+ ComputeNodeConfig.get_instance().add_node("llama", url, model_name)
+ ComputeNodeConfig.get_instance().save()
+ node = LocalLlama_ComputeNode(url, model_name)
+ node.start()
+ ComputeKernel.get_instance().add_compute_node(node)
elif sub_cmd == "rm":
- if len(args) < 2:
- return show_text
- if args[1] == "llama":
- if len(args) < 4:
- return show_text
-
- model_name = args[2]
- url = args[3]
- ComputeNodeConfig.get_instance().remove_node("llama", url, model_name)
- ComputeNodeConfig.get_instance().save()
- else:
+ if len(args) < 3:
return show_text
+
+ model_name = args[1]
+ url = args[2]
+ ComputeNodeConfig.get_instance().remove_node("llama", url, model_name)
+ ComputeNodeConfig.get_instance().save()
elif sub_cmd == "list":
print_formatted_text(ComputeNodeConfig.get_instance().list())
@@ -430,13 +466,69 @@ class AIOS_Shell:
match func_name:
case 'send':
show_text = FormattedText([("class:error", f'send args error,/send Tracy "Hello! It is a good day!" default')])
+ sender = None
if len(args) == 3:
target_id = args[0]
msg_content = args[1]
topic = args[2]
- resp = await self.send_msg(msg_content,target_id,topic,self.username)
- show_text = FormattedText([("class:title", f"{self.current_topic}@{self.current_target} >>> "),
- ("class:content", resp)])
+ sender = self.username
+ elif len(args) == 4:
+ target_id = args[0]
+ msg_content = args[1]
+ topic = args[2]
+ sender = args[3]
+
+ resp = await self.send_msg(msg_content,target_id,topic,sender)
+ show_text = FormattedText([("class:title", f"{self.current_topic}@{self.current_target} >>> "),
+ ("class:content", resp)])
+ return show_text
+ case 'send_img':
+ sender = None
+ if len(args) == 4:
+ target_id = args[0]
+ msg_content = args[1]
+ image_path = args[2]
+ topic = args[3]
+ sender = self.username
+ elif len(args) == 5:
+ target_id = args[0]
+ msg_content = args[1]
+ image_path = args[2]
+ topic = args[3]
+ sender = args[4]
+
+ ext = os.path.splitext(image_path)[1][1:]
+ resp = await self.send_msg(AgentMsg.create_image_body([image_path], msg_content),
+ target_id,
+ topic,
+ sender,
+ f"image/{ext}")
+ show_text = FormattedText([("class:title", f"{self.current_topic}@{self.current_target} >>> "),
+ ("class:content", resp)])
+ return show_text
+ case 'send_video':
+ sender = None
+ if len(args) == 4:
+ target_id = args[0]
+ msg_content = args[1]
+ video_path = args[2]
+ topic = args[3]
+ sender = self.username
+ elif len(args) == 5:
+ target_id = args[0]
+ msg_content = args[1]
+ video_path = args[2]
+ topic = args[3]
+ sender = args[4]
+
+ ext = os.path.splitext(video_path)[1][1:]
+ resp = await self.send_msg(AgentMsg.create_video_body(video_path, msg_content),
+ target_id,
+ topic,
+ sender,
+ f"video/{ext}")
+ show_text = FormattedText([("class:title", f"{self.current_topic}@{self.current_target} >>> "),
+ ("class:content", resp)])
return show_text
case 'set_config':
show_text = FormattedText([("class:error", f"set config args error,/set_config $config_item! ")])
@@ -463,7 +555,7 @@ class AIOS_Shell:
tunnel_type = "telegram"
else:
tunnel_type = args[1]
-
+ tunnel_type = tunnel_type.lower()
tunnel_config = await self.get_tunnel_config_from_input(tunnel_target,tunnel_type)
if tunnel_config:
if await AgentTunnel.load_tunnel_from_config(tunnel_config):
@@ -482,18 +574,34 @@ class AIOS_Shell:
the_agent = await AgentManager.get_instance().get(target_id)
if the_agent is not None:
await the_agent._do_think()
+ case 'wakeup':
+ if len(args) >= 1:
+ target_id = args[0]
+ the_agent = await AgentManager.get_instance().get(target_id)
+ if the_agent is not None:
+ the_agent.wake_up()
case 'open':
if len(args) >= 1:
target_id = args[0]
else:
show_text = FormattedText([("class:error", "/open Need Target Agent/Workflow ID! like /open Jarvis default")])
return show_text
-
+
if len(args) >= 2:
topic = args[1]
else:
topic = "default"
+ target_exist = False
+ if await AgentManager.get_instance().is_exist(target_id):
+ target_exist = True
+ # if await WorkflowManager.get_instance().is_exist(target_id):
+ # target_exist = True
+
+ if target_exist is False:
+ show_text = FormattedText([("class:error", f"Target {target_id} not exist!")])
+ return show_text
+
self.current_target = target_id
self.current_topic = topic
show_text = FormattedText([("class:title", f"current session switch to {topic}@{target_id}")])
@@ -520,11 +628,11 @@ class AIOS_Shell:
else:
show_text = FormattedText([("class:error", "/disable Need Feature Name! like /disable llama")])
return show_text
-
+
if not await AIStorage.get_instance().is_feature_enable(feature):
show_text = FormattedText([("class:title", f"Feature {feature} already disabled!")])
return show_text
-
+
await AIStorage.get_instance().disable_feature(feature)
show_text = FormattedText([("class:title", f"Feature {feature} disabled!")])
return show_text
@@ -546,8 +654,8 @@ class AIOS_Shell:
db_path = ""
if await self.is_agent(self.current_target):
db_path = AgentManager.get_instance().db_path
- else:
- db_path = WorkflowManager.get_instance().db_file
+ # else:
+ # db_path = WorkflowManager.get_instance().db_file
chatsession:AIChatSession = AIChatSession.get_session(self.current_target,f"{self.username}#{self.current_topic}",db_path,False)
if chatsession is not None:
msgs = chatsession.read_history(num,offset)
@@ -608,9 +716,9 @@ async def get_user_config_from_input(check_result:dict) -> bool:
continue
else:
True
-
- if len(user_input) > 0:
- AIStorage.get_instance().get_user_config().set_value(key,user_input)
+ if user_input:
+ if len(user_input) > 0:
+ AIStorage.get_instance().get_user_config().set_value(key,user_input)
await AIStorage.get_instance().get_user_config().save_to_user_config()
return True
@@ -661,9 +769,8 @@ def print_welcome_screen():
\033[1;94m\tGive your Agent a Telegram account :\033[0m /connect $agent_name
\033[1;94m\tAdd personal files to the AI Knowledge Base. \033[0m
\t\t1) Copy your file to ~/myai/data
-\t\t2) /knowlege add dir
\033[1;94m\tSearch your knowledge base :\033[0m /open Mia
-\033[1;94m\tCheck the progress of AI reading personal data :\033[0m /knowledge journal
+\033[1;94m\tCheck the progress of AI reading personal data :\033[0m /knowledge $pipeline journal
\033[1;94m\tQuery object with ID in knowledge base :\033[0m /knowledge query $object_id
\033[1;94m\tOpen AI Bash (For Developer Only):\033[0m /open ai_bash
\033[1;94m\tEnable AIGC Feature :\033[0m /enable aigc
@@ -701,8 +808,8 @@ async def main():
logging.basicConfig(handlers=[handler],
level=logging.INFO,
- format='%(asctime)s - %(name)s - %(levelname)s - %(message)s')
-
+ format='%(asctime)s - %(name)s - %(levelname)s - %(message)s')
+
is_daemon = False
logger.info(f"Check Host OS :{os.name}")
if os.name != 'nt':
@@ -723,7 +830,7 @@ async def main():
shell.username = AIStorage.get_instance().get_user_config().get_value("username")
init_result = await shell.initial()
proxy.apply_storage()
-
+
if init_result is False:
if is_daemon:
logger.error("aios shell initial failed!")
@@ -738,18 +845,21 @@ async def main():
return await main_daemon_loop(shell)
completer = WordCompleter(['/send $target $msg $topic',
+ '/send_img $target $msg $img_path $topic',
+ '/send_video $target &msg &video_path $topic',
'/open $target $topic',
'/history $num $offset',
'/connect $target',
'/contact $name',
- '/knowledge add email | dir',
- '/knowledge journal [$topn]',
+ '/knowledge pipelines',
+ '/knowledge journal $pipeline [$topn]',
'/knowledge query $object_id',
'/set_config $key',
'/enable $feature',
'/disable $feature',
- '/node add llama $model_name $url',
- '/node rm llama $model_name $url',
+ '/node add $model_name $url',
+ '/node create',
+ '/node rm $model_name $url',
'/node list',
'/show',
'/exit',
diff --git a/src/aios_kernel/compute_node_config.py b/src/service/aios_shell/compute_node_config.py
similarity index 92%
rename from src/aios_kernel/compute_node_config.py
rename to src/service/aios_shell/compute_node_config.py
index c3b2c96..6528d74 100644
--- a/src/aios_kernel/compute_node_config.py
+++ b/src/service/aios_shell/compute_node_config.py
@@ -1,4 +1,5 @@
"""
+compute node config
Configuration for nodes:
```
@@ -14,12 +15,18 @@ Configuration for nodes:
"""
import logging
from typing import List
+import sys
import os
import toml
-from .local_llama_compute_node import LocalLlama_ComputeNode
-from .storage import AIStorage
+
+from aios import AIStorage
+directory = os.path.dirname(__file__)
+sys.path.append(directory + '/../../component/')
+
+from llama_node import LocalLlama_ComputeNode
+
# define singleton class knowledge pipline
class ComputeNodeConfig:
diff --git a/src/service/aios_shell/local_compute_node_builder/__init__.py b/src/service/aios_shell/local_compute_node_builder/__init__.py
new file mode 100644
index 0000000..a04f822
--- /dev/null
+++ b/src/service/aios_shell/local_compute_node_builder/__init__.py
@@ -0,0 +1,38 @@
+import os
+from prompt_toolkit import HTML, PromptSession, print_formatted_text
+from prompt_toolkit.styles import Style
+from aios.storage.storage import AIStorage
+from service.aios_shell.local_compute_node_builder.local_llama_node_builder import LocalLlamaNodeBuilder
+from .local_compute_node_builder import BuilderState
+
+async def build(prompt_session: PromptSession, shell_style: Style) -> str or None:
+ # model_type = await prompt_session.prompt_async(f"Please select the node server type (default: llama.cpp):", style = shell_style)
+
+ model_type = 'llama.cpp'
+
+ download_dir = AIStorage.get_instance().get_download_dir()
+ if not os.path.exists(download_dir):
+ os.mkdir(download_dir)
+
+ state = BuilderState(prompt_session, shell_style)
+
+ match model_type:
+ case 'llama.cpp':
+ builder = LocalLlamaNodeBuilder(state)
+
+ while True:
+ param = builder.next_parameter()
+ if param is None:
+ return None
+
+ if state.last_result_prompt or param.desc:
+ print_formatted_text(f"{state.last_result_prompt}{param.desc}", style = state.shell_style)
+ value = await state.prompt_session.prompt_async(f"{param.prompt}:", style = state.shell_style)
+ if value:
+ value = value.strip()
+
+ state.params[param.name] = value
+ url = await param.applier.apply(state, param.name, value)
+
+ if url is not None:
+ return url
\ No newline at end of file
diff --git a/src/service/aios_shell/local_compute_node_builder/local_compute_node_builder.py b/src/service/aios_shell/local_compute_node_builder/local_compute_node_builder.py
new file mode 100644
index 0000000..33e1e36
--- /dev/null
+++ b/src/service/aios_shell/local_compute_node_builder/local_compute_node_builder.py
@@ -0,0 +1,40 @@
+from abc import abstractmethod
+
+from prompt_toolkit import PromptSession
+from prompt_toolkit.styles import Style
+
+class BuilderState:
+ def __init__(self, prompt_session: PromptSession, shell_style: Style):
+ self.prompt_session = prompt_session
+ self.shell_style = shell_style
+ self.next_step = 0
+ self.last_result_prompt = ""
+ self.params = {}
+
+# class ApplyResult:
+# def __init__(self, next_step: any, url: str or None = None, result_prompt: str or None = None) -> None:
+# self.next_step = next_step
+# self.url = url
+# self.result_prompt = result_prompt
+
+
+class ParameterApplier:
+ @abstractmethod
+ async def apply(self, state: BuilderState, name: str, value: str or None = None) -> str or None:
+ pass
+
+class BuildParameter:
+ def __init__(self, name: str, applier: ParameterApplier, prompt: str or None = None, desc: str or None = None, default_value: str or None = None):
+ self.name = name
+ self.prompt = prompt
+ self.desc = desc
+ self.default_value = default_value
+ self.applier = applier
+
+class LocalComputeNodeBuilder:
+ def __init__(self, state: BuilderState) -> None:
+ self.state = state
+
+ @abstractmethod
+ def next_parameter(self) -> BuildParameter or None:
+ pass
\ No newline at end of file
diff --git a/src/service/aios_shell/local_compute_node_builder/local_llama_node_builder.py b/src/service/aios_shell/local_compute_node_builder/local_llama_node_builder.py
new file mode 100644
index 0000000..163d6de
--- /dev/null
+++ b/src/service/aios_shell/local_compute_node_builder/local_llama_node_builder.py
@@ -0,0 +1,254 @@
+import os
+import random
+import subprocess
+import requests
+
+from prompt_toolkit import print_formatted_text
+from prompt_toolkit.shortcuts import ProgressBar
+from prompt_toolkit.formatted_text import FormattedText
+
+from aios.storage.storage import AIStorage
+from aios import ComputeKernel
+from component.llama_node.local_llama_compute_node import LocalLlama_ComputeNode
+from compute_node_config import ComputeNodeConfig
+from .local_compute_node_builder import BuildParameter, BuilderState, LocalComputeNodeBuilder, ParameterApplier
+
+class BuildParameterModelPath(ParameterApplier):
+ async def apply(self, state: BuilderState, name: str, value: str or None = None) -> str or None:
+ if value:
+ if os.path.exists(value):
+ state.next_step += 2
+ else:
+ print_formatted_text(FormattedText([("class:error", f"Model not exist at {value}")]), style = state.shell_style)
+ else:
+ state.next_step += 1
+
+
+class BuildParameterModelUrl(ParameterApplier):
+ async def apply(self, state: BuilderState, name: str, value: str or None = None) -> str or None:
+ if value is None:
+ value = "1"
+
+ url = value
+ recommend = _recommend_model_urls.get(value)
+ if recommend:
+ url = recommend["url"]
+
+ save_path = f"{AIStorage.get_instance().get_download_dir()}/{url.split('/').pop()}"
+
+ print_formatted_text(FormattedText([("class:prompt", f"Will save the model to {save_path}:\n")]), style = state.shell_style)
+
+ try:
+ # get file size
+ response = requests.head(url)
+ file_size = int(response.headers.get('content-length', 0))
+
+ # start download
+ response = requests.get(url, stream=True)
+
+ if response.status_code == 200:
+ with open(save_path, 'wb') as f, ProgressBar() as pb:
+ for data in pb(response.iter_content(1024), total = (file_size + 1023) // 1024):
+ f.write(data)
+
+ print_formatted_text(FormattedText([("class:prompt", f"Download model success, save at: {save_path}\n")]), style = state.shell_style)
+
+ state.params["model_path"] = save_path
+ state.next_step += 1
+ else:
+ print_formatted_text(FormattedText([("class:error", f"Download model failed, error: {response.status_code}\nYou can retry it or select another one.")]), style = state.shell_style)
+
+ except Exception as e:
+ print_formatted_text(FormattedText([("class:error", f"Download model failed: {e}\nYou can retry it or select another one.")]), style = state.shell_style)
+
+class ParameterNodeNameApplier(ParameterApplier):
+ async def apply(self, state: BuilderState, name: str, value: str or None = None) -> str or None:
+ value = value or os.path.basename(state.params["model_path"])
+ state.params["node_name"] = value
+ state.next_step += 1
+
+class ParameterPortApplier(ParameterApplier):
+ async def apply(self, state: BuilderState, name: str, value: str or None = None) -> str or None:
+ if value is None or value == "0" or value == "":
+ value = str(random.randint(10000, 60000))
+
+ state.params["port"] = value
+ state.next_step += 1
+
+class ParameterNGpuLayersApplier(ParameterApplier):
+ async def apply(self, state: BuilderState, name: str, value: str or None = None) -> str or None:
+ value = value or "83"
+ state.params["n_gpu_layers"] = value
+ state.next_step += 1
+
+class ParameterNCtxApplier(ParameterApplier):
+ async def apply(self, state: BuilderState, name: str, value: str or None = None) -> str or None:
+ value = value or "4096"
+ state.params["n_ctx"] = value
+ state.next_step += 1
+
+class ParameterChatFormatApplier(ParameterApplier):
+ async def apply(self, state: BuilderState, name: str, value: str or None = None) -> str or None:
+ value = value or "llama-2"
+ state.params["chat_format"] = value
+ state.next_step += 1
+
+class ParameterExternParamsApplier(ParameterApplier):
+ async def apply(self, state: BuilderState, name: str, value: str or None = None) -> str or None:
+ extern_params = value
+ docker_image = ""
+ gpu_options = []
+ state.next_step += 1
+
+ if False and state.params["n_gpu_layers"] == "0":
+ docker_image = "ghcr.io/abetlen/llama-cpp-python:latest"
+ else:
+ gpu_options = ["--gpus", "all"]
+ llama_cpp_python_repo_url = "https://github.com/abetlen/llama-cpp-python.git"
+ download_path = AIStorage.get_instance().get_download_dir()
+ llama_cpp_python_path = download_path + "/llama-cpp-python"
+
+ # update the `llama-cpp-python`
+ retry = True
+ while retry:
+ retry = False
+ result = None
+ if os.path.exists(llama_cpp_python_path):
+ result = subprocess.run(['git', 'pull'], cwd = llama_cpp_python_path, stdout = subprocess.PIPE, stderr = subprocess.PIPE, text = True)
+ else:
+ result = subprocess.run(['git', 'clone', llama_cpp_python_repo_url, llama_cpp_python_path], stdout = subprocess.PIPE, stderr = subprocess.PIPE, text = True)
+
+ if result.returncode:
+ print_formatted_text(FormattedText([("class:warn", result.stderr)]), style = state.shell_style)
+ while True:
+ sel = await state.prompt_session.prompt_async(f"Update 'llama-cpp-python' failed, you can press 'r' to retry, or 'c' to continue with the current version.", style = state.shell_style)
+ if sel == 'r':
+ retry = True
+ break
+ elif sel == 'c':
+ break
+ else:
+ pass # Select again
+ else:
+ break
+
+ # build the image
+ docker_image = 'llama-cpp-python-cuda'
+ retry = True
+ while retry:
+ retry = False
+ result = subprocess.run(['docker', 'rmi', docker_image], stdout = subprocess.PIPE, stderr = subprocess.PIPE, text = True)
+ result = subprocess.run(['docker', 'build', '-t', docker_image, f"{llama_cpp_python_path}/docker/cuda_simple/"], stdout = subprocess.PIPE, stderr = subprocess.PIPE, text = True)
+
+ if result.returncode:
+ print_formatted_text(FormattedText([("class:warn", result.stderr)]), style = state.shell_style)
+ while True:
+ sel = await state.prompt_session.prompt_async(f"Build the image failed, you can press 'r' to retry, or 'c' to continue with the current version.", style = state.shell_style)
+ if sel == 'r':
+ retry = True
+ break
+ elif sel == 'c':
+ break
+ else:
+ pass # Select again
+ else:
+ break
+
+ retry = True
+ while retry:
+ retry = False
+ run_options = ['docker', 'run', '-d']
+
+ if gpu_options:
+ run_options.extend(gpu_options)
+
+ run_options.extend([
+ '-p', f"{state.params['port']}:8000",
+ '-v', f"{os.path.dirname(state.params['model_path'])}:/models", '-e', f"MODEL=/models/{os.path.basename(state.params['model_path'])}",
+ docker_image,
+ 'python3', '-m', 'llama_cpp.server',
+ '--n_gpu_layers', state.params["n_gpu_layers"],
+ '--n_ctx', state.params["n_ctx"],
+ '--chat_format', state.params["chat_format"],
+ ])
+
+ if extern_params:
+ run_options.extend(extern_params.split(' '))
+
+ print_formatted_text(FormattedText([("class:prompt", f"Will start service with: {' '.join(run_options)}")]), style = state.shell_style)
+
+ result = subprocess.run(run_options, stdout = subprocess.PIPE, stderr = subprocess.PIPE, text = True)
+
+ if result.returncode:
+ print_formatted_text(FormattedText([("class:warn", result.stderr)]), style = state.shell_style)
+ while True:
+ sel = await state.prompt_session.prompt_async(f"Start the node service failed, you can press 'r' to retry, or 'a' to abort.", style = state.shell_style)
+ if sel == 'r':
+ retry = True
+ break
+ elif sel == 'a':
+ break
+ else:
+ pass # Select again
+ else:
+ local_url = f'http://localhost:{state.params["port"]}'
+ foreign_url = 'http://{your-host-address}:' + state.params["port"]
+ model_name = state.params['node_name']
+
+ ComputeNodeConfig.get_instance().add_node("llama", local_url, model_name)
+ ComputeNodeConfig.get_instance().save()
+ node = LocalLlama_ComputeNode(local_url, model_name)
+ node.start()
+ ComputeKernel.get_instance().add_compute_node(node)
+
+ print_formatted_text(FormattedText([(
+ "class:prompt",
+f"""
+Congratulations! The node ({model_name}) service successed.
+You can access it with follow url:
+{local_url}
+And 'http://{foreign_url}' in other computers.
+Now you can refer it in agents as `llm_model_name={model_name}`
+"""
+ )]), style = state.shell_style)
+ break
+
+_recommend_model_urls = {
+ "1": {
+ "model": "Llama-2-70B-Chat-GGUF",
+ "url": "https://huggingface.co/TheBloke/Llama-2-70B-chat-GGUF/resolve/main/llama-2-70b-chat.Q4_0.gguf"
+ },
+ "2": {
+ "model": "Llama-2-13B-Chat-GGUF",
+ "url": "https://huggingface.co/TheBloke/Llama-2-13B-chat-GGUF/resolve/main/llama-2-13b-chat.Q4_0.gguf"
+ },
+ "3": {
+ "model": "Llama-2-7B-Chat-GGUF",
+ "url": "https://huggingface.co/TheBloke/Llama-2-7B-Chat-GGUF/resolve/main/llama-2-7b-chat.Q4_K_M.gguf"
+ },
+}
+
+_recommend_model_url_table_str = ""
+for i in range(1, 999):
+ id = str(i)
+ info = _recommend_model_urls.get(id)
+ if info:
+ _recommend_model_url_table_str += f"\n\t{id}\t{info['model']}\t{info['url']}"
+ else:
+ break
+
+_params = [
+ BuildParameter("model_path", BuildParameterModelPath(), "Please input the model file path (Press 'Enter' if you need to download it)"),
+ BuildParameter("model_url", BuildParameterModelUrl(), "Please input (default: Llama-2-70B-chat)", f"Now you need input the url to download the model, or you can input the 'ID' in the follow table to select one:\n\tID\tmodel\t\turl{_recommend_model_url_table_str}"),
+ BuildParameter("node_name", ParameterNodeNameApplier(), "Please input name for your node, and you can set it in 'llm_model_name' of 'agent.toml' (default: the name of the model file)"),
+ BuildParameter("port", ParameterPortApplier(), "Please input the port which the node server will listen on (default: random)"),
+ BuildParameter("n_gpu_layers", ParameterNGpuLayersApplier(), "Please input layers offload to GPU (<=83 for Llama, 0 for CPU only, default: 83)"),
+ BuildParameter("n_ctx", ParameterNCtxApplier(), "Please input the content limit (default: 4096)"),
+ BuildParameter("chat_format", ParameterChatFormatApplier(), "Please input the chat format (default: llama-2)"),
+ BuildParameter("extern_params", ParameterExternParamsApplier(), "Please input other parameters refer to 'llama-cpp-python'(https://github.com/abetlen/llama-cpp-python), press 'Enter' to ignore it"),
+]
+
+class LocalLlamaNodeBuilder(LocalComputeNodeBuilder):
+ def next_parameter(self) -> BuildParameter or None:
+ if self.state.next_step < len(_params):
+ return _params[self.state.next_step]
diff --git a/src/service/aios_shell/proxy.py b/src/service/aios_shell/proxy.py
index 2ed8a82..208c1a6 100644
--- a/src/service/aios_shell/proxy.py
+++ b/src/service/aios_shell/proxy.py
@@ -9,7 +9,7 @@ import logging
directory = os.path.dirname(__file__)
sys.path.append(directory + '/../../')
-from aios_kernel import AIStorage
+from aios import AIStorage
logger = logging.getLogger(__name__)
diff --git a/src/service/app_manager/README b/src/service/app_manager/README
deleted file mode 100644
index 30404ce..0000000
--- a/src/service/app_manager/README
+++ /dev/null
@@ -1 +0,0 @@
-TODO
\ No newline at end of file
diff --git a/src/service/email_spider/converter.py b/src/service/email_spider/converter.py
deleted file mode 100644
index 193b220..0000000
--- a/src/service/email_spider/converter.py
+++ /dev/null
@@ -1,24 +0,0 @@
-from aios_kernel.knowledge import KnowledgeBase, EmailObject
-
-# define a email converter class
-
-class EmailConverter:
- # define init method
- def __init__(self, local_dir, knowledge_base: KnowledgeBase) -> None:
- pass
-
- async def run(self):
- # convert the email to knowledge object
- for email_dir in self._next():
- # convert the email to knowledge object
- knowledge_object = self._convert(email_dir)
- # insert the knowledge object to knowledge base
- await self.knowledge_base.insert(knowledge_object)
-
- def _next(self) -> str:
- pass
-
- def _convert(self, email_dir) -> EmailObject:
- pass
-
-
\ No newline at end of file
diff --git a/src/service/email_spider/main.py b/src/service/email_spider/main.py
deleted file mode 100644
index cb75abd..0000000
--- a/src/service/email_spider/main.py
+++ /dev/null
@@ -1,12 +0,0 @@
-import asyncio
-from .spider import EmailSpider, EmailConverter
-
-
-if __name__ == "__main__":
- spider = EmailSpider("smtp.163.com","user","pwd","./email")
- asyncio.run(spider.run())
-
- converter = EmailConverter("./email",KnowledgeBase())
- asyncio.run(converter.run())
-
-
diff --git a/src/service/email_spider/spider.py b/src/service/email_spider/spider.py
deleted file mode 100644
index 0b5a212..0000000
--- a/src/service/email_spider/spider.py
+++ /dev/null
@@ -1,17 +0,0 @@
-# define a email spider class
-
-class EmailSpider:
- def __init__(self, address, account, pwd, local_dir) -> None:
- pass
-
- async def run(self):
- # spide the email from the email server
- for email_link in self._next():
- # save the email to local directory
- self._save(email_link)
-
- def _next(self):
- pass
-
- def _save(self, email_link) -> str:
- pass
\ No newline at end of file
diff --git a/src/service/spider/email_spider.py b/src/service/spider/email_spider.py
deleted file mode 100644
index cd44097..0000000
--- a/src/service/spider/email_spider.py
+++ /dev/null
@@ -1,171 +0,0 @@
-"""
-Capture your email locally, and parse out the pictures in the email body and the pictures, videos and other files in the attachment. Subsequently, it supports vectorized analysis of your personal data and serves as a knowledge base to enable large language model answers. Better results.
-
-An example of a local file is as follows:
-├── data
-│ └── alex0072@gmail.com
-│ └── 5de3e52f3a6b90cabe6cbdd4ae3a5c5b
-│ ├── email.txt
-│ ├── meta.json
-│ ├── image
-│ │ ├── 0648B869@99C03070.DB94B354.jpg
-│ └── body_image
-│ ├── 11044884873.jpg
-│ ├── 282985198265470.gif
-│ └── dd-login-service-min.png
-
-"""
-
-import imaplib
-import os
-import toml
-import logging
-import mailparser
-import hashlib
-import json
-import base64
-from bs4 import BeautifulSoup
-import requests
-
-class EmailSpider:
- def __init__(self):
- # logger config
- self.logger = logging.getLogger('email spider')
- self.logger.setLevel(logging.DEBUG)
- ch = logging.StreamHandler()
- formatter = logging.Formatter('%(asctime)s [%(name)s] [%(levelname)s] %(message)s')
- ch.setFormatter(formatter)
- self.logger.addHandler(ch)
-
- # read config from toml file
- # and read from config config.local.toml if exists (config.local.toml is ignored by git)
- self.config = toml.load('./rootfs/email/config.toml')
- if os.path.exists('./rootfs/email/config.local.toml'):
- self.config = toml.load('./rootfs/email/config.local.toml')
-
- self.client = self.email_client()
-
- def email_client(self) -> imaplib.IMAP4_SSL:
- self.logger.info(f"read email config from {self.config.get('EMAIL_IMAP_SERVER')}")
- client = imaplib.IMAP4_SSL(
- host=self.config.get('EMAIL_IMAP_SERVER'),
- port=self.config.get('EMAIL_IMAP_PORT')
- )
- client.login(self.config.get('EMAIL_ADDRESS'), self.config.get('EMAIL_PASSWORD'))
- return client
-
- def list_box(self):
- _, mailbox_list = self.client.list()
- for mailbox in mailbox_list:
- print(mailbox.decode())
-
- def read_emails(self, folder: str = 'INBOX', imap_keyword: str = "UNSEEN"):
- self.client.select(folder)
- _, data = self.client.uid('search', None, imap_keyword)
-
- # get email uid list
- email_list = data[0].split()
- self.logger.info(f"got {len(email_list)} emails")
- email_list.reverse()
- for uid in email_list:
- if self.check_email_saved(uid):
- self.logger.info(f"email uid {uid} already saved")
- else:
- self.read_and_save_email(uid)
- self.logger.info(f"email uid {uid} saved")
-
- def read_and_save_email(self, uid: str):
- message_parts = "(BODY.PEEK[])"
- _, email_data = self.client.uid('fetch', uid, message_parts)
- mail = mailparser.parse_from_bytes(email_data[0][1])
- self.logger.info(f"got email subject [{mail.subject}]")
- self.save_email(mail)
-
- def get_local_dir_name(self, mail: mailparser.MailParser) -> str:
- dir = f"{self.config.get('LOCAL_DIR')}/{self.config.get('EMAIL_ADDRESS')}"
- name = f"{mail.subject}__{mail.date}"
- name = hashlib.md5(name.encode('utf-8')).hexdigest()
- return f"{dir}/{name}"
-
- def check_email_saved(self, uid: str):
- message_parts = "(BODY[HEADER])"
- _, email_data = self.client.uid('fetch', uid, message_parts)
- mail = mailparser.parse_from_bytes(email_data[0][1])
- self.logger.info(f"[{uid}]check email subject [{mail.subject}]")
- dir = self.get_local_dir_name(mail)
- self.logger.info(f"check email saved {dir}")
- file = f"{dir}/email.txt"
- if os.path.exists(file):
- return False
- return False
-
- # save email attachment(images)
- def save_email_attachment(self, mail: mailparser.MailParser, email_dir: str):
- for attachment in mail.attachments:
- if attachment['mail_content_type'] in ['image/png', 'image/jpeg', 'image/gif']:
- print('current mail have image attachment')
- img_dir = f"{email_dir}/image"
- if not os.path.exists(img_dir):
- os.makedirs(img_dir)
- filename = attachment['filename']
- filefullname = f"{img_dir}/{filename}"
- image_data = attachment['payload']
- try:
- image_data = base64.b64decode(image_data)
- except base64.binascii.Error:
- image_data = image_data.encode()
- with open(filefullname, 'wb') as f:
- f.write(image_data)
- self.logger.info(f"save email image {filename} success")
-
- # save email body images(html content)
- def save_body_images(self, html_content: str, email_dir: str):
- # get all image urls
- soup = BeautifulSoup(html_content, 'html.parser')
- img_tags = soup.find_all('img')
- img_urls = [img['src'] for img in img_tags if 'src' in img.attrs]
- self.logger.info(f'Found {len(img_urls)} images in email body')
-
- if not os.path.exists(email_dir):
- os.makedirs(email_dir)
-
- for img_url in img_urls:
- # keep the original image filename(last of url)
- img_filename = os.path.join(email_dir, img_url.split('/')[-1])
- # download image
- response = requests.get(img_url, stream=True)
- if response.status_code == 200:
- with open(img_filename, 'wb') as img_file:
- for chunk in response.iter_content(1024):
- img_file.write(chunk)
- self.logger.info(f'Downloaded {img_url} to {img_filename}')
- else:
- self.logger.info(f'Failed to download {img_url}')
-
- # save email content to local dir
- def save_email(self, mail: mailparser.MailParser):
- dir = f"{self.config.get('LOCAL_DIR')}/{self.config.get('EMAIL_ADDRESS')}"
- if not os.path.exists(dir):
- os.makedirs(dir)
- email_dir = self.get_local_dir_name(mail)
- self.logger.info(f"save email to {email_dir}")
- if not os.path.exists(email_dir):
- os.makedirs(email_dir)
- with open(f"{email_dir}/email.txt", "w") as f:
- f.write(mail.body)
- with open(f"{email_dir}/meta.json", "w", encoding='utf-8') as f:
- mail_dict = json.loads(mail.mail_json)
- if 'body' in mail_dict:
- del mail_dict['body']
- json.dump(mail_dict, f, ensure_ascii=False, indent=4)
- self.logger.info(f"save email meta info {f.name}")
-
- self.save_email_attachment(mail, email_dir)
- self.save_body_images(mail.body, f"{email_dir}/body_image")
-
-
-if __name__ == "__main__":
- spider = EmailSpider()
- folder = 'INBOX'
- imap_keyword = "ALL"
- spider.read_emails(folder, imap_keyword)
\ No newline at end of file
diff --git a/src/system.cfg.toml b/src/system.cfg.toml
new file mode 100644
index 0000000..d8afcea
--- /dev/null
+++ b/src/system.cfg.toml
@@ -0,0 +1,3 @@
+[LLMAgentMessageProcess]
+type="LLMAgentMessageProcess"
+
diff --git a/test/env_test.py b/test/env_test.py
index 694acbb..5ba2d6b 100644
--- a/test/env_test.py
+++ b/test/env_test.py
@@ -1,11 +1,9 @@
import asyncio
import os
import sys
-
directory = os.path.dirname(__file__)
sys.path.append(directory + '/../src')
-
-from aios_kernel import CalenderEnvironment,WorkflowEnvironment
+from aios_kernel import CalenderEnvironment,WorkflowEnvironment,ComputeKernel,OpenAI_ComputeNode,AIStorage
async def test_buildin_envs():
@@ -22,6 +20,29 @@ async def test_buildin_envs():
await asyncio.sleep(10)
+async def test_image_to_text():
+ # init the compute kernel and add the compute node
+ open_ai_node = OpenAI_ComputeNode.get_instance()
+ if await open_ai_node.initial() is not True:
+ print("openai node initial failed!")
+ return False
+ ComputeKernel.get_instance().add_compute_node(open_ai_node)
+ w_env = WorkflowEnvironment("workflow",os.path.abspath(directory + "/../rootfs/workflow_env.db"))
+ assert w_env.functions['image_2_text'] is not None
+ await ComputeKernel.get_instance().start()
+ fn = w_env.get_ai_function('image_2_text')
+ image_path = os.path.abspath(directory + "/test.png")
+ arguments = {
+ 'image_path': image_path
+ }
+
+ # execute the ai function
+ result = await fn.execute(**arguments)
+ assert result is not ""
+ print(result)
+
+ await asyncio.sleep(10)
+
if __name__ == "__main__":
#test_rstr = "abc is {abc}"
@@ -29,4 +50,5 @@ if __name__ == "__main__":
#new_str = test_rstr.format_map(values)
#print(new_str)
- asyncio.run(test_buildin_envs())
\ No newline at end of file
+ # asyncio.run(test_buildin_envs())
+ asyncio.run(test_image_to_text())
\ No newline at end of file
diff --git a/test/test.png b/test/test.png
new file mode 100644
index 0000000..7f6cb7a
Binary files /dev/null and b/test/test.png differ
diff --git a/test/test_dall_e_node.py b/test/test_dall_e_node.py
new file mode 100644
index 0000000..04c826d
--- /dev/null
+++ b/test/test_dall_e_node.py
@@ -0,0 +1,47 @@
+import os
+import time
+import uuid
+import io
+import asyncio
+import sys
+import logging
+import pytest
+directory = os.path.dirname(__file__)
+sys.path.append(directory + '/../src')
+from aios_kernel.dall_e_compute_node import DallEComputeNode
+from aios_kernel.compute_task import ComputeTaskType, ComputeTask, ComputeTaskState
+
+os.environ["TEXT2IMG_OUTPUT_DIR"] = "./"
+os.environ["openai_api_key"] = ""
+
+@pytest.mark.asyncio
+async def test_dall_e_node(propmt, model):
+ node = DallEComputeNode.get_instance()
+ if await node.initial() is not True:
+ print("node initial failed!")
+ return
+
+ task = ComputeTask()
+ task.task_type = ComputeTaskType.TEXT_2_IMAGE
+ task.create_time = time.time()
+ task.task_id = uuid.uuid4().hex
+ task.params['prompt'] = propmt
+ await node.push_task(task)
+
+ while True:
+ if task.state == ComputeTaskState.DONE:
+ local_file = task.result.result
+ print("local file is: ", local_file)
+ break
+ await asyncio.sleep(1)
+
+if __name__ == "__main__":
+ arg_len = len(os.sys.argv)
+ prompt = "a beautiful sunset"
+ model = "dall-e-3"
+ if arg_len >= 2:
+ prompt = os.sys.argv[1]
+ if arg_len == 3:
+ model = os.sys.argv[2]
+
+ asyncio.run(test_dall_e_node(prompt, model))
diff --git a/test/test_local_sd_node.py b/test/test_local_sd_node.py
index 21f8219..4404670 100644
--- a/test/test_local_sd_node.py
+++ b/test/test_local_sd_node.py
@@ -14,12 +14,12 @@ from aios_kernel.compute_task import ComputeTaskType, ComputeTask, ComputeTaskSt
#to launch a local stability node, please check:
#https://github.com/glen0125/stable-diffusion-webui-docker
-os.environ["LOCAL_STABILITY_URL"] = ""
-os.environ["TEXT2IMG_DEFAULT_MODEL"] = "v1-5-pruned-emaonly"
-os.environ["TEXT2IMG_OUTPUT_DIR"] = "./"
+#os.environ["LOCAL_STABILITY_URL"] = "http://aigc:7860"
+#os.environ["TEXT2IMG_DEFAULT_MODEL"] = "v1-5-pruned-emaonly"
+#os.environ["TEXT2IMG_OUTPUT_DIR"] = "./"
@pytest.mark.asyncio
-async def test_local_sd_node(propmt, model):
+async def test_local_sd_node(propmt, model, negative_prompt):
node = Local_Stability_ComputeNode.get_instance()
if await node.initial() is not True:
print("node initial failed!")
@@ -31,6 +31,7 @@ async def test_local_sd_node(propmt, model):
task.task_id = uuid.uuid4().hex
task.params['model_name'] = model
task.params['prompt'] = propmt
+ task.params['negative_prompt'] = negative_prompt
await node.push_task(task)
while True:
@@ -51,9 +52,12 @@ if __name__ == "__main__":
arg_len = len(os.sys.argv)
prompt = "a beautiful sunset"
model = "v1-5-pruned-emaonly"
+ negative_prompt = None
if arg_len >= 2:
prompt = os.sys.argv[1]
if arg_len == 3:
model = os.sys.argv[2]
+ if arg_len == 4:
+ negative_prompt = os.sys.argv[3]
- asyncio.run(test_local_sd_node(prompt, model))
+ asyncio.run(test_local_sd_node(prompt, model, negative_prompt))
diff --git a/test/test_workspace.py b/test/test_workspace.py
new file mode 100644
index 0000000..62929d7
--- /dev/null
+++ b/test/test_workspace.py
@@ -0,0 +1,28 @@
+import sys
+import os
+import logging
+import asyncio
+import time
+import unittest
+
+dir_path = os.path.dirname(os.path.realpath(__file__))
+
+sys.path.append("{}/../src/".format(dir_path))
+
+from aios_kernel import WorkspaceEnvironment
+
+async def test_workspace():
+ test_env = WorkspaceEnvironment("test")
+ test_env._add_document_dir(f"{dir_path}/../rootfs/test_doc")
+ test_env._start_scan_document()
+ catalogs = await test_env.get_knowledege_catalog()
+ print(catalogs)
+ asyncio.sleep(60*60)
+
+
+if __name__ == "__main__":
+ asyncio.run(test_workspace())
+ print("OK!")
+ time.sleep(60*60)
+
+
diff --git a/test/workflow_test.py b/test/workflow_test.py
index 3e5e586..37b9d44 100644
--- a/test/workflow_test.py
+++ b/test/workflow_test.py
@@ -4,10 +4,10 @@ import asyncio
directory = os.path.dirname(__file__)
sys.path.append(directory + '/../src')
-from aios_kernel import WorkspaceEnvironment
+from aios_kernel import ShellEnvironment
async def test_workflow():
- env = WorkspaceEnvironment("test")
+ env = ShellEnvironment("test")
test_code ="""
import toml