feat: add git-consistent memory gateway architecture
This commit is contained in:
7
.dockerignore
Normal file
7
.dockerignore
Normal file
@@ -0,0 +1,7 @@
|
||||
.git
|
||||
.gitignore
|
||||
README.md
|
||||
TASK-RESULT.md
|
||||
testdata
|
||||
knowledge
|
||||
*.log
|
||||
18
.env.example
Normal file
18
.env.example
Normal file
@@ -0,0 +1,18 @@
|
||||
# Required: git source-of-truth URL (ssh/https/local path)
|
||||
GIT_REMOTE_URL=git@github.com:your-org/your-memory-repo.git
|
||||
|
||||
# Optional runtime tuning
|
||||
DEFAULT_BRANCH=main
|
||||
QMD_TIMEOUT_SECONDS=300
|
||||
QMD_TOP_K=5
|
||||
QMD_INDEX_PREFIX=ws
|
||||
QMD_UPDATE_ON_LATEST_QUERY=true
|
||||
QMD_EMBED_ON_CHANGE=true
|
||||
|
||||
# GPU pinning for qmd container
|
||||
CUDA_DEVICE=1
|
||||
CUDA_VISIBLE_DEVICES=1
|
||||
NVIDIA_VISIBLE_DEVICES=1
|
||||
|
||||
# Optional warmup profile interval (seconds)
|
||||
WARMUP_INTERVAL_SECONDS=300
|
||||
32
.gitignore
vendored
Normal file
32
.gitignore
vendored
Normal file
@@ -0,0 +1,32 @@
|
||||
# local env
|
||||
.env
|
||||
|
||||
# python
|
||||
.venv/
|
||||
.pytest_cache/
|
||||
__pycache__/
|
||||
*.pyc
|
||||
|
||||
# editor/os
|
||||
.DS_Store
|
||||
|
||||
# runtime data
|
||||
logs/
|
||||
*.log
|
||||
|
||||
data/git-mirror/*
|
||||
!data/git-mirror/.gitkeep
|
||||
|
||||
data/workspaces/*
|
||||
!data/workspaces/.gitkeep
|
||||
|
||||
data/qmd-cache/*
|
||||
!data/qmd-cache/.gitkeep
|
||||
|
||||
data/qmd-config/*
|
||||
!data/qmd-config/.gitkeep
|
||||
|
||||
data/remote-memory.git/
|
||||
|
||||
# generated index/cache in repo root if any
|
||||
qmd/
|
||||
11
AGENTS.md
Normal file
11
AGENTS.md
Normal file
@@ -0,0 +1,11 @@
|
||||
# AGENTS.md
|
||||
|
||||
## Working Rules
|
||||
|
||||
- 先阅读 `README.md`,确认架构和 API 约定后再修改代码。
|
||||
- 修改 `memory-gateway`、`git` 同步逻辑或 `docker-compose.yml` 后,先运行测试再提交结果。
|
||||
- 不要破坏现有 `qmd` 容器能力(HTTP MCP、健康检查、模型配置与持久化)。
|
||||
- 优先保证功能正确性和可恢复性,不为了“优雅”牺牲可追溯和可运维性。
|
||||
- 每次查询返回必须可追溯到 `branch + commit_hash + synced_at`。
|
||||
- 禁止实现“单目录频繁 checkout 不同 branch”的方案;必须使用分支隔离 workspace。
|
||||
- 任何查询一致性逻辑必须遵循:`sync git -> qmd update/embed -> query`。
|
||||
40
Dockerfile
Normal file
40
Dockerfile
Normal file
@@ -0,0 +1,40 @@
|
||||
FROM node:22-bookworm-slim
|
||||
|
||||
ENV NODE_ENV=production \
|
||||
QMD_VERSION=1.0.7 \
|
||||
XDG_CACHE_HOME=/var/lib/qmd/cache \
|
||||
XDG_CONFIG_HOME=/var/lib/qmd/config \
|
||||
QMD_HTTP_PORT=8181
|
||||
|
||||
RUN apt-get update \
|
||||
&& apt-get install -y --no-install-recommends \
|
||||
ca-certificates \
|
||||
curl \
|
||||
git \
|
||||
cmake \
|
||||
socat \
|
||||
tini \
|
||||
python3 \
|
||||
make \
|
||||
g++ \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
RUN npm install -g "@tobilu/qmd@${QMD_VERSION}" \
|
||||
&& npm cache clean --force
|
||||
|
||||
WORKDIR /app
|
||||
|
||||
RUN mkdir -p \
|
||||
/app/scripts \
|
||||
/var/lib/qmd/cache \
|
||||
/var/lib/qmd/config \
|
||||
/data/knowledge \
|
||||
/data/workspaces \
|
||||
/data/testdata
|
||||
|
||||
COPY scripts/qmd-entrypoint.sh /app/scripts/qmd-entrypoint.sh
|
||||
RUN chmod +x /app/scripts/qmd-entrypoint.sh
|
||||
|
||||
EXPOSE 8181
|
||||
|
||||
ENTRYPOINT ["/usr/bin/tini", "--", "/app/scripts/qmd-entrypoint.sh"]
|
||||
266
README.md
Normal file
266
README.md
Normal file
@@ -0,0 +1,266 @@
|
||||
# Git 一致性优先的共享记忆检索系统
|
||||
|
||||
本项目在现有 QMD 基础上增加 `memory-gateway`,目标是让多 Agent 查询具备“查询前强制同步”的一致性,而不是只依赖后台轮询同步。
|
||||
|
||||
## 1. 项目背景
|
||||
|
||||
在多 Agent 协作里,若“同步”和“查询”解耦为两个松散后台动作,会出现:
|
||||
|
||||
- Agent A 读到旧记忆
|
||||
- Agent B 读到新记忆
|
||||
- 对任务目标、恢复点、执行上下文理解分叉
|
||||
|
||||
本项目的核心设计:
|
||||
|
||||
- Git 是 source of truth
|
||||
- QMD 只做索引和检索
|
||||
- memory-gateway 负责查询前同步与一致性协调
|
||||
|
||||
## 2. 架构与一致性事务
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
C[Remote Client\nCodex/Claude/OpenClaw] --> G[memory-gateway\nHTTP :8787]
|
||||
G --> S[Sync Transaction\nfetch -> workspace sync -> qmd update/embed]
|
||||
S --> Q[qmd search/query\nper-workspace index]
|
||||
Q --> G
|
||||
G --> C
|
||||
|
||||
G --> M[/data/git-mirror/repo.git]
|
||||
G --> W1[/data/workspaces/main]
|
||||
G --> W2[/data/workspaces/memory-2026-03]
|
||||
G --> W3[/data/workspaces/task-TASK-001]
|
||||
```
|
||||
|
||||
`/query` 的主链路:
|
||||
|
||||
1. 解析 `branch / memory_profile / query_type / query`
|
||||
2. 获取 workspace 锁(同 workspace 串行,不同 workspace 并行)
|
||||
3. `git fetch` + workspace 对齐到目标 branch 最新 commit
|
||||
4. `qmd update`(需要时 `embed`)
|
||||
5. 执行检索
|
||||
6. 返回结果 + `branch/commit_hash/synced_at/workspace`
|
||||
|
||||
## 3. 服务职责
|
||||
|
||||
### qmd
|
||||
|
||||
- 提供 MCP/HTTP 能力(`8181`)
|
||||
- 持久化 cache/config
|
||||
- 作为底层检索引擎,不作为远程客户端唯一入口
|
||||
|
||||
### memory-gateway
|
||||
|
||||
- 远程唯一入口(`8787`)
|
||||
- 提供 `GET /health`、`POST /query`、`POST /sync`、`GET /status`
|
||||
- 强制执行查询前同步
|
||||
|
||||
### warmup(可选)
|
||||
|
||||
- 定时轻量请求,降低冷启动抖动
|
||||
- 仅优化项,不提供一致性保证
|
||||
|
||||
## 4. 目录与关键路径
|
||||
|
||||
### 仓库目录(Host)
|
||||
|
||||
```text
|
||||
.
|
||||
├─ docker-compose.yml
|
||||
├─ .env.example
|
||||
├─ AGENTS.md
|
||||
├─ README.md
|
||||
├─ gateway/
|
||||
│ ├─ app/
|
||||
│ ├─ tests/
|
||||
│ ├─ requirements.txt
|
||||
│ └─ Dockerfile
|
||||
├─ scripts/
|
||||
│ ├─ qmd-entrypoint.sh
|
||||
│ ├─ bootstrap.sh
|
||||
│ ├─ warmup.sh
|
||||
│ └─ smoke-test.sh
|
||||
└─ data/
|
||||
├─ git-mirror/
|
||||
├─ workspaces/
|
||||
├─ qmd-cache/
|
||||
├─ qmd-config/
|
||||
└─ remote-memory.git/ # bootstrap 生成的本地 demo remote
|
||||
```
|
||||
|
||||
### 容器内路径
|
||||
|
||||
- Git mirror: `/data/git-mirror/repo.git`
|
||||
- Workspace 根目录: `/data/workspaces`
|
||||
- QMD cache: `/var/lib/qmd/cache`(qmd 容器)/ `/data/qmd-cache`(gateway 调 CLI)
|
||||
- QMD config: `/var/lib/qmd/config`(qmd 容器)/ `/data/qmd-config`(gateway 调 CLI)
|
||||
|
||||
## 5. 分支与 profile 规则
|
||||
|
||||
优先级:`branch > memory_profile > main`
|
||||
|
||||
- `stable` -> `main`
|
||||
- `monthly-YYYY-MM` -> `memory/YYYY-MM`
|
||||
- `task-<task-id>` -> `task/<task-id>`
|
||||
|
||||
建议分层:
|
||||
|
||||
- 长期稳定:`main`
|
||||
- 月度滚动:`memory/YYYY-MM`
|
||||
- 任务临时:`task/<task-id>`
|
||||
|
||||
## 6. 环境变量(关键配置)
|
||||
|
||||
### 必填
|
||||
|
||||
- `GIT_REMOTE_URL`:Git 真相源地址(ssh/https/file/path)
|
||||
|
||||
### 常用
|
||||
|
||||
- `DEFAULT_BRANCH`:默认分支(默认 `main`)
|
||||
- `QMD_TIMEOUT_SECONDS`:gateway 调 qmd CLI 超时秒数(默认 `300`)
|
||||
- `QMD_INDEX_PREFIX`:per-workspace 索引名前缀(默认 `ws`)
|
||||
- `QMD_UPDATE_ON_LATEST_QUERY`:`require_latest=true` 时是否强制 `qmd update`(默认 `true`)
|
||||
- `QMD_EMBED_ON_CHANGE`:HEAD 变化时是否尝试 embed(默认 `true`,可按性能改 `false`)
|
||||
|
||||
### GPU(qmd 容器)
|
||||
|
||||
- `CUDA_DEVICE`
|
||||
- `CUDA_VISIBLE_DEVICES`
|
||||
- `NVIDIA_VISIBLE_DEVICES`
|
||||
|
||||
### 模型覆盖(可选)
|
||||
|
||||
- `QMD_EMBED_MODEL_URI`
|
||||
- `QMD_RERANK_MODEL_URI`
|
||||
- `QMD_GENERATE_MODEL_URI`
|
||||
|
||||
示例:见 `[.env.example](/home/lingyuzeng/project/qmd-local/qmd-docker-http-mcp/.env.example)`。
|
||||
|
||||
## 7. 快速开始
|
||||
|
||||
### 7.1 初始化 demo remote 与目录
|
||||
|
||||
```bash
|
||||
cd /home/lingyuzeng/project/qmd-local/qmd-docker-http-mcp
|
||||
./scripts/bootstrap.sh
|
||||
```
|
||||
|
||||
### 7.2 启动
|
||||
|
||||
```bash
|
||||
docker compose build
|
||||
docker compose up -d
|
||||
```
|
||||
|
||||
### 7.3 健康检查
|
||||
|
||||
```bash
|
||||
curl http://127.0.0.1:8181/health
|
||||
curl http://127.0.0.1:8787/health
|
||||
```
|
||||
|
||||
## 8. API 使用
|
||||
|
||||
### GET /health
|
||||
|
||||
```bash
|
||||
curl -fsS http://127.0.0.1:8787/health
|
||||
```
|
||||
|
||||
### POST /sync
|
||||
|
||||
```bash
|
||||
curl -fsS -X POST http://127.0.0.1:8787/sync \
|
||||
-H 'content-type: application/json' \
|
||||
-d '{"branch":"main","require_latest":true}'
|
||||
```
|
||||
|
||||
### POST /query(主入口)
|
||||
|
||||
```bash
|
||||
curl -fsS -X POST http://127.0.0.1:8787/query \
|
||||
-H 'content-type: application/json' \
|
||||
-d '{
|
||||
"branch":"main",
|
||||
"query_type":"search",
|
||||
"query":"recovery strategy",
|
||||
"require_latest":true,
|
||||
"debug":true
|
||||
}'
|
||||
```
|
||||
|
||||
返回字段核心:
|
||||
|
||||
- `ok`
|
||||
- `branch`
|
||||
- `resolved_workspace`
|
||||
- `commit_hash`
|
||||
- `synced_at`
|
||||
- `query_type`
|
||||
- `results`
|
||||
- `qmd_collection`
|
||||
- `debug`(可选)
|
||||
|
||||
### GET /status
|
||||
|
||||
```bash
|
||||
curl -fsS http://127.0.0.1:8787/status
|
||||
```
|
||||
|
||||
## 9. 查询策略说明
|
||||
|
||||
当前最小兼容实现中:
|
||||
|
||||
- `query_type=search` -> `qmd search`
|
||||
- `query_type=vsearch` -> `qmd vsearch`
|
||||
- `query_type=query/deep_search` -> 为保证稳定与时延上界,默认走 `qmd search`(响应 `debug.qmd_command` 可追踪)
|
||||
|
||||
## 10. 测试与验收
|
||||
|
||||
### 自动化测试
|
||||
|
||||
```bash
|
||||
python3 -m venv .venv
|
||||
. .venv/bin/activate
|
||||
pip install -r gateway/requirements.txt pytest httpx
|
||||
PYTHONPATH=./gateway pytest -q gateway/tests
|
||||
```
|
||||
|
||||
### 手工验收(建议)
|
||||
|
||||
1. 构建与启动
|
||||
2. 服务健康
|
||||
3. 基础查询
|
||||
4. 修改远端后立即查询(验证查询前同步)
|
||||
5. 分支隔离
|
||||
6. 默认分支
|
||||
7. 并发查询
|
||||
8. 重启恢复
|
||||
|
||||
## 11. 常见问题
|
||||
|
||||
### 首次请求慢
|
||||
|
||||
首次模型/索引准备、或首次分支激活会慢。可用:
|
||||
|
||||
- `QMD_EMBED_ON_CHANGE=false`(降低请求阻塞)
|
||||
- warmup profile 预热
|
||||
|
||||
### 本地路径 remote 被 git safe.directory 拦截
|
||||
|
||||
gateway 已对本地 `GIT_REMOTE_URL` 自动添加 `safe.directory`。
|
||||
|
||||
## 12. 客户端接入建议
|
||||
|
||||
- 远程客户端只访问 `memory-gateway`,不要直连 qmd
|
||||
- 每次请求尽量显式传 `branch` 或 `memory_profile`
|
||||
- 记录响应中的 `branch + commit_hash` 用于审计与回放
|
||||
|
||||
## 13. 关键文件
|
||||
|
||||
- 网关入口:[gateway/app/main.py](/home/lingyuzeng/project/qmd-local/qmd-docker-http-mcp/gateway/app/main.py)
|
||||
- 同步事务:[gateway/app/sync_service.py](/home/lingyuzeng/project/qmd-local/qmd-docker-http-mcp/gateway/app/sync_service.py)
|
||||
- Git 管理:[gateway/app/git_manager.py](/home/lingyuzeng/project/qmd-local/qmd-docker-http-mcp/gateway/app/git_manager.py)
|
||||
- QMD 调用:[gateway/app/qmd_client.py](/home/lingyuzeng/project/qmd-local/qmd-docker-http-mcp/gateway/app/qmd_client.py)
|
||||
- Compose 编排:[docker-compose.yml](/home/lingyuzeng/project/qmd-local/qmd-docker-http-mcp/docker-compose.yml)
|
||||
157
TASK-RESULT.md
Normal file
157
TASK-RESULT.md
Normal file
@@ -0,0 +1,157 @@
|
||||
# TASK RESULT
|
||||
|
||||
## Research Notes
|
||||
|
||||
1. **当前最适合 pin 的 QMD 版本**
|
||||
- 选择 `v1.0.7`(`@tobilu/qmd@1.0.7`),是截至 2026-03-07 的最新非预发布 stable release(发布时间 2026-02-18)。
|
||||
|
||||
2. **是否已有官方 compose 参考**
|
||||
- 在 `tobi/qmd` 的 `v1.0.7` 源码树中未找到 `Dockerfile` / `docker-compose.yml` 参考,需要自行实现。
|
||||
|
||||
3. **QMD HTTP MCP 启动与健康检查方式**
|
||||
- 启动:`qmd mcp --http`(默认端口 8181)
|
||||
- 端点:`POST /mcp`、`GET /health`
|
||||
|
||||
4. **是否需要覆盖默认模型**
|
||||
- 第一版不覆盖,直接沿用源码默认模型(embeddinggemma / qwen3-reranker / qmd-query-expansion)。
|
||||
|
||||
5. **Node 与 Bun 在容器中选型**
|
||||
- 选 Node(`node:22-bookworm-slim`)。理由:
|
||||
- 上游 package `engines.node >=22`
|
||||
- `npm install -g @tobilu/qmd@1.0.7` 路径直接、可复现
|
||||
- 官方生态和镜像稳定性更高,长期维护成本更低
|
||||
|
||||
6. **缓存目录、索引目录、知识库目录挂载策略**
|
||||
- `XDG_CACHE_HOME=/var/lib/qmd/cache`(模型缓存 + 默认 sqlite 索引)
|
||||
- `XDG_CONFIG_HOME=/var/lib/qmd/config`(collection YAML 配置)
|
||||
- 业务知识库挂载到 `/data/knowledge`
|
||||
- 测试数据挂载到 `/data/testdata`
|
||||
|
||||
7. **smoke test 最小闭环覆盖步骤**
|
||||
- compose 启动服务
|
||||
- `/health` 检查通过
|
||||
- 添加 collections(notes/docs/meetings)
|
||||
- `qmd update` + `qmd embed`
|
||||
- 至少一次 `qmd search`
|
||||
- 至少一次 `qmd query`(deep search 等价能力)
|
||||
|
||||
## Implemented Files
|
||||
|
||||
```text
|
||||
qmd-docker-http-mcp/
|
||||
├─ Dockerfile
|
||||
├─ docker-compose.yml
|
||||
├─ .dockerignore
|
||||
├─ README.md
|
||||
├─ TASK-RESULT.md
|
||||
├─ scripts/
|
||||
│ ├─ entrypoint.sh
|
||||
│ ├─ warmup.sh
|
||||
│ └─ smoke-test.sh
|
||||
├─ knowledge/
|
||||
└─ testdata/
|
||||
├─ notes/
|
||||
├─ docs/
|
||||
└─ meetings/
|
||||
```
|
||||
|
||||
## Commands Executed (Real)
|
||||
|
||||
```bash
|
||||
# Research
|
||||
curl -fsSL https://api.github.com/repos/tobi/qmd/releases?per_page=5 | jq ...
|
||||
curl -fsSL https://api.github.com/repos/tobi/qmd/git/trees/v1.0.7?recursive=1 | jq -r '.tree[].path' | rg 'docker|compose|Dockerfile|docker-compose' || true
|
||||
curl -fsSL https://raw.githubusercontent.com/tobi/qmd/v1.0.7/README.md
|
||||
curl -fsSL https://raw.githubusercontent.com/tobi/qmd/v1.0.7/src/llm.ts
|
||||
|
||||
# Build / Run
|
||||
cd qmd-docker-http-mcp
|
||||
docker build -t hotwa/qmd:latest .
|
||||
docker compose up -d
|
||||
docker compose logs --no-color --tail=120 qmd
|
||||
curl -fsS http://127.0.0.1:8181/health
|
||||
|
||||
# Warmup / Smoke
|
||||
./scripts/warmup.sh
|
||||
./scripts/smoke-test.sh
|
||||
|
||||
# Extra MCP endpoint check
|
||||
curl -X POST http://127.0.0.1:8181/mcp -H 'content-type: application/json' -d '{}'
|
||||
```
|
||||
|
||||
## Build Result
|
||||
|
||||
- Image built successfully: `hotwa/qmd:latest`
|
||||
- Image inspect summary:
|
||||
- `hotwa/qmd:latest b5c811d89722 1.99GB`
|
||||
|
||||
## Compose Startup Result
|
||||
|
||||
- `docker compose up -d` 成功
|
||||
- `docker compose ps` 结果:
|
||||
- `qmd-http-mcp ... Up ... (healthy) ... 0.0.0.0:8181->8181/tcp`
|
||||
|
||||
## Healthcheck Result
|
||||
|
||||
- `GET /health` 返回成功:
|
||||
|
||||
```json
|
||||
{"status":"ok","uptime":244}
|
||||
```
|
||||
|
||||
- `POST /mcp` 路由可达(示例请求返回协议校验错误而非 404):
|
||||
- HTTP `406`
|
||||
- `Not Acceptable: Client must accept both application/json and text/event-stream`
|
||||
|
||||
## Warmup Result
|
||||
|
||||
- 成功创建并索引 3 个 collection:`notes` / `docs` / `meetings`
|
||||
- 成功执行 `qmd update`
|
||||
- 成功执行 `qmd embed`
|
||||
- 首次运行完成 embedding 模型下载并生成向量
|
||||
|
||||
## Smoke Test Result
|
||||
|
||||
`scripts/smoke-test.sh` 真实执行通过,覆盖:
|
||||
|
||||
- 服务启动与 `/health` 成功
|
||||
- collection 检查通过(3/3)
|
||||
- `qmd search "incident runbook" --json -n 5` 返回命中(含 `docs/incident-runbook.md`)
|
||||
- `qmd query "what is the RTO target for disaster recovery" --json -n 5` 返回命中(`docs/disaster-recovery.md` 得分最高)
|
||||
- 脚本最终输出:`[smoke] Smoke test passed`
|
||||
|
||||
## Problems Found and Fixes
|
||||
|
||||
1. **问题:容器外访问 8181 失败(连接被重置)**
|
||||
- 根因:QMD `v1.0.7` 的 HTTP MCP 在上游实现中绑定 `localhost`。
|
||||
- 修复:在 `entrypoint.sh` 引入 `socat` 转发,将容器 `0.0.0.0:8181` 转发到容器内 QMD 监听端口。
|
||||
|
||||
2. **问题:初版转发到 `127.0.0.1:8182` 仍失败**
|
||||
- 根因:QMD 在容器内实际监听 `::1`(IPv6 localhost),`127.0.0.1` 不通。
|
||||
- 修复:改为 `socat ... TCP6:[::1]:8182`。
|
||||
|
||||
3. **观察项:首次 query/embed 有 node-llama-cpp 的 CUDA 构建回退告警**
|
||||
- 现象:日志提示 `git/cmake not found`,随后回退并继续 CPU 路径,流程最终成功。
|
||||
- 结论:当前不阻塞功能闭环,但会增加首次请求噪声与时延。
|
||||
|
||||
## Final Acceptance
|
||||
|
||||
验收条件逐项结果:
|
||||
|
||||
- [x] 成功构建 `hotwa/qmd:latest`
|
||||
- [x] `docker compose up -d` 后服务启动
|
||||
- [x] `/health` 返回成功
|
||||
- [x] 至少一组测试数据被索引
|
||||
- [x] 至少一次关键词搜索成功
|
||||
- [x] 至少一次深检索/混合检索成功
|
||||
- [x] README 可指导复现
|
||||
- [x] 本文件记录真实测试过程(无伪造)
|
||||
|
||||
## Sources
|
||||
|
||||
- QMD releases: https://api.github.com/repos/tobi/qmd/releases
|
||||
- QMD v1.0.7 README: https://raw.githubusercontent.com/tobi/qmd/v1.0.7/README.md
|
||||
- QMD v1.0.7 llm.ts: https://raw.githubusercontent.com/tobi/qmd/v1.0.7/src/llm.ts
|
||||
- QMD v1.0.7 mcp.ts: https://raw.githubusercontent.com/tobi/qmd/v1.0.7/src/mcp.ts
|
||||
- Codex MCP docs: https://developers.openai.com/codex/mcp/
|
||||
- Claude Code MCP docs: https://docs.anthropic.com/en/docs/claude-code/mcp
|
||||
0
data/git-mirror/.gitkeep
Normal file
0
data/git-mirror/.gitkeep
Normal file
0
data/qmd-cache/.gitkeep
Normal file
0
data/qmd-cache/.gitkeep
Normal file
0
data/qmd-config/.gitkeep
Normal file
0
data/qmd-config/.gitkeep
Normal file
0
data/workspaces/.gitkeep
Normal file
0
data/workspaces/.gitkeep
Normal file
91
docker-compose.yml
Normal file
91
docker-compose.yml
Normal file
@@ -0,0 +1,91 @@
|
||||
services:
|
||||
qmd:
|
||||
image: hotwa/qmd:latest
|
||||
build:
|
||||
context: .
|
||||
dockerfile: Dockerfile
|
||||
container_name: qmd-http-mcp
|
||||
ports:
|
||||
- "8181:8181"
|
||||
deploy:
|
||||
resources:
|
||||
reservations:
|
||||
devices:
|
||||
- driver: nvidia
|
||||
device_ids: ["1"]
|
||||
capabilities: [gpu]
|
||||
environment:
|
||||
XDG_CACHE_HOME: /var/lib/qmd/cache
|
||||
XDG_CONFIG_HOME: /var/lib/qmd/config
|
||||
QMD_HTTP_PORT: "8181"
|
||||
CUDA_DEVICE: "${CUDA_DEVICE:-1}"
|
||||
CUDA_VISIBLE_DEVICES: "${CUDA_VISIBLE_DEVICES:-1}"
|
||||
NVIDIA_VISIBLE_DEVICES: "${NVIDIA_VISIBLE_DEVICES:-1}"
|
||||
NVIDIA_DRIVER_CAPABILITIES: "compute,utility"
|
||||
QMD_EMBED_MODEL_URI: "${QMD_EMBED_MODEL_URI:-hf:ggml-org/embeddinggemma-300M-GGUF/embeddinggemma-300M-Q8_0.gguf}"
|
||||
QMD_RERANK_MODEL_URI: "${QMD_RERANK_MODEL_URI:-hf:ggml-org/Qwen3-Reranker-0.6B-Q8_0-GGUF/qwen3-reranker-0.6b-q8_0.gguf}"
|
||||
QMD_GENERATE_MODEL_URI: "${QMD_GENERATE_MODEL_URI:-hf:tobil/qmd-query-expansion-1.7B-gguf/qmd-query-expansion-1.7B-q4_k_m.gguf}"
|
||||
volumes:
|
||||
- ./data/qmd-cache:/var/lib/qmd/cache
|
||||
- ./data/qmd-config:/var/lib/qmd/config
|
||||
- ./data/workspaces:/data/workspaces
|
||||
- ./models:/models
|
||||
- ./testdata:/data/testdata:ro
|
||||
healthcheck:
|
||||
test: ["CMD", "curl", "-fsS", "http://127.0.0.1:8181/health"]
|
||||
interval: 30s
|
||||
timeout: 5s
|
||||
retries: 5
|
||||
start_period: 30s
|
||||
restart: unless-stopped
|
||||
|
||||
memory-gateway:
|
||||
build:
|
||||
context: .
|
||||
dockerfile: gateway/Dockerfile
|
||||
container_name: memory-gateway
|
||||
depends_on:
|
||||
qmd:
|
||||
condition: service_healthy
|
||||
ports:
|
||||
- "8787:8787"
|
||||
environment:
|
||||
APP_ENV: "${APP_ENV:-prod}"
|
||||
DEFAULT_BRANCH: "${DEFAULT_BRANCH:-main}"
|
||||
GIT_REMOTE_URL: "${GIT_REMOTE_URL:-}"
|
||||
GIT_MIRROR_PATH: /data/git-mirror/repo.git
|
||||
WORKSPACES_ROOT: /data/workspaces
|
||||
WORKSPACE_STATE_DIR: /data/workspaces/.gateway-state
|
||||
XDG_CACHE_HOME: /data/qmd-cache
|
||||
XDG_CONFIG_HOME: /data/qmd-config
|
||||
QMD_BINARY: qmd
|
||||
QMD_TIMEOUT_SECONDS: "${QMD_TIMEOUT_SECONDS:-300}"
|
||||
QMD_TOP_K: "${QMD_TOP_K:-5}"
|
||||
QMD_INDEX_PREFIX: "${QMD_INDEX_PREFIX:-ws}"
|
||||
QMD_UPDATE_ON_LATEST_QUERY: "${QMD_UPDATE_ON_LATEST_QUERY:-true}"
|
||||
QMD_EMBED_ON_CHANGE: "${QMD_EMBED_ON_CHANGE:-true}"
|
||||
volumes:
|
||||
- ./data:/data
|
||||
- ./models:/models
|
||||
healthcheck:
|
||||
test: ["CMD", "curl", "-fsS", "http://127.0.0.1:8787/health"]
|
||||
interval: 15s
|
||||
timeout: 5s
|
||||
retries: 6
|
||||
start_period: 10s
|
||||
restart: unless-stopped
|
||||
|
||||
warmup:
|
||||
image: curlimages/curl:8.12.1
|
||||
container_name: memory-warmup
|
||||
depends_on:
|
||||
memory-gateway:
|
||||
condition: service_healthy
|
||||
entrypoint: ["/bin/sh", "-lc"]
|
||||
command: >-
|
||||
while true; do
|
||||
curl -fsS http://memory-gateway:8787/health >/dev/null || true;
|
||||
sleep ${WARMUP_INTERVAL_SECONDS:-300};
|
||||
done
|
||||
restart: unless-stopped
|
||||
profiles: ["warmup"]
|
||||
38
gateway/Dockerfile
Normal file
38
gateway/Dockerfile
Normal file
@@ -0,0 +1,38 @@
|
||||
FROM node:22-bookworm-slim
|
||||
|
||||
ENV PYTHONDONTWRITEBYTECODE=1 \
|
||||
PYTHONUNBUFFERED=1 \
|
||||
QMD_VERSION=1.0.7 \
|
||||
PATH=/opt/venv/bin:$PATH \
|
||||
XDG_CACHE_HOME=/var/lib/qmd/cache \
|
||||
XDG_CONFIG_HOME=/var/lib/qmd/config
|
||||
|
||||
RUN apt-get update \
|
||||
&& apt-get install -y --no-install-recommends \
|
||||
ca-certificates \
|
||||
curl \
|
||||
git \
|
||||
cmake \
|
||||
python3 \
|
||||
python3-pip \
|
||||
python3-venv \
|
||||
make \
|
||||
g++ \
|
||||
tini \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
RUN npm install -g "@tobilu/qmd@${QMD_VERSION}" \
|
||||
&& npm cache clean --force
|
||||
|
||||
WORKDIR /app
|
||||
|
||||
COPY gateway/requirements.txt /app/requirements.txt
|
||||
RUN python3 -m venv /opt/venv \
|
||||
&& pip install --no-cache-dir -r /app/requirements.txt
|
||||
|
||||
COPY gateway/app /app/app
|
||||
|
||||
EXPOSE 8787
|
||||
|
||||
ENTRYPOINT ["/usr/bin/tini", "--"]
|
||||
CMD ["python3", "-m", "uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8787"]
|
||||
1
gateway/app/__init__.py
Normal file
1
gateway/app/__init__.py
Normal file
@@ -0,0 +1 @@
|
||||
"""Memory gateway application package."""
|
||||
41
gateway/app/config.py
Normal file
41
gateway/app/config.py
Normal file
@@ -0,0 +1,41 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from functools import lru_cache
|
||||
from pathlib import Path
|
||||
|
||||
from pydantic import Field, model_validator
|
||||
from pydantic_settings import BaseSettings, SettingsConfigDict
|
||||
|
||||
|
||||
class Settings(BaseSettings):
|
||||
model_config = SettingsConfigDict(env_file=".env", env_file_encoding="utf-8", extra="ignore")
|
||||
|
||||
app_name: str = "memory-gateway"
|
||||
app_env: str = "dev"
|
||||
default_branch: str = "main"
|
||||
|
||||
git_remote_url: str = Field(default="", description="Upstream git remote URL or local path")
|
||||
git_mirror_path: Path = Path("/data/git-mirror/repo.git")
|
||||
workspaces_root: Path = Path("/data/workspaces")
|
||||
workspace_state_dir: Path | None = None
|
||||
|
||||
qmd_binary: str = "qmd"
|
||||
qmd_timeout_seconds: int = 300
|
||||
qmd_top_k: int = 5
|
||||
qmd_index_prefix: str = "ws"
|
||||
qmd_update_on_latest_query: bool = True
|
||||
qmd_embed_on_change: bool = True
|
||||
|
||||
xdg_cache_home: Path = Path("/var/lib/qmd/cache")
|
||||
xdg_config_home: Path = Path("/var/lib/qmd/config")
|
||||
|
||||
@model_validator(mode="after")
|
||||
def _normalize_paths(self) -> "Settings":
|
||||
if self.workspace_state_dir is None:
|
||||
self.workspace_state_dir = self.workspaces_root / ".gateway-state"
|
||||
return self
|
||||
|
||||
|
||||
@lru_cache(maxsize=1)
|
||||
def get_settings() -> Settings:
|
||||
return Settings()
|
||||
153
gateway/app/git_manager.py
Normal file
153
gateway/app/git_manager.py
Normal file
@@ -0,0 +1,153 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
import re
|
||||
import shlex
|
||||
import subprocess
|
||||
from urllib.parse import urlparse
|
||||
|
||||
|
||||
class GitCommandError(RuntimeError):
|
||||
def __init__(self, args: list[str], returncode: int, stdout: str, stderr: str) -> None:
|
||||
cmd = " ".join(shlex.quote(x) for x in args)
|
||||
message = f"git command failed ({returncode}): {cmd}\nstdout:\n{stdout}\nstderr:\n{stderr}"
|
||||
super().__init__(message)
|
||||
self.args_list = args
|
||||
self.returncode = returncode
|
||||
self.stdout = stdout
|
||||
self.stderr = stderr
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class SyncResult:
|
||||
previous_commit: str | None
|
||||
commit_hash: str
|
||||
changed: bool
|
||||
|
||||
|
||||
class GitManager:
|
||||
def __init__(self, remote_url: str, mirror_path: Path) -> None:
|
||||
self.remote_url = remote_url
|
||||
self.mirror_path = mirror_path
|
||||
|
||||
def ensure_mirror(self) -> None:
|
||||
if not self.remote_url:
|
||||
raise ValueError("git_remote_url is required")
|
||||
|
||||
self._allow_local_remote_if_needed()
|
||||
|
||||
head_file = self.mirror_path / "HEAD"
|
||||
if not head_file.exists():
|
||||
self.mirror_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
if self.mirror_path.exists() and any(self.mirror_path.iterdir()):
|
||||
raise ValueError(f"mirror path exists and is not empty: {self.mirror_path}")
|
||||
self._run(["git", "clone", "--mirror", self.remote_url, str(self.mirror_path)])
|
||||
else:
|
||||
self._run(
|
||||
[
|
||||
"git",
|
||||
f"--git-dir={self.mirror_path}",
|
||||
"remote",
|
||||
"set-url",
|
||||
"origin",
|
||||
self.remote_url,
|
||||
]
|
||||
)
|
||||
|
||||
def fetch_origin(self) -> None:
|
||||
self._run(["git", f"--git-dir={self.mirror_path}", "fetch", "--prune", "origin"])
|
||||
|
||||
def get_branch_commit(self, branch: str) -> str:
|
||||
refs_to_try = [
|
||||
f"refs/heads/{branch}",
|
||||
f"refs/remotes/origin/{branch}",
|
||||
]
|
||||
last_error: GitCommandError | None = None
|
||||
for ref in refs_to_try:
|
||||
try:
|
||||
return self._run(
|
||||
["git", f"--git-dir={self.mirror_path}", "rev-parse", "--verify", ref]
|
||||
).strip()
|
||||
except GitCommandError as exc:
|
||||
last_error = exc
|
||||
if last_error is None:
|
||||
raise RuntimeError("unexpected branch lookup failure without error")
|
||||
raise last_error
|
||||
|
||||
def get_workspace_head(self, workspace_path: Path) -> str | None:
|
||||
if not (workspace_path / ".git").exists():
|
||||
return None
|
||||
try:
|
||||
return self._run(["git", "-C", str(workspace_path), "rev-parse", "HEAD"]).strip()
|
||||
except GitCommandError:
|
||||
return None
|
||||
|
||||
def sync_workspace(self, workspace_path: Path, branch: str, commit_hash: str) -> SyncResult:
|
||||
previous = self.get_workspace_head(workspace_path)
|
||||
self._ensure_workspace_repo(workspace_path)
|
||||
|
||||
self._run(["git", "-C", str(workspace_path), "fetch", "--prune", "origin"])
|
||||
local_branch = self._local_branch_name(branch)
|
||||
self._run(
|
||||
[
|
||||
"git",
|
||||
"-C",
|
||||
str(workspace_path),
|
||||
"checkout",
|
||||
"-B",
|
||||
local_branch,
|
||||
commit_hash,
|
||||
]
|
||||
)
|
||||
self._run(["git", "-C", str(workspace_path), "reset", "--hard", commit_hash])
|
||||
self._run(["git", "-C", str(workspace_path), "clean", "-fd"])
|
||||
|
||||
current = self.get_workspace_head(workspace_path)
|
||||
if not current:
|
||||
raise RuntimeError(f"failed to read workspace HEAD after sync: {workspace_path}")
|
||||
|
||||
return SyncResult(previous_commit=previous, commit_hash=current, changed=(previous != current))
|
||||
|
||||
def _ensure_workspace_repo(self, workspace_path: Path) -> None:
|
||||
if not (workspace_path / ".git").exists():
|
||||
workspace_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
if workspace_path.exists() and any(workspace_path.iterdir()):
|
||||
raise ValueError(f"workspace exists and is not a git repo: {workspace_path}")
|
||||
self._run(["git", "clone", str(self.mirror_path), str(workspace_path)])
|
||||
|
||||
self._run(
|
||||
[
|
||||
"git",
|
||||
"-C",
|
||||
str(workspace_path),
|
||||
"remote",
|
||||
"set-url",
|
||||
"origin",
|
||||
str(self.mirror_path),
|
||||
]
|
||||
)
|
||||
|
||||
def _local_branch_name(self, branch: str) -> str:
|
||||
cleaned = re.sub(r"[^A-Za-z0-9_.-]+", "-", branch).strip("-")
|
||||
return f"gateway-{cleaned or 'default'}"
|
||||
|
||||
def _run(self, args: list[str]) -> str:
|
||||
proc = subprocess.run(args, check=False, capture_output=True, text=True)
|
||||
if proc.returncode != 0:
|
||||
raise GitCommandError(args=args, returncode=proc.returncode, stdout=proc.stdout, stderr=proc.stderr)
|
||||
return proc.stdout
|
||||
|
||||
def _allow_local_remote_if_needed(self) -> None:
|
||||
candidate: str | None = None
|
||||
if self.remote_url.startswith("/"):
|
||||
candidate = self.remote_url
|
||||
elif self.remote_url.startswith("file://"):
|
||||
parsed = urlparse(self.remote_url)
|
||||
candidate = parsed.path
|
||||
if not candidate:
|
||||
return
|
||||
if not Path(candidate).exists():
|
||||
return
|
||||
# Avoid git safety checks blocking local bind-mounted repositories.
|
||||
self._run(["git", "config", "--global", "--add", "safe.directory", candidate])
|
||||
37
gateway/app/locks.py
Normal file
37
gateway/app/locks.py
Normal file
@@ -0,0 +1,37 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from contextlib import contextmanager
|
||||
from dataclasses import dataclass
|
||||
import threading
|
||||
import time
|
||||
|
||||
|
||||
@dataclass
|
||||
class LockStats:
|
||||
key: str
|
||||
wait_ms: int
|
||||
|
||||
|
||||
class WorkspaceLockManager:
|
||||
def __init__(self) -> None:
|
||||
self._locks: dict[str, threading.Lock] = {}
|
||||
self._guard = threading.Lock()
|
||||
|
||||
def _get_lock(self, key: str) -> threading.Lock:
|
||||
with self._guard:
|
||||
lock = self._locks.get(key)
|
||||
if lock is None:
|
||||
lock = threading.Lock()
|
||||
self._locks[key] = lock
|
||||
return lock
|
||||
|
||||
@contextmanager
|
||||
def acquire(self, key: str):
|
||||
lock = self._get_lock(key)
|
||||
started = time.perf_counter()
|
||||
lock.acquire()
|
||||
wait_ms = int((time.perf_counter() - started) * 1000)
|
||||
try:
|
||||
yield LockStats(key=key, wait_ms=wait_ms)
|
||||
finally:
|
||||
lock.release()
|
||||
115
gateway/app/main.py
Normal file
115
gateway/app/main.py
Normal file
@@ -0,0 +1,115 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import datetime, timezone
|
||||
import json
|
||||
from typing import Any
|
||||
|
||||
from fastapi import FastAPI, HTTPException
|
||||
from fastapi.responses import JSONResponse
|
||||
|
||||
from .config import Settings, get_settings
|
||||
from .git_manager import GitCommandError, GitManager
|
||||
from .locks import WorkspaceLockManager
|
||||
from .models import HealthResponse, QueryRequest, QueryResponse, StatusResponse, SyncRequest, SyncResponse, WorkspaceStatusItem
|
||||
from .qmd_client import QMDClient, QMDCommandError
|
||||
from .query_service import QueryService
|
||||
from .sync_service import SyncService
|
||||
from .workspace_manager import WorkspaceManager
|
||||
|
||||
|
||||
def create_app(
|
||||
settings: Settings | None = None,
|
||||
*,
|
||||
qmd_client: QMDClient | None = None,
|
||||
git_manager: GitManager | None = None,
|
||||
lock_manager: WorkspaceLockManager | None = None,
|
||||
) -> FastAPI:
|
||||
app = FastAPI(title="memory-gateway", version="0.1.0")
|
||||
|
||||
resolved_settings = settings or get_settings()
|
||||
workspace_manager = WorkspaceManager(
|
||||
workspaces_root=resolved_settings.workspaces_root,
|
||||
default_branch=resolved_settings.default_branch,
|
||||
qmd_index_prefix=resolved_settings.qmd_index_prefix,
|
||||
)
|
||||
resolved_git_manager = git_manager or GitManager(
|
||||
remote_url=resolved_settings.git_remote_url,
|
||||
mirror_path=resolved_settings.git_mirror_path,
|
||||
)
|
||||
resolved_qmd_client = qmd_client or QMDClient(
|
||||
qmd_binary=resolved_settings.qmd_binary,
|
||||
timeout_seconds=resolved_settings.qmd_timeout_seconds,
|
||||
xdg_cache_home=resolved_settings.xdg_cache_home,
|
||||
xdg_config_home=resolved_settings.xdg_config_home,
|
||||
)
|
||||
resolved_lock_manager = lock_manager or WorkspaceLockManager()
|
||||
|
||||
sync_service = SyncService(
|
||||
settings=resolved_settings,
|
||||
workspace_manager=workspace_manager,
|
||||
git_manager=resolved_git_manager,
|
||||
qmd_client=resolved_qmd_client,
|
||||
)
|
||||
query_service = QueryService(
|
||||
sync_service=sync_service,
|
||||
qmd_client=resolved_qmd_client,
|
||||
lock_manager=resolved_lock_manager,
|
||||
)
|
||||
|
||||
app.state.settings = resolved_settings
|
||||
app.state.sync_service = sync_service
|
||||
app.state.query_service = query_service
|
||||
|
||||
@app.exception_handler(ValueError)
|
||||
def handle_value_error(_: Any, exc: ValueError) -> JSONResponse:
|
||||
return JSONResponse(status_code=400, content={"ok": False, "error": str(exc)})
|
||||
|
||||
@app.exception_handler(GitCommandError)
|
||||
def handle_git_error(_: Any, exc: GitCommandError) -> JSONResponse:
|
||||
return JSONResponse(status_code=500, content={"ok": False, "error": str(exc)})
|
||||
|
||||
@app.exception_handler(QMDCommandError)
|
||||
def handle_qmd_error(_: Any, exc: QMDCommandError) -> JSONResponse:
|
||||
return JSONResponse(status_code=500, content={"ok": False, "error": str(exc)})
|
||||
|
||||
@app.get("/health", response_model=HealthResponse)
|
||||
def health() -> HealthResponse:
|
||||
return HealthResponse(ok=True, service="memory-gateway", timestamp=datetime.now(timezone.utc))
|
||||
|
||||
@app.post("/query", response_model=QueryResponse)
|
||||
def query(request: QueryRequest) -> QueryResponse:
|
||||
return app.state.query_service.handle_query(request)
|
||||
|
||||
@app.post("/sync", response_model=SyncResponse)
|
||||
def sync(request: SyncRequest) -> SyncResponse:
|
||||
return app.state.query_service.handle_sync(request)
|
||||
|
||||
@app.get("/status", response_model=StatusResponse)
|
||||
def status() -> StatusResponse:
|
||||
raw_states = app.state.sync_service.list_workspace_states()
|
||||
items: list[WorkspaceStatusItem] = []
|
||||
for row in raw_states:
|
||||
synced_at_raw = row.get("synced_at")
|
||||
if not synced_at_raw:
|
||||
continue
|
||||
items.append(
|
||||
WorkspaceStatusItem(
|
||||
branch=row["branch"],
|
||||
workspace=row["workspace"],
|
||||
commit_hash=row["commit_hash"],
|
||||
synced_at=datetime.fromisoformat(synced_at_raw),
|
||||
qmd_collection=row["qmd_collection"],
|
||||
)
|
||||
)
|
||||
|
||||
return StatusResponse(
|
||||
ok=True,
|
||||
default_branch=app.state.settings.default_branch,
|
||||
mirror_path=str(app.state.settings.git_mirror_path),
|
||||
workspaces=items,
|
||||
)
|
||||
|
||||
return app
|
||||
|
||||
|
||||
app = create_app()
|
||||
73
gateway/app/models.py
Normal file
73
gateway/app/models.py
Normal file
@@ -0,0 +1,73 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import datetime
|
||||
from enum import Enum
|
||||
from typing import Any
|
||||
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
|
||||
class QueryType(str, Enum):
|
||||
search = "search"
|
||||
vsearch = "vsearch"
|
||||
query = "query"
|
||||
deep_search = "deep_search"
|
||||
|
||||
|
||||
class QueryRequest(BaseModel):
|
||||
branch: str | None = None
|
||||
memory_profile: str | None = None
|
||||
query_type: QueryType = QueryType.query
|
||||
query: str = Field(min_length=1)
|
||||
require_latest: bool = True
|
||||
n: int = Field(default=5, ge=1, le=50)
|
||||
debug: bool = False
|
||||
|
||||
|
||||
class SyncRequest(BaseModel):
|
||||
branch: str | None = None
|
||||
memory_profile: str | None = None
|
||||
require_latest: bool = True
|
||||
|
||||
|
||||
class QueryResponse(BaseModel):
|
||||
ok: bool
|
||||
branch: str
|
||||
resolved_workspace: str
|
||||
commit_hash: str
|
||||
synced_at: datetime
|
||||
query_type: QueryType
|
||||
results: Any
|
||||
qmd_collection: str
|
||||
debug: dict[str, Any] | None = None
|
||||
|
||||
|
||||
class SyncResponse(BaseModel):
|
||||
ok: bool
|
||||
branch: str
|
||||
resolved_workspace: str
|
||||
commit_hash: str
|
||||
synced_at: datetime
|
||||
qmd_collection: str
|
||||
debug: dict[str, Any] | None = None
|
||||
|
||||
|
||||
class HealthResponse(BaseModel):
|
||||
ok: bool
|
||||
service: str
|
||||
timestamp: datetime
|
||||
|
||||
|
||||
class WorkspaceStatusItem(BaseModel):
|
||||
branch: str
|
||||
workspace: str
|
||||
commit_hash: str
|
||||
synced_at: datetime
|
||||
qmd_collection: str
|
||||
|
||||
|
||||
class StatusResponse(BaseModel):
|
||||
ok: bool
|
||||
default_branch: str
|
||||
mirror_path: str
|
||||
workspaces: list[WorkspaceStatusItem]
|
||||
175
gateway/app/qmd_client.py
Normal file
175
gateway/app/qmd_client.py
Normal file
@@ -0,0 +1,175 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import re
|
||||
import shlex
|
||||
import subprocess
|
||||
from typing import Any
|
||||
|
||||
from .models import QueryType
|
||||
|
||||
|
||||
class QMDCommandError(RuntimeError):
|
||||
def __init__(self, args: list[str], returncode: int, stdout: str, stderr: str) -> None:
|
||||
cmd = " ".join(shlex.quote(x) for x in args)
|
||||
message = f"qmd command failed ({returncode}): {cmd}\nstdout:\n{stdout}\nstderr:\n{stderr}"
|
||||
super().__init__(message)
|
||||
self.args_list = args
|
||||
self.returncode = returncode
|
||||
self.stdout = stdout
|
||||
self.stderr = stderr
|
||||
|
||||
|
||||
class QMDClient:
|
||||
def __init__(
|
||||
self,
|
||||
qmd_binary: str,
|
||||
timeout_seconds: int,
|
||||
xdg_cache_home: Path,
|
||||
xdg_config_home: Path,
|
||||
) -> None:
|
||||
self.qmd_binary = qmd_binary
|
||||
self.timeout_seconds = timeout_seconds
|
||||
self.xdg_cache_home = xdg_cache_home
|
||||
self.xdg_config_home = xdg_config_home
|
||||
|
||||
def ensure_collection(self, index_name: str, collection_name: str, workspace_path: Path) -> bool:
|
||||
existing = self.list_collections(index_name=index_name)
|
||||
if collection_name in existing:
|
||||
return False
|
||||
|
||||
self._run(
|
||||
[
|
||||
self.qmd_binary,
|
||||
"--index",
|
||||
index_name,
|
||||
"collection",
|
||||
"add",
|
||||
str(workspace_path),
|
||||
"--name",
|
||||
collection_name,
|
||||
]
|
||||
)
|
||||
return True
|
||||
|
||||
def list_collections(self, index_name: str) -> set[str]:
|
||||
output = self._run([self.qmd_binary, "--index", index_name, "collection", "list"])
|
||||
names: set[str] = set()
|
||||
for line in output.splitlines():
|
||||
match = re.match(r"^([A-Za-z0-9_.-]+) \(qmd://", line.strip())
|
||||
if match:
|
||||
names.add(match.group(1))
|
||||
return names
|
||||
|
||||
def update_workspace(self, index_name: str) -> None:
|
||||
self._run([self.qmd_binary, "--index", index_name, "update"])
|
||||
|
||||
def embed_workspace_if_needed(self, index_name: str, should_embed: bool) -> bool:
|
||||
if not should_embed:
|
||||
return False
|
||||
self._run([self.qmd_binary, "--index", index_name, "embed"])
|
||||
return True
|
||||
|
||||
def run_query(
|
||||
self,
|
||||
*,
|
||||
index_name: str,
|
||||
collection_name: str,
|
||||
query_type: QueryType,
|
||||
query: str,
|
||||
n: int,
|
||||
) -> tuple[Any, str]:
|
||||
command_name = self._query_command_name(query_type)
|
||||
top_k = max(n, 10) if query_type == QueryType.deep_search else n
|
||||
args = self._query_args(
|
||||
index_name=index_name,
|
||||
command_name=command_name,
|
||||
query=query,
|
||||
top_k=top_k,
|
||||
collection_name=collection_name,
|
||||
)
|
||||
try:
|
||||
output = self._run(args)
|
||||
return self._parse_output(output), command_name
|
||||
except QMDCommandError:
|
||||
# Query/deep_search may fail when LLM stack is not ready; keep API available.
|
||||
if query_type not in {QueryType.query, QueryType.deep_search}:
|
||||
raise
|
||||
fallback_command = "search"
|
||||
fallback_args = self._query_args(
|
||||
index_name=index_name,
|
||||
command_name=fallback_command,
|
||||
query=query,
|
||||
top_k=top_k,
|
||||
collection_name=collection_name,
|
||||
)
|
||||
fallback_output = self._run(fallback_args)
|
||||
return self._parse_output(fallback_output), fallback_command
|
||||
|
||||
def _query_command_name(self, query_type: QueryType) -> str:
|
||||
if query_type in {QueryType.query, QueryType.deep_search}:
|
||||
# Keep request latency bounded when LLM stack is not prewarmed.
|
||||
return "search"
|
||||
return query_type.value
|
||||
|
||||
def _parse_output(self, output: str) -> Any:
|
||||
stripped = output.strip()
|
||||
if not stripped:
|
||||
return []
|
||||
try:
|
||||
return json.loads(stripped)
|
||||
except json.JSONDecodeError:
|
||||
return {"raw": stripped}
|
||||
|
||||
def _query_args(
|
||||
self,
|
||||
*,
|
||||
index_name: str,
|
||||
command_name: str,
|
||||
query: str,
|
||||
top_k: int,
|
||||
collection_name: str,
|
||||
) -> list[str]:
|
||||
return [
|
||||
self.qmd_binary,
|
||||
"--index",
|
||||
index_name,
|
||||
command_name,
|
||||
query,
|
||||
"--json",
|
||||
"-n",
|
||||
str(top_k),
|
||||
"-c",
|
||||
collection_name,
|
||||
]
|
||||
|
||||
def _run(self, args: list[str]) -> str:
|
||||
env = {
|
||||
**os.environ,
|
||||
"XDG_CACHE_HOME": str(self.xdg_cache_home),
|
||||
"XDG_CONFIG_HOME": str(self.xdg_config_home),
|
||||
}
|
||||
try:
|
||||
proc = subprocess.run(
|
||||
args,
|
||||
check=False,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
env=env,
|
||||
timeout=self.timeout_seconds,
|
||||
)
|
||||
except subprocess.TimeoutExpired as exc:
|
||||
stdout = exc.stdout if isinstance(exc.stdout, str) else ""
|
||||
stderr = exc.stderr if isinstance(exc.stderr, str) else ""
|
||||
raise QMDCommandError(
|
||||
args=args,
|
||||
returncode=124,
|
||||
stdout=stdout,
|
||||
stderr=f"{stderr}\nTimeout after {self.timeout_seconds}s",
|
||||
) from exc
|
||||
|
||||
if proc.returncode != 0:
|
||||
raise QMDCommandError(args=args, returncode=proc.returncode, stdout=proc.stdout, stderr=proc.stderr)
|
||||
return proc.stdout
|
||||
98
gateway/app/query_service.py
Normal file
98
gateway/app/query_service.py
Normal file
@@ -0,0 +1,98 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any
|
||||
|
||||
from .locks import WorkspaceLockManager
|
||||
from .models import QueryRequest, QueryResponse, SyncRequest, SyncResponse
|
||||
from .qmd_client import QMDClient
|
||||
from .sync_service import SyncService
|
||||
|
||||
|
||||
class QueryService:
|
||||
def __init__(
|
||||
self,
|
||||
*,
|
||||
sync_service: SyncService,
|
||||
qmd_client: QMDClient,
|
||||
lock_manager: WorkspaceLockManager,
|
||||
) -> None:
|
||||
self.sync_service = sync_service
|
||||
self.qmd_client = qmd_client
|
||||
self.lock_manager = lock_manager
|
||||
|
||||
def handle_query(self, request: QueryRequest) -> QueryResponse:
|
||||
resolved = self.sync_service.workspace_manager.resolve_workspace(
|
||||
branch=request.branch,
|
||||
memory_profile=request.memory_profile,
|
||||
)
|
||||
|
||||
with self.lock_manager.acquire(resolved.workspace_name) as lock_stats:
|
||||
sync_meta = self.sync_service.sync_for_query(
|
||||
branch=resolved.branch,
|
||||
memory_profile=None,
|
||||
require_latest=request.require_latest,
|
||||
)
|
||||
|
||||
results, qmd_command = self.qmd_client.run_query(
|
||||
index_name=sync_meta.qmd_index,
|
||||
collection_name=sync_meta.qmd_collection,
|
||||
query_type=request.query_type,
|
||||
query=request.query,
|
||||
n=request.n,
|
||||
)
|
||||
|
||||
debug: dict[str, Any] | None = None
|
||||
if request.debug:
|
||||
debug = {
|
||||
"lock_wait_ms": lock_stats.wait_ms,
|
||||
"workspace_changed": sync_meta.changed,
|
||||
"previous_commit": sync_meta.previous_commit,
|
||||
"qmd_index": sync_meta.qmd_index,
|
||||
"qmd_command": qmd_command,
|
||||
"update_ran": sync_meta.update_ran,
|
||||
"embed_ran": sync_meta.embed_ran,
|
||||
"embed_error": sync_meta.embed_error,
|
||||
}
|
||||
|
||||
return QueryResponse(
|
||||
ok=True,
|
||||
branch=sync_meta.branch,
|
||||
resolved_workspace=str(sync_meta.workspace_path),
|
||||
commit_hash=sync_meta.commit_hash,
|
||||
synced_at=sync_meta.synced_at,
|
||||
query_type=request.query_type,
|
||||
results=results,
|
||||
qmd_collection=sync_meta.qmd_collection,
|
||||
debug=debug,
|
||||
)
|
||||
|
||||
def handle_sync(self, request: SyncRequest) -> SyncResponse:
|
||||
resolved = self.sync_service.workspace_manager.resolve_workspace(
|
||||
branch=request.branch,
|
||||
memory_profile=request.memory_profile,
|
||||
)
|
||||
|
||||
with self.lock_manager.acquire(resolved.workspace_name) as lock_stats:
|
||||
sync_meta = self.sync_service.sync_explicit(
|
||||
branch=resolved.branch,
|
||||
memory_profile=None,
|
||||
require_latest=request.require_latest,
|
||||
)
|
||||
debug = {
|
||||
"lock_wait_ms": lock_stats.wait_ms,
|
||||
"workspace_changed": sync_meta.changed,
|
||||
"previous_commit": sync_meta.previous_commit,
|
||||
"qmd_index": sync_meta.qmd_index,
|
||||
"update_ran": sync_meta.update_ran,
|
||||
"embed_ran": sync_meta.embed_ran,
|
||||
"embed_error": sync_meta.embed_error,
|
||||
}
|
||||
return SyncResponse(
|
||||
ok=True,
|
||||
branch=sync_meta.branch,
|
||||
resolved_workspace=str(sync_meta.workspace_path),
|
||||
commit_hash=sync_meta.commit_hash,
|
||||
synced_at=sync_meta.synced_at,
|
||||
qmd_collection=sync_meta.qmd_collection,
|
||||
debug=debug,
|
||||
)
|
||||
161
gateway/app/sync_service.py
Normal file
161
gateway/app/sync_service.py
Normal file
@@ -0,0 +1,161 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import asdict, dataclass
|
||||
from datetime import datetime, timezone
|
||||
import json
|
||||
from pathlib import Path
|
||||
import threading
|
||||
from typing import Any
|
||||
|
||||
from .config import Settings
|
||||
from .git_manager import GitCommandError, GitManager
|
||||
from .qmd_client import QMDClient, QMDCommandError
|
||||
from .workspace_manager import ResolvedWorkspace, WorkspaceManager
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class SyncMetadata:
|
||||
branch: str
|
||||
workspace_name: str
|
||||
workspace_path: Path
|
||||
qmd_collection: str
|
||||
qmd_index: str
|
||||
commit_hash: str
|
||||
previous_commit: str | None
|
||||
changed: bool
|
||||
synced_at: datetime
|
||||
created_collection: bool
|
||||
update_ran: bool
|
||||
embed_ran: bool
|
||||
embed_error: str | None
|
||||
|
||||
|
||||
class SyncService:
|
||||
def __init__(
|
||||
self,
|
||||
settings: Settings,
|
||||
workspace_manager: WorkspaceManager,
|
||||
git_manager: GitManager,
|
||||
qmd_client: QMDClient,
|
||||
) -> None:
|
||||
self.settings = settings
|
||||
self.workspace_manager = workspace_manager
|
||||
self.git_manager = git_manager
|
||||
self.qmd_client = qmd_client
|
||||
self._mirror_lock = threading.Lock()
|
||||
|
||||
self.settings.workspace_state_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
def sync_for_query(
|
||||
self,
|
||||
*,
|
||||
branch: str | None,
|
||||
memory_profile: str | None,
|
||||
require_latest: bool,
|
||||
) -> SyncMetadata:
|
||||
resolved = self.workspace_manager.resolve_workspace(branch=branch, memory_profile=memory_profile)
|
||||
return self._sync_resolved_workspace(resolved=resolved, require_latest=require_latest)
|
||||
|
||||
def sync_explicit(
|
||||
self,
|
||||
*,
|
||||
branch: str | None,
|
||||
memory_profile: str | None,
|
||||
require_latest: bool,
|
||||
) -> SyncMetadata:
|
||||
resolved = self.workspace_manager.resolve_workspace(branch=branch, memory_profile=memory_profile)
|
||||
return self._sync_resolved_workspace(resolved=resolved, require_latest=require_latest)
|
||||
|
||||
def list_workspace_states(self) -> list[dict[str, Any]]:
|
||||
results: list[dict[str, Any]] = []
|
||||
for path in sorted(self.settings.workspace_state_dir.glob("*.json")):
|
||||
try:
|
||||
payload = json.loads(path.read_text(encoding="utf-8"))
|
||||
except (json.JSONDecodeError, OSError):
|
||||
continue
|
||||
results.append(payload)
|
||||
return results
|
||||
|
||||
def _sync_resolved_workspace(self, *, resolved: ResolvedWorkspace, require_latest: bool) -> SyncMetadata:
|
||||
with self._mirror_lock:
|
||||
self.git_manager.ensure_mirror()
|
||||
if require_latest:
|
||||
self.git_manager.fetch_origin()
|
||||
commit_hash = self._resolve_branch_commit(resolved.branch, force_fetch=not require_latest)
|
||||
|
||||
git_sync = self.git_manager.sync_workspace(
|
||||
workspace_path=resolved.workspace_path,
|
||||
branch=resolved.branch,
|
||||
commit_hash=commit_hash,
|
||||
)
|
||||
|
||||
created_collection = self.qmd_client.ensure_collection(
|
||||
index_name=resolved.qmd_index,
|
||||
collection_name=resolved.qmd_collection,
|
||||
workspace_path=resolved.workspace_path,
|
||||
)
|
||||
|
||||
update_ran = require_latest and self.settings.qmd_update_on_latest_query
|
||||
if update_ran:
|
||||
self.qmd_client.update_workspace(index_name=resolved.qmd_index)
|
||||
|
||||
should_embed = self.settings.qmd_embed_on_change and (git_sync.changed or created_collection)
|
||||
embed_ran = False
|
||||
embed_error: str | None = None
|
||||
if update_ran:
|
||||
try:
|
||||
embed_ran = self.qmd_client.embed_workspace_if_needed(
|
||||
index_name=resolved.qmd_index,
|
||||
should_embed=should_embed,
|
||||
)
|
||||
except QMDCommandError as exc:
|
||||
embed_ran = False
|
||||
embed_error = str(exc)
|
||||
|
||||
synced_at = datetime.now(timezone.utc)
|
||||
|
||||
metadata = SyncMetadata(
|
||||
branch=resolved.branch,
|
||||
workspace_name=resolved.workspace_name,
|
||||
workspace_path=resolved.workspace_path,
|
||||
qmd_collection=resolved.qmd_collection,
|
||||
qmd_index=resolved.qmd_index,
|
||||
commit_hash=git_sync.commit_hash,
|
||||
previous_commit=git_sync.previous_commit,
|
||||
changed=git_sync.changed,
|
||||
synced_at=synced_at,
|
||||
created_collection=created_collection,
|
||||
update_ran=update_ran,
|
||||
embed_ran=embed_ran,
|
||||
embed_error=embed_error,
|
||||
)
|
||||
self._write_state(metadata)
|
||||
return metadata
|
||||
|
||||
def _resolve_branch_commit(self, branch: str, force_fetch: bool) -> str:
|
||||
try:
|
||||
return self.git_manager.get_branch_commit(branch)
|
||||
except GitCommandError:
|
||||
if not force_fetch:
|
||||
raise
|
||||
self.git_manager.fetch_origin()
|
||||
return self.git_manager.get_branch_commit(branch)
|
||||
|
||||
def _write_state(self, metadata: SyncMetadata) -> None:
|
||||
path = self.settings.workspace_state_dir / f"{metadata.workspace_name}.json"
|
||||
payload = {
|
||||
"branch": metadata.branch,
|
||||
"workspace": metadata.workspace_name,
|
||||
"workspace_path": str(metadata.workspace_path),
|
||||
"qmd_collection": metadata.qmd_collection,
|
||||
"qmd_index": metadata.qmd_index,
|
||||
"commit_hash": metadata.commit_hash,
|
||||
"previous_commit": metadata.previous_commit,
|
||||
"changed": metadata.changed,
|
||||
"created_collection": metadata.created_collection,
|
||||
"update_ran": metadata.update_ran,
|
||||
"embed_ran": metadata.embed_ran,
|
||||
"embed_error": metadata.embed_error,
|
||||
"synced_at": metadata.synced_at.isoformat(),
|
||||
}
|
||||
path.write_text(json.dumps(payload, ensure_ascii=True, indent=2), encoding="utf-8")
|
||||
79
gateway/app/workspace_manager.py
Normal file
79
gateway/app/workspace_manager.py
Normal file
@@ -0,0 +1,79 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
import re
|
||||
|
||||
|
||||
_PROFILE_MONTHLY_RE = re.compile(r"^monthly-(\d{4}-\d{2})$")
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class ResolvedWorkspace:
|
||||
branch: str
|
||||
workspace_name: str
|
||||
workspace_path: Path
|
||||
qmd_collection: str
|
||||
qmd_index: str
|
||||
|
||||
|
||||
class WorkspaceManager:
|
||||
def __init__(self, workspaces_root: Path, default_branch: str = "main", qmd_index_prefix: str = "ws") -> None:
|
||||
self.workspaces_root = workspaces_root
|
||||
self.default_branch = default_branch
|
||||
self.qmd_index_prefix = qmd_index_prefix
|
||||
self.workspaces_root.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
def resolve_branch(self, branch: str | None, memory_profile: str | None) -> str:
|
||||
if branch and branch.strip():
|
||||
return branch.strip()
|
||||
if memory_profile and memory_profile.strip():
|
||||
return self._profile_to_branch(memory_profile.strip())
|
||||
return self.default_branch
|
||||
|
||||
def resolve_workspace(self, branch: str | None, memory_profile: str | None) -> ResolvedWorkspace:
|
||||
resolved_branch = self.resolve_branch(branch=branch, memory_profile=memory_profile)
|
||||
workspace_name = self.workspace_name_for_branch(resolved_branch)
|
||||
workspace_path = self.workspaces_root / workspace_name
|
||||
qmd_collection = self.collection_name_for_workspace(workspace_name)
|
||||
qmd_index = self.index_name_for_workspace(workspace_name)
|
||||
return ResolvedWorkspace(
|
||||
branch=resolved_branch,
|
||||
workspace_name=workspace_name,
|
||||
workspace_path=workspace_path,
|
||||
qmd_collection=qmd_collection,
|
||||
qmd_index=qmd_index,
|
||||
)
|
||||
|
||||
def _profile_to_branch(self, memory_profile: str) -> str:
|
||||
if memory_profile == "stable":
|
||||
return "main"
|
||||
monthly = _PROFILE_MONTHLY_RE.match(memory_profile)
|
||||
if monthly:
|
||||
return f"memory/{monthly.group(1)}"
|
||||
if memory_profile.startswith("task-") and len(memory_profile) > len("task-"):
|
||||
return f"task/{memory_profile[len('task-') :]}"
|
||||
raise ValueError(
|
||||
"unsupported memory_profile, expected stable | monthly-YYYY-MM | task-<task-id>"
|
||||
)
|
||||
|
||||
def workspace_name_for_branch(self, branch: str) -> str:
|
||||
if branch in {"main", "stable"}:
|
||||
return "main"
|
||||
if branch.startswith("memory/"):
|
||||
return f"memory-{branch.split('/', 1)[1].replace('/', '-') }"
|
||||
if branch.startswith("task/"):
|
||||
return f"task-{branch.split('/', 1)[1].replace('/', '-') }"
|
||||
|
||||
return "branch-" + self._slugify(branch)
|
||||
|
||||
def collection_name_for_workspace(self, workspace_name: str) -> str:
|
||||
return self._slugify(workspace_name)
|
||||
|
||||
def index_name_for_workspace(self, workspace_name: str) -> str:
|
||||
return f"{self.qmd_index_prefix}_{self._slugify(workspace_name)}"
|
||||
|
||||
def _slugify(self, value: str) -> str:
|
||||
cleaned = re.sub(r"[^A-Za-z0-9_.-]+", "-", value)
|
||||
cleaned = cleaned.strip("-")
|
||||
return cleaned or "default"
|
||||
3
gateway/requirements.txt
Normal file
3
gateway/requirements.txt
Normal file
@@ -0,0 +1,3 @@
|
||||
fastapi==0.116.2
|
||||
uvicorn[standard]==0.35.0
|
||||
pydantic-settings==2.11.0
|
||||
135
gateway/tests/conftest.py
Normal file
135
gateway/tests/conftest.py
Normal file
@@ -0,0 +1,135 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
import os
|
||||
from pathlib import Path
|
||||
import subprocess
|
||||
from typing import Any
|
||||
|
||||
import pytest
|
||||
from fastapi.testclient import TestClient
|
||||
|
||||
# Keep module-level app import from trying to write to /data on local tests.
|
||||
os.environ.setdefault("WORKSPACES_ROOT", "/tmp/qmd-gateway-tests/workspaces")
|
||||
os.environ.setdefault("WORKSPACE_STATE_DIR", "/tmp/qmd-gateway-tests/state")
|
||||
os.environ.setdefault("GIT_MIRROR_PATH", "/tmp/qmd-gateway-tests/git-mirror/repo.git")
|
||||
os.environ.setdefault("GIT_REMOTE_URL", "/tmp/qmd-gateway-tests/remote.git")
|
||||
|
||||
from app.config import Settings
|
||||
from app.main import create_app
|
||||
from app.models import QueryType
|
||||
|
||||
|
||||
class FakeQMDClient:
|
||||
def __init__(self) -> None:
|
||||
self.collections: dict[str, dict[str, Path]] = {}
|
||||
self.update_calls: list[str] = []
|
||||
self.embed_calls: list[str] = []
|
||||
|
||||
def ensure_collection(self, index_name: str, collection_name: str, workspace_path: Path) -> bool:
|
||||
bucket = self.collections.setdefault(index_name, {})
|
||||
if collection_name in bucket:
|
||||
return False
|
||||
bucket[collection_name] = workspace_path
|
||||
return True
|
||||
|
||||
def list_collections(self, index_name: str) -> set[str]:
|
||||
return set(self.collections.get(index_name, {}).keys())
|
||||
|
||||
def update_workspace(self, index_name: str) -> None:
|
||||
self.update_calls.append(index_name)
|
||||
|
||||
def embed_workspace_if_needed(self, index_name: str, should_embed: bool) -> bool:
|
||||
if should_embed:
|
||||
self.embed_calls.append(index_name)
|
||||
return True
|
||||
return False
|
||||
|
||||
def run_query(self, *, index_name: str, collection_name: str, query_type: QueryType, query: str, n: int):
|
||||
workspace = self.collections[index_name][collection_name]
|
||||
needle = query.lower()
|
||||
matches: list[dict[str, Any]] = []
|
||||
for path in sorted(workspace.rglob("*.md")):
|
||||
text = path.read_text(encoding="utf-8")
|
||||
if needle in text.lower() or needle in path.name.lower():
|
||||
matches.append({"file": str(path), "snippet": text[:120]})
|
||||
return matches[:n], query_type.value
|
||||
|
||||
|
||||
@dataclass
|
||||
class TestRepo:
|
||||
remote_path: Path
|
||||
seed_path: Path
|
||||
|
||||
def commit_on_branch(self, branch: str, rel_path: str, content: str, message: str) -> str:
|
||||
_run(["git", "-C", str(self.seed_path), "checkout", branch])
|
||||
file_path = self.seed_path / rel_path
|
||||
file_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
file_path.write_text(content, encoding="utf-8")
|
||||
_run(["git", "-C", str(self.seed_path), "add", rel_path])
|
||||
_run(["git", "-C", str(self.seed_path), "commit", "-m", message])
|
||||
_run(["git", "-C", str(self.seed_path), "push", "origin", branch])
|
||||
commit = _run(["git", "-C", str(self.seed_path), "rev-parse", "HEAD"]).strip()
|
||||
return commit
|
||||
|
||||
|
||||
@pytest.fixture()
|
||||
def repo(tmp_path: Path) -> TestRepo:
|
||||
remote = tmp_path / "remote.git"
|
||||
seed = tmp_path / "seed"
|
||||
|
||||
_run(["git", "init", "--bare", str(remote)])
|
||||
_run(["git", "clone", str(remote), str(seed)])
|
||||
_run(["git", "-C", str(seed), "config", "user.name", "Test User"])
|
||||
_run(["git", "-C", str(seed), "config", "user.email", "test@example.com"])
|
||||
|
||||
(seed / "README.md").write_text("main branch memory root\n", encoding="utf-8")
|
||||
(seed / "docs" / "main-only.md").parent.mkdir(parents=True, exist_ok=True)
|
||||
(seed / "docs" / "main-only.md").write_text("alpha-main-signal\n", encoding="utf-8")
|
||||
_run(["git", "-C", str(seed), "add", "README.md", "docs/main-only.md"])
|
||||
_run(["git", "-C", str(seed), "commit", "-m", "init main"])
|
||||
_run(["git", "-C", str(seed), "branch", "-M", "main"])
|
||||
_run(["git", "-C", str(seed), "push", "-u", "origin", "main"])
|
||||
|
||||
_run(["git", "-C", str(seed), "checkout", "-b", "memory/2026-03"])
|
||||
(seed / "docs" / "monthly-only.md").write_text(
|
||||
"beta-monthly-signal\nmonthly-exclusive-signal\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
_run(["git", "-C", str(seed), "add", "docs/monthly-only.md"])
|
||||
_run(["git", "-C", str(seed), "commit", "-m", "add monthly"])
|
||||
_run(["git", "-C", str(seed), "push", "-u", "origin", "memory/2026-03"])
|
||||
|
||||
_run(["git", "-C", str(seed), "checkout", "main"])
|
||||
(seed / "docs" / "main-exclusive.md").write_text("main-exclusive-signal\n", encoding="utf-8")
|
||||
_run(["git", "-C", str(seed), "add", "docs/main-exclusive.md"])
|
||||
_run(["git", "-C", str(seed), "commit", "-m", "add main exclusive"])
|
||||
_run(["git", "-C", str(seed), "push", "origin", "main"])
|
||||
|
||||
return TestRepo(remote_path=remote, seed_path=seed)
|
||||
|
||||
|
||||
@pytest.fixture()
|
||||
def fake_qmd() -> FakeQMDClient:
|
||||
return FakeQMDClient()
|
||||
|
||||
|
||||
@pytest.fixture()
|
||||
def client(tmp_path: Path, repo: TestRepo, fake_qmd: FakeQMDClient) -> TestClient:
|
||||
settings = Settings(
|
||||
git_remote_url=str(repo.remote_path),
|
||||
git_mirror_path=tmp_path / "git-mirror" / "repo.git",
|
||||
workspaces_root=tmp_path / "workspaces",
|
||||
workspace_state_dir=tmp_path / "state",
|
||||
xdg_cache_home=tmp_path / "cache",
|
||||
xdg_config_home=tmp_path / "config",
|
||||
)
|
||||
app = create_app(settings=settings, qmd_client=fake_qmd)
|
||||
return TestClient(app)
|
||||
|
||||
|
||||
def _run(args: list[str]) -> str:
|
||||
proc = subprocess.run(args, check=False, capture_output=True, text=True)
|
||||
if proc.returncode != 0:
|
||||
raise RuntimeError(f"command failed: {' '.join(args)}\nstdout={proc.stdout}\nstderr={proc.stderr}")
|
||||
return proc.stdout
|
||||
51
gateway/tests/test_branch_isolation.py
Normal file
51
gateway/tests/test_branch_isolation.py
Normal file
@@ -0,0 +1,51 @@
|
||||
from __future__ import annotations
|
||||
|
||||
|
||||
def test_branch_isolation(client):
|
||||
main_resp = client.post(
|
||||
"/query",
|
||||
json={"branch": "main", "query_type": "search", "query": "main-exclusive-signal", "require_latest": True},
|
||||
)
|
||||
assert main_resp.status_code == 200
|
||||
main_payload = main_resp.json()
|
||||
assert main_payload["branch"] == "main"
|
||||
assert main_payload["results"]
|
||||
|
||||
monthly_resp = client.post(
|
||||
"/query",
|
||||
json={"branch": "memory/2026-03", "query_type": "search", "query": "monthly-exclusive-signal", "require_latest": True},
|
||||
)
|
||||
assert monthly_resp.status_code == 200
|
||||
monthly_payload = monthly_resp.json()
|
||||
assert monthly_payload["branch"] == "memory/2026-03"
|
||||
assert monthly_payload["results"]
|
||||
|
||||
cross_main = client.post(
|
||||
"/query",
|
||||
json={"branch": "main", "query_type": "search", "query": "monthly-exclusive-signal", "require_latest": True},
|
||||
)
|
||||
assert cross_main.status_code == 200
|
||||
assert cross_main.json()["results"] == []
|
||||
|
||||
cross_monthly = client.post(
|
||||
"/query",
|
||||
json={"branch": "memory/2026-03", "query_type": "search", "query": "main-exclusive-signal", "require_latest": True},
|
||||
)
|
||||
assert cross_monthly.status_code == 200
|
||||
assert cross_monthly.json()["results"] == []
|
||||
|
||||
|
||||
def test_memory_profile_and_default_branch(client):
|
||||
profile_resp = client.post(
|
||||
"/query",
|
||||
json={"memory_profile": "monthly-2026-03", "query_type": "search", "query": "monthly-exclusive-signal", "require_latest": True},
|
||||
)
|
||||
assert profile_resp.status_code == 200
|
||||
assert profile_resp.json()["branch"] == "memory/2026-03"
|
||||
|
||||
default_resp = client.post(
|
||||
"/query",
|
||||
json={"query_type": "search", "query": "main-exclusive-signal", "require_latest": True},
|
||||
)
|
||||
assert default_resp.status_code == 200
|
||||
assert default_resp.json()["branch"] == "main"
|
||||
9
gateway/tests/test_health.py
Normal file
9
gateway/tests/test_health.py
Normal file
@@ -0,0 +1,9 @@
|
||||
from __future__ import annotations
|
||||
|
||||
|
||||
def test_health(client):
|
||||
resp = client.get("/health")
|
||||
assert resp.status_code == 200
|
||||
payload = resp.json()
|
||||
assert payload["ok"] is True
|
||||
assert payload["service"] == "memory-gateway"
|
||||
43
gateway/tests/test_query_flow.py
Normal file
43
gateway/tests/test_query_flow.py
Normal file
@@ -0,0 +1,43 @@
|
||||
from __future__ import annotations
|
||||
|
||||
|
||||
def test_query_flow_and_sync_before_query(client, repo):
|
||||
first = client.post(
|
||||
"/query",
|
||||
json={
|
||||
"branch": "main",
|
||||
"query_type": "query",
|
||||
"query": "alpha-main-signal",
|
||||
"require_latest": True,
|
||||
},
|
||||
)
|
||||
assert first.status_code == 200
|
||||
first_payload = first.json()
|
||||
assert first_payload["ok"] is True
|
||||
assert first_payload["branch"] == "main"
|
||||
assert first_payload["commit_hash"]
|
||||
assert first_payload["synced_at"]
|
||||
assert isinstance(first_payload["results"], list)
|
||||
assert first_payload["results"]
|
||||
|
||||
repo.commit_on_branch(
|
||||
"main",
|
||||
"docs/new-sync-note.md",
|
||||
"gamma-sync-proof\n",
|
||||
"add sync proof",
|
||||
)
|
||||
|
||||
second = client.post(
|
||||
"/query",
|
||||
json={
|
||||
"branch": "main",
|
||||
"query_type": "query",
|
||||
"query": "gamma-sync-proof",
|
||||
"require_latest": True,
|
||||
},
|
||||
)
|
||||
assert second.status_code == 200
|
||||
second_payload = second.json()
|
||||
assert second_payload["commit_hash"] != first_payload["commit_hash"]
|
||||
snippets = [row["snippet"] for row in second_payload["results"]]
|
||||
assert any("gamma-sync-proof" in snippet for snippet in snippets)
|
||||
55
gateway/tests/test_sync_logic.py
Normal file
55
gateway/tests/test_sync_logic.py
Normal file
@@ -0,0 +1,55 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from concurrent.futures import ThreadPoolExecutor
|
||||
|
||||
|
||||
def test_sync_endpoint_and_status(client):
|
||||
sync_resp = client.post("/sync", json={"branch": "main", "require_latest": True})
|
||||
assert sync_resp.status_code == 200
|
||||
sync_payload = sync_resp.json()
|
||||
assert sync_payload["ok"] is True
|
||||
assert sync_payload["branch"] == "main"
|
||||
assert sync_payload["commit_hash"]
|
||||
assert sync_payload["synced_at"]
|
||||
|
||||
status_resp = client.get("/status")
|
||||
assert status_resp.status_code == 200
|
||||
status_payload = status_resp.json()
|
||||
assert status_payload["ok"] is True
|
||||
assert any(item["branch"] == "main" for item in status_payload["workspaces"])
|
||||
|
||||
|
||||
def test_concurrent_branch_queries(client):
|
||||
def run(body: dict):
|
||||
return client.post("/query", json=body)
|
||||
|
||||
with ThreadPoolExecutor(max_workers=2) as pool:
|
||||
fut_main = pool.submit(
|
||||
run,
|
||||
{"branch": "main", "query_type": "search", "query": "main-exclusive-signal", "require_latest": True},
|
||||
)
|
||||
fut_month = pool.submit(
|
||||
run,
|
||||
{
|
||||
"branch": "memory/2026-03",
|
||||
"query_type": "search",
|
||||
"query": "monthly-exclusive-signal",
|
||||
"require_latest": True,
|
||||
},
|
||||
)
|
||||
|
||||
resp_main = fut_main.result()
|
||||
resp_month = fut_month.result()
|
||||
|
||||
assert resp_main.status_code == 200
|
||||
assert resp_month.status_code == 200
|
||||
|
||||
payload_main = resp_main.json()
|
||||
payload_month = resp_month.json()
|
||||
|
||||
assert payload_main["branch"] == "main"
|
||||
assert payload_month["branch"] == "memory/2026-03"
|
||||
assert payload_main["resolved_workspace"] != payload_month["resolved_workspace"]
|
||||
|
||||
assert payload_main["results"]
|
||||
assert payload_month["results"]
|
||||
60
scripts/bootstrap.sh
Executable file
60
scripts/bootstrap.sh
Executable file
@@ -0,0 +1,60 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
ROOT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||
cd "${ROOT_DIR}"
|
||||
|
||||
mkdir -p data/git-mirror data/workspaces data/qmd-cache data/qmd-config
|
||||
|
||||
REMOTE_BARE="${ROOT_DIR}/data/remote-memory.git"
|
||||
if [[ ! -d "${REMOTE_BARE}" ]]; then
|
||||
echo "[bootstrap] initializing demo remote repo at ${REMOTE_BARE}"
|
||||
git init --bare "${REMOTE_BARE}"
|
||||
|
||||
tmp_dir="$(mktemp -d)"
|
||||
trap 'rm -rf "${tmp_dir}"' EXIT
|
||||
|
||||
git clone "${REMOTE_BARE}" "${tmp_dir}/seed"
|
||||
git -C "${tmp_dir}/seed" config user.name "Memory Gateway Bot"
|
||||
git -C "${tmp_dir}/seed" config user.email "memory-gateway@example.local"
|
||||
|
||||
mkdir -p "${tmp_dir}/seed/docs"
|
||||
cat > "${tmp_dir}/seed/docs/main.md" <<'EOF'
|
||||
# Main Memory
|
||||
|
||||
router firmware build recovery strategy in main branch.
|
||||
EOF
|
||||
|
||||
git -C "${tmp_dir}/seed" add docs/main.md
|
||||
git -C "${tmp_dir}/seed" commit -m "init main memory"
|
||||
git -C "${tmp_dir}/seed" branch -M main
|
||||
git -C "${tmp_dir}/seed" push -u origin main
|
||||
|
||||
git -C "${tmp_dir}/seed" checkout -b memory/2026-03
|
||||
cat > "${tmp_dir}/seed/docs/monthly.md" <<'EOF'
|
||||
# Monthly Memory 2026-03
|
||||
|
||||
monthly branch note for March 2026.
|
||||
EOF
|
||||
git -C "${tmp_dir}/seed" add docs/monthly.md
|
||||
git -C "${tmp_dir}/seed" commit -m "add monthly memory"
|
||||
git -C "${tmp_dir}/seed" push -u origin memory/2026-03
|
||||
|
||||
git -C "${tmp_dir}/seed" checkout -b task/TASK-001 main
|
||||
cat > "${tmp_dir}/seed/docs/task-TASK-001.md" <<'EOF'
|
||||
# Task Memory TASK-001
|
||||
|
||||
task specific recovery checkpoint.
|
||||
EOF
|
||||
git -C "${tmp_dir}/seed" add docs/task-TASK-001.md
|
||||
git -C "${tmp_dir}/seed" commit -m "add task memory"
|
||||
git -C "${tmp_dir}/seed" push -u origin task/TASK-001
|
||||
fi
|
||||
|
||||
if [[ ! -f .env ]]; then
|
||||
cp .env.example .env
|
||||
sed -i "s|^GIT_REMOTE_URL=.*$|GIT_REMOTE_URL=/data/remote-memory.git|" .env
|
||||
echo "[bootstrap] generated .env with local demo remote: /data/remote-memory.git"
|
||||
fi
|
||||
|
||||
echo "[bootstrap] done"
|
||||
4
scripts/entrypoint.sh
Executable file
4
scripts/entrypoint.sh
Executable file
@@ -0,0 +1,4 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
exec /app/scripts/qmd-entrypoint.sh "$@"
|
||||
65
scripts/qmd-entrypoint.sh
Executable file
65
scripts/qmd-entrypoint.sh
Executable file
@@ -0,0 +1,65 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
CACHE_HOME="${XDG_CACHE_HOME:-/var/lib/qmd/cache}"
|
||||
CONFIG_HOME="${XDG_CONFIG_HOME:-/var/lib/qmd/config}"
|
||||
PORT="${QMD_HTTP_PORT:-8181}"
|
||||
INTERNAL_PORT="${QMD_INTERNAL_PORT:-8182}"
|
||||
QMD_LLM_FILE="${QMD_LLM_FILE:-/usr/local/lib/node_modules/@tobilu/qmd/dist/llm.js}"
|
||||
|
||||
mkdir -p "${CACHE_HOME}/qmd" "${CONFIG_HOME}/qmd"
|
||||
|
||||
apply_model_overrides() {
|
||||
if [[ -z "${QMD_EMBED_MODEL_URI:-}" && -z "${QMD_RERANK_MODEL_URI:-}" && -z "${QMD_GENERATE_MODEL_URI:-}" ]]; then
|
||||
return 0
|
||||
fi
|
||||
|
||||
if [[ ! -f "${QMD_LLM_FILE}" ]]; then
|
||||
echo "WARN: llm.js not found at ${QMD_LLM_FILE}, skip model overrides." >&2
|
||||
return 0
|
||||
fi
|
||||
|
||||
QMD_LLM_FILE="${QMD_LLM_FILE}" node - <<'NODE'
|
||||
const fs = require("fs");
|
||||
const llmFile = process.env.QMD_LLM_FILE;
|
||||
let src = fs.readFileSync(llmFile, "utf8");
|
||||
|
||||
const replacements = [
|
||||
["DEFAULT_EMBED_MODEL", process.env.QMD_EMBED_MODEL_URI],
|
||||
["DEFAULT_RERANK_MODEL", process.env.QMD_RERANK_MODEL_URI],
|
||||
["DEFAULT_GENERATE_MODEL", process.env.QMD_GENERATE_MODEL_URI],
|
||||
];
|
||||
|
||||
for (const [name, value] of replacements) {
|
||||
if (!value) continue;
|
||||
const pattern = new RegExp(`const ${name} = \\"[^\\"]+\\";`);
|
||||
if (!pattern.test(src)) {
|
||||
console.error(`WARN: failed to find ${name} in ${llmFile}`);
|
||||
continue;
|
||||
}
|
||||
src = src.replace(pattern, `const ${name} = ${JSON.stringify(value)};`);
|
||||
}
|
||||
|
||||
fs.writeFileSync(llmFile, src, "utf8");
|
||||
NODE
|
||||
}
|
||||
|
||||
apply_model_overrides
|
||||
|
||||
qmd mcp --http --port "${INTERNAL_PORT}" &
|
||||
qmd_pid=$!
|
||||
|
||||
socat "TCP-LISTEN:${PORT},fork,reuseaddr,bind=0.0.0.0" "TCP6:[::1]:${INTERNAL_PORT}" &
|
||||
proxy_pid=$!
|
||||
|
||||
cleanup() {
|
||||
kill "${proxy_pid}" "${qmd_pid}" 2>/dev/null || true
|
||||
wait "${proxy_pid}" "${qmd_pid}" 2>/dev/null || true
|
||||
}
|
||||
|
||||
trap cleanup SIGINT SIGTERM
|
||||
|
||||
wait -n "${qmd_pid}" "${proxy_pid}"
|
||||
status=$?
|
||||
cleanup
|
||||
exit "${status}"
|
||||
29
scripts/smoke-test.sh
Executable file
29
scripts/smoke-test.sh
Executable file
@@ -0,0 +1,29 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
ROOT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||
cd "${ROOT_DIR}"
|
||||
|
||||
log() {
|
||||
printf '[smoke] %s\n' "$*"
|
||||
}
|
||||
|
||||
log "Bootstrapping demo repo"
|
||||
./scripts/bootstrap.sh
|
||||
|
||||
log "Starting services"
|
||||
docker compose up -d --build
|
||||
|
||||
log "Checking health"
|
||||
curl -fsS http://127.0.0.1:8181/health >/dev/null
|
||||
curl -fsS http://127.0.0.1:8787/health >/dev/null
|
||||
|
||||
log "Running query through memory-gateway"
|
||||
query_output="$(curl -fsS -X POST http://127.0.0.1:8787/query \
|
||||
-H 'content-type: application/json' \
|
||||
-d '{"branch":"main","query_type":"search","query":"router","require_latest":true,"debug":true}')"
|
||||
printf '%s\n' "${query_output}"
|
||||
printf '%s\n' "${query_output}" | grep -F 'commit_hash' >/dev/null
|
||||
printf '%s\n' "${query_output}" | grep -F 'branch' >/dev/null
|
||||
|
||||
log "Smoke test passed"
|
||||
27
scripts/warmup.sh
Executable file
27
scripts/warmup.sh
Executable file
@@ -0,0 +1,27 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
ROOT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||
cd "${ROOT_DIR}"
|
||||
|
||||
BASE_URL="${GATEWAY_BASE_URL:-http://127.0.0.1:8787}"
|
||||
|
||||
wait_health() {
|
||||
local attempts=60
|
||||
for _ in $(seq 1 "${attempts}"); do
|
||||
if curl -fsS "${BASE_URL}/health" >/dev/null 2>&1; then
|
||||
return 0
|
||||
fi
|
||||
sleep 2
|
||||
done
|
||||
echo "ERROR: gateway did not become healthy at ${BASE_URL}/health" >&2
|
||||
return 1
|
||||
}
|
||||
|
||||
wait_health
|
||||
|
||||
curl -fsS -X POST "${BASE_URL}/query" \
|
||||
-H 'content-type: application/json' \
|
||||
-d '{"branch":"main","query_type":"search","query":"router","require_latest":true,"debug":true}' >/dev/null
|
||||
|
||||
echo "Warmup completed."
|
||||
6
testdata/docs/disaster-recovery.md
vendored
Normal file
6
testdata/docs/disaster-recovery.md
vendored
Normal file
@@ -0,0 +1,6 @@
|
||||
# Disaster Recovery Policy
|
||||
|
||||
- Recovery Time Objective (RTO): 2 hours.
|
||||
- Recovery Point Objective (RPO): 15 minutes.
|
||||
- Test failover monthly.
|
||||
- Keep indexed meeting notes for decision traceability.
|
||||
8
testdata/docs/incident-runbook.md
vendored
Normal file
8
testdata/docs/incident-runbook.md
vendored
Normal file
@@ -0,0 +1,8 @@
|
||||
# Incident Runbook
|
||||
|
||||
When the QMD HTTP MCP service is degraded:
|
||||
|
||||
- Check `GET /health`.
|
||||
- Validate vector index status with `qmd status`.
|
||||
- Restart container only after collecting logs.
|
||||
- Escalate if query quality drops after embedding drift.
|
||||
8
testdata/meetings/2026-03-01-architecture.md
vendored
Normal file
8
testdata/meetings/2026-03-01-architecture.md
vendored
Normal file
@@ -0,0 +1,8 @@
|
||||
# Architecture Sync 2026-03-01
|
||||
|
||||
Decisions:
|
||||
|
||||
- Use QMD as internal shared MCP endpoint.
|
||||
- Keep HTTP endpoint on port 8181.
|
||||
- Persist cache and index to avoid repeated cold starts.
|
||||
- Confirm disaster recovery RTO target remains 2 hours.
|
||||
6
testdata/meetings/2026-03-02-ops.md
vendored
Normal file
6
testdata/meetings/2026-03-02-ops.md
vendored
Normal file
@@ -0,0 +1,6 @@
|
||||
# Ops Review 2026-03-02
|
||||
|
||||
Action items:
|
||||
|
||||
- Add smoke test covering health, collection add, embed, search, and query.
|
||||
- Validate that deep query can locate runbook and recovery policy.
|
||||
6
testdata/notes/ops-checklist.md
vendored
Normal file
6
testdata/notes/ops-checklist.md
vendored
Normal file
@@ -0,0 +1,6 @@
|
||||
# Ops Checklist
|
||||
|
||||
1. Verify MCP service health endpoint before release.
|
||||
2. Confirm embeddings are up to date.
|
||||
3. Keep cache volume persistent to avoid repeated model downloads.
|
||||
4. Keep runbook links in docs/incident-runbook.md.
|
||||
6
testdata/notes/personal-productivity.md
vendored
Normal file
6
testdata/notes/personal-productivity.md
vendored
Normal file
@@ -0,0 +1,6 @@
|
||||
# Personal Productivity Notes
|
||||
|
||||
- Daily review at 09:00.
|
||||
- Focus blocks: 90 minutes.
|
||||
- Keep a short incident checklist for local services.
|
||||
- Search keyword target: lightweight knowledge retrieval.
|
||||
Reference in New Issue
Block a user