572 lines
14 KiB
Markdown
572 lines
14 KiB
Markdown
# Woodpecker CI - S3 对象存储上传指南
|
||
|
||
本指南介绍如何在 Woodpecker CI 中使用 S3 插件将文件和文件夹上传到对象存储（兼容 S3 API 的存储服务，如 AWS S3、MinIO、RustFS 等）。
|
||
|
||
## 📋 目录
|
||
|
||
- [前置准备](#前置准备)
|
||
- [配置密钥](#配置密钥)
|
||
- [基础用法](#基础用法)
|
||
- [上传单个文件](#上传单个文件)
|
||
- [上传多个文件](#上传多个文件)
|
||
- [上传整个文件夹](#上传整个文件夹)
|
||
- [高级用法](#高级用法)
|
||
- [完整示例](#完整示例)
|
||
- [故障排查](#故障排查)
|
||
|
||
---
|
||
|
||
## 前置准备
|
||
|
||
### 1. 安装 plugin-s3
|
||
|
||
确保您的 Woodpecker Agent 环境中已安装 `plugin-s3`：
|
||
|
||
```bash
|
||
# 使用 Homebrew (macOS)
|
||
brew install plugin-s3
|
||
|
||
# 或使用 Go 安装
|
||
go install github.com/woodpecker-ci/plugin-s3@latest
|
||
|
||
# 验证安装
|
||
which plugin-s3
|
||
```
|
||
|
||
### 2. 准备 S3 凭证
|
||
|
||
您需要以下信息：
|
||
- **Access Key ID**: S3 访问密钥 ID
|
||
- **Secret Access Key**: S3 密钥
|
||
- **Bucket 名称**: 目标存储桶
|
||
- **Endpoint**: S3 服务端点（如使用 AWS S3 可省略）
|
||
- **Region**: 区域（如 `us-east-1`）
|
||
|
||
---
|
||
|
||
## 配置密钥
|
||
|
||
### 通过 Woodpecker UI 配置
|
||
|
||
1. 进入您的仓库页面
|
||
2. 点击 **Settings** → **Secrets**
|
||
3. 添加以下密钥：
|
||
|
||
| 密钥名称 | 说明 | 示例值 |
|
||
|---------|------|--------|
|
||
| `AWS_ACCESS_KEY_ID` | S3 访问密钥 ID | `AKIAIOSFODNN7EXAMPLE` |
|
||
| `AWS_SECRET_ACCESS_KEY` | S3 密钥 | `wJalrXUtnFEMI/K7MDENG/...` |
|
||
| `S3_BUCKET` | 存储桶名称 | `my-bucket` |
|
||
| `S3_ENDPOINT` | S3 端点（自建服务） | `https://s3.example.com:9000` |
|
||
| `AWS_DEFAULT_REGION` | AWS 区域 | `us-east-1` |
|
||
|
||
4. 确保勾选适当的事件类型（如 `push`、`manual`、`pull_request`）
|
||
|
||
### 通过 CLI 配置
|
||
|
||
```bash
|
||
# 添加 Access Key
|
||
woodpecker-cli repo secret add \
|
||
--repository your-org/your-repo \
|
||
--name AWS_ACCESS_KEY_ID \
|
||
--value "your-access-key-id"
|
||
|
||
# 添加 Secret Key
|
||
woodpecker-cli repo secret add \
|
||
--repository your-org/your-repo \
|
||
--name AWS_SECRET_ACCESS_KEY \
|
||
--value "your-secret-access-key"
|
||
|
||
# 添加 Bucket
|
||
woodpecker-cli repo secret add \
|
||
--repository your-org/your-repo \
|
||
--name S3_BUCKET \
|
||
--value "my-bucket"
|
||
|
||
# 添加 Endpoint (可选，用于自建 S3 服务)
|
||
woodpecker-cli repo secret add \
|
||
--repository your-org/your-repo \
|
||
--name S3_ENDPOINT \
|
||
--value "https://s3.example.com:9000"
|
||
|
||
# 添加 Region
|
||
woodpecker-cli repo secret add \
|
||
--repository your-org/your-repo \
|
||
--name AWS_DEFAULT_REGION \
|
||
--value "us-east-1"
|
||
```
|
||
|
||
---
|
||
|
||
## 基础用法
|
||
|
||
### 上传单个文件
|
||
|
||
上传单个文件到 S3 存储桶：
|
||
|
||
```yaml
|
||
steps:
|
||
- name: upload-single-file
|
||
image: /bin/zsh
|
||
environment:
|
||
AWS_ACCESS_KEY_ID:
|
||
from_secret: AWS_ACCESS_KEY_ID
|
||
AWS_SECRET_ACCESS_KEY:
|
||
from_secret: AWS_SECRET_ACCESS_KEY
|
||
S3_BUCKET:
|
||
from_secret: S3_BUCKET
|
||
S3_ENDPOINT:
|
||
from_secret: S3_ENDPOINT
|
||
commands:
|
||
- |
|
||
# 创建测试文件
|
||
echo "Hello from Woodpecker CI" > hello.txt
|
||
|
||
# 上传到 S3
|
||
export PLUGIN_SOURCE="hello.txt"
|
||
export PLUGIN_BUCKET="$S3_BUCKET"
|
||
export PLUGIN_TARGET="uploads/"
|
||
export PLUGIN_ENDPOINT="$S3_ENDPOINT"
|
||
export PLUGIN_PATH_STYLE=true
|
||
plugin-s3
|
||
```
|
||
|
||
**结果**: `hello.txt` 将被上传到 `s3://your-bucket/uploads/hello.txt`
|
||
|
||
---
|
||
|
||
### 上传多个文件
|
||
|
||
使用通配符上传多个文件：
|
||
|
||
```yaml
|
||
steps:
|
||
- name: upload-multiple-files
|
||
image: /bin/zsh
|
||
environment:
|
||
AWS_ACCESS_KEY_ID:
|
||
from_secret: AWS_ACCESS_KEY_ID
|
||
AWS_SECRET_ACCESS_KEY:
|
||
from_secret: AWS_SECRET_ACCESS_KEY
|
||
S3_BUCKET:
|
||
from_secret: S3_BUCKET
|
||
S3_ENDPOINT:
|
||
from_secret: S3_ENDPOINT
|
||
commands:
|
||
- |
|
||
# 创建多个测试文件
|
||
echo "File 1" > file1.txt
|
||
echo "File 2" > file2.txt
|
||
echo "File 3" > file3.log
|
||
|
||
# 上传所有 .txt 文件
|
||
export PLUGIN_SOURCE="*.txt"
|
||
export PLUGIN_BUCKET="$S3_BUCKET"
|
||
export PLUGIN_TARGET="logs/"
|
||
export PLUGIN_ENDPOINT="$S3_ENDPOINT"
|
||
export PLUGIN_PATH_STYLE=true
|
||
plugin-s3
|
||
```
|
||
|
||
**结果**: 所有 `.txt` 文件将被上传到 `s3://your-bucket/logs/`
|
||
|
||
---
|
||
|
||
### 上传整个文件夹
|
||
|
||
递归上传整个目录及其子目录：
|
||
|
||
```yaml
|
||
steps:
|
||
- name: upload-folder
|
||
image: /bin/zsh
|
||
environment:
|
||
AWS_ACCESS_KEY_ID:
|
||
from_secret: AWS_ACCESS_KEY_ID
|
||
AWS_SECRET_ACCESS_KEY:
|
||
from_secret: AWS_SECRET_ACCESS_KEY
|
||
S3_BUCKET:
|
||
from_secret: S3_BUCKET
|
||
S3_ENDPOINT:
|
||
from_secret: S3_ENDPOINT
|
||
commands:
|
||
- |
|
||
# 创建目录结构
|
||
mkdir -p dist/css dist/js dist/images
|
||
echo "body { color: red; }" > dist/css/style.css
|
||
echo "console.log('hello');" > dist/js/app.js
|
||
echo "placeholder" > dist/images/logo.png
|
||
echo "<html>Hello</html>" > dist/index.html
|
||
|
||
# 上传整个 dist 文件夹
|
||
export PLUGIN_SOURCE="dist/**/*"
|
||
export PLUGIN_BUCKET="$S3_BUCKET"
|
||
export PLUGIN_TARGET="website/"
|
||
export PLUGIN_ENDPOINT="$S3_ENDPOINT"
|
||
export PLUGIN_PATH_STYLE=true
|
||
export PLUGIN_STRIP_PREFIX="dist/"
|
||
plugin-s3
|
||
```
|
||
|
||
**结果**:
|
||
```
|
||
s3://your-bucket/website/css/style.css
|
||
s3://your-bucket/website/js/app.js
|
||
s3://your-bucket/website/images/logo.png
|
||
s3://your-bucket/website/index.html
|
||
```
|
||
|
||
---
|
||
|
||
## 高级用法
|
||
|
||
### 设置文件访问权限
|
||
|
||
```yaml
|
||
export PLUGIN_ACL="public-read" # 公开可读
|
||
# 可选值: private, public-read, public-read-write, authenticated-read
|
||
```
|
||
|
||
### 设置缓存控制
|
||
|
||
```yaml
|
||
export PLUGIN_CACHE_CONTROL="max-age=3600" # 缓存 1 小时
|
||
```
|
||
|
||
### 设置内容类型
|
||
|
||
```yaml
|
||
export PLUGIN_CONTENT_TYPE="text/html"
|
||
export PLUGIN_CONTENT_ENCODING="gzip"
|
||
```
|
||
|
||
### 删除目标文件夹中的旧文件
|
||
|
||
```yaml
|
||
export PLUGIN_DELETE=true # 上传前删除目标路径的现有文件
|
||
```
|
||
|
||
### 使用服务器端加密
|
||
|
||
```yaml
|
||
export PLUGIN_ENCRYPTION="AES256" # 或 "aws:kms"
|
||
```
|
||
|
||
### 完整高级示例
|
||
|
||
```yaml
|
||
steps:
|
||
- name: deploy-website
|
||
image: /bin/zsh
|
||
environment:
|
||
AWS_ACCESS_KEY_ID:
|
||
from_secret: AWS_ACCESS_KEY_ID
|
||
AWS_SECRET_ACCESS_KEY:
|
||
from_secret: AWS_SECRET_ACCESS_KEY
|
||
S3_BUCKET:
|
||
from_secret: S3_BUCKET
|
||
S3_ENDPOINT:
|
||
from_secret: S3_ENDPOINT
|
||
commands:
|
||
- |
|
||
export PLUGIN_SOURCE="public/**/*"
|
||
export PLUGIN_BUCKET="$S3_BUCKET"
|
||
export PLUGIN_TARGET="production/"
|
||
export PLUGIN_ENDPOINT="$S3_ENDPOINT"
|
||
export PLUGIN_PATH_STYLE=true
|
||
export PLUGIN_STRIP_PREFIX="public/"
|
||
export PLUGIN_ACL="public-read"
|
||
export PLUGIN_CACHE_CONTROL="max-age=31536000"
|
||
export PLUGIN_DELETE=true
|
||
plugin-s3
|
||
```
|
||
|
||
---
|
||
|
||
## 完整示例
|
||
|
||
### 最小化示例 - `.woodpecker.yml`
|
||
|
||
这是一个完整的 Woodpecker 配置文件，演示如何上传文件和文件夹：
|
||
|
||
```yaml
|
||
# .woodpecker.yml
|
||
labels:
|
||
platform: linux/amd64
|
||
|
||
when:
|
||
event: [push, manual]
|
||
|
||
steps:
|
||
# 第一步：构建项目（示例）
|
||
- name: build
|
||
image: node:18-alpine
|
||
commands:
|
||
- echo "Building project..."
|
||
- mkdir -p dist/assets
|
||
- echo '<!DOCTYPE html><html><body><h1>Hello World</h1></body></html>' > dist/index.html
|
||
- echo 'body { font-family: Arial; }' > dist/assets/style.css
|
||
- echo 'console.log("App loaded");' > dist/assets/app.js
|
||
- echo "Build complete!"
|
||
- ls -R dist/
|
||
|
||
# 第二步：上传单个日志文件
|
||
- name: upload-build-log
|
||
image: alpine:latest
|
||
environment:
|
||
AWS_ACCESS_KEY_ID:
|
||
from_secret: AWS_ACCESS_KEY_ID
|
||
AWS_SECRET_ACCESS_KEY:
|
||
from_secret: AWS_SECRET_ACCESS_KEY
|
||
S3_BUCKET:
|
||
from_secret: S3_BUCKET
|
||
S3_ENDPOINT:
|
||
from_secret: S3_ENDPOINT
|
||
commands:
|
||
- |
|
||
# 创建构建日志
|
||
date > build.log
|
||
echo "Build completed successfully" >> build.log
|
||
|
||
# 上传日志文件
|
||
export PLUGIN_SOURCE="build.log"
|
||
export PLUGIN_BUCKET="$S3_BUCKET"
|
||
export PLUGIN_TARGET="logs/build-${CI_COMMIT_SHA:0:8}.log"
|
||
export PLUGIN_ENDPOINT="$S3_ENDPOINT"
|
||
export PLUGIN_PATH_STYLE=true
|
||
plugin-s3
|
||
|
||
echo "✅ Build log uploaded to s3://$S3_BUCKET/logs/"
|
||
|
||
# 第三步：上传整个 dist 文件夹
|
||
- name: upload-dist-folder
|
||
image: alpine:latest
|
||
environment:
|
||
AWS_ACCESS_KEY_ID:
|
||
from_secret: AWS_ACCESS_KEY_ID
|
||
AWS_SECRET_ACCESS_KEY:
|
||
from_secret: AWS_SECRET_ACCESS_KEY
|
||
S3_BUCKET:
|
||
from_secret: S3_BUCKET
|
||
S3_ENDPOINT:
|
||
from_secret: S3_ENDPOINT
|
||
commands:
|
||
- |
|
||
# 上传整个 dist 目录
|
||
export PLUGIN_SOURCE="dist/**/*"
|
||
export PLUGIN_BUCKET="$S3_BUCKET"
|
||
export PLUGIN_TARGET="website/${CI_COMMIT_BRANCH}/"
|
||
export PLUGIN_ENDPOINT="$S3_ENDPOINT"
|
||
export PLUGIN_PATH_STYLE=true
|
||
export PLUGIN_STRIP_PREFIX="dist/"
|
||
export PLUGIN_ACL="public-read"
|
||
plugin-s3
|
||
|
||
echo "✅ Website deployed to s3://$S3_BUCKET/website/${CI_COMMIT_BRANCH}/"
|
||
|
||
# 第四步：上传构建产物（压缩包）
|
||
- name: upload-artifacts
|
||
image: alpine:latest
|
||
environment:
|
||
AWS_ACCESS_KEY_ID:
|
||
from_secret: AWS_ACCESS_KEY_ID
|
||
AWS_SECRET_ACCESS_KEY:
|
||
from_secret: AWS_SECRET_ACCESS_KEY
|
||
S3_BUCKET:
|
||
from_secret: S3_BUCKET
|
||
S3_ENDPOINT:
|
||
from_secret: S3_ENDPOINT
|
||
commands:
|
||
- |
|
||
# 创建压缩包
|
||
apk add --no-cache zip
|
||
zip -r dist-${CI_COMMIT_SHA:0:8}.zip dist/
|
||
|
||
# 上传压缩包
|
||
export PLUGIN_SOURCE="dist-*.zip"
|
||
export PLUGIN_BUCKET="$S3_BUCKET"
|
||
export PLUGIN_TARGET="releases/${CI_COMMIT_BRANCH}/"
|
||
export PLUGIN_ENDPOINT="$S3_ENDPOINT"
|
||
export PLUGIN_PATH_STYLE=true
|
||
plugin-s3
|
||
|
||
echo "✅ Artifacts uploaded to s3://$S3_BUCKET/releases/${CI_COMMIT_BRANCH}/"
|
||
```
|
||
|
||
---
|
||
|
||
### macOS 本地 Agent 示例
|
||
|
||
如果您使用本地 macOS agent（如您的 Mac mini），配置略有不同：
|
||
|
||
```yaml
|
||
# .woodpecker.yml (macOS Local Agent)
|
||
labels:
|
||
host: Mac-mini.local
|
||
platform: darwin/arm64
|
||
|
||
when:
|
||
event: [push, manual]
|
||
|
||
steps:
|
||
- name: build-app
|
||
image: /bin/zsh
|
||
commands:
|
||
- echo "Building on macOS..."
|
||
- mkdir -p build/output
|
||
- echo "Binary placeholder" > build/output/app
|
||
- echo "Config file" > build/output/config.json
|
||
- ls -R build/
|
||
|
||
- name: upload-single-file
|
||
image: /bin/zsh
|
||
environment:
|
||
AWS_ACCESS_KEY_ID:
|
||
from_secret: AWS_ACCESS_KEY_ID
|
||
AWS_SECRET_ACCESS_KEY:
|
||
from_secret: AWS_SECRET_ACCESS_KEY
|
||
S3_BUCKET:
|
||
from_secret: S3_BUCKET
|
||
S3_ENDPOINT:
|
||
from_secret: S3_ENDPOINT
|
||
commands:
|
||
- |
|
||
echo "📦 Uploading config file..."
|
||
export PLUGIN_SOURCE="build/output/config.json"
|
||
export PLUGIN_BUCKET="$S3_BUCKET"
|
||
export PLUGIN_TARGET="configs/"
|
||
export PLUGIN_ENDPOINT="$S3_ENDPOINT"
|
||
export PLUGIN_PATH_STYLE=true
|
||
plugin-s3
|
||
echo "✅ Config uploaded"
|
||
|
||
- name: upload-build-folder
|
||
image: /bin/zsh
|
||
environment:
|
||
AWS_ACCESS_KEY_ID:
|
||
from_secret: AWS_ACCESS_KEY_ID
|
||
AWS_SECRET_ACCESS_KEY:
|
||
from_secret: AWS_SECRET_ACCESS_KEY
|
||
S3_BUCKET:
|
||
from_secret: S3_BUCKET
|
||
S3_ENDPOINT:
|
||
from_secret: S3_ENDPOINT
|
||
commands:
|
||
- |
|
||
echo "📦 Uploading entire build folder..."
|
||
export PLUGIN_SOURCE="build/output/**/*"
|
||
export PLUGIN_BUCKET="$S3_BUCKET"
|
||
export PLUGIN_TARGET="builds/macos-$(date +%Y%m%d-%H%M%S)/"
|
||
export PLUGIN_ENDPOINT="$S3_ENDPOINT"
|
||
export PLUGIN_PATH_STYLE=true
|
||
export PLUGIN_STRIP_PREFIX="build/output/"
|
||
plugin-s3
|
||
echo "✅ Build folder uploaded"
|
||
```
|
||
|
||
---
|
||
|
||
## 故障排查
|
||
|
||
### 问题 1: 密钥未传递
|
||
|
||
**错误信息**: `No S3 credentials found`
|
||
|
||
**解决方案**:
|
||
- ✅ 确保使用 `from_secret:` 语法而不是 `${SECRET_NAME}`
|
||
- ✅ 检查 Woodpecker UI 中密钥名称是否完全匹配
|
||
- ✅ 确认密钥事件类型包含当前触发事件（如 `manual`）
|
||
|
||
```yaml
|
||
# ❌ 错误
|
||
environment:
|
||
AWS_ACCESS_KEY_ID: ${AWS_ACCESS_KEY_ID}
|
||
|
||
# ✅ 正确
|
||
environment:
|
||
AWS_ACCESS_KEY_ID:
|
||
from_secret: AWS_ACCESS_KEY_ID
|
||
```
|
||
|
||
### 问题 2: plugin-s3 未找到
|
||
|
||
**错误信息**: `plugin-s3: command not found`
|
||
|
||
**解决方案**:
|
||
```bash
|
||
# 安装 plugin-s3
|
||
brew install plugin-s3
|
||
|
||
# 或添加到 PATH
|
||
export PATH=$PATH:/usr/local/bin
|
||
```
|
||
|
||
### 问题 3: 端点连接失败
|
||
|
||
**错误信息**: `connection refused` 或 `timeout`
|
||
|
||
**解决方案**:
|
||
- ✅ 检查 `S3_ENDPOINT` 格式（需包含协议，如 `https://`）
|
||
- ✅ 确认防火墙规则允许访问
|
||
- ✅ 验证 SSL 证书（自签名证书可能需要额外配置）
|
||
|
||
### 问题 4: 权限被拒绝
|
||
|
||
**错误信息**: `Access Denied` 或 `403 Forbidden`
|
||
|
||
**解决方案**:
|
||
- ✅ 验证 Access Key 和 Secret Key 是否正确
|
||
- ✅ 检查 IAM 策略是否允许 `s3:PutObject` 权限
|
||
- ✅ 确认 bucket 存在且有写入权限
|
||
|
||
### 调试技巧
|
||
|
||
添加调试输出以检查环境变量：
|
||
|
||
```yaml
|
||
commands:
|
||
- |
|
||
echo "=== 调试信息 ==="
|
||
echo "Bucket: ${S3_BUCKET}"
|
||
echo "Endpoint: ${S3_ENDPOINT}"
|
||
echo "Access Key (前10位): ${AWS_ACCESS_KEY_ID:0:10}..."
|
||
echo "================="
|
||
|
||
# 继续执行上传...
|
||
```
|
||
|
||
---
|
||
|
||
## 环境变量参考
|
||
|
||
| 变量名 | 说明 | 必需 | 示例 |
|
||
|--------|------|------|------|
|
||
| `PLUGIN_SOURCE` | 源文件路径（支持通配符） | ✅ | `dist/**/*` |
|
||
| `PLUGIN_BUCKET` | 目标 bucket | ✅ | `my-bucket` |
|
||
| `PLUGIN_TARGET` | 目标路径前缀 | ❌ | `uploads/` |
|
||
| `PLUGIN_ENDPOINT` | S3 端点 URL | ❌ | `https://s3.example.com` |
|
||
| `PLUGIN_PATH_STYLE` | 使用路径风格 URL | ❌ | `true` |
|
||
| `PLUGIN_STRIP_PREFIX` | 移除源路径前缀 | ❌ | `dist/` |
|
||
| `PLUGIN_ACL` | 访问控制列表 | ❌ | `public-read` |
|
||
| `PLUGIN_CACHE_CONTROL` | 缓存控制头 | ❌ | `max-age=3600` |
|
||
| `PLUGIN_DELETE` | 上传前删除目标文件 | ❌ | `true` |
|
||
| `PLUGIN_ENCRYPTION` | 服务器端加密 | ❌ | `AES256` |
|
||
|
||
---
|
||
|
||
## 总结
|
||
|
||
本指南涵盖了在 Woodpecker CI 中使用 S3 上传的所有常见场景：
|
||
|
||
✅ **单文件上传**: 适用于日志、配置文件等
|
||
✅ **多文件上传**: 使用通配符批量上传
|
||
✅ **文件夹上传**: 递归上传整个目录结构
|
||
✅ **高级配置**: ACL、缓存、加密等
|
||
|
||
如有问题，请参考 [Woodpecker 官方文档](https://woodpecker-ci.org/docs/) 或提交 Issue。
|
||
|
||
---
|
||
|
||
**License**: MIT
|
||
**维护者**: Your Team
|
||
**最后更新**: 2025-10-12 |