Initial commit
This commit is contained in:
3
.gitignore
vendored
Normal file
3
.gitignore
vendored
Normal file
@@ -0,0 +1,3 @@
|
||||
.venv/
|
||||
__pycache__/
|
||||
*.pyc
|
||||
21
LICENSE
Normal file
21
LICENSE
Normal file
@@ -0,0 +1,21 @@
|
||||
MIT License
|
||||
|
||||
Copyright (c) 2024 mm644706215
|
||||
|
||||
Permission is hereby granted, free of charge, to any person obtaining a copy
|
||||
of this software and associated documentation files (the "Software"), to deal
|
||||
in the Software without restriction, including without limitation the rights
|
||||
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
||||
copies of the Software, and to permit persons to whom the Software is
|
||||
furnished to do so, subject to the following conditions:
|
||||
|
||||
The above copyright notice and this permission notice shall be included in all
|
||||
copies or substantial portions of the Software.
|
||||
|
||||
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
||||
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
||||
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
||||
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
||||
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
||||
SOFTWARE.
|
||||
370
README.md
Normal file
370
README.md
Normal file
@@ -0,0 +1,370 @@
|
||||
# RustFS S3 Storage Toolkit
|
||||
|
||||
一个专为 RustFS 设计的 S3 兼容对象存储工具包,同时支持其他各种 S3 兼容存储服务。提供完整的文件和目录操作功能,具有高度的可复用性和工具性。
|
||||
|
||||
## 🧪 测试环境
|
||||
|
||||
本工具包已在以下 RustFS 版本上完成全面测试:
|
||||
|
||||
```
|
||||
rustfs 1.0.0-alpha.34
|
||||
build time : 2025-07-21 08:13:28 +00:00
|
||||
build profile: release
|
||||
build os : linux-x86_64
|
||||
rust version : rustc 1.88.0 (6b00bc388 2025-06-23)
|
||||
rust channel : stable-x86_64-unknown-linux-gnu
|
||||
git commit : 3f095e75cb1276adf47a05472c8cc608eaa51504
|
||||
git tag : 1.0.0-alpha.34
|
||||
```
|
||||
|
||||
**注意**: 虽然理论上兼容其他 S3 兼容服务,但建议在使用前进行自行测试以确保兼容性。
|
||||
|
||||
## 🚀 核心功能
|
||||
|
||||
本工具包提供以下 9 个核心功能,全部测试通过:
|
||||
|
||||
- ✅ **Connection (连接测试)**: 验证 S3 服务连接和存储桶访问权限
|
||||
- ✅ **Upload File (上传文件)**: 上传单个文件到 S3 存储
|
||||
- ✅ **Create Folder (创建文件夹)**: 在 S3 中创建文件夹结构
|
||||
- ✅ **Upload Directory (上传目录)**: 递归上传整个目录及其子目录
|
||||
- ✅ **Download File (下载文件)**: 从 S3 下载单个文件
|
||||
- ✅ **Download Directory (下载目录)**: 下载整个目录及其所有文件
|
||||
- ✅ **List Files (列出文件)**: 列出存储桶中的文件和目录
|
||||
- ✅ **Delete File (删除文件)**: 删除 S3 中的单个文件
|
||||
- ✅ **Delete Directory (删除目录)**: 删除整个目录及其所有文件
|
||||
|
||||
## 🎯 设计特点
|
||||
|
||||
- **易于使用**: 简单的类实例化和方法调用
|
||||
- **高度可复用**: 一次配置,多次使用
|
||||
- **工具性强**: 专注于核心功能,无冗余依赖
|
||||
- **兼容性好**: 支持各种 S3 兼容存储服务
|
||||
- **错误处理**: 完善的异常处理和错误信息返回
|
||||
- **灵活配置**: 自动适配不同 S3 服务的签名要求
|
||||
|
||||
## 📦 安装
|
||||
|
||||
### 从 PyPI 安装 (推荐)
|
||||
|
||||
```bash
|
||||
pip install rustfs-s3-toolkit
|
||||
```
|
||||
|
||||
### 开发版本安装
|
||||
|
||||
```bash
|
||||
# 克隆项目
|
||||
git clone https://github.com/mm644706215/rustfs-s3-toolkit.git
|
||||
cd rustfs-s3-toolkit
|
||||
|
||||
# 安装开发版本
|
||||
pip install -e .
|
||||
```
|
||||
|
||||
## 🔧 支持的存储服务
|
||||
|
||||
### 已测试服务
|
||||
- **RustFS 1.0.0-alpha.34**: 主要测试目标,所有功能完全兼容 ✅
|
||||
|
||||
### 理论兼容服务(需自行测试)
|
||||
- **AWS S3**: 亚马逊云存储服务
|
||||
- **MinIO**: 开源对象存储服务
|
||||
- **Alibaba Cloud OSS**: 阿里云对象存储
|
||||
- **Tencent Cloud COS**: 腾讯云对象存储
|
||||
- **其他 S3 兼容服务**: 任何支持 S3 API 的存储服务
|
||||
|
||||
**重要提示**: 除 RustFS 1.0.0-alpha.34 外,其他服务虽然理论上兼容,但建议在生产环境使用前进行充分测试。
|
||||
|
||||
## 📖 快速开始
|
||||
|
||||
### 基本使用
|
||||
|
||||
```python
|
||||
from rustfs_s3_toolkit import S3StorageToolkit
|
||||
|
||||
# 初始化工具包(以 RustFS 为例)
|
||||
toolkit = S3StorageToolkit(
|
||||
endpoint_url="https://your-rustfs-endpoint.com",
|
||||
access_key_id="your-access-key-id",
|
||||
secret_access_key="your-secret-access-key",
|
||||
bucket_name="your-bucket-name",
|
||||
region_name="us-east-1" # 可选,默认 us-east-1
|
||||
)
|
||||
|
||||
# 测试连接
|
||||
result = toolkit.test_connection()
|
||||
if result['success']:
|
||||
print(f"连接成功: {result['message']}")
|
||||
else:
|
||||
print(f"连接失败: {result['error']}")
|
||||
```
|
||||
|
||||
## 📚 详细使用方法
|
||||
|
||||
### 1. 连接测试 (Connection)
|
||||
|
||||
```python
|
||||
# 测试 S3 连接和存储桶访问权限
|
||||
result = toolkit.test_connection()
|
||||
|
||||
# 返回结果
|
||||
{
|
||||
"success": True,
|
||||
"bucket_count": 5,
|
||||
"bucket_names": ["bucket1", "bucket2", ...],
|
||||
"target_bucket_exists": True,
|
||||
"message": "连接成功!找到 5 个存储桶"
|
||||
}
|
||||
```
|
||||
|
||||
### 2. 上传文件 (Upload File)
|
||||
|
||||
```python
|
||||
# 上传单个文件
|
||||
result = toolkit.upload_file(
|
||||
local_file_path="/path/to/local/file.txt",
|
||||
remote_key="remote/path/file.txt",
|
||||
metadata={"author": "user", "type": "document"} # 可选
|
||||
)
|
||||
|
||||
# 返回结果
|
||||
{
|
||||
"success": True,
|
||||
"bucket": "your-bucket",
|
||||
"key": "remote/path/file.txt",
|
||||
"public_url": "https://endpoint/bucket/remote/path/file.txt",
|
||||
"file_size": 1024,
|
||||
"upload_time": "2024-01-01T12:00:00"
|
||||
}
|
||||
```
|
||||
|
||||
### 3. 创建文件夹 (Create Folder)
|
||||
|
||||
```python
|
||||
# 创建文件夹(S3 中通过空对象实现)
|
||||
result = toolkit.create_folder("my-folder/sub-folder/")
|
||||
|
||||
# 返回结果
|
||||
{
|
||||
"success": True,
|
||||
"bucket": "your-bucket",
|
||||
"folder_path": "my-folder/sub-folder/",
|
||||
"create_time": "2024-01-01T12:00:00"
|
||||
}
|
||||
```
|
||||
|
||||
### 4. 上传目录 (Upload Directory)
|
||||
|
||||
```python
|
||||
# 递归上传整个目录
|
||||
result = toolkit.upload_directory(
|
||||
local_dir="/path/to/local/directory",
|
||||
remote_prefix="remote/directory/"
|
||||
)
|
||||
|
||||
# 返回结果
|
||||
{
|
||||
"success": True,
|
||||
"bucket": "your-bucket",
|
||||
"local_directory": "/path/to/local/directory",
|
||||
"remote_prefix": "remote/directory/",
|
||||
"uploaded_files": ["remote/directory/file1.txt", "remote/directory/sub/file2.txt"],
|
||||
"file_count": 2,
|
||||
"upload_time": "2024-01-01T12:00:00"
|
||||
}
|
||||
```
|
||||
|
||||
### 5. 下载文件 (Download File)
|
||||
|
||||
```python
|
||||
# 下载单个文件
|
||||
result = toolkit.download_file(
|
||||
remote_key="remote/path/file.txt",
|
||||
local_file_path="/path/to/save/file.txt"
|
||||
)
|
||||
|
||||
# 返回结果
|
||||
{
|
||||
"success": True,
|
||||
"bucket": "your-bucket",
|
||||
"key": "remote/path/file.txt",
|
||||
"local_path": "/path/to/save/file.txt",
|
||||
"file_size": 1024,
|
||||
"download_time": "2024-01-01T12:00:00"
|
||||
}
|
||||
```
|
||||
|
||||
### 6. 下载目录 (Download Directory)
|
||||
|
||||
```python
|
||||
# 下载整个目录
|
||||
result = toolkit.download_directory(
|
||||
remote_prefix="remote/directory/",
|
||||
local_dir="/path/to/save/directory"
|
||||
)
|
||||
|
||||
# 返回结果
|
||||
{
|
||||
"success": True,
|
||||
"bucket": "your-bucket",
|
||||
"remote_prefix": "remote/directory/",
|
||||
"local_directory": "/path/to/save/directory",
|
||||
"downloaded_files": ["/path/to/save/directory/file1.txt", ...],
|
||||
"file_count": 2,
|
||||
"download_time": "2024-01-01T12:00:00"
|
||||
}
|
||||
```
|
||||
|
||||
### 7. 列出文件 (List Files)
|
||||
|
||||
```python
|
||||
# 列出存储桶中的文件
|
||||
result = toolkit.list_files(
|
||||
prefix="my-folder/", # 可选,过滤前缀
|
||||
max_keys=100 # 可选,最大返回数量
|
||||
)
|
||||
|
||||
# 返回结果
|
||||
{
|
||||
"success": True,
|
||||
"bucket": "your-bucket",
|
||||
"prefix": "my-folder/",
|
||||
"files": [
|
||||
{
|
||||
"key": "my-folder/file1.txt",
|
||||
"size": 1024,
|
||||
"last_modified": "2024-01-01T12:00:00",
|
||||
"public_url": "https://endpoint/bucket/my-folder/file1.txt"
|
||||
}
|
||||
],
|
||||
"file_count": 1,
|
||||
"list_time": "2024-01-01T12:00:00"
|
||||
}
|
||||
```
|
||||
|
||||
### 8. 删除文件 (Delete File)
|
||||
|
||||
```python
|
||||
# 删除单个文件
|
||||
result = toolkit.delete_file("remote/path/file.txt")
|
||||
|
||||
# 返回结果
|
||||
{
|
||||
"success": True,
|
||||
"bucket": "your-bucket",
|
||||
"key": "remote/path/file.txt",
|
||||
"delete_time": "2024-01-01T12:00:00"
|
||||
}
|
||||
```
|
||||
|
||||
### 9. 删除目录 (Delete Directory)
|
||||
|
||||
```python
|
||||
# 删除整个目录及其所有文件
|
||||
result = toolkit.delete_directory("remote/directory/")
|
||||
|
||||
# 返回结果
|
||||
{
|
||||
"success": True,
|
||||
"bucket": "your-bucket",
|
||||
"remote_prefix": "remote/directory/",
|
||||
"deleted_count": 5,
|
||||
"delete_time": "2024-01-01T12:00:00"
|
||||
}
|
||||
```
|
||||
|
||||
## 🔧 配置说明
|
||||
|
||||
### 基本配置参数
|
||||
|
||||
| 参数 | 类型 | 必需 | 说明 |
|
||||
|------|------|------|------|
|
||||
| `endpoint_url` | str | ✅ | S3 服务端点 URL |
|
||||
| `access_key_id` | str | ✅ | 访问密钥 ID |
|
||||
| `secret_access_key` | str | ✅ | 访问密钥 |
|
||||
| `bucket_name` | str | ✅ | 存储桶名称 |
|
||||
| `region_name` | str | ❌ | 区域名称,默认 "us-east-1" |
|
||||
|
||||
### 常见配置示例
|
||||
|
||||
#### AWS S3
|
||||
```python
|
||||
toolkit = S3StorageToolkit(
|
||||
endpoint_url="https://s3.amazonaws.com",
|
||||
access_key_id="AKIAIOSFODNN7EXAMPLE",
|
||||
secret_access_key="wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
|
||||
bucket_name="my-bucket",
|
||||
region_name="us-west-2"
|
||||
)
|
||||
```
|
||||
|
||||
#### MinIO
|
||||
```python
|
||||
toolkit = S3StorageToolkit(
|
||||
endpoint_url="http://localhost:9000",
|
||||
access_key_id="minioadmin",
|
||||
secret_access_key="minioadmin",
|
||||
bucket_name="my-bucket"
|
||||
)
|
||||
```
|
||||
|
||||
#### RustFS (已测试版本 1.0.0-alpha.34)
|
||||
```python
|
||||
toolkit = S3StorageToolkit(
|
||||
endpoint_url="https://your-rustfs-endpoint.com",
|
||||
access_key_id="your-access-key",
|
||||
secret_access_key="your-secret-key",
|
||||
bucket_name="your-bucket-name"
|
||||
)
|
||||
```
|
||||
|
||||
## 🧪 测试
|
||||
|
||||
运行完整的功能测试:
|
||||
|
||||
```bash
|
||||
# 运行测试套件
|
||||
python tests/test_toolkit.py
|
||||
|
||||
# 运行基本使用示例
|
||||
python examples/basic_usage.py
|
||||
```
|
||||
|
||||
测试将验证所有 9 个核心功能是否正常工作。
|
||||
|
||||
## 📁 项目结构
|
||||
|
||||
```
|
||||
rustfs-s3-toolkit/
|
||||
├── README.md # 项目文档
|
||||
├── pyproject.toml # 项目配置
|
||||
├── LICENSE # MIT 许可证
|
||||
├── install.py # 安装脚本
|
||||
├── build.py # 构建脚本
|
||||
├── src/
|
||||
│ └── rustfs_s3_toolkit/
|
||||
│ ├── __init__.py # 包初始化
|
||||
│ └── s3_client.py # 核心工具类
|
||||
├── tests/
|
||||
│ └── test_toolkit.py # 测试套件
|
||||
└── examples/
|
||||
└── basic_usage.py # 使用示例
|
||||
```
|
||||
|
||||
## 🤝 贡献
|
||||
|
||||
欢迎提交 Issue 和 Pull Request!
|
||||
|
||||
## 📄 许可证
|
||||
|
||||
MIT License
|
||||
|
||||
## 🔗 相关链接
|
||||
|
||||
- [GitHub 仓库](https://github.com/mm644706215/rustfs-s3-toolkit)
|
||||
- [PyPI 包](https://pypi.org/project/rustfs-s3-toolkit/)
|
||||
- [问题反馈](https://github.com/mm644706215/rustfs-s3-toolkit/issues)
|
||||
|
||||
---
|
||||
|
||||
**RustFS S3 Storage Toolkit** - 专为 RustFS 设计,让 S3 对象存储操作变得简单高效!
|
||||
104
build.py
Normal file
104
build.py
Normal file
@@ -0,0 +1,104 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
RustFS S3 Storage Toolkit 构建和发布脚本
|
||||
"""
|
||||
|
||||
import subprocess
|
||||
import sys
|
||||
import os
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
def run_command(command, description):
|
||||
"""运行命令并显示结果"""
|
||||
print(f"🔧 {description}...")
|
||||
try:
|
||||
result = subprocess.run(command, shell=True, check=True, capture_output=True, text=True)
|
||||
print(f"✅ {description}成功")
|
||||
if result.stdout.strip():
|
||||
print(f"输出: {result.stdout.strip()}")
|
||||
return True
|
||||
except subprocess.CalledProcessError as e:
|
||||
print(f"❌ {description}失败: {e}")
|
||||
if e.stdout:
|
||||
print(f"输出: {e.stdout}")
|
||||
if e.stderr:
|
||||
print(f"错误: {e.stderr}")
|
||||
return False
|
||||
|
||||
|
||||
def main():
|
||||
"""主构建流程"""
|
||||
print("🚀 RustFS S3 Storage Toolkit 构建脚本")
|
||||
print("=" * 50)
|
||||
|
||||
# 检查当前目录
|
||||
current_dir = Path.cwd()
|
||||
if not (current_dir / "pyproject.toml").exists():
|
||||
print("❌ 请在项目根目录运行此脚本")
|
||||
sys.exit(1)
|
||||
|
||||
print(f"✅ 项目目录: {current_dir}")
|
||||
|
||||
# 清理旧的构建文件
|
||||
print("\n🧹 清理旧的构建文件...")
|
||||
cleanup_commands = [
|
||||
"rm -rf dist/",
|
||||
"rm -rf build/",
|
||||
"rm -rf *.egg-info/",
|
||||
"find . -name '__pycache__' -type d -exec rm -rf {} + 2>/dev/null || true",
|
||||
"find . -name '*.pyc' -delete 2>/dev/null || true"
|
||||
]
|
||||
|
||||
for command in cleanup_commands:
|
||||
subprocess.run(command, shell=True, capture_output=True)
|
||||
|
||||
print("✅ 清理完成")
|
||||
|
||||
# 安装构建依赖
|
||||
build_deps = [
|
||||
"pip install --upgrade pip",
|
||||
"pip install --upgrade build twine"
|
||||
]
|
||||
|
||||
for command in build_deps:
|
||||
if not run_command(command, f"安装构建依赖"):
|
||||
print("❌ 安装构建依赖失败")
|
||||
sys.exit(1)
|
||||
|
||||
# 构建包
|
||||
if not run_command("python -m build", "构建包"):
|
||||
print("❌ 构建失败")
|
||||
sys.exit(1)
|
||||
|
||||
# 检查构建结果
|
||||
dist_dir = current_dir / "dist"
|
||||
if not dist_dir.exists():
|
||||
print("❌ 构建目录不存在")
|
||||
sys.exit(1)
|
||||
|
||||
built_files = list(dist_dir.glob("*"))
|
||||
if not built_files:
|
||||
print("❌ 没有找到构建文件")
|
||||
sys.exit(1)
|
||||
|
||||
print(f"\n📦 构建完成!生成的文件:")
|
||||
for file in built_files:
|
||||
print(f" - {file.name}")
|
||||
|
||||
# 验证包
|
||||
if not run_command("python -m twine check dist/*", "验证包"):
|
||||
print("❌ 包验证失败")
|
||||
sys.exit(1)
|
||||
|
||||
print("\n🎉 构建和验证完成!")
|
||||
print("\n📖 发布到 PyPI:")
|
||||
print("1. 测试发布: python -m twine upload --repository testpypi dist/*")
|
||||
print("2. 正式发布: python -m twine upload dist/*")
|
||||
|
||||
print("\n💡 本地安装测试:")
|
||||
print("pip install dist/*.whl")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
193
examples/basic_usage.py
Normal file
193
examples/basic_usage.py
Normal file
@@ -0,0 +1,193 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
RustFS S3 Storage Toolkit 基本使用示例
|
||||
演示所有 9 个核心功能的使用方法
|
||||
已在 RustFS 1.0.0-alpha.34 上完成测试
|
||||
"""
|
||||
|
||||
import tempfile
|
||||
from pathlib import Path
|
||||
from datetime import datetime
|
||||
|
||||
from rustfs_s3_toolkit import S3StorageToolkit
|
||||
|
||||
|
||||
def main():
|
||||
"""RustFS S3 Storage Toolkit 使用示例"""
|
||||
|
||||
# 1. 初始化工具包
|
||||
print("🚀 RustFS S3 Storage Toolkit 使用示例")
|
||||
print("🧪 测试环境: RustFS 1.0.0-alpha.34")
|
||||
print("=" * 50)
|
||||
|
||||
# 配置信息 - 请根据实际 RustFS 情况修改
|
||||
config = {
|
||||
"endpoint_url": "https://your-rustfs-endpoint.com",
|
||||
"access_key_id": "your-access-key-id",
|
||||
"secret_access_key": "your-secret-access-key",
|
||||
"bucket_name": "your-bucket-name",
|
||||
"region_name": "us-east-1"
|
||||
}
|
||||
|
||||
# 创建工具包实例
|
||||
toolkit = S3StorageToolkit(**config)
|
||||
print("✅ 工具包初始化完成")
|
||||
|
||||
# 2. 测试连接
|
||||
print("\n📡 测试连接...")
|
||||
result = toolkit.test_connection()
|
||||
if result['success']:
|
||||
print(f"✅ 连接成功: {result['message']}")
|
||||
print(f"📊 存储桶数量: {result['bucket_count']}")
|
||||
print(f"🎯 目标存储桶存在: {result['target_bucket_exists']}")
|
||||
else:
|
||||
print(f"❌ 连接失败: {result['error']}")
|
||||
return
|
||||
|
||||
# 3. 上传文件
|
||||
print("\n📤 上传文件...")
|
||||
|
||||
# 创建测试文件
|
||||
test_content = f"Hello RustFS S3 Storage Toolkit!\n创建时间: {datetime.now().isoformat()}"
|
||||
with tempfile.NamedTemporaryFile(mode='w', delete=False, suffix='.txt', encoding='utf-8') as f:
|
||||
f.write(test_content)
|
||||
test_file = f.name
|
||||
|
||||
# 上传文件
|
||||
timestamp = datetime.now().strftime("%Y%m%d_%H%M%S")
|
||||
remote_key = f"examples/test_file_{timestamp}.txt"
|
||||
|
||||
result = toolkit.upload_file(test_file, remote_key)
|
||||
if result['success']:
|
||||
print(f"✅ 文件上传成功")
|
||||
print(f"📁 远程路径: {result['key']}")
|
||||
print(f"🔗 公开链接: {result['public_url']}")
|
||||
print(f"📊 文件大小: {result['file_size']} 字节")
|
||||
else:
|
||||
print(f"❌ 文件上传失败: {result['error']}")
|
||||
|
||||
# 4. 创建文件夹
|
||||
print("\n📁 创建文件夹...")
|
||||
folder_path = f"examples/test_folder_{timestamp}/"
|
||||
|
||||
result = toolkit.create_folder(folder_path)
|
||||
if result['success']:
|
||||
print(f"✅ 文件夹创建成功: {result['folder_path']}")
|
||||
else:
|
||||
print(f"❌ 文件夹创建失败: {result['error']}")
|
||||
|
||||
# 5. 上传目录
|
||||
print("\n📂 上传目录...")
|
||||
|
||||
# 创建测试目录
|
||||
test_dir = tempfile.mkdtemp(prefix='s3_example_')
|
||||
test_dir_path = Path(test_dir)
|
||||
|
||||
# 创建一些测试文件
|
||||
(test_dir_path / "file1.txt").write_text("这是文件1", encoding='utf-8')
|
||||
(test_dir_path / "file2.txt").write_text("这是文件2", encoding='utf-8')
|
||||
|
||||
# 创建子目录
|
||||
sub_dir = test_dir_path / "subdir"
|
||||
sub_dir.mkdir()
|
||||
(sub_dir / "file3.txt").write_text("这是子目录中的文件", encoding='utf-8')
|
||||
|
||||
# 上传目录
|
||||
remote_prefix = f"examples/test_directory_{timestamp}/"
|
||||
result = toolkit.upload_directory(test_dir, remote_prefix)
|
||||
if result['success']:
|
||||
print(f"✅ 目录上传成功")
|
||||
print(f"📁 本地目录: {result['local_directory']}")
|
||||
print(f"🌐 远程前缀: {result['remote_prefix']}")
|
||||
print(f"📊 文件数量: {result['file_count']}")
|
||||
print("📄 上传的文件:")
|
||||
for file_key in result['uploaded_files']:
|
||||
print(f" - {file_key}")
|
||||
else:
|
||||
print(f"❌ 目录上传失败: {result['error']}")
|
||||
|
||||
# 6. 列出文件
|
||||
print("\n📋 列出文件...")
|
||||
result = toolkit.list_files(prefix="examples/", max_keys=10)
|
||||
if result['success']:
|
||||
print(f"✅ 文件列表获取成功")
|
||||
print(f"📊 文件数量: {result['file_count']}")
|
||||
print("📄 文件列表:")
|
||||
for file_info in result['files']:
|
||||
size_mb = file_info['size'] / (1024 * 1024)
|
||||
print(f" - {file_info['key']} ({size_mb:.3f} MB)")
|
||||
else:
|
||||
print(f"❌ 文件列表获取失败: {result['error']}")
|
||||
|
||||
# 7. 下载文件
|
||||
print("\n📥 下载文件...")
|
||||
download_path = tempfile.mktemp(suffix='_downloaded.txt')
|
||||
|
||||
result = toolkit.download_file(remote_key, download_path)
|
||||
if result['success']:
|
||||
print(f"✅ 文件下载成功")
|
||||
print(f"📁 本地路径: {result['local_path']}")
|
||||
print(f"📊 文件大小: {result['file_size']} 字节")
|
||||
|
||||
# 验证下载的内容
|
||||
with open(download_path, 'r', encoding='utf-8') as f:
|
||||
downloaded_content = f.read()
|
||||
|
||||
if test_content == downloaded_content:
|
||||
print("✅ 文件内容验证成功")
|
||||
else:
|
||||
print("❌ 文件内容验证失败")
|
||||
else:
|
||||
print(f"❌ 文件下载失败: {result['error']}")
|
||||
|
||||
# 8. 下载目录
|
||||
print("\n📂 下载目录...")
|
||||
download_dir = tempfile.mkdtemp(prefix='s3_download_')
|
||||
|
||||
result = toolkit.download_directory(remote_prefix, download_dir)
|
||||
if result['success']:
|
||||
print(f"✅ 目录下载成功")
|
||||
print(f"📁 本地目录: {result['local_directory']}")
|
||||
print(f"🌐 远程前缀: {result['remote_prefix']}")
|
||||
print(f"📊 文件数量: {result['file_count']}")
|
||||
print("📄 下载的文件:")
|
||||
for file_path in result['downloaded_files']:
|
||||
print(f" - {file_path}")
|
||||
else:
|
||||
print(f"❌ 目录下载失败: {result['error']}")
|
||||
|
||||
# 9. 删除文件
|
||||
print("\n🗑️ 删除文件...")
|
||||
result = toolkit.delete_file(remote_key)
|
||||
if result['success']:
|
||||
print(f"✅ 文件删除成功: {result['key']}")
|
||||
else:
|
||||
print(f"❌ 文件删除失败: {result['error']}")
|
||||
|
||||
# 10. 删除目录
|
||||
print("\n🗑️ 删除目录...")
|
||||
result = toolkit.delete_directory(remote_prefix)
|
||||
if result['success']:
|
||||
print(f"✅ 目录删除成功")
|
||||
print(f"📊 删除文件数量: {result['deleted_count']}")
|
||||
else:
|
||||
print(f"❌ 目录删除失败: {result['error']}")
|
||||
|
||||
# 清理本地文件
|
||||
import os
|
||||
import shutil
|
||||
|
||||
try:
|
||||
os.unlink(test_file)
|
||||
os.unlink(download_path)
|
||||
shutil.rmtree(test_dir)
|
||||
shutil.rmtree(download_dir)
|
||||
print("\n🧹 本地临时文件清理完成")
|
||||
except:
|
||||
pass
|
||||
|
||||
print("\n🎉 示例演示完成!")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
96
install.py
Normal file
96
install.py
Normal file
@@ -0,0 +1,96 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
RustFS S3 Storage Toolkit 安装脚本
|
||||
"""
|
||||
|
||||
import subprocess
|
||||
import sys
|
||||
import os
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
def run_command(command, description):
|
||||
"""运行命令并显示结果"""
|
||||
print(f"🔧 {description}...")
|
||||
try:
|
||||
result = subprocess.run(command, shell=True, check=True, capture_output=True, text=True)
|
||||
print(f"✅ {description}成功")
|
||||
return True
|
||||
except subprocess.CalledProcessError as e:
|
||||
print(f"❌ {description}失败: {e}")
|
||||
if e.stdout:
|
||||
print(f"输出: {e.stdout}")
|
||||
if e.stderr:
|
||||
print(f"错误: {e.stderr}")
|
||||
return False
|
||||
|
||||
|
||||
def main():
|
||||
"""主安装流程"""
|
||||
print("🚀 RustFS S3 Storage Toolkit 安装脚本")
|
||||
print("=" * 50)
|
||||
|
||||
# 检查 Python 版本
|
||||
if sys.version_info < (3, 9):
|
||||
print("❌ 需要 Python 3.9 或更高版本")
|
||||
sys.exit(1)
|
||||
|
||||
print(f"✅ Python 版本: {sys.version}")
|
||||
|
||||
# 检查当前目录
|
||||
current_dir = Path.cwd()
|
||||
if not (current_dir / "pyproject.toml").exists():
|
||||
print("❌ 请在项目根目录运行此脚本")
|
||||
sys.exit(1)
|
||||
|
||||
print(f"✅ 项目目录: {current_dir}")
|
||||
|
||||
# 安装包
|
||||
install_commands = [
|
||||
("pip install -e .", "安装 RustFS S3 Storage Toolkit"),
|
||||
]
|
||||
|
||||
for command, description in install_commands:
|
||||
if not run_command(command, description):
|
||||
print(f"❌ 安装失败,请检查错误信息")
|
||||
sys.exit(1)
|
||||
|
||||
# 验证安装
|
||||
print("\n🧪 验证安装...")
|
||||
try:
|
||||
from rustfs_s3_toolkit import S3StorageToolkit
|
||||
print("✅ S3StorageToolkit 导入成功")
|
||||
|
||||
# 显示版本信息
|
||||
import rustfs_s3_toolkit
|
||||
print(f"✅ 版本: {rustfs_s3_toolkit.__version__}")
|
||||
|
||||
except ImportError as e:
|
||||
print(f"❌ 导入失败: {e}")
|
||||
sys.exit(1)
|
||||
|
||||
print("\n🎉 安装完成!")
|
||||
print("\n📖 使用方法:")
|
||||
print("1. 查看示例: python examples/basic_usage.py")
|
||||
print("2. 运行测试: python tests/test_toolkit.py")
|
||||
print("3. 查看文档: cat README.md")
|
||||
|
||||
print("\n💡 快速开始:")
|
||||
print("""
|
||||
from rustfs_s3_toolkit import S3StorageToolkit
|
||||
|
||||
toolkit = S3StorageToolkit(
|
||||
endpoint_url="https://your-rustfs-endpoint.com",
|
||||
access_key_id="your-key",
|
||||
secret_access_key="your-secret",
|
||||
bucket_name="your-bucket"
|
||||
)
|
||||
|
||||
# 测试连接
|
||||
result = toolkit.test_connection()
|
||||
print(result)
|
||||
""")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
48
pyproject.toml
Normal file
48
pyproject.toml
Normal file
@@ -0,0 +1,48 @@
|
||||
[project]
|
||||
name = "rustfs-s3-toolkit"
|
||||
version = "0.2.0"
|
||||
description = "RustFS S3 Storage Toolkit - A simple and powerful toolkit for RustFS and other S3-compatible object storage operations"
|
||||
readme = "README.md"
|
||||
authors = [
|
||||
{ name = "mm644706215", email = "ze.ga@qq.com" }
|
||||
]
|
||||
requires-python = ">=3.9"
|
||||
dependencies = [
|
||||
"boto3>=1.39.0",
|
||||
]
|
||||
keywords = ["rustfs", "s3", "object-storage", "file-management", "cloud-storage", "toolkit"]
|
||||
classifiers = [
|
||||
"Development Status :: 4 - Beta",
|
||||
"Intended Audience :: Developers",
|
||||
"License :: OSI Approved :: MIT License",
|
||||
"Programming Language :: Python :: 3",
|
||||
"Programming Language :: Python :: 3.9",
|
||||
"Programming Language :: Python :: 3.10",
|
||||
"Programming Language :: Python :: 3.11",
|
||||
"Programming Language :: Python :: 3.12",
|
||||
"Programming Language :: Python :: 3.13",
|
||||
"Topic :: Software Development :: Libraries :: Python Modules",
|
||||
"Topic :: System :: Archiving",
|
||||
"Topic :: Internet :: File Transfer Protocol (FTP)",
|
||||
"Topic :: System :: Filesystems",
|
||||
]
|
||||
|
||||
[project.urls]
|
||||
Homepage = "https://github.com/mm644706215/rustfs-s3-toolkit"
|
||||
Repository = "https://github.com/mm644706215/rustfs-s3-toolkit"
|
||||
Documentation = "https://github.com/mm644706215/rustfs-s3-toolkit#readme"
|
||||
Issues = "https://github.com/mm644706215/rustfs-s3-toolkit/issues"
|
||||
|
||||
[project.optional-dependencies]
|
||||
dev = [
|
||||
"pytest>=7.0.0",
|
||||
"pytest-asyncio>=0.21.0",
|
||||
"black>=23.0.0",
|
||||
"isort>=5.12.0",
|
||||
"flake8>=6.0.0",
|
||||
"mypy>=1.0.0",
|
||||
]
|
||||
|
||||
[build-system]
|
||||
requires = ["hatchling"]
|
||||
build-backend = "hatchling.build"
|
||||
14
src/rustfs_s3_toolkit/__init__.py
Normal file
14
src/rustfs_s3_toolkit/__init__.py
Normal file
@@ -0,0 +1,14 @@
|
||||
"""
|
||||
RustFS S3 Storage Toolkit
|
||||
一个专为 RustFS 设计的 S3 兼容对象存储工具包,同时支持其他各种 S3 兼容存储
|
||||
"""
|
||||
|
||||
__version__ = "0.2.0"
|
||||
__author__ = "mm644706215"
|
||||
__email__ = "ze.ga@qq.com"
|
||||
|
||||
from .s3_client import S3StorageToolkit
|
||||
|
||||
__all__ = [
|
||||
"S3StorageToolkit",
|
||||
]
|
||||
617
src/rustfs_s3_toolkit/s3_client.py
Normal file
617
src/rustfs_s3_toolkit/s3_client.py
Normal file
@@ -0,0 +1,617 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
S3 Storage Toolkit
|
||||
一个易于使用的 S3 兼容对象存储工具包,支持 RustFS 等各种 S3 兼容存储
|
||||
"""
|
||||
|
||||
import os
|
||||
import shutil
|
||||
import tempfile
|
||||
import time
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
from typing import Dict, List, Optional, Union
|
||||
|
||||
import boto3
|
||||
from botocore.client import Config
|
||||
from botocore.exceptions import ClientError, NoCredentialsError
|
||||
|
||||
|
||||
class S3StorageToolkit:
|
||||
"""S3 兼容对象存储工具包
|
||||
|
||||
支持以下功能:
|
||||
1. 连接测试
|
||||
2. 文件上传
|
||||
3. 文件夹创建
|
||||
4. 目录上传
|
||||
5. 文件下载
|
||||
6. 目录下载
|
||||
7. 文件列表
|
||||
8. 文件删除
|
||||
9. 目录删除
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
endpoint_url: str,
|
||||
access_key_id: str,
|
||||
secret_access_key: str,
|
||||
bucket_name: str,
|
||||
region_name: str = "us-east-1"
|
||||
):
|
||||
"""
|
||||
初始化 S3 存储工具包
|
||||
|
||||
Args:
|
||||
endpoint_url: S3 服务端点 URL
|
||||
access_key_id: 访问密钥 ID
|
||||
secret_access_key: 访问密钥
|
||||
bucket_name: 存储桶名称
|
||||
region_name: 区域名称,默认 us-east-1
|
||||
"""
|
||||
self.endpoint_url = endpoint_url
|
||||
self.access_key_id = access_key_id
|
||||
self.secret_access_key = secret_access_key
|
||||
self.bucket_name = bucket_name
|
||||
self.region_name = region_name
|
||||
|
||||
# 存储不同操作类型的客户端
|
||||
self._clients = {}
|
||||
|
||||
def _create_s3_client(self, operation_type: str = "general"):
|
||||
"""创建 S3 客户端,根据操作类型选择最佳配置"""
|
||||
if operation_type in self._clients:
|
||||
return self._clients[operation_type]
|
||||
|
||||
try:
|
||||
if operation_type == "upload":
|
||||
# 对于上传操作,使用 V2 签名(避免 SHA256 问题)
|
||||
configs_to_try = [
|
||||
{
|
||||
'signature_version': 's3', # V2 签名
|
||||
's3': {'addressing_style': 'path'}
|
||||
},
|
||||
{
|
||||
'signature_version': 's3v4',
|
||||
's3': {
|
||||
'addressing_style': 'path',
|
||||
'payload_signing_enabled': False
|
||||
}
|
||||
}
|
||||
]
|
||||
else:
|
||||
# 对于列表、删除等操作,使用 V4 签名
|
||||
configs_to_try = [
|
||||
{
|
||||
'signature_version': 's3v4',
|
||||
's3': {
|
||||
'addressing_style': 'path',
|
||||
'payload_signing_enabled': False
|
||||
}
|
||||
},
|
||||
{
|
||||
'signature_version': 's3', # V2 签名作为备选
|
||||
's3': {'addressing_style': 'path'}
|
||||
}
|
||||
]
|
||||
|
||||
for config in configs_to_try:
|
||||
try:
|
||||
s3_client = boto3.client(
|
||||
's3',
|
||||
endpoint_url=self.endpoint_url,
|
||||
aws_access_key_id=self.access_key_id,
|
||||
aws_secret_access_key=self.secret_access_key,
|
||||
region_name=self.region_name,
|
||||
config=Config(**config)
|
||||
)
|
||||
|
||||
# 测试连接
|
||||
s3_client.list_buckets()
|
||||
self._clients[operation_type] = s3_client
|
||||
return s3_client
|
||||
|
||||
except Exception:
|
||||
continue
|
||||
|
||||
raise Exception("所有配置都失败了")
|
||||
|
||||
except Exception as e:
|
||||
raise RuntimeError(f"创建 S3 客户端失败: {str(e)}")
|
||||
|
||||
def test_connection(self) -> Dict[str, Union[bool, str, List[str]]]:
|
||||
"""测试 S3 连接
|
||||
|
||||
Returns:
|
||||
包含连接测试结果的字典
|
||||
"""
|
||||
try:
|
||||
s3_client = self._create_s3_client()
|
||||
|
||||
# 尝试列出存储桶
|
||||
response = s3_client.list_buckets()
|
||||
buckets = response.get('Buckets', [])
|
||||
bucket_names = [bucket['Name'] for bucket in buckets]
|
||||
|
||||
# 检查目标存储桶是否存在
|
||||
bucket_exists = self.bucket_name in bucket_names
|
||||
|
||||
return {
|
||||
"success": True,
|
||||
"bucket_count": len(buckets),
|
||||
"bucket_names": bucket_names,
|
||||
"target_bucket_exists": bucket_exists,
|
||||
"message": f"连接成功!找到 {len(buckets)} 个存储桶"
|
||||
}
|
||||
|
||||
except Exception as e:
|
||||
return {
|
||||
"success": False,
|
||||
"error": str(e),
|
||||
"message": f"连接测试失败: {str(e)}"
|
||||
}
|
||||
|
||||
def upload_file(
|
||||
self,
|
||||
local_file_path: str,
|
||||
remote_key: str,
|
||||
metadata: Optional[Dict[str, str]] = None
|
||||
) -> Dict[str, Union[bool, str, int]]:
|
||||
"""
|
||||
上传文件到 S3
|
||||
|
||||
Args:
|
||||
local_file_path: 本地文件路径
|
||||
remote_key: 远程对象键
|
||||
metadata: 可选的元数据
|
||||
|
||||
Returns:
|
||||
包含上传信息的字典
|
||||
"""
|
||||
try:
|
||||
local_path = Path(local_file_path)
|
||||
if not local_path.exists():
|
||||
return {
|
||||
"success": False,
|
||||
"error": f"本地文件不存在: {local_file_path}"
|
||||
}
|
||||
|
||||
s3_client = self._create_s3_client("upload")
|
||||
|
||||
# 读取文件内容
|
||||
with open(local_path, 'rb') as f:
|
||||
file_content = f.read()
|
||||
|
||||
# 上传文件
|
||||
upload_args = {
|
||||
'Bucket': self.bucket_name,
|
||||
'Key': remote_key,
|
||||
'Body': file_content,
|
||||
'ContentType': 'application/octet-stream'
|
||||
}
|
||||
|
||||
if metadata:
|
||||
upload_args['Metadata'] = metadata
|
||||
|
||||
s3_client.put_object(**upload_args)
|
||||
|
||||
# 生成公开访问 URL
|
||||
public_url = f"{self.endpoint_url}/{self.bucket_name}/{remote_key}"
|
||||
|
||||
return {
|
||||
"success": True,
|
||||
"bucket": self.bucket_name,
|
||||
"key": remote_key,
|
||||
"public_url": public_url,
|
||||
"file_size": local_path.stat().st_size,
|
||||
"upload_time": datetime.now().isoformat()
|
||||
}
|
||||
|
||||
except Exception as e:
|
||||
return {
|
||||
"success": False,
|
||||
"error": f"上传文件失败: {str(e)}"
|
||||
}
|
||||
|
||||
def create_folder(self, folder_path: str) -> Dict[str, Union[bool, str]]:
|
||||
"""
|
||||
创建文件夹(通过上传空对象)
|
||||
|
||||
Args:
|
||||
folder_path: 文件夹路径,会自动添加 '/' 后缀
|
||||
|
||||
Returns:
|
||||
包含创建结果的字典
|
||||
"""
|
||||
try:
|
||||
s3_client = self._create_s3_client("upload")
|
||||
|
||||
# 确保文件夹路径以 '/' 结尾
|
||||
if not folder_path.endswith('/'):
|
||||
folder_path += '/'
|
||||
|
||||
# S3 中通过上传空对象来创建"文件夹"
|
||||
s3_client.put_object(
|
||||
Bucket=self.bucket_name,
|
||||
Key=folder_path,
|
||||
Body=b''
|
||||
)
|
||||
|
||||
return {
|
||||
"success": True,
|
||||
"bucket": self.bucket_name,
|
||||
"folder_path": folder_path,
|
||||
"create_time": datetime.now().isoformat()
|
||||
}
|
||||
|
||||
except Exception as e:
|
||||
return {
|
||||
"success": False,
|
||||
"error": f"创建文件夹失败: {str(e)}"
|
||||
}
|
||||
|
||||
def upload_directory(
|
||||
self,
|
||||
local_dir: str,
|
||||
remote_prefix: str = ""
|
||||
) -> Dict[str, Union[bool, str, int, List[str]]]:
|
||||
"""
|
||||
上传整个目录到 S3
|
||||
|
||||
Args:
|
||||
local_dir: 本地目录路径
|
||||
remote_prefix: 远程路径前缀
|
||||
|
||||
Returns:
|
||||
包含上传结果的字典
|
||||
"""
|
||||
try:
|
||||
local_path = Path(local_dir)
|
||||
if not local_path.exists() or not local_path.is_dir():
|
||||
return {
|
||||
"success": False,
|
||||
"error": f"本地目录不存在或不是目录: {local_dir}"
|
||||
}
|
||||
|
||||
s3_client = self._create_s3_client("upload")
|
||||
uploaded_files = []
|
||||
|
||||
# 递归上传目录中的所有文件
|
||||
for root, dirs, files in os.walk(local_path):
|
||||
for file in files:
|
||||
local_file_path = os.path.join(root, file)
|
||||
relative_path = os.path.relpath(local_file_path, local_path)
|
||||
|
||||
# 构建远程键
|
||||
if remote_prefix:
|
||||
remote_key = f"{remote_prefix.rstrip('/')}/{relative_path.replace(os.sep, '/')}"
|
||||
else:
|
||||
remote_key = relative_path.replace(os.sep, '/')
|
||||
|
||||
# 上传文件
|
||||
s3_client.upload_file(local_file_path, self.bucket_name, remote_key)
|
||||
uploaded_files.append(remote_key)
|
||||
|
||||
return {
|
||||
"success": True,
|
||||
"bucket": self.bucket_name,
|
||||
"local_directory": str(local_path),
|
||||
"remote_prefix": remote_prefix,
|
||||
"uploaded_files": uploaded_files,
|
||||
"file_count": len(uploaded_files),
|
||||
"upload_time": datetime.now().isoformat()
|
||||
}
|
||||
|
||||
except Exception as e:
|
||||
return {
|
||||
"success": False,
|
||||
"error": f"上传目录失败: {str(e)}"
|
||||
}
|
||||
|
||||
def download_file(
|
||||
self,
|
||||
remote_key: str,
|
||||
local_file_path: str
|
||||
) -> Dict[str, Union[bool, str, int]]:
|
||||
"""
|
||||
从 S3 下载文件
|
||||
|
||||
Args:
|
||||
remote_key: 远程对象键
|
||||
local_file_path: 本地保存路径
|
||||
|
||||
Returns:
|
||||
包含下载信息的字典
|
||||
"""
|
||||
try:
|
||||
s3_client = self._create_s3_client("upload") # 使用上传客户端进行下载
|
||||
local_path = Path(local_file_path)
|
||||
|
||||
# 确保本地目录存在
|
||||
local_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
# 下载文件
|
||||
s3_client.download_file(
|
||||
self.bucket_name,
|
||||
remote_key,
|
||||
str(local_path)
|
||||
)
|
||||
|
||||
return {
|
||||
"success": True,
|
||||
"bucket": self.bucket_name,
|
||||
"key": remote_key,
|
||||
"local_path": str(local_path),
|
||||
"file_size": local_path.stat().st_size,
|
||||
"download_time": datetime.now().isoformat()
|
||||
}
|
||||
|
||||
except Exception as e:
|
||||
return {
|
||||
"success": False,
|
||||
"error": f"下载文件失败: {str(e)}"
|
||||
}
|
||||
|
||||
def download_directory(
|
||||
self,
|
||||
remote_prefix: str,
|
||||
local_dir: str
|
||||
) -> Dict[str, Union[bool, str, int, List[str]]]:
|
||||
"""
|
||||
从 S3 下载整个目录
|
||||
|
||||
Args:
|
||||
remote_prefix: 远程路径前缀
|
||||
local_dir: 本地保存目录
|
||||
|
||||
Returns:
|
||||
包含下载结果的字典
|
||||
"""
|
||||
try:
|
||||
s3_client = self._create_s3_client("list")
|
||||
local_path = Path(local_dir)
|
||||
local_path.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
# 列出远程目录中的所有文件
|
||||
response = s3_client.list_objects_v2(
|
||||
Bucket=self.bucket_name,
|
||||
Prefix=remote_prefix
|
||||
)
|
||||
|
||||
downloaded_files = []
|
||||
for obj in response.get('Contents', []):
|
||||
key = obj['Key']
|
||||
|
||||
# 跳过文件夹对象
|
||||
if key.endswith('/'):
|
||||
continue
|
||||
|
||||
# 计算本地文件路径
|
||||
relative_path = key[len(remote_prefix):].lstrip('/')
|
||||
if not relative_path:
|
||||
continue
|
||||
|
||||
local_file_path = local_path / relative_path
|
||||
local_file_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
# 下载文件
|
||||
s3_client.download_file(self.bucket_name, key, str(local_file_path))
|
||||
downloaded_files.append(str(local_file_path))
|
||||
|
||||
return {
|
||||
"success": True,
|
||||
"bucket": self.bucket_name,
|
||||
"remote_prefix": remote_prefix,
|
||||
"local_directory": str(local_path),
|
||||
"downloaded_files": downloaded_files,
|
||||
"file_count": len(downloaded_files),
|
||||
"download_time": datetime.now().isoformat()
|
||||
}
|
||||
|
||||
except Exception as e:
|
||||
return {
|
||||
"success": False,
|
||||
"error": f"下载目录失败: {str(e)}"
|
||||
}
|
||||
|
||||
def list_files(
|
||||
self,
|
||||
prefix: str = "",
|
||||
max_keys: int = 1000
|
||||
) -> Dict[str, Union[bool, str, int, List[Dict]]]:
|
||||
"""
|
||||
列出 S3 中的文件
|
||||
|
||||
Args:
|
||||
prefix: 文件前缀过滤
|
||||
max_keys: 最大返回数量
|
||||
|
||||
Returns:
|
||||
包含文件列表的字典
|
||||
"""
|
||||
try:
|
||||
s3_client = self._create_s3_client("list")
|
||||
|
||||
# 尝试多种列表方法
|
||||
list_methods = [
|
||||
lambda: s3_client.list_objects_v2(
|
||||
Bucket=self.bucket_name,
|
||||
Prefix=prefix,
|
||||
MaxKeys=max_keys
|
||||
),
|
||||
lambda: s3_client.list_objects(
|
||||
Bucket=self.bucket_name,
|
||||
Prefix=prefix,
|
||||
MaxKeys=max_keys
|
||||
)
|
||||
]
|
||||
|
||||
response = None
|
||||
for method in list_methods:
|
||||
try:
|
||||
response = method()
|
||||
break
|
||||
except Exception:
|
||||
continue
|
||||
|
||||
if response is None:
|
||||
return {
|
||||
"success": False,
|
||||
"error": "所有列表方法都失败了"
|
||||
}
|
||||
|
||||
files = []
|
||||
for obj in response.get('Contents', []):
|
||||
files.append({
|
||||
"key": obj['Key'],
|
||||
"size": obj['Size'],
|
||||
"last_modified": obj['LastModified'].isoformat(),
|
||||
"public_url": f"{self.endpoint_url}/{self.bucket_name}/{obj['Key']}"
|
||||
})
|
||||
|
||||
return {
|
||||
"success": True,
|
||||
"bucket": self.bucket_name,
|
||||
"prefix": prefix,
|
||||
"files": files,
|
||||
"file_count": len(files),
|
||||
"list_time": datetime.now().isoformat()
|
||||
}
|
||||
|
||||
except Exception as e:
|
||||
return {
|
||||
"success": False,
|
||||
"error": f"列出文件失败: {str(e)}"
|
||||
}
|
||||
|
||||
def delete_file(self, remote_key: str) -> Dict[str, Union[bool, str]]:
|
||||
"""
|
||||
删除 S3 中的文件
|
||||
|
||||
Args:
|
||||
remote_key: 远程对象键
|
||||
|
||||
Returns:
|
||||
包含删除结果的字典
|
||||
"""
|
||||
try:
|
||||
s3_client = self._create_s3_client("delete")
|
||||
|
||||
# 删除文件
|
||||
s3_client.delete_object(
|
||||
Bucket=self.bucket_name,
|
||||
Key=remote_key
|
||||
)
|
||||
|
||||
# 验证文件已被删除
|
||||
try:
|
||||
s3_client.head_object(Bucket=self.bucket_name, Key=remote_key)
|
||||
return {
|
||||
"success": False,
|
||||
"error": "文件删除验证失败:文件仍然存在"
|
||||
}
|
||||
except ClientError as e:
|
||||
if e.response['Error']['Code'] == '404':
|
||||
return {
|
||||
"success": True,
|
||||
"bucket": self.bucket_name,
|
||||
"key": remote_key,
|
||||
"delete_time": datetime.now().isoformat()
|
||||
}
|
||||
else:
|
||||
raise e
|
||||
|
||||
except Exception as e:
|
||||
return {
|
||||
"success": False,
|
||||
"error": f"删除文件失败: {str(e)}"
|
||||
}
|
||||
|
||||
def delete_directory(self, remote_prefix: str) -> Dict[str, Union[bool, str, int, List[str]]]:
|
||||
"""
|
||||
删除 S3 中的整个目录
|
||||
|
||||
Args:
|
||||
remote_prefix: 远程路径前缀
|
||||
|
||||
Returns:
|
||||
包含删除结果的字典
|
||||
"""
|
||||
try:
|
||||
s3_client = self._create_s3_client("delete")
|
||||
|
||||
# 首先列出目录中的所有文件
|
||||
response = s3_client.list_objects_v2(
|
||||
Bucket=self.bucket_name,
|
||||
Prefix=remote_prefix
|
||||
)
|
||||
|
||||
objects = response.get('Contents', [])
|
||||
if not objects:
|
||||
return {
|
||||
"success": True,
|
||||
"message": "目录为空或不存在",
|
||||
"deleted_count": 0
|
||||
}
|
||||
|
||||
# 尝试批量删除
|
||||
try:
|
||||
delete_response = s3_client.delete_objects(
|
||||
Bucket=self.bucket_name,
|
||||
Delete={'Objects': [{'Key': obj['Key']} for obj in objects]}
|
||||
)
|
||||
|
||||
deleted_count = len(delete_response.get('Deleted', []))
|
||||
errors = delete_response.get('Errors', [])
|
||||
|
||||
if errors:
|
||||
error_messages = [f"{error['Key']}: {error['Message']}" for error in errors]
|
||||
return {
|
||||
"success": False,
|
||||
"error": f"批量删除部分失败: {error_messages}",
|
||||
"deleted_count": deleted_count,
|
||||
"failed_count": len(errors)
|
||||
}
|
||||
|
||||
return {
|
||||
"success": True,
|
||||
"bucket": self.bucket_name,
|
||||
"remote_prefix": remote_prefix,
|
||||
"deleted_count": deleted_count,
|
||||
"delete_time": datetime.now().isoformat()
|
||||
}
|
||||
|
||||
except Exception:
|
||||
# 如果批量删除失败,尝试逐个删除
|
||||
deleted_count = 0
|
||||
failed_files = []
|
||||
|
||||
for obj in objects:
|
||||
try:
|
||||
s3_client.delete_object(Bucket=self.bucket_name, Key=obj['Key'])
|
||||
deleted_count += 1
|
||||
except Exception as e:
|
||||
failed_files.append(f"{obj['Key']}: {str(e)}")
|
||||
|
||||
if failed_files:
|
||||
return {
|
||||
"success": False,
|
||||
"error": f"逐个删除部分失败: {failed_files}",
|
||||
"deleted_count": deleted_count,
|
||||
"failed_count": len(failed_files)
|
||||
}
|
||||
|
||||
return {
|
||||
"success": True,
|
||||
"bucket": self.bucket_name,
|
||||
"remote_prefix": remote_prefix,
|
||||
"deleted_count": deleted_count,
|
||||
"delete_time": datetime.now().isoformat()
|
||||
}
|
||||
|
||||
except Exception as e:
|
||||
return {
|
||||
"success": False,
|
||||
"error": f"删除目录失败: {str(e)}"
|
||||
}
|
||||
243
tests/test_toolkit.py
Normal file
243
tests/test_toolkit.py
Normal file
@@ -0,0 +1,243 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
RustFS S3 Storage Toolkit 测试文件
|
||||
基于成功的 test_s3_flexible.py 创建的测试套件
|
||||
已在 RustFS 1.0.0-alpha.34 上完成测试
|
||||
"""
|
||||
|
||||
import os
|
||||
import tempfile
|
||||
import shutil
|
||||
from pathlib import Path
|
||||
from datetime import datetime
|
||||
|
||||
from rustfs_s3_toolkit import S3StorageToolkit
|
||||
|
||||
|
||||
# 测试配置 - 请根据实际情况修改
|
||||
TEST_CONFIG = {
|
||||
"endpoint_url": "https://rfs.jmsu.top",
|
||||
"access_key_id": "lingyuzeng",
|
||||
"secret_access_key": "rustAdminlingyuzeng",
|
||||
"bucket_name": "test",
|
||||
"region_name": "us-east-1"
|
||||
}
|
||||
|
||||
|
||||
def create_test_files():
|
||||
"""创建测试文件和目录"""
|
||||
test_dir = tempfile.mkdtemp(prefix='s3_toolkit_test_')
|
||||
|
||||
# 创建测试文件
|
||||
test_files = {
|
||||
"test.txt": "这是一个测试文件\n时间: " + datetime.now().isoformat(),
|
||||
"data.json": '{"name": "test", "value": 123, "timestamp": "' + datetime.now().isoformat() + '"}',
|
||||
"readme.md": "# 测试文件\n\n这是一个测试用的 Markdown 文件。\n\n- 项目: S3 存储测试\n- 时间: " + datetime.now().isoformat(),
|
||||
}
|
||||
|
||||
# 创建子目录和文件
|
||||
sub_dir = Path(test_dir) / "subdir"
|
||||
sub_dir.mkdir()
|
||||
|
||||
for filename, content in test_files.items():
|
||||
file_path = Path(test_dir) / filename
|
||||
with open(file_path, 'w', encoding='utf-8') as f:
|
||||
f.write(content)
|
||||
|
||||
# 在子目录中创建文件
|
||||
sub_file = sub_dir / "sub_test.txt"
|
||||
with open(sub_file, 'w', encoding='utf-8') as f:
|
||||
f.write("子目录中的测试文件\n时间: " + datetime.now().isoformat())
|
||||
|
||||
return test_dir
|
||||
|
||||
|
||||
def test_all_functions():
|
||||
"""测试所有 9 个核心功能"""
|
||||
print("🚀 RustFS S3 Storage Toolkit 完整功能测试")
|
||||
print("🧪 测试环境: RustFS 1.0.0-alpha.34")
|
||||
print("=" * 60)
|
||||
|
||||
# 初始化工具包
|
||||
try:
|
||||
toolkit = S3StorageToolkit(**TEST_CONFIG)
|
||||
print("✅ 工具包初始化成功")
|
||||
except Exception as e:
|
||||
print(f"❌ 工具包初始化失败: {e}")
|
||||
return False
|
||||
|
||||
test_results = {}
|
||||
|
||||
# 1. 测试连接
|
||||
print("\n1. 测试连接...")
|
||||
result = toolkit.test_connection()
|
||||
test_results['connection'] = result['success']
|
||||
if result['success']:
|
||||
print(f"✅ 连接成功: {result['message']}")
|
||||
else:
|
||||
print(f"❌ 连接失败: {result['error']}")
|
||||
return test_results
|
||||
|
||||
# 2. 测试上传文件
|
||||
print("\n2. 测试上传文件...")
|
||||
test_content = f"测试文件内容\n时间: {datetime.now().isoformat()}"
|
||||
with tempfile.NamedTemporaryFile(mode='w', delete=False, suffix='.txt', encoding='utf-8') as f:
|
||||
f.write(test_content)
|
||||
test_file = f.name
|
||||
|
||||
timestamp = datetime.now().strftime("%Y%m%d_%H%M%S")
|
||||
remote_key = f"test_files/single_file_{timestamp}.txt"
|
||||
|
||||
result = toolkit.upload_file(test_file, remote_key)
|
||||
test_results['upload_file'] = result['success']
|
||||
if result['success']:
|
||||
print(f"✅ 文件上传成功: {remote_key}")
|
||||
uploaded_file_key = remote_key
|
||||
else:
|
||||
print(f"❌ 文件上传失败: {result['error']}")
|
||||
uploaded_file_key = None
|
||||
|
||||
os.unlink(test_file)
|
||||
|
||||
# 3. 测试创建文件夹
|
||||
print("\n3. 测试创建文件夹...")
|
||||
folder_path = f"test_folders/folder_{timestamp}/"
|
||||
result = toolkit.create_folder(folder_path)
|
||||
test_results['create_folder'] = result['success']
|
||||
if result['success']:
|
||||
print(f"✅ 文件夹创建成功: {folder_path}")
|
||||
else:
|
||||
print(f"❌ 文件夹创建失败: {result['error']}")
|
||||
|
||||
# 4. 测试上传目录
|
||||
print("\n4. 测试上传目录...")
|
||||
test_dir = create_test_files()
|
||||
remote_prefix = f"test_directories/dir_{timestamp}/"
|
||||
|
||||
result = toolkit.upload_directory(test_dir, remote_prefix)
|
||||
test_results['upload_directory'] = result['success']
|
||||
if result['success']:
|
||||
print(f"✅ 目录上传成功: 上传了 {result['file_count']} 个文件")
|
||||
uploaded_dir_prefix = remote_prefix
|
||||
else:
|
||||
print(f"❌ 目录上传失败: {result['error']}")
|
||||
uploaded_dir_prefix = None
|
||||
|
||||
shutil.rmtree(test_dir)
|
||||
|
||||
# 5. 测试下载文件
|
||||
print("\n5. 测试下载文件...")
|
||||
if uploaded_file_key:
|
||||
download_path = tempfile.mktemp(suffix='_downloaded.txt')
|
||||
result = toolkit.download_file(uploaded_file_key, download_path)
|
||||
test_results['download_file'] = result['success']
|
||||
if result['success']:
|
||||
print(f"✅ 文件下载成功: {download_path}")
|
||||
# 验证内容
|
||||
with open(download_path, 'r', encoding='utf-8') as f:
|
||||
downloaded_content = f.read()
|
||||
if test_content == downloaded_content:
|
||||
print("✅ 文件内容验证成功")
|
||||
else:
|
||||
print("❌ 文件内容验证失败")
|
||||
os.unlink(download_path)
|
||||
else:
|
||||
print(f"❌ 文件下载失败: {result['error']}")
|
||||
else:
|
||||
test_results['download_file'] = False
|
||||
print("❌ 跳过文件下载测试(没有可下载的文件)")
|
||||
|
||||
# 6. 测试下载目录
|
||||
print("\n6. 测试下载目录...")
|
||||
if uploaded_dir_prefix:
|
||||
download_dir = tempfile.mkdtemp(prefix='s3_download_')
|
||||
result = toolkit.download_directory(uploaded_dir_prefix, download_dir)
|
||||
test_results['download_directory'] = result['success']
|
||||
if result['success']:
|
||||
print(f"✅ 目录下载成功: 下载了 {result['file_count']} 个文件")
|
||||
else:
|
||||
print(f"❌ 目录下载失败: {result['error']}")
|
||||
shutil.rmtree(download_dir)
|
||||
else:
|
||||
test_results['download_directory'] = False
|
||||
print("❌ 跳过目录下载测试(没有可下载的目录)")
|
||||
|
||||
# 7. 测试列出文件
|
||||
print("\n7. 测试列出文件...")
|
||||
result = toolkit.list_files(max_keys=20)
|
||||
test_results['list_files'] = result['success']
|
||||
if result['success']:
|
||||
print(f"✅ 文件列表获取成功: 找到 {result['file_count']} 个文件")
|
||||
if result['files']:
|
||||
print("📄 前几个文件:")
|
||||
for i, file_info in enumerate(result['files'][:5], 1):
|
||||
size_mb = file_info['size'] / (1024 * 1024)
|
||||
print(f" {i}. {file_info['key']} ({size_mb:.2f} MB)")
|
||||
else:
|
||||
print(f"❌ 文件列表获取失败: {result['error']}")
|
||||
|
||||
# 8. 测试删除文件
|
||||
print("\n8. 测试删除文件...")
|
||||
if uploaded_file_key:
|
||||
result = toolkit.delete_file(uploaded_file_key)
|
||||
test_results['delete_file'] = result['success']
|
||||
if result['success']:
|
||||
print(f"✅ 文件删除成功: {uploaded_file_key}")
|
||||
else:
|
||||
print(f"❌ 文件删除失败: {result['error']}")
|
||||
else:
|
||||
test_results['delete_file'] = False
|
||||
print("❌ 跳过文件删除测试(没有可删除的文件)")
|
||||
|
||||
# 9. 测试删除目录
|
||||
print("\n9. 测试删除目录...")
|
||||
if uploaded_dir_prefix:
|
||||
result = toolkit.delete_directory(uploaded_dir_prefix)
|
||||
test_results['delete_directory'] = result['success']
|
||||
if result['success']:
|
||||
print(f"✅ 目录删除成功: 删除了 {result['deleted_count']} 个文件")
|
||||
else:
|
||||
print(f"❌ 目录删除失败: {result['error']}")
|
||||
else:
|
||||
test_results['delete_directory'] = False
|
||||
print("❌ 跳过目录删除测试(没有可删除的目录)")
|
||||
|
||||
# 打印测试摘要
|
||||
print("\n" + "="*60)
|
||||
print("🎯 测试结果摘要")
|
||||
print("="*60)
|
||||
|
||||
test_names = {
|
||||
'connection': 'Connection',
|
||||
'upload_file': 'Upload File',
|
||||
'create_folder': 'Create Folder',
|
||||
'upload_directory': 'Upload Directory',
|
||||
'download_file': 'Download File',
|
||||
'download_directory': 'Download Directory',
|
||||
'list_files': 'List Files',
|
||||
'delete_file': 'Delete File',
|
||||
'delete_directory': 'Delete Directory'
|
||||
}
|
||||
|
||||
passed_tests = 0
|
||||
for test_key, test_name in test_names.items():
|
||||
status = "✅ 通过" if test_results.get(test_key, False) else "❌ 失败"
|
||||
print(f"{test_name:20} : {status}")
|
||||
if test_results.get(test_key, False):
|
||||
passed_tests += 1
|
||||
|
||||
print("-" * 60)
|
||||
print(f"总计: {passed_tests}/{len(test_names)} 个测试通过")
|
||||
|
||||
if passed_tests == len(test_names):
|
||||
print("🎉 所有测试都通过了!RustFS S3 Storage Toolkit 工作正常。")
|
||||
elif passed_tests > 0:
|
||||
print("⚠️ 部分测试通过。请检查失败的测试项。")
|
||||
else:
|
||||
print("❌ 所有测试都失败了。请检查 RustFS 配置和网络连接。")
|
||||
|
||||
return test_results
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
test_all_functions()
|
||||
Reference in New Issue
Block a user