Initial commit

This commit is contained in:
mm644706215
2025-07-23 22:44:33 +08:00
commit 5d6ed09fd5
10 changed files with 1709 additions and 0 deletions

3
.gitignore vendored Normal file
View File

@@ -0,0 +1,3 @@
.venv/
__pycache__/
*.pyc

21
LICENSE Normal file
View File

@@ -0,0 +1,21 @@
MIT License
Copyright (c) 2024 mm644706215
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.

370
README.md Normal file
View File

@@ -0,0 +1,370 @@
# RustFS S3 Storage Toolkit
一个专为 RustFS 设计的 S3 兼容对象存储工具包,同时支持其他各种 S3 兼容存储服务。提供完整的文件和目录操作功能,具有高度的可复用性和工具性。
## 🧪 测试环境
本工具包已在以下 RustFS 版本上完成全面测试:
```
rustfs 1.0.0-alpha.34
build time : 2025-07-21 08:13:28 +00:00
build profile: release
build os : linux-x86_64
rust version : rustc 1.88.0 (6b00bc388 2025-06-23)
rust channel : stable-x86_64-unknown-linux-gnu
git commit : 3f095e75cb1276adf47a05472c8cc608eaa51504
git tag : 1.0.0-alpha.34
```
**注意**: 虽然理论上兼容其他 S3 兼容服务,但建议在使用前进行自行测试以确保兼容性。
## 🚀 核心功能
本工具包提供以下 9 个核心功能,全部测试通过:
-**Connection (连接测试)**: 验证 S3 服务连接和存储桶访问权限
-**Upload File (上传文件)**: 上传单个文件到 S3 存储
-**Create Folder (创建文件夹)**: 在 S3 中创建文件夹结构
-**Upload Directory (上传目录)**: 递归上传整个目录及其子目录
-**Download File (下载文件)**: 从 S3 下载单个文件
-**Download Directory (下载目录)**: 下载整个目录及其所有文件
-**List Files (列出文件)**: 列出存储桶中的文件和目录
-**Delete File (删除文件)**: 删除 S3 中的单个文件
-**Delete Directory (删除目录)**: 删除整个目录及其所有文件
## 🎯 设计特点
- **易于使用**: 简单的类实例化和方法调用
- **高度可复用**: 一次配置,多次使用
- **工具性强**: 专注于核心功能,无冗余依赖
- **兼容性好**: 支持各种 S3 兼容存储服务
- **错误处理**: 完善的异常处理和错误信息返回
- **灵活配置**: 自动适配不同 S3 服务的签名要求
## 📦 安装
### 从 PyPI 安装 (推荐)
```bash
pip install rustfs-s3-toolkit
```
### 开发版本安装
```bash
# 克隆项目
git clone https://github.com/mm644706215/rustfs-s3-toolkit.git
cd rustfs-s3-toolkit
# 安装开发版本
pip install -e .
```
## 🔧 支持的存储服务
### 已测试服务
- **RustFS 1.0.0-alpha.34**: 主要测试目标,所有功能完全兼容 ✅
### 理论兼容服务(需自行测试)
- **AWS S3**: 亚马逊云存储服务
- **MinIO**: 开源对象存储服务
- **Alibaba Cloud OSS**: 阿里云对象存储
- **Tencent Cloud COS**: 腾讯云对象存储
- **其他 S3 兼容服务**: 任何支持 S3 API 的存储服务
**重要提示**: 除 RustFS 1.0.0-alpha.34 外,其他服务虽然理论上兼容,但建议在生产环境使用前进行充分测试。
## 📖 快速开始
### 基本使用
```python
from rustfs_s3_toolkit import S3StorageToolkit
# 初始化工具包(以 RustFS 为例)
toolkit = S3StorageToolkit(
endpoint_url="https://your-rustfs-endpoint.com",
access_key_id="your-access-key-id",
secret_access_key="your-secret-access-key",
bucket_name="your-bucket-name",
region_name="us-east-1" # 可选,默认 us-east-1
)
# 测试连接
result = toolkit.test_connection()
if result['success']:
print(f"连接成功: {result['message']}")
else:
print(f"连接失败: {result['error']}")
```
## 📚 详细使用方法
### 1. 连接测试 (Connection)
```python
# 测试 S3 连接和存储桶访问权限
result = toolkit.test_connection()
# 返回结果
{
"success": True,
"bucket_count": 5,
"bucket_names": ["bucket1", "bucket2", ...],
"target_bucket_exists": True,
"message": "连接成功!找到 5 个存储桶"
}
```
### 2. 上传文件 (Upload File)
```python
# 上传单个文件
result = toolkit.upload_file(
local_file_path="/path/to/local/file.txt",
remote_key="remote/path/file.txt",
metadata={"author": "user", "type": "document"} # 可选
)
# 返回结果
{
"success": True,
"bucket": "your-bucket",
"key": "remote/path/file.txt",
"public_url": "https://endpoint/bucket/remote/path/file.txt",
"file_size": 1024,
"upload_time": "2024-01-01T12:00:00"
}
```
### 3. 创建文件夹 (Create Folder)
```python
# 创建文件夹S3 中通过空对象实现)
result = toolkit.create_folder("my-folder/sub-folder/")
# 返回结果
{
"success": True,
"bucket": "your-bucket",
"folder_path": "my-folder/sub-folder/",
"create_time": "2024-01-01T12:00:00"
}
```
### 4. 上传目录 (Upload Directory)
```python
# 递归上传整个目录
result = toolkit.upload_directory(
local_dir="/path/to/local/directory",
remote_prefix="remote/directory/"
)
# 返回结果
{
"success": True,
"bucket": "your-bucket",
"local_directory": "/path/to/local/directory",
"remote_prefix": "remote/directory/",
"uploaded_files": ["remote/directory/file1.txt", "remote/directory/sub/file2.txt"],
"file_count": 2,
"upload_time": "2024-01-01T12:00:00"
}
```
### 5. 下载文件 (Download File)
```python
# 下载单个文件
result = toolkit.download_file(
remote_key="remote/path/file.txt",
local_file_path="/path/to/save/file.txt"
)
# 返回结果
{
"success": True,
"bucket": "your-bucket",
"key": "remote/path/file.txt",
"local_path": "/path/to/save/file.txt",
"file_size": 1024,
"download_time": "2024-01-01T12:00:00"
}
```
### 6. 下载目录 (Download Directory)
```python
# 下载整个目录
result = toolkit.download_directory(
remote_prefix="remote/directory/",
local_dir="/path/to/save/directory"
)
# 返回结果
{
"success": True,
"bucket": "your-bucket",
"remote_prefix": "remote/directory/",
"local_directory": "/path/to/save/directory",
"downloaded_files": ["/path/to/save/directory/file1.txt", ...],
"file_count": 2,
"download_time": "2024-01-01T12:00:00"
}
```
### 7. 列出文件 (List Files)
```python
# 列出存储桶中的文件
result = toolkit.list_files(
prefix="my-folder/", # 可选,过滤前缀
max_keys=100 # 可选,最大返回数量
)
# 返回结果
{
"success": True,
"bucket": "your-bucket",
"prefix": "my-folder/",
"files": [
{
"key": "my-folder/file1.txt",
"size": 1024,
"last_modified": "2024-01-01T12:00:00",
"public_url": "https://endpoint/bucket/my-folder/file1.txt"
}
],
"file_count": 1,
"list_time": "2024-01-01T12:00:00"
}
```
### 8. 删除文件 (Delete File)
```python
# 删除单个文件
result = toolkit.delete_file("remote/path/file.txt")
# 返回结果
{
"success": True,
"bucket": "your-bucket",
"key": "remote/path/file.txt",
"delete_time": "2024-01-01T12:00:00"
}
```
### 9. 删除目录 (Delete Directory)
```python
# 删除整个目录及其所有文件
result = toolkit.delete_directory("remote/directory/")
# 返回结果
{
"success": True,
"bucket": "your-bucket",
"remote_prefix": "remote/directory/",
"deleted_count": 5,
"delete_time": "2024-01-01T12:00:00"
}
```
## 🔧 配置说明
### 基本配置参数
| 参数 | 类型 | 必需 | 说明 |
|------|------|------|------|
| `endpoint_url` | str | ✅ | S3 服务端点 URL |
| `access_key_id` | str | ✅ | 访问密钥 ID |
| `secret_access_key` | str | ✅ | 访问密钥 |
| `bucket_name` | str | ✅ | 存储桶名称 |
| `region_name` | str | ❌ | 区域名称,默认 "us-east-1" |
### 常见配置示例
#### AWS S3
```python
toolkit = S3StorageToolkit(
endpoint_url="https://s3.amazonaws.com",
access_key_id="AKIAIOSFODNN7EXAMPLE",
secret_access_key="wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
bucket_name="my-bucket",
region_name="us-west-2"
)
```
#### MinIO
```python
toolkit = S3StorageToolkit(
endpoint_url="http://localhost:9000",
access_key_id="minioadmin",
secret_access_key="minioadmin",
bucket_name="my-bucket"
)
```
#### RustFS (已测试版本 1.0.0-alpha.34)
```python
toolkit = S3StorageToolkit(
endpoint_url="https://your-rustfs-endpoint.com",
access_key_id="your-access-key",
secret_access_key="your-secret-key",
bucket_name="your-bucket-name"
)
```
## 🧪 测试
运行完整的功能测试:
```bash
# 运行测试套件
python tests/test_toolkit.py
# 运行基本使用示例
python examples/basic_usage.py
```
测试将验证所有 9 个核心功能是否正常工作。
## 📁 项目结构
```
rustfs-s3-toolkit/
├── README.md # 项目文档
├── pyproject.toml # 项目配置
├── LICENSE # MIT 许可证
├── install.py # 安装脚本
├── build.py # 构建脚本
├── src/
│ └── rustfs_s3_toolkit/
│ ├── __init__.py # 包初始化
│ └── s3_client.py # 核心工具类
├── tests/
│ └── test_toolkit.py # 测试套件
└── examples/
└── basic_usage.py # 使用示例
```
## 🤝 贡献
欢迎提交 Issue 和 Pull Request
## 📄 许可证
MIT License
## 🔗 相关链接
- [GitHub 仓库](https://github.com/mm644706215/rustfs-s3-toolkit)
- [PyPI 包](https://pypi.org/project/rustfs-s3-toolkit/)
- [问题反馈](https://github.com/mm644706215/rustfs-s3-toolkit/issues)
---
**RustFS S3 Storage Toolkit** - 专为 RustFS 设计,让 S3 对象存储操作变得简单高效!

104
build.py Normal file
View File

@@ -0,0 +1,104 @@
#!/usr/bin/env python3
"""
RustFS S3 Storage Toolkit 构建和发布脚本
"""
import subprocess
import sys
import os
from pathlib import Path
def run_command(command, description):
"""运行命令并显示结果"""
print(f"🔧 {description}...")
try:
result = subprocess.run(command, shell=True, check=True, capture_output=True, text=True)
print(f"{description}成功")
if result.stdout.strip():
print(f"输出: {result.stdout.strip()}")
return True
except subprocess.CalledProcessError as e:
print(f"{description}失败: {e}")
if e.stdout:
print(f"输出: {e.stdout}")
if e.stderr:
print(f"错误: {e.stderr}")
return False
def main():
"""主构建流程"""
print("🚀 RustFS S3 Storage Toolkit 构建脚本")
print("=" * 50)
# 检查当前目录
current_dir = Path.cwd()
if not (current_dir / "pyproject.toml").exists():
print("❌ 请在项目根目录运行此脚本")
sys.exit(1)
print(f"✅ 项目目录: {current_dir}")
# 清理旧的构建文件
print("\n🧹 清理旧的构建文件...")
cleanup_commands = [
"rm -rf dist/",
"rm -rf build/",
"rm -rf *.egg-info/",
"find . -name '__pycache__' -type d -exec rm -rf {} + 2>/dev/null || true",
"find . -name '*.pyc' -delete 2>/dev/null || true"
]
for command in cleanup_commands:
subprocess.run(command, shell=True, capture_output=True)
print("✅ 清理完成")
# 安装构建依赖
build_deps = [
"pip install --upgrade pip",
"pip install --upgrade build twine"
]
for command in build_deps:
if not run_command(command, f"安装构建依赖"):
print("❌ 安装构建依赖失败")
sys.exit(1)
# 构建包
if not run_command("python -m build", "构建包"):
print("❌ 构建失败")
sys.exit(1)
# 检查构建结果
dist_dir = current_dir / "dist"
if not dist_dir.exists():
print("❌ 构建目录不存在")
sys.exit(1)
built_files = list(dist_dir.glob("*"))
if not built_files:
print("❌ 没有找到构建文件")
sys.exit(1)
print(f"\n📦 构建完成!生成的文件:")
for file in built_files:
print(f" - {file.name}")
# 验证包
if not run_command("python -m twine check dist/*", "验证包"):
print("❌ 包验证失败")
sys.exit(1)
print("\n🎉 构建和验证完成!")
print("\n📖 发布到 PyPI:")
print("1. 测试发布: python -m twine upload --repository testpypi dist/*")
print("2. 正式发布: python -m twine upload dist/*")
print("\n💡 本地安装测试:")
print("pip install dist/*.whl")
if __name__ == "__main__":
main()

193
examples/basic_usage.py Normal file
View File

@@ -0,0 +1,193 @@
#!/usr/bin/env python3
"""
RustFS S3 Storage Toolkit 基本使用示例
演示所有 9 个核心功能的使用方法
已在 RustFS 1.0.0-alpha.34 上完成测试
"""
import tempfile
from pathlib import Path
from datetime import datetime
from rustfs_s3_toolkit import S3StorageToolkit
def main():
"""RustFS S3 Storage Toolkit 使用示例"""
# 1. 初始化工具包
print("🚀 RustFS S3 Storage Toolkit 使用示例")
print("🧪 测试环境: RustFS 1.0.0-alpha.34")
print("=" * 50)
# 配置信息 - 请根据实际 RustFS 情况修改
config = {
"endpoint_url": "https://your-rustfs-endpoint.com",
"access_key_id": "your-access-key-id",
"secret_access_key": "your-secret-access-key",
"bucket_name": "your-bucket-name",
"region_name": "us-east-1"
}
# 创建工具包实例
toolkit = S3StorageToolkit(**config)
print("✅ 工具包初始化完成")
# 2. 测试连接
print("\n📡 测试连接...")
result = toolkit.test_connection()
if result['success']:
print(f"✅ 连接成功: {result['message']}")
print(f"📊 存储桶数量: {result['bucket_count']}")
print(f"🎯 目标存储桶存在: {result['target_bucket_exists']}")
else:
print(f"❌ 连接失败: {result['error']}")
return
# 3. 上传文件
print("\n📤 上传文件...")
# 创建测试文件
test_content = f"Hello RustFS S3 Storage Toolkit!\n创建时间: {datetime.now().isoformat()}"
with tempfile.NamedTemporaryFile(mode='w', delete=False, suffix='.txt', encoding='utf-8') as f:
f.write(test_content)
test_file = f.name
# 上传文件
timestamp = datetime.now().strftime("%Y%m%d_%H%M%S")
remote_key = f"examples/test_file_{timestamp}.txt"
result = toolkit.upload_file(test_file, remote_key)
if result['success']:
print(f"✅ 文件上传成功")
print(f"📁 远程路径: {result['key']}")
print(f"🔗 公开链接: {result['public_url']}")
print(f"📊 文件大小: {result['file_size']} 字节")
else:
print(f"❌ 文件上传失败: {result['error']}")
# 4. 创建文件夹
print("\n📁 创建文件夹...")
folder_path = f"examples/test_folder_{timestamp}/"
result = toolkit.create_folder(folder_path)
if result['success']:
print(f"✅ 文件夹创建成功: {result['folder_path']}")
else:
print(f"❌ 文件夹创建失败: {result['error']}")
# 5. 上传目录
print("\n📂 上传目录...")
# 创建测试目录
test_dir = tempfile.mkdtemp(prefix='s3_example_')
test_dir_path = Path(test_dir)
# 创建一些测试文件
(test_dir_path / "file1.txt").write_text("这是文件1", encoding='utf-8')
(test_dir_path / "file2.txt").write_text("这是文件2", encoding='utf-8')
# 创建子目录
sub_dir = test_dir_path / "subdir"
sub_dir.mkdir()
(sub_dir / "file3.txt").write_text("这是子目录中的文件", encoding='utf-8')
# 上传目录
remote_prefix = f"examples/test_directory_{timestamp}/"
result = toolkit.upload_directory(test_dir, remote_prefix)
if result['success']:
print(f"✅ 目录上传成功")
print(f"📁 本地目录: {result['local_directory']}")
print(f"🌐 远程前缀: {result['remote_prefix']}")
print(f"📊 文件数量: {result['file_count']}")
print("📄 上传的文件:")
for file_key in result['uploaded_files']:
print(f" - {file_key}")
else:
print(f"❌ 目录上传失败: {result['error']}")
# 6. 列出文件
print("\n📋 列出文件...")
result = toolkit.list_files(prefix="examples/", max_keys=10)
if result['success']:
print(f"✅ 文件列表获取成功")
print(f"📊 文件数量: {result['file_count']}")
print("📄 文件列表:")
for file_info in result['files']:
size_mb = file_info['size'] / (1024 * 1024)
print(f" - {file_info['key']} ({size_mb:.3f} MB)")
else:
print(f"❌ 文件列表获取失败: {result['error']}")
# 7. 下载文件
print("\n📥 下载文件...")
download_path = tempfile.mktemp(suffix='_downloaded.txt')
result = toolkit.download_file(remote_key, download_path)
if result['success']:
print(f"✅ 文件下载成功")
print(f"📁 本地路径: {result['local_path']}")
print(f"📊 文件大小: {result['file_size']} 字节")
# 验证下载的内容
with open(download_path, 'r', encoding='utf-8') as f:
downloaded_content = f.read()
if test_content == downloaded_content:
print("✅ 文件内容验证成功")
else:
print("❌ 文件内容验证失败")
else:
print(f"❌ 文件下载失败: {result['error']}")
# 8. 下载目录
print("\n📂 下载目录...")
download_dir = tempfile.mkdtemp(prefix='s3_download_')
result = toolkit.download_directory(remote_prefix, download_dir)
if result['success']:
print(f"✅ 目录下载成功")
print(f"📁 本地目录: {result['local_directory']}")
print(f"🌐 远程前缀: {result['remote_prefix']}")
print(f"📊 文件数量: {result['file_count']}")
print("📄 下载的文件:")
for file_path in result['downloaded_files']:
print(f" - {file_path}")
else:
print(f"❌ 目录下载失败: {result['error']}")
# 9. 删除文件
print("\n🗑️ 删除文件...")
result = toolkit.delete_file(remote_key)
if result['success']:
print(f"✅ 文件删除成功: {result['key']}")
else:
print(f"❌ 文件删除失败: {result['error']}")
# 10. 删除目录
print("\n🗑️ 删除目录...")
result = toolkit.delete_directory(remote_prefix)
if result['success']:
print(f"✅ 目录删除成功")
print(f"📊 删除文件数量: {result['deleted_count']}")
else:
print(f"❌ 目录删除失败: {result['error']}")
# 清理本地文件
import os
import shutil
try:
os.unlink(test_file)
os.unlink(download_path)
shutil.rmtree(test_dir)
shutil.rmtree(download_dir)
print("\n🧹 本地临时文件清理完成")
except:
pass
print("\n🎉 示例演示完成!")
if __name__ == "__main__":
main()

96
install.py Normal file
View File

@@ -0,0 +1,96 @@
#!/usr/bin/env python3
"""
RustFS S3 Storage Toolkit 安装脚本
"""
import subprocess
import sys
import os
from pathlib import Path
def run_command(command, description):
"""运行命令并显示结果"""
print(f"🔧 {description}...")
try:
result = subprocess.run(command, shell=True, check=True, capture_output=True, text=True)
print(f"{description}成功")
return True
except subprocess.CalledProcessError as e:
print(f"{description}失败: {e}")
if e.stdout:
print(f"输出: {e.stdout}")
if e.stderr:
print(f"错误: {e.stderr}")
return False
def main():
"""主安装流程"""
print("🚀 RustFS S3 Storage Toolkit 安装脚本")
print("=" * 50)
# 检查 Python 版本
if sys.version_info < (3, 9):
print("❌ 需要 Python 3.9 或更高版本")
sys.exit(1)
print(f"✅ Python 版本: {sys.version}")
# 检查当前目录
current_dir = Path.cwd()
if not (current_dir / "pyproject.toml").exists():
print("❌ 请在项目根目录运行此脚本")
sys.exit(1)
print(f"✅ 项目目录: {current_dir}")
# 安装包
install_commands = [
("pip install -e .", "安装 RustFS S3 Storage Toolkit"),
]
for command, description in install_commands:
if not run_command(command, description):
print(f"❌ 安装失败,请检查错误信息")
sys.exit(1)
# 验证安装
print("\n🧪 验证安装...")
try:
from rustfs_s3_toolkit import S3StorageToolkit
print("✅ S3StorageToolkit 导入成功")
# 显示版本信息
import rustfs_s3_toolkit
print(f"✅ 版本: {rustfs_s3_toolkit.__version__}")
except ImportError as e:
print(f"❌ 导入失败: {e}")
sys.exit(1)
print("\n🎉 安装完成!")
print("\n📖 使用方法:")
print("1. 查看示例: python examples/basic_usage.py")
print("2. 运行测试: python tests/test_toolkit.py")
print("3. 查看文档: cat README.md")
print("\n💡 快速开始:")
print("""
from rustfs_s3_toolkit import S3StorageToolkit
toolkit = S3StorageToolkit(
endpoint_url="https://your-rustfs-endpoint.com",
access_key_id="your-key",
secret_access_key="your-secret",
bucket_name="your-bucket"
)
# 测试连接
result = toolkit.test_connection()
print(result)
""")
if __name__ == "__main__":
main()

48
pyproject.toml Normal file
View File

@@ -0,0 +1,48 @@
[project]
name = "rustfs-s3-toolkit"
version = "0.2.0"
description = "RustFS S3 Storage Toolkit - A simple and powerful toolkit for RustFS and other S3-compatible object storage operations"
readme = "README.md"
authors = [
{ name = "mm644706215", email = "ze.ga@qq.com" }
]
requires-python = ">=3.9"
dependencies = [
"boto3>=1.39.0",
]
keywords = ["rustfs", "s3", "object-storage", "file-management", "cloud-storage", "toolkit"]
classifiers = [
"Development Status :: 4 - Beta",
"Intended Audience :: Developers",
"License :: OSI Approved :: MIT License",
"Programming Language :: Python :: 3",
"Programming Language :: Python :: 3.9",
"Programming Language :: Python :: 3.10",
"Programming Language :: Python :: 3.11",
"Programming Language :: Python :: 3.12",
"Programming Language :: Python :: 3.13",
"Topic :: Software Development :: Libraries :: Python Modules",
"Topic :: System :: Archiving",
"Topic :: Internet :: File Transfer Protocol (FTP)",
"Topic :: System :: Filesystems",
]
[project.urls]
Homepage = "https://github.com/mm644706215/rustfs-s3-toolkit"
Repository = "https://github.com/mm644706215/rustfs-s3-toolkit"
Documentation = "https://github.com/mm644706215/rustfs-s3-toolkit#readme"
Issues = "https://github.com/mm644706215/rustfs-s3-toolkit/issues"
[project.optional-dependencies]
dev = [
"pytest>=7.0.0",
"pytest-asyncio>=0.21.0",
"black>=23.0.0",
"isort>=5.12.0",
"flake8>=6.0.0",
"mypy>=1.0.0",
]
[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"

View File

@@ -0,0 +1,14 @@
"""
RustFS S3 Storage Toolkit
一个专为 RustFS 设计的 S3 兼容对象存储工具包,同时支持其他各种 S3 兼容存储
"""
__version__ = "0.2.0"
__author__ = "mm644706215"
__email__ = "ze.ga@qq.com"
from .s3_client import S3StorageToolkit
__all__ = [
"S3StorageToolkit",
]

View File

@@ -0,0 +1,617 @@
#!/usr/bin/env python3
"""
S3 Storage Toolkit
一个易于使用的 S3 兼容对象存储工具包,支持 RustFS 等各种 S3 兼容存储
"""
import os
import shutil
import tempfile
import time
from datetime import datetime
from pathlib import Path
from typing import Dict, List, Optional, Union
import boto3
from botocore.client import Config
from botocore.exceptions import ClientError, NoCredentialsError
class S3StorageToolkit:
"""S3 兼容对象存储工具包
支持以下功能:
1. 连接测试
2. 文件上传
3. 文件夹创建
4. 目录上传
5. 文件下载
6. 目录下载
7. 文件列表
8. 文件删除
9. 目录删除
"""
def __init__(
self,
endpoint_url: str,
access_key_id: str,
secret_access_key: str,
bucket_name: str,
region_name: str = "us-east-1"
):
"""
初始化 S3 存储工具包
Args:
endpoint_url: S3 服务端点 URL
access_key_id: 访问密钥 ID
secret_access_key: 访问密钥
bucket_name: 存储桶名称
region_name: 区域名称,默认 us-east-1
"""
self.endpoint_url = endpoint_url
self.access_key_id = access_key_id
self.secret_access_key = secret_access_key
self.bucket_name = bucket_name
self.region_name = region_name
# 存储不同操作类型的客户端
self._clients = {}
def _create_s3_client(self, operation_type: str = "general"):
"""创建 S3 客户端,根据操作类型选择最佳配置"""
if operation_type in self._clients:
return self._clients[operation_type]
try:
if operation_type == "upload":
# 对于上传操作,使用 V2 签名(避免 SHA256 问题)
configs_to_try = [
{
'signature_version': 's3', # V2 签名
's3': {'addressing_style': 'path'}
},
{
'signature_version': 's3v4',
's3': {
'addressing_style': 'path',
'payload_signing_enabled': False
}
}
]
else:
# 对于列表、删除等操作,使用 V4 签名
configs_to_try = [
{
'signature_version': 's3v4',
's3': {
'addressing_style': 'path',
'payload_signing_enabled': False
}
},
{
'signature_version': 's3', # V2 签名作为备选
's3': {'addressing_style': 'path'}
}
]
for config in configs_to_try:
try:
s3_client = boto3.client(
's3',
endpoint_url=self.endpoint_url,
aws_access_key_id=self.access_key_id,
aws_secret_access_key=self.secret_access_key,
region_name=self.region_name,
config=Config(**config)
)
# 测试连接
s3_client.list_buckets()
self._clients[operation_type] = s3_client
return s3_client
except Exception:
continue
raise Exception("所有配置都失败了")
except Exception as e:
raise RuntimeError(f"创建 S3 客户端失败: {str(e)}")
def test_connection(self) -> Dict[str, Union[bool, str, List[str]]]:
"""测试 S3 连接
Returns:
包含连接测试结果的字典
"""
try:
s3_client = self._create_s3_client()
# 尝试列出存储桶
response = s3_client.list_buckets()
buckets = response.get('Buckets', [])
bucket_names = [bucket['Name'] for bucket in buckets]
# 检查目标存储桶是否存在
bucket_exists = self.bucket_name in bucket_names
return {
"success": True,
"bucket_count": len(buckets),
"bucket_names": bucket_names,
"target_bucket_exists": bucket_exists,
"message": f"连接成功!找到 {len(buckets)} 个存储桶"
}
except Exception as e:
return {
"success": False,
"error": str(e),
"message": f"连接测试失败: {str(e)}"
}
def upload_file(
self,
local_file_path: str,
remote_key: str,
metadata: Optional[Dict[str, str]] = None
) -> Dict[str, Union[bool, str, int]]:
"""
上传文件到 S3
Args:
local_file_path: 本地文件路径
remote_key: 远程对象键
metadata: 可选的元数据
Returns:
包含上传信息的字典
"""
try:
local_path = Path(local_file_path)
if not local_path.exists():
return {
"success": False,
"error": f"本地文件不存在: {local_file_path}"
}
s3_client = self._create_s3_client("upload")
# 读取文件内容
with open(local_path, 'rb') as f:
file_content = f.read()
# 上传文件
upload_args = {
'Bucket': self.bucket_name,
'Key': remote_key,
'Body': file_content,
'ContentType': 'application/octet-stream'
}
if metadata:
upload_args['Metadata'] = metadata
s3_client.put_object(**upload_args)
# 生成公开访问 URL
public_url = f"{self.endpoint_url}/{self.bucket_name}/{remote_key}"
return {
"success": True,
"bucket": self.bucket_name,
"key": remote_key,
"public_url": public_url,
"file_size": local_path.stat().st_size,
"upload_time": datetime.now().isoformat()
}
except Exception as e:
return {
"success": False,
"error": f"上传文件失败: {str(e)}"
}
def create_folder(self, folder_path: str) -> Dict[str, Union[bool, str]]:
"""
创建文件夹(通过上传空对象)
Args:
folder_path: 文件夹路径,会自动添加 '/' 后缀
Returns:
包含创建结果的字典
"""
try:
s3_client = self._create_s3_client("upload")
# 确保文件夹路径以 '/' 结尾
if not folder_path.endswith('/'):
folder_path += '/'
# S3 中通过上传空对象来创建"文件夹"
s3_client.put_object(
Bucket=self.bucket_name,
Key=folder_path,
Body=b''
)
return {
"success": True,
"bucket": self.bucket_name,
"folder_path": folder_path,
"create_time": datetime.now().isoformat()
}
except Exception as e:
return {
"success": False,
"error": f"创建文件夹失败: {str(e)}"
}
def upload_directory(
self,
local_dir: str,
remote_prefix: str = ""
) -> Dict[str, Union[bool, str, int, List[str]]]:
"""
上传整个目录到 S3
Args:
local_dir: 本地目录路径
remote_prefix: 远程路径前缀
Returns:
包含上传结果的字典
"""
try:
local_path = Path(local_dir)
if not local_path.exists() or not local_path.is_dir():
return {
"success": False,
"error": f"本地目录不存在或不是目录: {local_dir}"
}
s3_client = self._create_s3_client("upload")
uploaded_files = []
# 递归上传目录中的所有文件
for root, dirs, files in os.walk(local_path):
for file in files:
local_file_path = os.path.join(root, file)
relative_path = os.path.relpath(local_file_path, local_path)
# 构建远程键
if remote_prefix:
remote_key = f"{remote_prefix.rstrip('/')}/{relative_path.replace(os.sep, '/')}"
else:
remote_key = relative_path.replace(os.sep, '/')
# 上传文件
s3_client.upload_file(local_file_path, self.bucket_name, remote_key)
uploaded_files.append(remote_key)
return {
"success": True,
"bucket": self.bucket_name,
"local_directory": str(local_path),
"remote_prefix": remote_prefix,
"uploaded_files": uploaded_files,
"file_count": len(uploaded_files),
"upload_time": datetime.now().isoformat()
}
except Exception as e:
return {
"success": False,
"error": f"上传目录失败: {str(e)}"
}
def download_file(
self,
remote_key: str,
local_file_path: str
) -> Dict[str, Union[bool, str, int]]:
"""
从 S3 下载文件
Args:
remote_key: 远程对象键
local_file_path: 本地保存路径
Returns:
包含下载信息的字典
"""
try:
s3_client = self._create_s3_client("upload") # 使用上传客户端进行下载
local_path = Path(local_file_path)
# 确保本地目录存在
local_path.parent.mkdir(parents=True, exist_ok=True)
# 下载文件
s3_client.download_file(
self.bucket_name,
remote_key,
str(local_path)
)
return {
"success": True,
"bucket": self.bucket_name,
"key": remote_key,
"local_path": str(local_path),
"file_size": local_path.stat().st_size,
"download_time": datetime.now().isoformat()
}
except Exception as e:
return {
"success": False,
"error": f"下载文件失败: {str(e)}"
}
def download_directory(
self,
remote_prefix: str,
local_dir: str
) -> Dict[str, Union[bool, str, int, List[str]]]:
"""
从 S3 下载整个目录
Args:
remote_prefix: 远程路径前缀
local_dir: 本地保存目录
Returns:
包含下载结果的字典
"""
try:
s3_client = self._create_s3_client("list")
local_path = Path(local_dir)
local_path.mkdir(parents=True, exist_ok=True)
# 列出远程目录中的所有文件
response = s3_client.list_objects_v2(
Bucket=self.bucket_name,
Prefix=remote_prefix
)
downloaded_files = []
for obj in response.get('Contents', []):
key = obj['Key']
# 跳过文件夹对象
if key.endswith('/'):
continue
# 计算本地文件路径
relative_path = key[len(remote_prefix):].lstrip('/')
if not relative_path:
continue
local_file_path = local_path / relative_path
local_file_path.parent.mkdir(parents=True, exist_ok=True)
# 下载文件
s3_client.download_file(self.bucket_name, key, str(local_file_path))
downloaded_files.append(str(local_file_path))
return {
"success": True,
"bucket": self.bucket_name,
"remote_prefix": remote_prefix,
"local_directory": str(local_path),
"downloaded_files": downloaded_files,
"file_count": len(downloaded_files),
"download_time": datetime.now().isoformat()
}
except Exception as e:
return {
"success": False,
"error": f"下载目录失败: {str(e)}"
}
def list_files(
self,
prefix: str = "",
max_keys: int = 1000
) -> Dict[str, Union[bool, str, int, List[Dict]]]:
"""
列出 S3 中的文件
Args:
prefix: 文件前缀过滤
max_keys: 最大返回数量
Returns:
包含文件列表的字典
"""
try:
s3_client = self._create_s3_client("list")
# 尝试多种列表方法
list_methods = [
lambda: s3_client.list_objects_v2(
Bucket=self.bucket_name,
Prefix=prefix,
MaxKeys=max_keys
),
lambda: s3_client.list_objects(
Bucket=self.bucket_name,
Prefix=prefix,
MaxKeys=max_keys
)
]
response = None
for method in list_methods:
try:
response = method()
break
except Exception:
continue
if response is None:
return {
"success": False,
"error": "所有列表方法都失败了"
}
files = []
for obj in response.get('Contents', []):
files.append({
"key": obj['Key'],
"size": obj['Size'],
"last_modified": obj['LastModified'].isoformat(),
"public_url": f"{self.endpoint_url}/{self.bucket_name}/{obj['Key']}"
})
return {
"success": True,
"bucket": self.bucket_name,
"prefix": prefix,
"files": files,
"file_count": len(files),
"list_time": datetime.now().isoformat()
}
except Exception as e:
return {
"success": False,
"error": f"列出文件失败: {str(e)}"
}
def delete_file(self, remote_key: str) -> Dict[str, Union[bool, str]]:
"""
删除 S3 中的文件
Args:
remote_key: 远程对象键
Returns:
包含删除结果的字典
"""
try:
s3_client = self._create_s3_client("delete")
# 删除文件
s3_client.delete_object(
Bucket=self.bucket_name,
Key=remote_key
)
# 验证文件已被删除
try:
s3_client.head_object(Bucket=self.bucket_name, Key=remote_key)
return {
"success": False,
"error": "文件删除验证失败:文件仍然存在"
}
except ClientError as e:
if e.response['Error']['Code'] == '404':
return {
"success": True,
"bucket": self.bucket_name,
"key": remote_key,
"delete_time": datetime.now().isoformat()
}
else:
raise e
except Exception as e:
return {
"success": False,
"error": f"删除文件失败: {str(e)}"
}
def delete_directory(self, remote_prefix: str) -> Dict[str, Union[bool, str, int, List[str]]]:
"""
删除 S3 中的整个目录
Args:
remote_prefix: 远程路径前缀
Returns:
包含删除结果的字典
"""
try:
s3_client = self._create_s3_client("delete")
# 首先列出目录中的所有文件
response = s3_client.list_objects_v2(
Bucket=self.bucket_name,
Prefix=remote_prefix
)
objects = response.get('Contents', [])
if not objects:
return {
"success": True,
"message": "目录为空或不存在",
"deleted_count": 0
}
# 尝试批量删除
try:
delete_response = s3_client.delete_objects(
Bucket=self.bucket_name,
Delete={'Objects': [{'Key': obj['Key']} for obj in objects]}
)
deleted_count = len(delete_response.get('Deleted', []))
errors = delete_response.get('Errors', [])
if errors:
error_messages = [f"{error['Key']}: {error['Message']}" for error in errors]
return {
"success": False,
"error": f"批量删除部分失败: {error_messages}",
"deleted_count": deleted_count,
"failed_count": len(errors)
}
return {
"success": True,
"bucket": self.bucket_name,
"remote_prefix": remote_prefix,
"deleted_count": deleted_count,
"delete_time": datetime.now().isoformat()
}
except Exception:
# 如果批量删除失败,尝试逐个删除
deleted_count = 0
failed_files = []
for obj in objects:
try:
s3_client.delete_object(Bucket=self.bucket_name, Key=obj['Key'])
deleted_count += 1
except Exception as e:
failed_files.append(f"{obj['Key']}: {str(e)}")
if failed_files:
return {
"success": False,
"error": f"逐个删除部分失败: {failed_files}",
"deleted_count": deleted_count,
"failed_count": len(failed_files)
}
return {
"success": True,
"bucket": self.bucket_name,
"remote_prefix": remote_prefix,
"deleted_count": deleted_count,
"delete_time": datetime.now().isoformat()
}
except Exception as e:
return {
"success": False,
"error": f"删除目录失败: {str(e)}"
}

243
tests/test_toolkit.py Normal file
View File

@@ -0,0 +1,243 @@
#!/usr/bin/env python3
"""
RustFS S3 Storage Toolkit 测试文件
基于成功的 test_s3_flexible.py 创建的测试套件
已在 RustFS 1.0.0-alpha.34 上完成测试
"""
import os
import tempfile
import shutil
from pathlib import Path
from datetime import datetime
from rustfs_s3_toolkit import S3StorageToolkit
# 测试配置 - 请根据实际情况修改
TEST_CONFIG = {
"endpoint_url": "https://rfs.jmsu.top",
"access_key_id": "lingyuzeng",
"secret_access_key": "rustAdminlingyuzeng",
"bucket_name": "test",
"region_name": "us-east-1"
}
def create_test_files():
"""创建测试文件和目录"""
test_dir = tempfile.mkdtemp(prefix='s3_toolkit_test_')
# 创建测试文件
test_files = {
"test.txt": "这是一个测试文件\n时间: " + datetime.now().isoformat(),
"data.json": '{"name": "test", "value": 123, "timestamp": "' + datetime.now().isoformat() + '"}',
"readme.md": "# 测试文件\n\n这是一个测试用的 Markdown 文件。\n\n- 项目: S3 存储测试\n- 时间: " + datetime.now().isoformat(),
}
# 创建子目录和文件
sub_dir = Path(test_dir) / "subdir"
sub_dir.mkdir()
for filename, content in test_files.items():
file_path = Path(test_dir) / filename
with open(file_path, 'w', encoding='utf-8') as f:
f.write(content)
# 在子目录中创建文件
sub_file = sub_dir / "sub_test.txt"
with open(sub_file, 'w', encoding='utf-8') as f:
f.write("子目录中的测试文件\n时间: " + datetime.now().isoformat())
return test_dir
def test_all_functions():
"""测试所有 9 个核心功能"""
print("🚀 RustFS S3 Storage Toolkit 完整功能测试")
print("🧪 测试环境: RustFS 1.0.0-alpha.34")
print("=" * 60)
# 初始化工具包
try:
toolkit = S3StorageToolkit(**TEST_CONFIG)
print("✅ 工具包初始化成功")
except Exception as e:
print(f"❌ 工具包初始化失败: {e}")
return False
test_results = {}
# 1. 测试连接
print("\n1. 测试连接...")
result = toolkit.test_connection()
test_results['connection'] = result['success']
if result['success']:
print(f"✅ 连接成功: {result['message']}")
else:
print(f"❌ 连接失败: {result['error']}")
return test_results
# 2. 测试上传文件
print("\n2. 测试上传文件...")
test_content = f"测试文件内容\n时间: {datetime.now().isoformat()}"
with tempfile.NamedTemporaryFile(mode='w', delete=False, suffix='.txt', encoding='utf-8') as f:
f.write(test_content)
test_file = f.name
timestamp = datetime.now().strftime("%Y%m%d_%H%M%S")
remote_key = f"test_files/single_file_{timestamp}.txt"
result = toolkit.upload_file(test_file, remote_key)
test_results['upload_file'] = result['success']
if result['success']:
print(f"✅ 文件上传成功: {remote_key}")
uploaded_file_key = remote_key
else:
print(f"❌ 文件上传失败: {result['error']}")
uploaded_file_key = None
os.unlink(test_file)
# 3. 测试创建文件夹
print("\n3. 测试创建文件夹...")
folder_path = f"test_folders/folder_{timestamp}/"
result = toolkit.create_folder(folder_path)
test_results['create_folder'] = result['success']
if result['success']:
print(f"✅ 文件夹创建成功: {folder_path}")
else:
print(f"❌ 文件夹创建失败: {result['error']}")
# 4. 测试上传目录
print("\n4. 测试上传目录...")
test_dir = create_test_files()
remote_prefix = f"test_directories/dir_{timestamp}/"
result = toolkit.upload_directory(test_dir, remote_prefix)
test_results['upload_directory'] = result['success']
if result['success']:
print(f"✅ 目录上传成功: 上传了 {result['file_count']} 个文件")
uploaded_dir_prefix = remote_prefix
else:
print(f"❌ 目录上传失败: {result['error']}")
uploaded_dir_prefix = None
shutil.rmtree(test_dir)
# 5. 测试下载文件
print("\n5. 测试下载文件...")
if uploaded_file_key:
download_path = tempfile.mktemp(suffix='_downloaded.txt')
result = toolkit.download_file(uploaded_file_key, download_path)
test_results['download_file'] = result['success']
if result['success']:
print(f"✅ 文件下载成功: {download_path}")
# 验证内容
with open(download_path, 'r', encoding='utf-8') as f:
downloaded_content = f.read()
if test_content == downloaded_content:
print("✅ 文件内容验证成功")
else:
print("❌ 文件内容验证失败")
os.unlink(download_path)
else:
print(f"❌ 文件下载失败: {result['error']}")
else:
test_results['download_file'] = False
print("❌ 跳过文件下载测试(没有可下载的文件)")
# 6. 测试下载目录
print("\n6. 测试下载目录...")
if uploaded_dir_prefix:
download_dir = tempfile.mkdtemp(prefix='s3_download_')
result = toolkit.download_directory(uploaded_dir_prefix, download_dir)
test_results['download_directory'] = result['success']
if result['success']:
print(f"✅ 目录下载成功: 下载了 {result['file_count']} 个文件")
else:
print(f"❌ 目录下载失败: {result['error']}")
shutil.rmtree(download_dir)
else:
test_results['download_directory'] = False
print("❌ 跳过目录下载测试(没有可下载的目录)")
# 7. 测试列出文件
print("\n7. 测试列出文件...")
result = toolkit.list_files(max_keys=20)
test_results['list_files'] = result['success']
if result['success']:
print(f"✅ 文件列表获取成功: 找到 {result['file_count']} 个文件")
if result['files']:
print("📄 前几个文件:")
for i, file_info in enumerate(result['files'][:5], 1):
size_mb = file_info['size'] / (1024 * 1024)
print(f" {i}. {file_info['key']} ({size_mb:.2f} MB)")
else:
print(f"❌ 文件列表获取失败: {result['error']}")
# 8. 测试删除文件
print("\n8. 测试删除文件...")
if uploaded_file_key:
result = toolkit.delete_file(uploaded_file_key)
test_results['delete_file'] = result['success']
if result['success']:
print(f"✅ 文件删除成功: {uploaded_file_key}")
else:
print(f"❌ 文件删除失败: {result['error']}")
else:
test_results['delete_file'] = False
print("❌ 跳过文件删除测试(没有可删除的文件)")
# 9. 测试删除目录
print("\n9. 测试删除目录...")
if uploaded_dir_prefix:
result = toolkit.delete_directory(uploaded_dir_prefix)
test_results['delete_directory'] = result['success']
if result['success']:
print(f"✅ 目录删除成功: 删除了 {result['deleted_count']} 个文件")
else:
print(f"❌ 目录删除失败: {result['error']}")
else:
test_results['delete_directory'] = False
print("❌ 跳过目录删除测试(没有可删除的目录)")
# 打印测试摘要
print("\n" + "="*60)
print("🎯 测试结果摘要")
print("="*60)
test_names = {
'connection': 'Connection',
'upload_file': 'Upload File',
'create_folder': 'Create Folder',
'upload_directory': 'Upload Directory',
'download_file': 'Download File',
'download_directory': 'Download Directory',
'list_files': 'List Files',
'delete_file': 'Delete File',
'delete_directory': 'Delete Directory'
}
passed_tests = 0
for test_key, test_name in test_names.items():
status = "✅ 通过" if test_results.get(test_key, False) else "❌ 失败"
print(f"{test_name:20} : {status}")
if test_results.get(test_key, False):
passed_tests += 1
print("-" * 60)
print(f"总计: {passed_tests}/{len(test_names)} 个测试通过")
if passed_tests == len(test_names):
print("🎉 所有测试都通过了RustFS S3 Storage Toolkit 工作正常。")
elif passed_tests > 0:
print("⚠️ 部分测试通过。请检查失败的测试项。")
else:
print("❌ 所有测试都失败了。请检查 RustFS 配置和网络连接。")
return test_results
if __name__ == "__main__":
test_all_functions()