A serverless semantic search engine using AWS S3 Vectors for vector storage and Amazon Bedrock for text embedding generation. This project provides a RESTful API for semantic search capabilities without requiring a dedicated vector database.

This application enables semantic search functionality by leveraging AWS services:

• Text Embedding Generation: Using Amazon Bedrock to convert text into vector embeddings
• Vector Storage and Retrieval: Using AWS S3 Vectors for efficient vector storage and similarity search
• RESTful API: FastAPI-based endpoints for embedding generation, vector storage, and semantic search
• Serverless Deployment: AWS Lambda and API Gateway for scalable, serverless operation
• Infrastructure as Code: AWS CDK for application infrastructure + Python scripts for S3 Vector management
• Flexible Resource Management: Separate S3 Vector bucket management via dedicated scripts

┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
│   API Client    │───▶│   API Gateway   │───▶│   Lambda        │
│                 │    │                 │    │                 │
└─────────────────┘    └─────────────────┘    └─────────────────┘
                                                       │
                                                       ▼
┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
│   S3 Vectors    │◀───│   Bedrock       │◀───│   FastAPI App   │
└─────────────────┘    └─────────────────┘    └─────────────────┘

• Generate vector embeddings from text using Amazon Bedrock
• Store and retrieve vectors using AWS S3 Vectors
• Perform semantic similarity search
• Filter search results by metadata
• Batch processing for multiple texts
• Comprehensive API documentation with OpenAPI/Swagger

• Python 3.12+ installed
• AWS account with appropriate permissions for S3 Vectors and CDK
• AWS CLI configured with credentials
• Node.js 18+ for AWS CDK
• Make for Windows (if not already installed)
• Access to S3 Vectors supported regions (us-east-1, us-east-2, us-west-2, eu-central-1, ap-southeast-2)

git clone <repository-url>
cd s3-vector

This project uses two separate virtual environments: one for the application and one for the CDK infrastructure.

# Create both environments using the Makefile
make setup

# Or create them individually
make setup-app    # For application environment
make setup-cdk    # For CDK environment

# Install all dependencies using the Makefile
make install

# Or install them individually
make install-app  # For application dependencies
make install-cdk  # For CDK dependencies

Important: S3 Vector buckets are now managed separately from the CDK stack due to lack of native CloudFormation support.

# Install script dependencies
cd scripts
pip install -r requirements.txt

# Set up S3 Vector bucket and index for development
python setup_s3_vectors.py setup --environment dev --region us-east-1

Copy the sample environment file and modify with the S3 Vector bucket name:

cp sample.env .env

Edit the .env file with your specific configuration:

# Application settings
APP_NAME=s3-vector-search
APP_VERSION=0.1.0
DEBUG=False

# AWS settings
AWS_REGION=us-east-1
# AWS_PROFILE=default  # Uncomment and set if using a specific AWS profile

# S3 Vector settings (set after running setup_s3_vectors.py)
S3_BUCKET_NAME=s3-vector-search-dev

# Vector settings
VECTOR_INDEX_NAME=default-index  # Default index name used when not specified in API requests
VECTOR_DIMENSION=1024

# Bedrock settings
BEDROCK_MODEL_ID=amazon.titan-embed-text-v2:0

# API settings
API_PREFIX=/api/v1

Ensure everything is working correctly by running the tests:

make test

Deploy the AWS infrastructure using CDK:

cd infrastructure
.\venv\Scripts\activate.bat
cdk deploy --context environment=dev

# Activate the virtual environment
.\venv\Scripts\activate.bat

# Run the FastAPI application with uvicorn
python -m uvicorn src.main:app --reload

The API will be available at http://localhost:8000

When running locally, access the interactive API documentation at:

• Swagger UI: http://localhost:8000/docs
• ReDoc: http://localhost:8000/redoc

├── docs/                  # Project documentation
├── infrastructure/        # AWS CDK infrastructure code
│   ├── stacks/            # CDK stack definitions
│   └── tests/             # CDK tests
├── scripts/               # S3 Vector management scripts
│   ├── s3_vector_manager.py    # Core S3 Vector operations
│   ├── setup_s3_vectors.py     # Environment setup script
│   ├── requirements.txt        # Script dependencies
│   └── README.md               # Script documentation
├── src/                   # Application source code
│   ├── config/            # Configuration settings
│   ├── models/            # Pydantic data models
│   ├── services/          # Business logic services
│   └── utils/             # Utility functions
├── tests/                 # Application tests
│   ├── integration/       # Integration tests
│   └── unit/              # Unit tests
├── .env                   # Environment variables (create from sample.env)
├── DEPLOYMENT.md          # Detailed deployment guide
├── Makefile               # Build automation
├── pytest.ini            # Pytest configuration
├── requirements.txt      # Application dependencies
└── sample.env            # Sample environment variables

For detailed deployment instructions, see DEPLOYMENT.md.

# 1. Set up S3 Vector resources
cd scripts
pip install -r requirements.txt
python setup_s3_vectors.py setup --environment dev --region us-east-1

# 2. Update .env file with the bucket name shown in output
# S3_BUCKET_NAME=s3-vector-search-dev

# 3. Deploy CDK stack
cd infrastructure
npm install
cdk deploy --context environment=dev

# 4. Test the deployment
cd ../scripts
python setup_s3_vectors.py verify --environment dev --region us-east-1

This project uses dedicated Python scripts to manage S3 Vector resources:

• Setup: scripts/setup_s3_vectors.py - Environment-based setup
• Management: scripts/s3_vector_manager.py - Direct resource management
• Documentation: scripts/README.md - Detailed script usage

1. Set up S3 Vector resources for your environment
2. Write tests for new features
3. Implement the feature code
4. Run tests to verify functionality
5. Format code with make format
6. Run linting with make lint
7. Commit changes
8. Deploy infrastructure changes if needed

MIT License

Contributions are welcome! Please feel free to submit a Pull Request.