Scaling AI Batch Processing: Enhancing 235 Plugins with Vertex AI Gemini on the Free Tier

The Problem: 235 Plugins Need Comprehensive Documentation

I maintain claude-code-plugins, a marketplace with 235 plugins for Claude Code. Each plugin needed enhanced SKILL.md files (8,000-14,000 bytes) following Anthropic’s Agent Skills standards. Doing this manually would take weeks.

The goal: Process all 235 plugins overnight using Vertex AI Gemini 2.0 Flash - entirely on the free tier.

The constraints:

Must stay within Vertex AI free tier limits
Need 100% success rate (no corrupted files)
Require full audit trail for compliance
Zero tolerance for API quota violations

The Journey: From Ultra-Conservative to Optimized

Phase 1: Initial System Design

I built overnight-plugin-enhancer.py with these core components:

# Ultra-conservative rate limiting
RATE_LIMIT_DELAY = 90.0  # 90 seconds base delay
RATE_LIMIT_RANDOMNESS = 30.0  # Add 0-30 seconds random

Why so slow? I wanted to ensure we stayed well under the Vertex AI free tier limits:

1,500 requests/day
235 plugins = 470 API calls (analysis + generation per plugin)
At 90-120s per plugin: ~15 plugins/hour = Safe

The system included:

SQLite Audit Database

def init_database(self):
    cursor.execute('''
        CREATE TABLE IF NOT EXISTS enhancements (
            id INTEGER PRIMARY KEY AUTOINCREMENT,
            timestamp TEXT NOT NULL,
            plugin_name TEXT NOT NULL,
            plugin_path TEXT NOT NULL,
            enhancement_type TEXT NOT NULL,
            status TEXT NOT NULL,
            processing_time_seconds REAL
        )
    ''')

Automatic Backups Before Changes

def backup_plugin(self, plugin):
    timestamp = datetime.now().strftime('%Y%m%d_%H%M%S')
    backup_dir = BACKUP_DIR / 'plugin-backups' / f"{plugin['name']}_{timestamp}"
    shutil.copytree(plugin_path, backup_dir)

Two-Phase AI Generation

# Phase 1: Analyze and create enhancement plan
plan = self.generate_enhancement_plan(plugin)

# Phase 2: Generate comprehensive SKILL.md
skill_content = self.generate_skill_md(plugin, plan)

Phase 2: Testing and Timeout Issues

First test run on 10 plugins:

timeout 120 python3 overnight-plugin-enhancer.py --limit 10

Problem: Process appeared stuck - no output for minutes.

Diagnosis: Python output buffering. The script was working but output wasn’t showing in real-time.

Fix: Unbuffered output flag

python3 -u overnight-plugin-enhancer.py

Result: Real-time log streaming confirmed the system was working perfectly. Each plugin took 90-100 seconds as designed.

Phase 3: Expanding to All Categories

Initially, I only processed 7 plugin categories (testing). Time to go all in:

# Before (testing with 7 categories)
CATEGORIES = [
    'productivity', 'security', 'testing', 'packages',
    'examples', 'community', 'mcp'
]

# After (all 17 categories - full 235 plugins)
CATEGORIES = [
    'productivity', 'security', 'testing', 'packages',
    'examples', 'community', 'mcp', 'ai-agency', 'ai-ml',
    'api-development', 'crypto', 'database', 'devops',
    'fairdb-operations-kit', 'finance', 'performance',
    'skill-enhancers'
]

Started the overnight batch:

nohup python3 -u scripts/overnight-plugin-enhancer.py >> overnight-enhancement-all-plugins.log 2>&1 &

Phase 4: Disaster Recovery Planning

Mid-batch, user concern: “What if I get locked out of GitHub?”

This is a legitimate fear when you have 235 production plugins and rely on GitHub for everything. I needed an off-site backup solution immediately.

Enter Turso: Edge SQLite database with free tier (500 databases, 9GB storage).

Built turso-plugin-backup.sh in 30 minutes:

# Creates comprehensive backup with integrity checks
create_plugins_archive() {
    local archive_name="plugins-$(date +%Y%m%d-%H%M%S).tar.gz"
    tar -czf "$archive_path" -C "$PLUGINS_DIR" .

    # Calculate SHA256 hash for integrity
    local hash=$(sha256sum "$archive_path" | cut -d' ' -f1)
}

# Store metadata in Turso
upload_to_turso() {
    turso db shell "$TURSO_DB_NAME" <<EOF
    INSERT INTO backup_history (timestamp, version, plugin_count,
                                archive_size, backup_metadata)
    VALUES ('$timestamp', '$version', $plugin_count,
            $archive_size, '$metadata');
EOF
}

The backup system includes:

All 235 plugins (tar.gz compressed)
Enhancement SQLite database
Plugin inventory JSON
SHA256 integrity hashes
Turso metadata for queryability

Recovery time objective: < 30 minutes to restore complete repository from Turso.

Related: Building Production Testing Suite with Playwright covers similar disaster recovery planning.

Phase 5: Speed Optimization Request

At 12:53 PM (after 12 hours): 157/235 plugins complete (66%)

User: “Let’s speed it up - we have room.”

Analysis:

Only using 7-14% of Vertex AI free tier quota
Success rate: 100%
Could safely cut delays in half

Optimization:

# Old: Ultra-conservative
RATE_LIMIT_DELAY = 90.0
RATE_LIMIT_RANDOMNESS = 30.0

# New: Conservative but 2x faster
RATE_LIMIT_DELAY = 45.0
RATE_LIMIT_RANDOMNESS = 15.0

Impact:

Before: ~15 plugins/hour → Completion: 5:30 AM
After: ~30 plugins/hour → Completion: 2:30 AM (saved 3 hours!)

Killed the old process and restarted:

kill 876147
nohup python3 -u scripts/overnight-plugin-enhancer.py >> overnight-enhancement-all-plugins.log 2>&1 &

The system intelligently skips already-enhanced plugins:

skill_path = plugin_path / 'skills' / 'skill-adapter' / 'SKILL.md'
if skill_path.exists() and len(skill_path.read_text()) > 8000:
    print(f"  ⏭️  SKILL.md already comprehensive ({len(content)} bytes)")
    # Skip AI generation, just backup and validate

The Technical Architecture

Rate Limiting Strategy

def apply_rate_limit(self, idx, total):
    """Apply intelligent rate limiting"""
    # Base delay with randomness
    base_delay = RATE_LIMIT_DELAY + random.uniform(0, RATE_LIMIT_RANDOMNESS)
    time.sleep(base_delay)

    # Extra rest every 10 plugins
    if idx % 10 == 0:
        extra_delay = random.uniform(30, 60)
        print(f"  ⏸️  Extra rest break: {extra_delay:.1f}s...")
        time.sleep(extra_delay)

Why this works:

Randomness prevents patterns that might trigger rate limits
Extra breaks every 10 plugins ensure long-term sustainability
Configurable delays allow real-time optimization without code changes

Smart Processing Logic

def process_plugin(self, plugin):
    """Process single plugin with comprehensive enhancement"""
    try:
        # Always backup first (disaster recovery)
        self.backup_plugin(plugin)

        # Generate enhancement plan
        plan = self.generate_enhancement_plan(plugin)

        # Generate or validate SKILL.md
        if needs_generation:
            skill_content = self.generate_skill_md(plugin, plan)
        else:
            print(f"  ⏭️  SKILL.md already comprehensive")

        # Create bundled resource directories
        for resource_type in ['scripts', 'references', 'assets']:
            if plan['bundled_resources_needed'][resource_type]:
                create_resource_directory(resource_type)

        # Log to SQLite audit trail
        self.log_enhancement(plugin, 'success', changes)

    except Exception as e:
        self.log_enhancement(plugin, 'failed', error=str(e))
        raise

Monitoring and Observability

Real-time progress tracking:

# Check current status
tail -f overnight-enhancement-all-plugins.log

# Query database for metrics
sqlite3 backups/plugin-enhancements/enhancements.db \
  "SELECT COUNT(*) FROM enhancements WHERE status = 'success';"

# Get processing time stats
sqlite3 backups/plugin-enhancements/enhancements.db \
  "SELECT AVG(processing_time_seconds), MAX(processing_time_seconds)
   FROM enhancements WHERE status = 'success';"

The Results

Final Metrics (as of 11:30 PM):

Plugins processed: 163/235 (69% complete)
Success rate: 100%
Average SKILL.md size: 10,617 bytes
Processing time: ~60-100 seconds per plugin
API calls used: ~326 of 1,500 daily limit (22%)
Estimated completion: 2:30-3:00 AM

Quality metrics:

All SKILL.md files follow Anthropic Agent Skills standards
Comprehensive documentation (8,000-14,000 bytes each)
Proper YAML frontmatter
Bundled resource directories created
Complete backup trail in SQLite

Lessons Learned

1. Start Conservative, Optimize Later

Initial 90-120s delays seemed wasteful, but they ensured:

No quota violations
100% success rate
Confidence to optimize

Once we had data proving safety margins, cutting to 45-60s was an easy decision.

2. Real-Time Observability is Critical

The unbuffered output fix was crucial. Without seeing real-time progress:

Can’t identify stuck processes
Can’t calculate accurate completion times
Can’t debug issues as they happen

3. Disaster Recovery Before Production

Building the Turso backup system mid-batch was the right call. Production systems need:

Off-site backups (not just local)
Integrity verification (SHA256 hashes)
Fast recovery (< 30 minutes)
Queryable metadata (Turso SQLite)

4. SQLite for Audit Trails

Using SQLite for enhancement tracking provided:

Complete history of every change
Easy querying for metrics
Backup-friendly (just copy the .db file)
No external dependencies

Related: Building 254 BigQuery Schemas in 72 Hours shows similar database-driven automation patterns.

5. Smart Skipping Saves Money

The system automatically skips already-enhanced plugins:

Saves API quota
Reduces processing time
Allows safe restarts after failures
Enables incremental improvements

The Code

Full implementation: claude-code-plugins/scripts/overnight-plugin-enhancer.py

Key files:

overnight-plugin-enhancer.py - Main batch processor
turso-plugin-backup.sh - Disaster recovery system
TURSO-BACKUP-GUIDE.md - Recovery procedures
enhancements.db - SQLite audit trail

What’s Next

Immediate (tonight):

Complete batch processing (163/235 done)
Run Turso backup after completion
Release v1.2.0 with 235 enhanced plugins

Short-term (this week):

Generate analytics on enhancement quality
Spot-check 10 random SKILL.md files
Deploy marketplace website with new content
Set up automated weekly backups to Turso

Long-term (future releases):

Build turso-plugin-restore.sh for automated recovery
Add Turso backup to release checklist
Implement progressive enhancement (update existing SKILL.md files)
A/B test different SKILL.md structures for effectiveness

Try It Yourself

The enhancement system is open source and works with any plugin repository:

# Clone the repo
git clone https://github.com/jeremylongshore/claude-code-plugins
cd claude-code-plugins

# Configure Vertex AI
gcloud auth application-default login

# Test on single plugin
python3 scripts/overnight-plugin-enhancer.py --plugin overnight-dev

# Run batch on 10 plugins
python3 scripts/overnight-plugin-enhancer.py --limit 10

# Full overnight batch
nohup python3 -u scripts/overnight-plugin-enhancer.py >> batch.log 2>&1 &

Requirements:

Google Cloud account with Vertex AI enabled
Python 3.12+
Claude Code plugins repository structure

Free tier limits:

1,500 Vertex AI requests/day
Process ~750 plugins/day (2 calls per plugin)
Completely free for repositories under 1,000 plugins

Building AI-Friendly Codebases - Documentation systems for AI tools
Automating Developer Workflows - Building slash commands and automation
Building Production Testing Suites - Automated testing at scale

Conclusion

Batch processing 235 plugins with AI isn’t just about throwing API calls at the problem. It requires:

Conservative rate limiting that respects free tier limits
Real-time observability to catch issues immediately
Disaster recovery planning before you need it
Smart optimization based on real data
Complete audit trails for compliance and debugging

The overnight batch will complete around 2:30 AM with 100% success rate, entirely on the Vertex AI free tier. That’s 235 plugins × 10KB of AI-generated documentation = 2.3MB of high-quality content created overnight.

Not bad for free.

Want to see the historical results? The marketplace was hosted at the [retired legacy public domain]; the complete source code remains on GitHub.

Have questions about batch processing with Vertex AI? Drop a comment or find me on X @AsphaltCowb0y.