feat: Implement comprehensive code indexing system with multi-language support

## Overview

This commit introduces a complete structured code indexing system (version 3.0) that provides deep code analysis, relationship tracking, and enhanced search capabilities through a JSON-based index format. The system supports 8+ specialized language analyzers plus generic analysis for 20+ additional languages.

## Core Architecture

### IndexBuilder Pipeline
```
IndexBuilder
├── ProjectScanner (file discovery & categorization)
├── LanguageAnalyzerManager (parallel code analysis)
├── RelationshipTracker (cross-file relationships)
└── IndexStructureBuilder (final assembly)
```

### Data Models
- **FileInfo**: Basic file metadata with language detection
- **FunctionInfo**: Function definitions with parameters, decorators, async flags
- **ClassInfo**: Class definitions with inheritance and method lists
- **ImportInfo**: Import statements with module relationships
- **CodeIndex**: Complete structured index with lookups and relationships

## Language Analysis Support

### Specialized Analyzers (8 languages)
- **Python**: AST-based parsing with decorators, async functions, class inheritance
- **JavaScript/TypeScript**: ES6 features, arrow functions, imports/exports
- **Java**: Annotations, interfaces, package imports, method overrides
- **Go**: Structs, goroutines, interfaces, methods
- **C/C++**: Functions, classes, namespaces, templates, inheritance
- **C#**: Properties, LINQ, async/await, attributes
- **Objective-C**: Interfaces, implementations, protocols, categories

### Enhanced Generic Analyzer
- **Pattern-Based Analysis**: Extracts functions, classes, imports using common patterns
- **Language Detection**: Basic heuristics for Ruby, Rust, Swift, PHP, etc.
- **Graceful Fallback**: Provides meaningful analysis for unsupported languages
- **Error Resilience**: Continues processing when specific analyzers fail

### File Type Coverage
**Total Support**: 60+ file extensions including:
- Programming languages: .py, .js, .java, .go, .c, .cpp, .cs, .m, .rb, .php, .swift, .kt, .rs, .scala
- Web technologies: .html, .css, .scss, .vue, .svelte, .astro
- Configuration: .json, .yaml, .xml, .toml, .ini
- Documentation: .md, .rst, .txt
- Database: .sql, .cql, .cypher

## Index Structure

### Unified Extension Management
- Centralized `SUPPORTED_EXTENSIONS` in constants.py
- Consistent filtering across ProjectScanner and LanguageAnalyzerManager
- Only files with supported extensions are indexed
- Clean separation between code files and special files

### JSON Index Format
```json
{
  "project_metadata": {
    "name": "project-name",
    "indexed_at": "timestamp",
    "total_files": "count",
    "total_lines": "count"
  },
  "index_metadata": {
    "version": "3.0",
    "analysis_time_ms": "timing",
    "files_with_errors": [],
    "languages_analyzed": ["detected_languages"]
  },
  "directory_tree": { /* Only supported file types */ },
  "files": [ /* Detailed analysis results with line numbers */ ],
  "lookups": { /* Forward lookup tables */ },
  "reverse_lookups": { /* Relationship queries */ },
  "special_files": { /* File categorization */ }
}
```

## Advanced Features

### Relationship Tracking
- **Function Calls**: Tracks which functions call other functions
- **Class Usage**: Monitors class instantiation relationships
- **Import Dependencies**: Maps module import relationships
- **Reverse Lookups**: Efficient querying of "who calls this function"

### Performance Optimization
- **Parallel Processing**: Concurrent file analysis using ThreadPoolExecutor
- **Memory Management**: On-demand content reading, structured caching
- **Error Handling**: Graceful degradation with comprehensive error reporting
- **Incremental Analysis**: Efficient processing of large codebases

### Special File Detection
- **Entry Points**: main.py, index.js, Main.java, etc.
- **Configuration**: package.json, requirements.txt, pom.xml, etc.
- **Documentation**: README files, LICENSE, CHANGELOG
- **Build Files**: Dockerfile, Jenkinsfile, CI/CD configurations

## Implementation Details

### Key Components
- **constants.py**: Centralized supported extensions and configuration
- **ProjectScanner**: File discovery with strict extension filtering
- **LanguageAnalyzerManager**: Coordinates analyzers with fallback logic
- **GenericAnalyzer**: Pattern-matching for common language constructs
- **RelationshipTracker**: Cross-file relationship analysis
- **IndexBuilder**: Main pipeline coordinator with error handling

### Testing & Validation
- Comprehensive test suite for all language analyzers
- Integration tests with real-world multi-language projects
- Performance validation and error scenario testing
- Mock-based unit testing for complex components

## Backward Compatibility

- **API Consistency**: Existing server.py uses same extension list
- **Version Detection**: 3.0 format with automatic migration support
- **Graceful Degradation**: Enhanced error handling and fallback mechanisms
- **Legacy Support**: Maintains compatibility with existing MCP server functionality

This implementation provides a solid foundation for advanced code analysis and search capabilities while maintaining excellent performance, extensibility, and strict scope control over indexed content.

Author: johnhuang316

demo_index.json

@@ -0,0 +1,228 @@
+#!/usr/bin/env python3
+"""
+Demo script for the new code indexing system.
+
+This script demonstrates the capabilities of the new structured indexing system,
+showing how it analyzes code structure, relationships, and provides rich metadata.
+"""
+
+import sys
+import os
+import json
+from pathlib import Path
+
+# Add src to path
+sys.path.insert(0, os.path.join(os.path.dirname(__file__), 'src'))
+
+from code_index_mcp.indexing import IndexBuilder
+
+
+def demo_indexing():
+    """Demonstrate the indexing system on the current project."""
+    print("🔍 Code Indexing System Demo")
+    print("=" * 50)
+    
+    # Build index for current project
+    project_path = "."
+    print(f"📁 Analyzing project: {os.path.abspath(project_path)}")
+    
+    builder = IndexBuilder()
+    index = builder.build_index(project_path)
+    
+    # Display project metadata
+    print(f"\n📊 Project Metadata:")
+    print(f"   Name: {index.project_metadata['name']}")
+    print(f"   Total Files: {index.project_metadata['total_files']}")
+    print(f"   Total Lines: {index.project_metadata['total_lines']}")
+    print(f"   Indexed At: {index.project_metadata['indexed_at']}")
+    
+    # Display index metadata
+    print(f"\n🔧 Index Metadata:")
+    print(f"   Version: {index.index_metadata['version']}")
+    print(f"   Analysis Time: {index.index_metadata['analysis_time_ms']}ms")
+    print(f"   Languages: {', '.join(index.index_metadata['languages_analyzed'])}")
+    # Removed supports field as it was not useful
+    
+    # Display file analysis
+    print(f"\n📄 File Analysis:")
+    python_files = [f for f in index.files if f['language'] == 'python']
+    print(f"   Python files: {len(python_files)}")
+    
+    # Show some Python files with their functions and classes
+    for file_info in python_files[:3]:  # Show first 3 Python files
+        print(f"   📝 {file_info['path']}:")
+        if file_info['functions']:
+            func_names = [f['name'] for f in file_info['functions']]
+            print(f"      Functions: {', '.join(func_names[:5])}")  # Show first 5
+        if file_info['classes']:
+            class_names = [c['name'] for c in file_info['classes']]
+            print(f"      Classes: {', '.join(class_names)}")
+    
+    # Display special files
+    print(f"\n📋 Special Files:")
+    for category, files in index.special_files.items():
+        if files:
+            print(f"   {category.replace('_', ' ').title()}: {len(files)} files")
+            for file_path in files[:3]:  # Show first 3 files in each category
+                print(f"      - {file_path}")
+    
+    # Display directory structure (simplified)
+    print(f"\n🌳 Directory Structure:")
+    def print_tree(tree, indent=0):
+        for name, subtree in tree.items():
+            print("  " * indent + f"├── {name}")
+            if isinstance(subtree, dict):
+                print_tree(subtree, indent + 1)
+    
+    # Show only first level to avoid too much output
+    for name, subtree in list(index.directory_tree.items())[:5]:
+        print(f"├── {name}")
+        if isinstance(subtree, dict) and subtree:
+            for subname in list(subtree.keys())[:3]:
+                print(f"│   ├── {subname}")
+    
+    # Display some lookup examples
+    print(f"\n🔍 Lookup Examples:")
+    print(f"   Total path mappings: {len(index.lookups['path_to_id'])}")
+    print(f"   Total function mappings: {len(index.lookups['function_to_file_id'])}")
+    print(f"   Total class mappings: {len(index.lookups['class_to_file_id'])}")
+    
+    # Show some function examples
+    if index.lookups['function_to_file_id']:
+        print(f"   Sample functions:")
+        for func_name in list(index.lookups['function_to_file_id'].keys())[:5]:
+            file_id = index.lookups['function_to_file_id'][func_name]
+            file_path = next(f['path'] for f in index.files if f['id'] == file_id)
+            print(f"      {func_name} → {file_path}")
+    
+    # Display relationship examples
+    print(f"\n🔗 Relationships:")
+    reverse_lookups = index.reverse_lookups
+    
+    if reverse_lookups.get('function_callers'):
+        print(f"   Function call relationships: {len(reverse_lookups['function_callers'])}")
+        for func_name, callers in list(reverse_lookups['function_callers'].items())[:3]:
+            caller_names = [c['caller'] for c in callers]
+            print(f"      {func_name} ← called by: {', '.join(caller_names)}")
+    
+    if reverse_lookups.get('imports_module'):
+        print(f"   Import relationships: {len(reverse_lookups['imports_module'])}")
+        for module, file_ids in list(reverse_lookups['imports_module'].items())[:3]:
+            print(f"      {module} ← imported by {len(file_ids)} files")
+    
+    # Show errors if any
+    if index.index_metadata.get('files_with_errors'):
+        print(f"\n⚠️  Files with errors: {len(index.index_metadata['files_with_errors'])}")
+        for error_file in index.index_metadata['files_with_errors'][:3]:
+            print(f"      - {error_file}")
+    
+    print(f"\n✅ Indexing complete! Index contains {len(index.files)} files.")
+    
+    # Optionally save the index to a file
+    save_index = input("\n💾 Save index to file? (y/N): ").lower().strip()
+    if save_index == 'y':
+        output_file = "demo_index.json"
+        with open(output_file, 'w', encoding='utf-8') as f:
+            f.write(index.to_json())
+        print(f"📁 Index saved to {output_file}")
+        print(f"   File size: {os.path.getsize(output_file)} bytes")
+
+
+def analyze_specific_file():
+    """Analyze a specific file in detail."""
+    print("\n🔬 Detailed File Analysis")
+    print("=" * 30)
+    
+    # Let's analyze the main server file
+    server_file = "src/code_index_mcp/server.py"
+    if not os.path.exists(server_file):
+        print(f"❌ File not found: {server_file}")
+        return
+    
+    # Build index and find the server file
+    builder = IndexBuilder()
+    index = builder.build_index(".")
+    
+    server_info = None
+    for file_info in index.files:
+        if file_info['path'] == server_file.replace('\\', '/'):
+            server_info = file_info
+            break
+    
+    if not server_info:
+        print(f"❌ File not found in index: {server_file}")
+        return
+    
+    print(f"📄 File: {server_info['path']}")
+    print(f"   Language: {server_info['language']}")
+    print(f"   Size: {server_info['size']} bytes")
+    print(f"   Lines: {server_info['line_count']}")
+    
+    print(f"\n🔧 Functions ({len(server_info['functions'])}):")
+    for func in server_info['functions'][:10]:  # Show first 10 functions
+        params = ', '.join(func['parameters'][:3])  # Show first 3 params
+        if len(func['parameters']) > 3:
+            params += '...'
+        async_marker = "async " if func['is_async'] else ""
+        decorators = f"@{', @'.join(func['decorators'])} " if func['decorators'] else ""
+        print(f"   {decorators}{async_marker}{func['name']}({params}) [lines {func['line_start']}-{func['line_end']}]")
+        
+        if func['calls']:
+            print(f"      → calls: {', '.join(func['calls'][:3])}")
+        if func['called_by']:
+            print(f"      ← called by: {', '.join(func['called_by'][:3])}")
+    
+    print(f"\n🏗️  Classes ({len(server_info['classes'])}):")
+    for cls in server_info['classes']:
+        inheritance = f" extends {cls['inherits_from']}" if cls['inherits_from'] else ""
+        print(f"   {cls['name']}{inheritance} [lines {cls['line_start']}-{cls['line_end']}]")
+        if cls['methods']:
+            print(f"      Methods: {', '.join(cls['methods'])}")
+        if cls['instantiated_by']:
+            print(f"      Instantiated by: {', '.join(cls['instantiated_by'])}")
+    
+    print(f"\n📦 Imports ({len(server_info['imports'])}):")
+    for imp in server_info['imports'][:10]:  # Show first 10 imports
+        if imp['imported_names']:
+            names = ', '.join(imp['imported_names'][:3])
+            if len(imp['imported_names']) > 3:
+                names += '...'
+            print(f"   from {imp['module']} import {names}")
+        else:
+            print(f"   import {imp['module']}")
+    
+    # Show language-specific features
+    if server_info['language_specific']:
+        print(f"\n🐍 Python-specific features:")
+        python_features = server_info['language_specific'].get('python', {})
+        
+        if python_features.get('decorators'):
+            print(f"   Decorators:")
+            for func_name, decorators in python_features['decorators'].items():
+                print(f"      {func_name}: {', '.join(decorators)}")
+        
+        if python_features.get('async_functions'):
+            print(f"   Async functions: {', '.join(python_features['async_functions'])}")
+        
+        if python_features.get('class_inheritance'):
+            print(f"   Class inheritance:")
+            for cls_name, base in python_features['class_inheritance'].items():
+                if base:
+                    print(f"      {cls_name} → {base}")
+
+
+if __name__ == "__main__":
+    try:
+        demo_indexing()
+        
+        # Ask if user wants detailed file analysis
+        detail_analysis = input("\n🔬 Run detailed file analysis? (y/N): ").lower().strip()
+        if detail_analysis == 'y':
+            analyze_specific_file()
+            
+    except KeyboardInterrupt:
+        print("\n\n👋 Demo interrupted by user")
+    except Exception as e:
+        print(f"\n❌ Error during demo: {e}")
+        import traceback
+        traceback.print_exc()

demo_indexing.py

@@ -5,5 +5,72 @@
 # Directory and file names
 SETTINGS_DIR = "code_indexer"
 CONFIG_FILE = "config.json"
-INDEX_FILE = "file_index.pickle"
-CACHE_FILE = "content_cache.pickle"
+INDEX_FILE = "index.json"
+# CACHE_FILE removed - no longer needed with new indexing system
+
+# Supported file extensions for code analysis
+# This is the authoritative list used by both old and new indexing systems
+SUPPORTED_EXTENSIONS = [
+    # Core programming languages
+    '.py', '.pyw',                    # Python
+    '.js', '.jsx', '.ts', '.tsx',     # JavaScript/TypeScript
+    '.mjs', '.cjs',                   # Modern JavaScript
+    '.java',                          # Java
+    '.c', '.cpp', '.h', '.hpp',       # C/C++
+    '.cxx', '.cc', '.hxx', '.hh',     # C++ variants
+    '.cs',                            # C#
+    '.go',                            # Go
+    '.m', '.mm',                      # Objective-C
+    '.rb',                            # Ruby
+    '.php',                           # PHP
+    '.swift',                         # Swift
+    '.kt', '.kts',                    # Kotlin
+    '.rs',                            # Rust
+    '.scala',                         # Scala
+    '.sh', '.bash', '.zsh',           # Shell scripts
+    '.ps1',                           # PowerShell
+    '.bat', '.cmd',                   # Windows batch
+    '.r', '.R',                       # R
+    '.pl', '.pm',                     # Perl
+    '.lua',                           # Lua
+    '.dart',                          # Dart
+    '.hs',                            # Haskell
+    '.ml', '.mli',                    # OCaml
+    '.fs', '.fsx',                    # F#
+    '.clj', '.cljs',                  # Clojure
+    '.vim',                           # Vim script
+    '.zig',                           # Zig
+    
+    # Web and markup
+    '.html', '.htm',                  # HTML
+    '.css', '.scss', '.sass',         # Stylesheets
+    '.less', '.stylus', '.styl',      # Style languages
+    '.md', '.mdx',                    # Markdown
+    '.json', '.jsonc',                # JSON
+    '.xml',                           # XML
+    '.yml', '.yaml',                  # YAML
+    
+    # Frontend frameworks
+    '.vue',                           # Vue.js
+    '.svelte',                        # Svelte
+    '.astro',                         # Astro
+    
+    # Template engines
+    '.hbs', '.handlebars',            # Handlebars
+    '.ejs',                           # EJS
+    '.pug',                           # Pug
+    
+    # Database and SQL
+    '.sql', '.ddl', '.dml',           # SQL
+    '.mysql', '.postgresql', '.psql', # Database-specific SQL
+    '.sqlite', '.mssql', '.oracle',   # More databases
+    '.ora', '.db2',                   # Oracle and DB2
+    '.proc', '.procedure',            # Stored procedures
+    '.func', '.function',             # Functions
+    '.view', '.trigger', '.index',    # Database objects
+    '.migration', '.seed', '.fixture', # Migration files
+    '.schema',                        # Schema files
+    '.cql', '.cypher', '.sparql',     # NoSQL query languages
+    '.gql',                           # GraphQL
+    '.liquibase', '.flyway',          # Migration tools
+]

src/code_index_mcp/constants.py

@@ -0,0 +1,31 @@
+"""
+Code indexing system for the MCP server.
+
+This module provides structured code analysis, relationship tracking,
+and enhanced search capabilities through a JSON-based index format.
+"""
+
+from .models import (
+    FileInfo,
+    FunctionInfo,
+    ClassInfo,
+    ImportInfo,
+    FileAnalysisResult,
+    CodeIndex
+)
+
+from .builder import IndexBuilder
+from .scanner import ProjectScanner
+from .analyzers import LanguageAnalyzerManager
+
+__all__ = [
+    'FileInfo',
+    'FunctionInfo', 
+    'ClassInfo',
+    'ImportInfo',
+    'FileAnalysisResult',
+    'CodeIndex',
+    'IndexBuilder',
+    'ProjectScanner',
+    'LanguageAnalyzerManager'
+]

src/code_index_mcp/indexing/__init__.py

@@ -0,0 +1,30 @@
+"""
+Language analyzers for code structure extraction.
+
+This module provides language-specific analyzers that extract functions, classes,
+imports, and other code structures from source files.
+"""
+
+from .base import LanguageAnalyzer
+from .manager import LanguageAnalyzerManager
+from .python_analyzer import PythonAnalyzer
+from .javascript_analyzer import JavaScriptAnalyzer
+from .java_analyzer import JavaAnalyzer
+from .go_analyzer import GoAnalyzer
+from .c_analyzer import CAnalyzer
+from .cpp_analyzer import CppAnalyzer
+from .csharp_analyzer import CSharpAnalyzer
+from .objective_c_analyzer import ObjectiveCAnalyzer
+
+__all__ = [
+    'LanguageAnalyzer',
+    'LanguageAnalyzerManager',
+    'PythonAnalyzer',
+    'JavaScriptAnalyzer', 
+    'JavaAnalyzer',
+    'GoAnalyzer',
+    'CAnalyzer',
+    'CppAnalyzer',
+    'CSharpAnalyzer',
+    'ObjectiveCAnalyzer'
+]