Feature/research optimization #537
Draft
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Add KuCoin Spot, Futures, and Futures Testnet support - Implement KuCoinMain base class with API integration - Add timeframe conversion utilities for KuCoin API - Support up to 1500 candles per request with 10 req/sec rate limit - Include proper error handling and retry logic - Add geographic restriction handling Supported exchanges: - KuCoin Spot (api.kucoin.com) - KuCoin Futures (api-futures.kucoin.com) - KuCoin Futures Testnet (api-sandbox-futures.kucoin.com) Supported timeframes: 1m, 3m, 5m, 15m, 30m, 1h, 2h, 4h, 6h, 8h, 12h, 1D, 1W, 1M
- Add KUCOIN_TIMEFRAMES with 14 supported timeframes - Add KuCoin Spot, Futures, and Futures Testnet to exchange_info - Configure proper fees, URLs, and trading modes - Enable backtesting for Spot and Futures - Enable live trading for all KuCoin exchanges - Set appropriate required_live_plan levels Frontend will now display KuCoin exchanges in: - Exchange selection dropdown - Backtesting mode - Live trading mode - Candle import interface
- Replace custom API implementation with CCXT library - Add proper symbol format conversion (BTC-USDT <-> BTC/USDT) - Improve error handling and fallback mechanisms - Add CCXT dependency to requirements.txt - Fix get_starting_time method with better error handling - Support for both spot and futures markets - Tested successfully with BTC-USDT symbol Features: - 1305+ available symbols detected - Reliable candle fetching (tested with 1440 candles) - Proper timeframe conversion - Robust error handling with fallbacks - Support for sandbox/testnet environments
- Add CSV Data Provider for loading tick data from CSV files - Add CSV Parser for various CSV formats with auto-detection - Add CSV Controller with REST API endpoints - Update Data Provider to support custom CSV sources - Update Candle service to handle CSV data - Add comprehensive documentation and tests - Support for aggregating tick data to OHLCV candles - API endpoints for managing CSV data sources - Support for all standard Jesse timeframes - Memory-efficient processing of large CSV files
- Add Custom CSV exchange to Jesse enums and exchange info - Create CustomCSV driver for import_candles_mode with full Jesse compatibility - Implement CSV data provider with tick data aggregation to OHLCV candles - Add support for SYMBOL-USDT format symbols throughout the system - Create API endpoints for CSV data management - Add comprehensive test suite for all new functionality - Support batch data loading with progress tracking - Integrate with Jesse database using batch insertion for large datasets - Add symbol mapping for common suffixes (-USDT, -USDC, -BTC, -ETH) - Ensure all candle data includes required fields: id, timestamp, open, close, high, low, volume, symbol, exchange, timeframe Features: - 193+ CSV symbols available in SYMBOL-USDT format - Full backtesting and hyperparameter optimization support - Real-time data import through Jesse dashboard - Programmatic data loading via API - Comprehensive error handling and logging - Memory-efficient batch processing for large datasets Files added: - CustomCSV driver and supporting files - CSV data provider with aggregation logic - API controllers for CSV management - Test scripts and documentation - Data loading utilities and examples
- Added optional end_date parameter to run function for specifying the end date of candle imports. - Implemented validation for end_date to ensure it is after start_date and not in the future. - Updated logic to use end_date when provided, improving flexibility in candle data imports. - Adjusted success message to reflect the specified end_date or default to today.
- Added a new optimization function to facilitate hyperparameter tuning for trading strategies. - Implemented isolated optimization logic to support multiprocessing without pickling issues. - Included detailed examples in the function docstring for user guidance. - Added internal functions for random hyperparameter generation and fitness evaluation. - Ensured compatibility with existing backtesting infrastructure and validation mechanisms. - Enhanced error handling for invalid input configurations and candle data requirements.
- Add fill_missing_candles configuration option (enabled by default) - Modify generate_candle_from_one_minutes to fill missing data with empty candles - Modify _get_generated_candles to handle insufficient data scenarios - Add warning logs when filling missing candles - Create comprehensive tests for the functionality - Fix backtest errors when insufficient data for higher timeframes This resolves the issue where backtests would fail with errors like: 'Sent only 8 candles but 15 is required to create a 15m candle.'
…ionality - Deleted `test_backtest_missing_candles.py` and `test_fill_missing_candles.py` as they are no longer needed. - These tests have been superseded by more comprehensive testing strategies implemented in recent updates. This cleanup helps maintain a streamlined codebase and reduces redundancy in testing.
- Add bounds checking for start_idx and end_idx in array slicing - Use max(0, start_idx) to prevent negative indices - Use min(end_idx, len(trading_candles)) to prevent exceeding array bounds - Add additional condition to prevent duplicate processing This fixes the error: 'index 1380779 is out of bounds for axis 0 with size 1380769'
- Fix indexing in _step_simulator for candle generation - Fix indexing in _simulate_new_candles for fast mode - Fix indexing in _update_all_routes_a_partial_candle - Fix indexing in get_candles_from_pipeline - Add bounds checking for all array slicing operations - Add proper error handling for out of bounds access This fixes the recurring error: 'index 1380779 is out of bounds for axis 0 with size 1380769' that was occurring during backtest simulation for various timeframes.
…errors - Fix array slicing in _simulate_new_candles with proper bounds checking - Add bounds checking in main simulation loop for first_candles_set access - Improve error messages in _simulation_minutes_length and _prepare_times_before_simulation - Ensure all array operations respect array boundaries - Add fallback logic when accessing out-of-bounds indices This should completely resolve the recurring 'index 1380779 is out of bounds for axis 0 with size 1380769' error that was occurring during backtest simulation for various timeframes and symbols.
- Fix _simulate_new_candles to properly handle array bounds when candles_step extends beyond array size - Add max_available_step calculation to prevent requesting candles beyond array bounds - Add bounds checking for candles array assignment operations - Fix _skip_simulator loop to respect actual array length vs calculated length - Add empty array checking in _simulate_price_change_effect_multiple_candles - Add bounds checking for previous candle access in jumped candle fixes This addresses the remaining 'index 1380779 is out of bounds for axis 0 with size 1380769' errors that were occurring in fast backtest mode for various symbols and timeframes.
…les option - Modify _get_candles_from_db to fill missing candles at the end of data range - Add support for filling missing candles at the beginning of data range - Use fill_missing_candles config option to control behavior - Create empty candles with open=close=last_price and volume=0 - Add warning logs when filling missing data gaps - Maintain backward compatibility when fill_missing_candles is disabled This resolves the 'Missing recent candles' error that was preventing backtests from running when data ended earlier than the requested finish date.
- Update the naming convention for the Custom CSV exchange from 'Custom CSV' to 'CustomCSV' in enums and related classes. - Modify the data provider initialization path for consistency across environments. This change improves clarity and consistency in the codebase.
- Update references in test files to use 'CustomCSV' instead of 'Custom CSV' - Modify enum definition for exchanges to reflect the new naming convention - Ensure consistency in driver imports and comments related to CustomCSV - Adjust API request parameters and print statements for clarity This change improves uniformity in the codebase and prevents potential issues with naming discrepancies.
- Updated CUSTOM_CSV enum value from 'Custom CSV' to 'CustomCSV' for consistency in naming conventions.
- Implement logic to prevent filling absent candles for KuCoin Futures, avoiding the creation of zero-volume fake candles. - Add detailed logging for candle fetching and processing, including zero volume and same price analysis. - Override global `_fill_absent_candles` function to ensure only real candles are returned. - Introduce methods for symbol conversion and market status checks to enhance usability and reliability. - Update rate limits for both KuCoin Futures and Testnet to ensure compliance with API restrictions.
- Remove KuCoin Futures and KuCoin Futures Testnet support - Add clear error messages directing users to use KuCoin Spot instead - Revert conditional processing in _fill_absent_candles function - Clean up unused imports and code KuCoin Futures had issues with data quality (86% zero volume candles) due to _fill_absent_candles creating fake candles. Instead of fixing the complex data filling logic, we disable futures support entirely and recommend users use the stable KuCoin Spot exchange.
…o feature/custom-csv-data
- Introduce BatchCSVLoader for batch loading symbols from CSV files with progress reporting and statistics. - Add CustomCSV driver for handling local CSV files, including symbol normalization and error handling. - Create a factory for CSV parsers to support different formats, starting with KucoinCSVParser. - Implement base CSV parser interface to ensure compatibility with CustomCSV driver. - Enhance error messages for missing data and provide detailed logging for data loading operations. - Add support for loading and saving symbols to the database, including batch operations and performance metrics. This update significantly improves the handling of CSV data within the Jesse framework, enabling users to efficiently manage and analyze their trading data.
- Resolved conflicts in jesse/enums/__init__.py (kept both CUSTOM_CSV and CUSTOM_CSV_RAW) - Resolved conflicts in CustomCSV.py (kept unlimited candles functionality) - Successfully merged fill-missing-candles features with custom CSV functionality
- Introduce batch_size parameter to control the number of symbols processed in each batch, defaulting to 20. - Implement logic to split symbols into batches for improved performance during database saving. - Enhance logging to provide insights on batch processing, including the number of batches created and their completion status. - Maintain existing functionality for both sequential and parallel saving within batches. This update optimizes the loading process for CSV data, allowing for more efficient handling of large datasets.
- Update _format_config function to accept an optional n_trials parameter for trial configuration. - Modify the configuration dictionary to include trials if n_trials is provided, improving flexibility for optimization processes. This change allows researchers to specify the number of trials directly in the configuration, streamlining the optimization setup.
- Update set_config to default trials to 20 if not specified in the configuration. - Modify _format_config to accept an optional n_trials parameter, allowing for dynamic trial configuration. - Clear cache for is_optimizing to ensure accurate optimization state. These changes enhance the flexibility and reliability of the optimization setup, ensuring that default values are applied correctly when not explicitly defined.
- Removed unnecessary exclusions from the runtime environment in the optimizer class. - Streamlined the configuration for better clarity and maintainability. This change simplifies the optimization setup by eliminating redundant paths and files from the ignore list, enhancing the overall configuration management.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.