- Normal dist - big changes from mean are uncommon
- Fat-tail dist - exists in financial markets, where big changes (from mean) have hi probabilities
Generate a basic strategy
- Use a chart to generate an idea
- https://www.tradingview.com/chart/RARYjeai/
- Insert some details at topright panel:
- Symbol search: BINANCE:BTCUSDT
- Time interval: 1 hr
- Chart style: Candles
- Indicators: Bollinger Bands (BB). Doubleclick to setup:
- Length: 20 (no. of days used to calc moving average)
- StdDev: 2
- Indicators: Relative Strength Index (RSI). Doubleclick to setup:
- Length: 14 (no. of past candles)
- Observe. Tendency is price at lower BB will return to avg BB, and price at higher BB will return to avg BB.
- So, the basic idea is to buy when price = lower BB (at time T1), and sell when price = avg BB or above (at time T2 > T1). But, sometimes BB can keep trending down, and no chance to sell at a higher price. (It's oversold, and continue being oversold for a long time). How to buy when BB stops going down and begin its trend up?
- When BB is about to begin trending up, it represents a change from "oversold" to "overbought".
- RSI looks at the past 14 candles. The bigger the green candles are relative to the red candles, the more "overbought" it is. The bigger the red candles are relative to the green candles, the more "oversold" it is. RSI takes a value between 0 (extremely oversold) and 100 (extremely overbought). RSI between 30 and 70 is desirable (stable). Oversold is 0-30, overbought is 70-100. When to buy? When it's not being oversold (i.e. when RSI >30). (When RSI is low e.g. 20... thinking is: hey something bad's going on right now, best not to buy now)
- So, price hits lower BB... Is it a good time to buy? BB still looks like it's still falling. When will BB start rising? Check RSI. If RSI<30, then BB's are still falling. If RSI>30, then BB are potentially trending up, so, buy. Summary: If (price < lower BB) AND (RSI>30), then buy.
- When to sell? If (price > avg BB), then sell.
- So, basic strategy:
- Buy if (price<lower BB) AND RSI>30
- Sell if (price>avg BB)
- Later, to optimise the strategy:
- Buying... Lower BB: use how many SD ? (note: bigger SD means fewer but more profitable trades)
- Buying... RSI: is 30 the best value? maybe 20? maybe 40?
- Selling... use avg BB? could we use 1 SD below avgBB ? or 2 SD above avgBB?
Automate to what Extent?
- If you don't have time to watch the market, then fully automate the buy & sell decisions.
- If you have time to read news that affects the fundamentals (vs technical), u cld automate to help u time the entry & exit.
Code the Strategy
- 1. Start VM (Oracle VM VirtualBox > Start a saved machine)
- 2. In Ubuntu, open terminal (Ctrl+Alt+T)
- 3. Activate the virtual environment
- user1@computer1:~$ cd freqtrade
- user1@computer1:~/freqtrade$ source .env/bin/activate
- 4. Load Atom within freqtrade folder
- (.env) user1@computer1:~/freqtrade$ atom .
- 5. Duplicate the default strategy file and rename it e.g. bbrsi.py (you'll be working on this new file)
- freqtrade\freqtrade\strategy\default_strategy.py > RHS click >duplicate> rename as "bb_rsi.py"
- 6. In the new strategy file (bb_rsi.py), change the class name:
- "class DefaultStrategy(IStrategy):" to "class BBRSI(IStrategy):"
- 7. Tell the bot where to find the strategy file (bb_rsi.py) and strategy class (BBRSI) in the init file:
- freqtrade\freqtrade\strategy\__init__.py
- Duplicate the line "from freqtrade.strategy.default_strategy import DefaultStrategy"
- Change, in the new line: "default_strategy" to "bbrsi", and "DefaultStrategy" to "BBRSI".
- So now u have "from freqtrade.strategy.bbrsi import BBRSI"
- 8. Minimal ROI: Tells the bot to sell a coin after a certain time if unable to detect a sell signal. It works independently of the sell signal. Default values (before optimization) are:
- minimal_roi = {
- "40": 0.0,
- "30": 0.01,
- "20": 0.02,
- "0": 0.04
} - Note: 0 to <20th min, sell if at least 4% profit, 20th to <30th min sell if at least 2% profit, etc.
- 9. Stop Loss: Default value (before optimization) is 10%:
- 10. Ticker Interval: 1h ("1m" gets many more trades compared to "1h")
11. Define all indicators used in the "populate_indicators" function. This time we only need #RSI and #BollingerBands, delete all else.
- def populate_indicators(self, dataframe: DataFrame, metadata: dict) -> DataFrame:
- # RSI
- dataframe['rsi'] = ta.RSI(dataframe)
- # Bollinger bands
- bollinger1 = qtpylib.bollinger_bands(qtpylib.typical_price(dataframe), window=20, stds=1)
- dataframe['bb_lowerband1'] = bollinger1['lower']
- dataframe['bb_middleband1'] = bollinger1['mid']
- dataframe['bb_upperband1'] = bollinger1['upper']
- bollinger2 = qtpylib.bollinger_bands(qtpylib.typical_price(dataframe), window=20, stds=2)
- dataframe['bb_lowerband2'] = bollinger2['lower']
- dataframe['bb_middleband2'] = bollinger2['mid']
- dataframe['bb_upperband2'] = bollinger2['upper']
- bollinger3 = qtpylib.bollinger_bands(qtpylib.typical_price(dataframe), window=20, stds=3)
- dataframe['bb_lowerband3'] = bollinger3['lower']
- dataframe['bb_middleband3'] = bollinger3['mid']
- dataframe['bb_upperband3'] = bollinger3['upper']
- return dataframe
12. Buy decision: edit the "populate_buy_trend" function. Buy if RSI>30 AND closing price < lowerBB
- def populate_buy_trend(self, dataframe: DataFrame, metadata: dict) -> DataFrame:
- dataframe.loc[
- (
- (dataframe['rsi'] < 35) &
- (dataframe['close'] < dataframe['bb_lowerband3'])
- ),
- 'buy'] = 1
- return dataframe
13. Sell decision: edit the "populate_sell_trend" function. Sell if closing price > avg BB
- def populate_sell_trend(self, dataframe: DataFrame, metadata: dict) -> DataFrame:
- dataframe.loc[
- (
- (dataframe['rsi'] > 81) &
- (dataframe['close'] > dataframe['bb_middleband1'])
- ),
- 'sell'] = 1
- return dataframe
Setup the config file1. freqtrade/config.json
2. Config file is more general, strategy file is more specific. If an identical parameter is found in both files, the bot will adopt parameters as per the specific strategy file
3. Set the bot to do up to 15 trades simultaneously, using 0.01 BTC per trade (smallest allowed by the exchange). Cos when going live, we will have 0.15 BTC to trade. Display profit/loss in USD. Ticker interval was agreed at 1hour
- "max_open_trades": 15,
- "stake_currency": "BTC",
- "stake_amount": 0.01, (smallest amount allowed by the exchange)
- "fiat_display_currency": "USD",
- "ticker_interval" : "1h",
- "dry_run": true, (we're backtesting i.e. running the strategy on past data)
4. Backtest using as many coins as possible
"pair_whitelist": [
"ADA/BTC",
"AAVE/BTC",
"AION/BTC",
"AMB/BTC",
"APPC/BTC",
"BAND/BTC",
"BLZ/BTC",
"BNB/BTC",
"BRD/BTC",
"BZRX/BTC",
"DASH/BTC",
"EOS/BTC",
"ETC/BTC",
"ETH/BTC",
"FIL/BTC",
"HARD/BTC",
"ICX/BTC",
"KMD/BTC",
"LINK/BTC",
"LSK/BTC",
"LTC/BTC",
"MANA/BTC",
"NEO/BTC",
"ONT/BTC",
"ORN/BTC",
"POWR/BTC",
"PPT/BTC",
"QLC/BTC",
"RDN/BTC",
"REN/BTC",
"RLC/BTC",
"SNGLS/BTC",
"STX/BTC",
"SYS/BTC",
"THETA/BTC",
"TRX/BTC",
"UNI/BTC",
"VIA/BTC",
"WAVES/BTC",
"WBTC/BTC",
"WTC/BTC",
"XEM/BTC",
"XLM/BTC",
"XMR/BTC",
"XTZ/BTC",
"XRP/BTC",
"YFI/BTC",
"ZEC/BTC",
"ZIL/BTC"
],
5. "Experimental" section: Old version of freqtrade would sell to achieve a certain profit. New version allows to sell when a sell signal is detected. So, turn on "use_sell_signal"; turn off Sell only when achieve profit. - "experimental": {
- "use_sell_signal": true,
- "sell_profit_only": false,
- "ignore_roi_if_buy_signal": false
- },
Quick check to ensure basic strategy code works (not backtesting yet)1. Run freqtrade (main.py) with specific config file (-c) and specific strategy (-s) on the top 50 pairs with most volume (--dynamic-whitelist) on the exchange.
(.env) user1@computer1:~/freqtrade$ python3 ./freqtrade/main.py -c config.json -s BBRSI --dynamic-whitelist 50
2. What's happening: Each round, the bot will try to find buy triggers among the top 50 coins (with most volume) on the exchange. For coins already bought, bot will try to find sell signal.
3. This command is just to ensure code works without errors. If code appears to be running, hit Ctrl+C to stop it, and proceed to next section. To do backtest (and optimization), must do it on the coins in the pair_whitelist. Next: Backtesting.
Backtesting
1a. Execute a backtest (backtesting) by getting new data about the pairs in the pairs_whitelist (--refresh-pair) like this, and save backtest results to user_data/backtest/backtest-result.json:
(.env) user1@computer1:~/freqtrade$ python3 ./freqtrade/main.py -c config.json -s BBRSI backtesting --refresh-pair --export trades --export-filename user_data/backtest_data/backtest-result.json
1b. Execute a backtest (backtesting) using existing pairs data (assuming it was downloaded earlier)
(.env) user1@computer1:~/freqtrade$ python3 ./freqtrade/main.py -c config.json -s BBRSI backtesting
2. Notes:
- If first time running a backtest, binance will give you 30 days of data (u can dload it). The longer u work on it, the more data u can addon to your current dloaded data. Backtesting with more data results in better bot.
- Warnings such as: "XXX/YYY" has missing frames ==> several reasons: coin didn't trade during that period, or exchange was under maintenance during that period.
- Downloaded data (e.g. ETH_BTC-1h.json) is stored in /freqtrade/user_data/data
- Inside the data file, the format is like this:
- [1544536800000, 0.025903, 0.025942, 0.025825, 0.025872, 7481.045]
- => [Unix timestamp, open, high, low, close, volume]
3. Backtest results:
....
2020-12-10 21:29:31,016 - freqtrade.optimize.backtesting - INFO - Measuring data from 2020-11-10T14:00:00+00:00 up to 2020-12-10T12:00:00+00:00 (29 days)..
Result for strategy BBRSI
================================================== BACKTESTING REPORT =================================================
| pair | buy count | avg profit % | cum profit % | total profit BTC | avg duration | profit | loss |
|:---------|------------:|---------------:|---------------:|-------------------:|:---------------|---------:|-------:|
| ETH/BTC | 12 | -0.85 | -10.16 | -0.00101655 | 19:35:00 | 5 | 7 |
| LTC/BTC | 17 | 0.64 | 10.94 | 0.00109497 | 11:11:00 | 13 | 4 |
| ETC/BTC | 14 | -0.07 | -0.94 | -0.00009413 | 16:39:00 | 7 | 7 |
| DASH/BTC | 10 | -0.57 | -5.72 | -0.00057255 | 18:12:00 | 6 | 4 |
| ZEC/BTC | 18 | 0.92 | 16.65 | 0.00166632 | 13:00:00 | 12 | 6 |
| XLM/BTC | 13 | -0.36 | -4.70 | -0.00047067 | 18:14:00 | 6 | 7 |
| POWR/BTC | 15 | -0.81 | -12.12 | -0.00121320 | 16:00:00 | 8 | 7 |
| ADA/BTC | 12 | -1.22 | -14.63 | -0.00146494 | 17:35:00 | 5 | 7 |
| XMR/BTC | 9 | -0.03 | -0.31 | -0.00003128 | 15:07:00 | 5 | 4 |
| BNB/BTC | 11 | -0.92 | -10.12 | -0.00101252 | 17:11:00 | 3 | 8 |
| QLC/BTC | 17 | 0.20 | 3.36 | 0.00033610 | 13:21:00 | 9 | 8 |
| KMD/BTC | 13 | -1.21 | -15.76 | -0.00157717 | 16:46:00 | 5 | 8 |
| WTC/BTC | 16 | -0.17 | -2.69 | -0.00026907 | 14:56:00 | 9 | 7 |
| TRX/BTC | 15 | -0.47 | -7.12 | -0.00071259 | 13:08:00 | 4 | 11 |
| XRP/BTC | 8 | -1.77 | -14.14 | -0.00141558 | 21:45:00 | 2 | 6 |
| EOS/BTC | 13 | 0.43 | 5.62 | 0.00056288 | 12:28:00 | 10 | 3 |
| XVG/BTC | 44 | -0.29 | -12.79 | -0.00128081 | 5:45:00 | 14 | 30 |
| NEO/BTC | 14 | 0.11 | 1.51 | 0.00015073 | 13:30:00 | 8 | 6 |
| LINK/BTC | 12 | -0.74 | -8.92 | -0.00089329 | 14:40:00 | 7 | 5 |
| ONT/BTC | 16 | 0.71 | 11.40 | 0.00114147 | 13:45:00 | 10 | 6 |
| XEM/BTC | 10 | -0.12 | -1.16 | -0.00011656 | 18:12:00 | 5 | 5 |
| VET/BTC | 20 | 0.50 | 9.91 | 0.00099211 | 8:57:00 | 9 | 11 |
| ICX/BTC | 12 | -0.03 | -0.35 | -0.00003467 | 13:20:00 | 7 | 5 |
| TOTAL | 341 | -0.18 | -62.25 | -0.00623100 | 13:40:00 | 169 | 172 |
...
Interpretation:
- Backtest of strategy BBRSI was performed on 29 days worth of data on the pairs found in the "pair_whitelist" list
- Over the 29 days,
- made a total 341 trades (covers both cases whereby (1) buy + sold (bcos detected sell signal); and (2) buy + forced sell (bcos ran out of days). Average profit per trade of -0.18% (therefore a loss). (Ideally, avg profit per trade > 0, then automated trading can grow your cumulative profit more quickly). Also, each trade took an avg 13h40m00s (so this is not a quick strategy, it takes hours to get in & out).
- 169 trades were profitable, 172 trades were losses. So, u won 49.6% and lost 50.4% of all trades. (Ideally u want a >50% success rate. e.g. 180 won, 161 lost. Automated trading helps ppl complete the losing trades, so that they can enjoy the overall gains).
- The biggest winner was ZEC, with 16.65% cum profit. Biggest loser was KMD, with 15.76% loss
...
================================================== SELL REASON STATS ==================================================
| Sell Reason | Count |
|:--------------|--------:|
| sell_signal | 332 |
| force_sell | 9 |
=============================================== LEFT OPEN TRADES REPORT ===============================================
| pair | buy count | avg profit % | cum profit % | total profit BTC | avg duration | profit | loss |
|:---------|------------:|---------------:|---------------:|-------------------:|:---------------|---------:|-------:|
| ETH/BTC | 1 | -0.55 | -0.55 | -0.00005459 | 5:00:00 | 0 | 1 |
| LTC/BTC | 1 | -0.20 | -0.20 | -0.00002000 | 6:00:00 | 0 | 1 |
| ZEC/BTC | 1 | 0.21 | 0.21 | 0.00002057 | 5:00:00 | 1 | 0 |
| POWR/BTC | 1 | -0.39 | -0.39 | -0.00003892 | 1:00:00 | 0 | 1 |
| KMD/BTC | 1 | -0.55 | -0.55 | -0.00005481 | 5:00:00 | 0 | 1 |
| WTC/BTC | 1 | -0.76 | -0.76 | -0.00007581 | 1:00:00 | 0 | 1 |
| NEO/BTC | 1 | 0.47 | 0.47 | 0.00004667 | 7:00:00 | 1 | 0 |
| VET/BTC | 1 | -0.20 | -0.20 | -0.00002000 | 5:00:00 | 0 | 1 |
| ICX/BTC | 1 | -0.01 | -0.01 | -0.00000083 | 10:00:00 | 0 | 1 |
| TOTAL | 9 | -0.22 | -1.98 | -0.00019772 | 5:00:00 | 2 | 7 |
...
Interpretation:
- Out of the 341 trades (in the first table), bot sold 332 bcos detected sell signal, and forced sold 9 bcos ran out of days.
- When the bot forced sold those 9 trades, it caused a change in cum profit of -1.98%. This contributed to the overall cum profit in first table of -62.25% (meaning the -62.25% already includes the effect of the forced sold 9 coins).
4. For better visualization, plot a selected coin's price vs time, to see all the detected buy and sell signals
Example: Use freqtrade's plotter (plot_dataframe.py) to see the strategy's (-s) Bb_Rsi buy/sell signals for the pair (-p) ZEC/BTC, generated by the indicators (bollingerbands & RSI). We choose the bollinger bands indicators to be on the main plot (--indicators1 bb_lowerband,bb_middleband,bb_upperband) and RSI to be on separate plot (--indicators2 rsi):
(.env) user1@computer1:~/freqtrade$ python3 ./scripts/plot_dataframe.py -s Bb_Rsi -p ZEC/BTC --indicators1 bb_lowerband,bb_middleband,bb_upperband --indicators2 rsi
- If you get the error ModuleNotFoundError: No module named 'plotly' , then try to install plotly first like this: (.env) user1@computer1:~/freqtrade$ pip install plotly
- If you get the error FileNotFoundError: [Errno 2] No such file or directory: 'user_data/backtest_data/backtest-result.json' , then see here https://github.com/freqtrade/freqtrade/issues/2081
- The name of indicators (e.g. bb_lowerband) must be defined in the strategy (i.e. bb_rsi.py > class "Bb_Rsi" > def "populate_indicators" > dataframe['bb_lowerband'] = bollinger['lower'])
- The BB shaded area (vs BB lines) are shown by default.
- Resultant plot appears in FIREFOX
- Shows all possible buy & sell signals. Note: (1) If bot already bought, it won't buy again (need to sell current position). (2) If bot already sold, it won't sell again (need to buy into a new position). (3) If "max_open_trades" (in config.json) has been achieved, then the bot won't buy more.
- Admittedly, TradingView charts look & feel more usable. https://www.tradingview.com/chart/RARYjeai/
Optimization
- Why?
- You want a strategy where u can let the bot run by itself while u sleep peacefully at night
- Maximizing profit without care for size of risk (volatility or StdDev) taken is nuts
- You want to maximise the Sharpe Ratio (returns relative to risk)
- If bot A makes 40% profit but faced 40% volatility, its SR=1. If bot B makes 10% profit by facing 1% volatility, its SR=10. We prefer to run bot B (SR is higher). Running bot A means it could buy or sell at a very unfavourable prices; it might be very profitable one night, but the next night might lose everything. By running bot B, u can give it $1m and it would return $1.1m safely.
- Objective: Maximise the Sharpe Ratio by adjusting available parameters
- Use hyperopt:
- will change the value of 1 parameter (e.g. RSI=30) (among several available) and run an iteration on available price data. If iteration gives an improved Sharpe Ratio, then it will try to develop that result further by changing that same parameter (e.g. RSI=29); if iteration gave lousier result then it will go back to the previous node and try something else. Eventually it will move to the next parameter (e.g. BB's std dev=3) and run iterations, measure results and adjust (e.g. BB std dev=2).
- Important to allow a sufficient number of iterations to test all parameters with all reasonable values, so that results will reach a "convergence" or a local minima (to obtain the best possible Sharpe Ratio with a specific value of RSI and BB)
- Initial developer team's idea was to minimize: result (= trade_loss + profit_loss + duration_loss) (see old "calculate_loss" function which has been commented out)
- In our case, we have modified the "calculate_loss" function to incorporate and calculate the annualized Sharpe Ratio
- considers first and last day of trade, to calc annualized values (SR = (return/risk)*(365)^(1/2))
- considers slippage.
- Backtest may get great results, but reality is: (1) when u buy, u may not get the price u want (2) your presence has an impact on the price; u're an extra buyer (3) when u buy, u don't buy at the last price; u buy at the ask price (33) when u sell, u don't sell at the last price but u sell at the bid price.
- To address slippage, we deduct 0.05% of the profit from each trade. With this extra loss, we face better odds when trading live.
- sharp_ratio = expected_average_return/np.std(total_profit)*np.sqrt(365) (a negative value bcos the original approach was a minimization function)
- Then get the max SR by taking the negative of the result.
- How to write the optimization:
- Test only parameters and range of values that make sense.
- [OPTIONAL READING] In the file default_hyperopt.py, what matters to us:
- "populate_indicators" function must have same indicators as strategy file (bb_rsi.py)
- in the "populate_buy_trend" function, define the rships like this:
- #GUARDS AND TRENDS (must be true then only go to "#TRIGGERS")
- if 'rsi-enabled' in params and params['rsi-enabled']:
- conditions.append(dataframe['rsi'] < params['rsi-value'])
- Note: indicator (rsi) is either below or above a certain value "rsi-value". This value is constantly changing over time.
- #TRIGGERS
- if 'trigger' in params:
- if params['trigger'] == 'bb_lower':
- conditions.append(dataframe['close'] < dataframe['bb_lowerband'])
- if params['trigger'] == 'macd_cross_signal':
- conditions.append(qtpylib.crossed_above(
- dataframe['macd'], dataframe['macdsignal']
- ))
- if params['trigger'] == 'sar_reversal':
- conditions.append(qtpylib.crossed_above(
- dataframe['close'], dataframe['sar']
- Note: triggered when closing price is below lowerBB
- in the "indicator_space" function, define the values to test:
- Integer(20, 40, name='rsi-value'),
- Note: Test different integer values of 'rsi-value' between 20 and 40 (must make sense; don't give the entire range 0-100)
- Categorical([True, False], name='rsi-enabled'),
- Note: Turn on/off the indicator, just to see how it affects SR
- Categorical(['bb_lower', 'macd_cross_signal', 'sar_reversal'], name='trigger')
- Note: Test each indicator and try to find out which indicator is the best trigger. Must match indicators in #TRIGGERS.
- in the "stoploss_space" function:
- Real(-0.5, -0.02, name='stoploss'),
- Note: Try real values (vs integers) between -50% and -2%, to find the best stoploss value which maximises Sharpe Ratio. % of that particular trade (not percentage of your entire account).
- in the "roi_space" function:
- Integer(10, 120, name='roi_t1'),
- Integer(10, 60, name='roi_t2'),
- Integer(10, 40, name='roi_t3'),
- Real(0.01, 0.04, name='roi_p1'),
- Real(0.01, 0.07, name='roi_p2'),
- Real(0.01, 0.20, name='roi_p3'),
- Note1: Find an integer btwn 10 and 120 minutes for time t1
- Note2: and find a real number btwn 1% and 4% as profit p1
- Note3: which maximizes the SR
- DOING THE OPTIMIZATION:
- Duplicate freqtrade/freqtrade/optimize/default_hyperopt.py then rename to e.g. bbris_opt.py then save
- Replace class name "DefaultHyperOpts" with your strategy name e.g. BBRSI_OPT
- "populate_indicators" function
- def populate_indicators(dataframe: DataFrame, metadata: dict) -> DataFrame:
- # RSI
- dataframe['rsi'] = ta.RSI(dataframe)
- # Bollinger bands
- bollinger1 = qtpylib.bollinger_bands(qtpylib.typical_price(dataframe), window=20, stds=1)
- dataframe['bb_upperband1'] = bollinger1['upper']
- dataframe['bb_middleband1'] = bollinger1['mid']
- dataframe['bb_lowerband1'] = bollinger1['lower']
- bollinger2 = qtpylib.bollinger_bands(qtpylib.typical_price(dataframe), window=20, stds=2)
- dataframe['bb_upperband2'] = bollinger2['upper']
- dataframe['bb_middleband2'] = bollinger2['mid']
- dataframe['bb_lowerband2'] = bollinger2['lower']
- bollinger3 = qtpylib.bollinger_bands(qtpylib.typical_price(dataframe), window=20, stds=3)
- dataframe['bb_upperband3'] = bollinger3['upper']
- dataframe['bb_middleband3'] = bollinger3['mid']
- dataframe['bb_lowerband3'] = bollinger3['lower']
- bollinger4 = qtpylib.bollinger_bands(qtpylib.typical_price(dataframe), window=20, stds=4)
- dataframe['bb_upperband4'] = bollinger4['upper']
- dataframe['bb_middleband4'] = bollinger4['mid']
- dataframe['bb_lowerband4'] = bollinger4['lower']
- return dataframe
- Notes:
- Want to find best RSI to buy and best RSI to sell (rsi). Hyperopt will automatically try different values to find the best RSIs.
- When buying, is it best to buy at lowerBB of 1 SD? 2 SD? 3SD? etc?Later will store the results of "bb_lowerband1", "bb_lowerband2", "bb_lowerband3" and "bb_lowerband4" in a dictionary. Gotta tell hyperopt to manually change the values of SD.
- "populate_buy_trend" function
- def populate_buy_trend(dataframe: DataFrame, metadata: dict) -> DataFrame:
- conditions = []
- # GUARDS AND TRENDS
- if 'rsi-enabled' in params and params['rsi-enabled']:
- conditions.append(dataframe['rsi'] > params['rsi-value'])
- # TRIGGERS
- if 'trigger' in params:
- if params['trigger'] == 'bb_lower1':
- conditions.append(dataframe['close'] < dataframe['bb_lowerband1'])
- if params['trigger'] == 'bb_lower2':
- conditions.append(dataframe['close'] < dataframe['bb_lowerband2'])
- if params['trigger'] == 'bb_lower3':
- conditions.append(dataframe['close'] < dataframe['bb_lowerband3'])
- if params['trigger'] == 'bb_lower4':
- conditions.append(dataframe['close'] < dataframe['bb_lowerband4'])
- dataframe.loc[
- reduce(lambda x, y: x & y, conditions),
- 'buy'] = 1
- return dataframe
- NOTES: RSI: want to find the best "rsi-value" for the buy rsi (rsi) (current "best" value = 30. Maybe there's a better value to be found).
- "indicator_space" function
- def indicator_space() -> List[Dimension]:
- return [
- Integer(5, 50, name='rsi-value'),
- Categorical([True, False], name='rsi-enabled'),
- Categorical(['bb_lower1', 'bb_lower2', 'bb_lower3', 'bb_lower4'], name='trigger')
- ]
- NOTE:
- choose the best integer for "rsi-value". Try integers between 5 and 50. Why? bcos dowan to buy when extremely oversold(0) or overbought (>70).
- choose best category for "rsi-enabled". Should RSI be enabled (true) or not (false)?
- choose best category for "trigger"... i.e. pick the best from among 'bb_lower1', 'bb_lower2', 'bb_lower3', 'bb_lower4'
- "populate_sell_trend" function
- def populate_sell_trend(dataframe: DataFrame, metadata: dict) -> DataFrame:
- # print(params)
- conditions = []
- # GUARDS AND TRENDS
- if 'sell-rsi-enabled' in params and params['sell-rsi-enabled']:
- conditions.append(dataframe['rsi'] > params['sell-rsi-value'])
- # TRIGGERS
- if 'sell-trigger' in params:
- if params['sell-trigger'] == 'sell-bb_lower1':
- conditions.append(dataframe['close'] > dataframe['bb_lowerband1'])
- if params['sell-trigger'] == 'sell-bb_middle1':
- conditions.append(dataframe['close'] > dataframe['bb_middleband1'])
- if params['sell-trigger'] == 'sell-bb_upper1':
- conditions.append(dataframe['close'] > dataframe['bb_upperband1'])
- dataframe.loc[
- reduce(lambda x, y: x & y, conditions),
- 'sell'] = 1
- return dataframe
- NOTE:
- Personally guru doesn't believe need RSI to sell, but just let bot decide as an exercise.
- When selling, we just look at BB with 1 SD (bcos waiting to sell at upperBB with 3 SD is too greedy). So, test: (1) sell when closing price hit 1SD lowerBB i.e. "bb_lowerband1" (for a "buy cheapest, sell cheap" strategy), (2) sell when close price hit 1sd middleBB, (3) sell when close price hit 1sd upperBB.
- "sell_indicator_space" function
- def sell_indicator_space() -> List[Dimension]:
- return [
- Integer(60, 100, name='sell-rsi-value'),
- Categorical([True, False], name='sell-rsi-enabled'),
- Categorical(['sell-bb_lower1', 'sell-bb_middle1', 'sell-bb_upper1'], name='sell-trigger')
- ]
- NOTE: (1) Find the best integer btwn 60 and 100 for "sell-rsi-value". Why? Just for the exercise. Usually be patient to buy-in, but wanna be quick (don't want to make it a hassle (i.e. wait for RSI) to get out. (2) Pick best category for sell-rsi-enabled: Use it (true) or don't use it (false) ? (3) choose best category for "sell-trigger"... i.e. pick the best from among 'sell-bb_lower1', 'sell-bb_middle1', 'sell-bb_upper1'.
- "populate_buy_trend" and "populate_sell_trend" functions:
- Copy both the entire functions from bbrsi.py and paste as the last part in bbrsi_opt.py
- Finally, register your file-to-be-optimized in the init file:
- /freqtrade/freqtrade/optimize/__init__.py
- Duplicate line "from freqtrade.optimize.default_hyperopt import DefaultHyperOpts" so now u have 2 consecutive identical lines. Newer line:
- replace "default_hyperopt" with your strat file i.e. "bbrsi_opt"
- replace classname "DefaultHyperOpts" with yours i.e. BBRSI_OPT
- Run the optimization:
- Get custom opt (--customhyperopt) to run the optimization class called "BBRSI_OPT" (in "class BBRSI_OPT(IHyperOpt):"), and get "hyperopt" to iterate (-e) 100 times, onto a timerange (--timerange) that must be equal to that used in the backtest (20201110-20201216)
- (.env) user1@computer1:~/freqtrade$ python3 ./freqtrade/main.py -c config.json --customhyperopt BBRSI_OPT hyperopt -e 1000
- (Note: try run in batches of 500-1000 iterations; any higher tends to crash my PC somehow. Completing a hyperopt will save the results, & u can run the same command again which builds on those saved results. When a hyperopt crashes midway, u lose the results for that round.)
- Note: Give it the biggest number of iterations to find the best combination that maximises SR. E.g. Leave it to run overnight. Or while u go out. (100 is insufficient).
- Hit Ctrl+S to pause
- How to read, example:
- 758/1000: 212 trades. Avg profit 0.35%. Total profit 0.00732600 BTC (73.1891Σ%). Avg duration 699.9 mins.. Loss -29.24738
- ...
- Note: 758th iteration (changed a parameter & ran 758th time). Made 212 trades. Total profit was +73.2% over 40 days (==> +668% profit/year). Sharpe Ratio = +29.2 (= -1 x -29.24738)
- Note: Each dot "." is 1 iteration (or "epoch"). Bot only displays full description of iteration when it discovers an improvement (i.e. increase) in the Sharpe Ratio. If u specified thousands of iterations and all u see are dots after hundreds of iterations, u can hit Ctrl+C to end the process, and bot will print the best results obtained thus far (and u can use that).
- 'rsi-enabled': False,
- 'rsi-value': 26,
- Note: Using the RSI to buy is not recommended (even if the recommended value was 26)
- 'sell-rsi-enabled': True,
- 'sell-rsi-value': 76,
- Note: Using the RSI to sell is recommended. Recommended value is 76
- 'sell-trigger': 'sell-bb_lower1',
- Note: Best to sell when closing price hits lowerBB @ 1SD
- 'stoploss': -0.18757341633907693,
- Note: The best stoploss is -18%
- 'trigger': 'bb_lower3'}
- Note: The best buy trigger is when price hits lowerBB @ 3 SD
- { 0: 0.1038762896661849,
- 26: 0.07887892828698762,
- 48: 0.013130689064203501,
- 108: 0}
- Note: [Above "ROI table" is a strategy based on exit time (unrelated to SR)]. To maximize profit: 0-25th minute: exit if u have at least 10.4% profit. If can't achieve that 10.4%, then in 26-47th minute, exit with at least 7.9% profit. If can't achieve that 7.9%, then in 48th-107th minute, exit with at least 1.3% profit. If cant achieve that 1.3%, then exit with at least 0% profit.
- Saved iterations
- Start a new strategy and do 100 iterations; the discoveries are saved here:
- /freqtrade/user_data/hyperopt_results.pickle
- /freqtrade/user_data/hyperopt_tickerdata.pkl
- Continue with exact same strategy and do another 50 iterations, the bot will read the previous work (from the 2 pickle files above), then continue where it left off.
- If u decide to change anything in the strategy, must delete the above 2 files (bcos the previous work done should not be referred any longer, bcos variables have changed). Then only run new iterations.
- Once you've run sufficient iterations and happy with the results, copy them aside, example:
- Best result:
- 212 trades. Avg profit 0.35%. Total profit 0.00732600 BTC (73.1891Σ%). Avg duration 699.9 mins.
- 'rsi-enabled': False,
- 'rsi-value': 33,
- 'sell-rsi-enabled': True,
- 'sell-rsi-value': 81,
- 'sell-trigger': 'sell-bb_middle1',
- 'stoploss': -0.4884759339579788,
- 'trigger': 'bb_lower3'}
- { 0: 0.14804935708164307,
- 13: 0.04033164340222185,
- 39: 0.01698340789831671,
- 80: 0}
- Results must make sense (because, it cld be true for this data, but may not work during live runs): Buy when closing price hits bb_lower3 (i.e. oversold) and rsi-value=33 (i.e. price is falling but drop rate is slowing) (==> buy at a low good price). Sell when closing price hits sell-bb_middle1 (i.e. middle band of 1 SD), and sell-rsi-value=81 (i.e. overbought, on a spike up). Strategy makes sense bcos buy at very cheap, and sell at average price.
- Accuracy test: Does backtesting the strategy while using the optimized parameter values (e.g. RSI values, bollinger SD, stoploss, etc) give the same results (i.e. no. of trades, avg profit, total profit %, etc)?
- Copy optimized values onscreen & paste into original strategy file (bbrsi.py)
- ROI table (e.g. 20: 0.032379109428157866,) --> shd appear in "minimal_roi" dictionary as ("20": 0.032379109428157866,)
- stoploss
- Buy BB:
- 'trigger': 'bb_lower2' --> "populate_indicators" fn >>> bollinger = qtpylib.bollinger_bands(qtpylib.typical_price(dataframe), window=20, stds=2)
- Sell BB:
- 'sell-trigger': 'sell-bb_middle1', --> "populate_sell_trend" fn >>> (dataframe['close'] > dataframe['bb_middleband'])
- Buy RSI:
- 'rsi-value': 41, --> "populate_buy_trend" fn >>> (dataframe['rsi'] >41)
- Sell RSI:
- 'sell-rsi-value': 96, --> "populate_sell_trend" fn >>> & (dataframe['rsi'] >96)
- Note: entire line "& (dataframe['rsi'] >96)" added in manually
- Save updated strategy file (bbrsi.py)
- Execute the backtest to run in the same environment as hyperopt (i.e. DO NOT REFRESH PAIRS bcos will change the dataset that the hyperopt ran with earlier):
(.env) user1@computer1:~/freqtrade$ python3 ./freqtrade/main.py -c config.json -s BBRSI backtesting --export trades --export-filename user_data/backtest_data/backtest-result.json --eps --dmmp
Note: This will save the backtest results to user_data/backtest/backtest-result.json
- Backtest should result in "TOTAL" row showing same number of trades and cum profit % as per results from the optimization process. If identical, this means strategy is working correctly, ready to go to next step (Sandbox: i.e. test strategy in live market using fake money).
- Not identical? Check that, before starting optimization:
- (1) Ensure "trailing_stop": true in config.json.
- (2) The contents of the 3 functions (populate_indicators, populate_buy_trend and populate_sell_trend) should be identical word-for-word in "bbrsi.py" and "bbrsi_opt.py".
- (3) Ensure when backtesting you used "--eps" and "--dmmp". "Enable position stacking" means allow bot to buy the same pair multiple times. "Disable max market positions" means disable the max open trades during a backtest.
- Run strategy in Sandbox
- Why? Let strategy trade using fake money, on live data which it has never seen before (out-of-sample data) to see if strategy still performs (i.e. are results similar to the backtest results of the optimized strategy?). Does it underperform due to overfitting? Let strategy complete 100 trades before deciding.
- Command: run freqtrade, using a specific (-c) config file in the same freqtrade folder (config.json), using a specific strategy (-s) BBRSI, on the top 100 coin pairs on the exchange with most volume
- (.env) user1@computer1:~/freqtrade$ python3 ./freqtrade/main.py -c config.json -s BBRSI --dynamic-whitelist 100
- Note: A max of 100 pairs is still ok, not many will have low volume (i.e. hi chance of slippage).
- If strategy involves frequent buy-sell within short span of time, better to avoid low liquidity coins (i.e. pick a lower number of pairs e.g. 50)
- Slippage: u r an active member in the orderbook, so u may not get result as good as in the backtest.
- Once execute the command, u can monitor the telegram messages on phone:
- Status: running
- Warning: Dry run is enabled. All trades are simulated.
- Exchange: binance
- Stake per trade: 0.01 BTC
- Minimum ROI: {xxxxxxxxxx}
- Ticker interval: 1h
- Strategy: BBRSI
- Status: Searching for BTC pairs to buy and sell based on VolumePairList - top 100 volume pairs
- Bot will keep looking for buy signals each round. Based on backtest results: data spanned 35 days, total trades=219 trades, so u expect an average of 6 trades per day.
- When buy signal is found
- Telegram:
- Binance: Buying XXX/BTC with limit 0.00000xxx
- (0.01 BTC, 36.594 USD)
- Terminal:
- Buy signal found: about to create a new trade with stake_amount: 0.01 BTC
- Using Last Ask / Last Price
- Sending rpc message: {xxxxxx} (tells binance)
- Wallets synced
- Found open order for Trade(id=1, pair=XXX/BTC, amount=xxx...)
- Applying fee on amount for Trade(id=1, ....)
- Updating trade (id=1)..
- LIMIT_BUY has been fulfilled for Trade(id=1, ...) (the bot simulates a trade)
- Telegram commands:
- Hit the "/daily" button: see total profit per day for past (default) 7 days. To see past N days, type "/daily N".
- Hit the "/profit" button:
- Hit the "/balance" button: see coins remaining in your exchange's wallet
- Hit the "/status" button: see "tradeID", pair name, when open (e.g. 2 mins ago), amount, open rate (i.e. buy price), close rate, current rate (i.e. current price), close profit (i.e. profit if sold the pair), current profit (%) (note: usually negative initially bcos u bought while price was dropping). Note: (1) In live mode, if a buy order was placed, but timeout bcos unable to fulfill the order, bot will cancel the current order (e.g. TradeID=8) and place a new buy order with updated id (TradeID=9). (2) In live mode, if OpenRate = CurrentRate, then CurrentProfit= trade fee (e.g. -0.20%).
- Hit the "/status table" button: (nice table). In live mode, profit includes the trading fee. Status of open trades. See pairs which, (1) failed to fulfill any "sell signal", and also (2) did not fulfill the minimum ROI strategy.
- Hit the "/performance" button: See all closed trades. Format: Pair, Profit%, Number of trades
- Hit the "/count" button: see current open positions (e.g. 4), and max positions allowed (i.e. 15), total stake = 0.04 (=4 positions x 0.01BTC)
- Hit the "/help" button: see all commands u can use
- e.g. type "/forcesell 4": sell the pair with tradeID=4.
- errors?
- Error msg (use "sudo systemctl status freqtrade.service" to see error msg):
- freqtrade.exceptions.TemporaryError: Could not get balance due to InvalidNonce. Message: binance {"code":-1021,"msg":"Timestamp for this reque st is outside of the recvWindow.
- Do this:
- sudo apt-get update
- sudo apt-get install ntpdate
- sudo ntpdate ntp.ubuntu.com
- Hit "/reload_config": reload config if any changes were made to config or strategy file
- errors?
- Telegram msg stuck at "Status: reload_config" ==> use "sudo systemctl status freqtrade.service" to confirm the error is "freqtrade.exceptions.TemporaryError: Could not get balance due to InvalidNonce. Message: binance {"code":-1021,"msg":"Timestamp for this reque st is outside of the recvWindow." ==> solution: "sudo ntpdate ntp.ubuntu.com"
- Observe visually:
- Open TradingView https://www.tradingview.com/chart/RARYjeai/ & view your pair
- set e.g.: ARPA/BTC , 1h , BB@3sd
- Buy occurs when a candle hits a lowerBB, but why did buy price>BB and not =BB ?
- Look at order book (use e.g. https://www.binance.com/en/trade/ARPA_BTC > TradingView > 1h > insert indicators): Bought at last ask price. Means bought at higher price. This is slippage. When pairs are less popular => less buyers & sellers => spread is bigger => ask price is higher => slippage bigger.
- Conclusion: Trade pairs with higher volumes so that spreads are smaller, to minimize slippage.
- Note: When u have a strategy with good results (i.e. high Sharpe Ratio), don't think *i have to change things to see more trades done frequently*. It's performance that counts. BUT if you like to see more trades, just play with the time interval i.e. instead of using 1h candles, try using 15m candles.
- Once results are good (i.e. similar no. of trades, profit % per period etc to the hyperopt), then u're ready to go live.
- Run strategy live
- config.json: change "dry_run": true, to "dry_run": false,
- Minimise losses due to wide spreads:
- Pairs with low volume will have larger spreads.
- Example:
- Ask price=0.00008150
- Bid price= 0.00008130
- Spread=20 satoshis (=8150-8130)
- By default, when the bot buys at the last price, it assumes last price = ask price (i.e. the higher price), and will buy at 8150
- In the low volume example above, when bot buys at ask price, it loses 0.24% (=(8130-8150)/8130 x 100%)
- Solution: Buy at the bid (lower) price
- Cons: price might go up and u miss the purchase, BUT
- Pros: when u consider the 0.24% for all the trades done... it adds up n u get overall more profit. Since it's automated, u can let bot do it (u don't have to be there at the computer patiently waiting...). So, try to buy at bid price to get more profit.
- How?
- config.json > bid strategy (i.e. your bot's buying strategy) > use_order_book: change false to true
- This change is not done during sandbox mode bcos u don't have actual bid & ask prices to work with. Only do it for live mode. In sandbox mode, assumption is bot buys at the worst price (high ask price), and sells at the worst price (low bid price)
- config.json > bid strategy > ask_last_balance: change 0.0 to 1.0 (from "buy at (higher) ask price" to "buy at (lower) bid price")
- Note about: config.json > ask strategy (i.e. your bot's selling strategy)
- Don't use order book. Why? Order book adds another layer of decisions. We want to be able to get out of the trade quick if possible. (Careful to buy, quick to sell)
- Prepare starting capital
- Binance > Wallet > Fiat & Spot https://www.binance.com/en/my/wallet/account/main
- Start with 0.15109228 BTC in wallet (ideally 0.15 BTC)
- Have e.g. 12.81661160 BNB (=0.03357568 BTC) (Binance gives discount for trading fee if we keep BNB in wallet; Binance deduct from this BNB)
- Note starting total BTC value = 0.18466796 BTC (=0.15109228+0.03357568)
- Run live
- (.env) user1@computer1:~/freqtrade$ python3 ./freqtrade/main.py -c config.json -s BBRSI --dynamic-whitelist 100
- Once execute the command, can monitor the telegram messages on phone:
- Status: running
- Exchange: binance (No more "Dry run is enabled. All trades are simulated.")
- etc
- Telegram messages shd correspond to the exchange's orders.
- See open orders here: Binance > Orders > Spot Orders (https://www.binance.com/en/my/orders/exchange/openorder)
- To see your open (i.e. unfulfilled) order in the order book:
- Binance > Trade > Advanced (https://www.binance.com/en/trade/BTC_USDT?layout=pro) > enter the crypto ticker > your open order shd have a yellow arrow in the order book. If you did the "Minimise losses due to wide spreads" above, the arrow should point at the bid price.
- To close all positions, normally just type "forcesell" button in Telegram. If accidentally hit "/stop" button in Telegram, then goto https://www.binance.com/en/trade/BTC_USDT?layout=pro and click "Cancel" on each open (unfulfilled) buy/sell order.
- Reset your wallet to only BTC and BNB:
- In case of hardware failure (e.g. blackout), u can just restart the bot and freqtrade will continue where it left. Consider using spare PC backup battery, Raspberry Pi, VPS, etc.
- Calculate your absolute performance:
- Note the starting total BTC value (before began trading) (e.g. 0.18537136 BTC)
- https://www.binance.com/en/my/wallet/account/main > Get ending total BTC value (e.g. 0.19820787 BTC)
- Profit = (0.19820787/0.18537136-1)x100% = 6.9% over 4 days => amazing
- Extra note: U can use your BTC capital more efficiently by changing the "stake" amount, so that all BTC in the wallet is unused (to avoid small fractions of BTC sit unused in the exchange wallet). e.g. instead of 0.01BTC per trade, try use 0.015 BTC. Maximise BTC used to maximise return.
Walk-forward Analysis
- Usual way to get better results: 1. Strategy that makes sense 2. LARGE sample size (many different pairs)(& more historical prices per pair) 3. WFA 4. Sandbox then Live (using small money)
- WFA
- Aim: To avoid overfitting. Example: Split past data into 2 parts: for training the strategy, and for testing the trained strategy. If the strategy still yields good results, then its not overfitted.
- Lets say u have 3 months of data: Jan, Feb, Mar
- Jan-Feb: use this data to get optimized strategy ("in-sample data")
- Mar: use your optimized strategy to perform a backtest on Mar data ("out-of-sample data")
- How
- Open data file e.g.: freqtrade\user_data\data\binance\XLM_BTC-1h.json
- Get start date:
- See 1st row > copy the first 10 digits of the Unix timestamp e.g. 1544310000
- Convert to normal timestamp: https://www.unixtimestamp.com/
- Get normal time: e.g. 12/08/2018 @ 11:00pm (UTC) ==> Day 1 is Dec 8th, 2018
- Get end date:
- Do the same for the last row: e.g. 1549846800
- ==> 02/11/2019 @ 1:00am (UTC) ==> Last day is Feb 11th, 2019
- Split the days
- Total days = 65 (can use https://www.timeanddate.com/date/duration.html)
- Lets optimise using first 2/3 of total = approx 40 days (so, in freqtrade timerange format: 20181208-20190118)
- Lets walkforward using last 1/3 of total = approx 25 days (so: 20190119-20190211)
- Note: If you have a bot running a sandbox on one terminal, u can always open a new terminal to run something else:
- 1. New terminal: Ctrl+Alt+T
- 2. Activate virtual environment
- user1@computer1:~$ cd freqtrade
- user1@computer1:~/freqtrade$ source .env/bin/activate
- 3. Then proceed to do the other activity.
- Run the optimization on the first 2/3 "in-sample" data:
- user1@computer1:~/freqtrade$ python3 ./freqtrade/main.py -c config.json --customhyperopt BBRSI_OPT hyperopt -e 1000 --timerange=20181208-20190118
- Once optimised results appear, insert them into the strategy (i.e. bbrsi.py): e.g. minimal roi, stopless, buy signals, sell signals. Backtest against the SAME "in-sample" data if u want:
- (.env) user1@computer1:~/freqtrade$ python3 ./freqtrade/main.py -c config.json -s BBRSI backtesting --export trades --export-filename user_data/backtest_data/backtest-result.json --eps --dmmp --timerange=20181208-20190118
- Run the backtest on the last 1/3 "out-of-sample"data:
- user1@computer1:~/freqtrade$ python3 ./freqtrade/main.py -c config.json -s BBRSI backtesting --export trades --export-filename user_data/backtest_data/backtest-result.json --timerange=20190119-20190211
- Compare results
- cumulative profit % still positive? => can do sandbox => go live.
Get More Strategies
- Why? Build new strategies by learning from other strategies, borrow from them, combine indicators, etc.
- Google "github freqtrade strategies":
- Select a strategy (e.g. AverageStrategy.py)
- Summary: Uses macd. When MA-ST (=8 periods) cross above MA-LT (=21 periods) => buy. Vice versa.
- > click RAW button (code has no hidden html) > remember the strategy's class e.g. "AverageStrategy" in "class AverageStrategy(IStrategy):" > select all > copy
- Make new strategy file: freqtrade\freqtrade\strategy\default_strategy.py > RHS click >duplicate> rename as "macrossover.py" > paste into macrossover.py > save
- Register new strategy file: freqtrade\freqtrade\strategy\__init__.py > Duplicate the line "from freqtrade.strategy.default_strategy import DefaultStrategy" > Change, in the new line: "default_strategy" to "macrossover", and "DefaultStrategy" to the class name "AverageStrategy". > So now u have "from freqtrade.strategy.macrossover import AverageStrategy"
- Take it on a test run (sandbox mode)
- Ensure: config.json > "dry_run": true
- (.env) user1@computer1:~/freqtrade$ python3 ./freqtrade/main.py -c config.json -s AverageStrategy --dynamic-whitelist 100
- Try to get a feel for the strategy, understand what it's doing in the background.
- Try to backtest it, optimise it, do walkforward analysis, sandbox, live.
Edge Positioning
- Why? Get a possible extra edge in the strategy by optimizing the stoploss for each pair. Great if you have many trades for that pair (i.e. LOTS of data for that pair). Not good if you have only a few trades for that pair (e.g. <5 trades done for that pair in a backtest), because then it's overfitting the stoploss for that pair.
- How?
- Setup
- Enable edge. Perform edge once every 3600 seconds. When it runs, it performs calculations based on past 7 days. Edge will update your strategy's stoploss dynamically when running sandbox or live mode.
- Config file:
- "max_open_trades": -1,
- "edge": {
- "enabled": true,
- "process_throttle_secs": 3600,
- "calculate_since_number_of_days": 7,
- "allowed_risk": 0.02,
- "stoploss_range_min": -0.01,
- "stoploss_range_max": -0.90,
- "stoploss_range_step": -0.01,
- "minimum_winrate": 0.0,
- "minimum_expectancy": 0.00,
- "min_trade_number": 1,
- "max_trade_duration_minute": 1440,
- "remove_pumps": false
- },
- Run edge module, with specific strategy (--strategy BBRSI), on a specific period (--timerange=20181110-20181113)
- freqtrade edge --strategy BBRSI --timerange=20201015-20201220
Impact of Cheap Coins on Backtests
- Don't use cheap coins in backtests. Leads to biased "Total cum profit %" in backtest, which lead to wrong decisions.
- Why?
- "Regular price" coin e.g. DASH/BTC
- "Cheap" coin e.g. HOT/BTC. (those worth single- or double-digit satoshis)
- See order book: https://www.binance.com/en/trade/HOT_BTC
- If a trade sees price change from 0.00000003 BTC to 0.00000004 BTC (i.e. from 3 satoshi to 4 satoshi), then profit is +33.3% per trade
- Assuming 100 trades in the backtest, then cum profit % = +3333%
- In a backtest containing many "regular priced" coins and 1 "cheap" coin, if somehow most regular priced coins resulted in a negative cum profit % and 1 cheap coin gave +3333%, then TOTAL cum profit % becomes positive and u would have decided to take the strategy live and get hammered with losses from the regular priced coins. U wont be able to trade the cheap coins effectively bcos there are many professional traders who just focus on buy/sell these coins to profit from this alllll day, and if u r buying at bid (i.e. lower) price to save costs, it can take forever to get your buy order filled (maybe just once in 24 hours).
- So, AVOID the cheap coins: https://coinmarketcap.com/ > sort by price (lowest first) > identify coins with 1-2 digit satoshis.
- ONE/BTC
- ETN/BTC
- QKC/BTC
- IOST/BTC
- IOTX/BTC
- XVG/BTC
- TMTG/BTC
- ANKR/BTC
- REV/BTC
- RVN/BTC
- CHZ/BTC
- TFUEL/BTC
- VET/BTC
- FTM/BTC
- MATIC/BTC
- RSR/BTC
- DGB/BTC
- HOT/BTC
- YOYOW/BTC
UPDATE TO LATEST VERSION OF FREQTRADE
- After learning & understanding old version of freqtrade, highly recommended to get latest version.
- Auto-updates tend to cause issues. If this happens, best to get the very latest versions and reinstall:
- VirtualBox-6.1.17-141968-Win.exe (https://www.virtualbox.org/wiki/Testbuilds)
- ubuntu-20.04.1-desktop-amd64.iso (https://ubuntu.com/download/desktop)
- VBoxGuestAdditions_6.1.17-141968.iso ((https://www.virtualbox.org/wiki/Testbuilds)(install instructions: https://itsfoss.com/virtualbox-guest-additions-ubuntu/)
- How to get latest version
- Create virtual machine: virtualbox.org > download VirtualBox > install > New > Name(VM1)> Type(Linux) > Version(Ubuntu 64-bit) > NEXT > MemorySize(8000mb) > Create virtual harddisk now > VirtualBox disk Image > Dynamically allocated > 10GB > CREATE
- Install OS: Ubuntu.com > download ubuntu (e.g. 20.04.1 Desktop LTS) > RHS click on VM1 > Settings > Storage > ControllerIDE(empty)> choose virtual optical disk > add disk image > select downloaded file > CHOOSE > Ok > click "START" to bootup the virtual machine > "Install Ubuntu" (once installed, can delete the .iso file) > normal installation + download updates during installation + install 3rd party software > erase disk & install ubuntu > Install Now > YourName(FirstLast) + ComputerName(computer1) + Username(user1) + Pwd(xx) + LoginAutomatically > go> installs... > Restart > "Pls remove installation medium">(remove cd by: Devices>OpticalDrive>select downloaded file > Remove disk from virtual drive) > Force unmount > ENTER.
- Ignore any software update request from the "Software Updater" popup.
- Install Guest Additions (to enable fullscreen, copypaste, etc):
$ sudo add-apt-repository multiverseTry this first (bcos latest virtualbox (as at 12jan2021) uses kernel 5.8 which has bug. Must switch back to kernal 5.4):Start VM and immediately hit ESC twice hold left shift to access GRUB menuGrub menu: "Advanced options for Ubuntu" > "Ubuntu, with Linux 5.4.0-42-generic"
Try next: Ubuntu > firefox > https://www.virtualbox.org/wiki/Linux_Downloads > download "virtualbox-6.1_6.1.16-140961~Ubuntu~eoan_amd64.deb" into Downloads folderTerminal:cd DownloadsUninstall old dkms (sudo apt-get remove --auto-remove dkms)Remove all virtualbox* pockets ($ sudo apt-get purge virtualbox*)Launch install in terminal ($ sudo dpkg -i virtualbox-6.1_6.1.16-140961_Ubuntu_eoan_amd64.deb)
$ sudo apt install virtualbox-guest-dkms virtualbox-guest-x11if using Virtualbox with kernal 5.8, will resulted in error msg:
$ sudo reboot- Installation instructions https://itsfoss.com/virtualbox-guest-additions-ubuntu/
- Install Python3.7+: (required by freqtrade)
- Ubuntu 20.04 has Python 3.8 installed by default
- python3 --version
- Install pip for Python 3.8 (required by freqtrade) (~5minutes)
- sudo apt update
- sudo apt install python3-pip
- Errors?
- debconf: delaying package configuration, since apt-utils is not installed
- Selecting previously unselected package libfftw3-double3:armhf.
- dpkg: unrecoverable fatal error, aborting:
- files list file for package 'gcc-8' is missing final newline
- E: Sub-process /usr/bin/dpkg returned an error code (2)
- Solution:
- sudo rm /var/lib/dpkg/info/gcc-8*
- sudo apt-get install --reinstall gcc-8
- pip3 --version
- Install git (required by freqtrade) (~ 3 minutes + 16 minutes)
- sudo apt install git-all
- git --version
- Install virtualenv(recommended by freqtrade) (~2 minutes)
- sudo apt-get install python3-venv
- sudo apt install python3-virtualenv
- virtualenv --help
- Install LATEST version of FREQTRADE (~40 minutes)
- git clone https://github.com/freqtrade/freqtrade.git
- cd freqtrade
- ./setup.sh --install
- reset git branch? yes
- install dependecies for dev? no
- install plotting dependencies? yes
- install hyperopt dependencies? yes
- VM1 Desktop > Ubuntu Software: (if error arises when installing any app below, see solution here: https://askubuntu.com/questions/1246668/ubuntu-software-error-error-opening-directory-usr-share-appdata-no-such-fil)
- install "Telegram Desktop"
- install "Atom" (snappyatom) > Packages > Tree View > Focus > Rightclick left panel > Add Project Folder > home\user1\freqtrade\
- Changes in location of main files:
- config.json : no change (/home/user1/freqtrade/config.json)
- bbrsi.py :
- From: /home/user1/freqtrade/freqtrade/strategy/bbrsi.py
- To: /home/user1/freqtrade/user_data/strategies/bbrsi.py
- bbrsi_opt:
- From: /home/user1/freqtrade/freqtrade/optimize/bbrsi_opt.py
- To: /home/user1/freqtrade/user_data/hyperopts/bbrsi_optx.py
Some commands for latest version of freqtrade (/freqtrade/docs)
- Verify freqtrade successfully installed:
- user1@user1-computer1:~/freqtrade$ source .env/bin/activate
- freqtrade --version
- Get help
- Generate new config file
- freqtrade new-config -c config.json
- Run the bot
- source .env/bin/activate
- freqtrade <subcommand>
- Download data
- 1st time: for the intervals (-t) "15m" and "1h", for the past 90 days (--days 90), for the exchange and pairs as specified (-c) in config.json, (and before starting the d/load, to erase all existing data for the earlier-mentioned exchange/pairs/timeframes (--erase))
- user1@computer1:~/freqtrade$ freqtrade download-data -t {15m,1h} --days 90 -c config_001.json --erase
- Subsequent times: If need to fetch past 10 days data (don't do it if <10 days, entire dataset might be erased), then fetch slightly over 10 days e.g. 15 days:
- user1@computer1:~/freqtrade$ freqtrade download-data -t {15m,1h} --days 15 -c config.json
- Update config.json:
- pair_whitelist
- "exchange": {
- "name": "binance",
- "key": "your_exchange_key",
- "secret": "your_exchange_secret",
- "telegram": {
- "enabled": true,
- "token": "your_telegram_token",
- "chat_id": "your_telegram_chat_id"
- Strategy
- Create new strategy from scratch:
- (.env) user1@user1-computer1:~/freqtrade$ freqtrade new-strategy --strategy BBRSI_001
- New strategy file saved in: user_data/strategies/BBRSI_001.py
- Backtest
- Run backtest, using custom strategy class "BBRSI_001" in file freqtrade/user_data/strategies/bbrsi.py (-s BBRSI_001), on specific pairs at a specific exchange as described in a specific config file (--config config_001.json), using time interval 15m (--timeframe 15m), across a specific period (--timerange=20201015-20201220), then save backtest results for plotting (--export trades), to a specific file (--export-filename=backtest_bbrsi.json)
- (.env) user1@user1-computer1:~/freqtrade$ freqtrade backtesting --config config_001.json -s BBRSI_001 --timeframe 15m --timerange=20191222-20200821 --export trades
- Errors?
- TypeError: 'Series' object is not callable
- Add braces: from "dataframe['close'] > 1.8335 * dataframe['ema9']" to "(dataframe['close'] > 1.8335 * dataframe['ema9'])"
- When performing backtest to verify the hyperopt results: If backtest results are very different from hyperopt results:
- try to run backtest together with "--eps" and/or "--dmmp". Ideally, don't use "--eps" & "--dmmp" bcos sandbox and live modes do not use it. eps means buy the same pair multiple times. dmmp means bot can have infinite open trades.
- check they are correct: timeframe, timerange
- check that the sell settings in strategy file are identical to "ask_strategy" in config file:
- use_sell_signal = True
- sell_profit_only = False
- ignore_roi_if_buy_signal = False
- check in the "buy" and "sell" sections of the strategy file: are there any statements (e.g. guards or triggers) that are irrelevant / should be commented out?
- Backtest results saved to: /freqtrade/user_data/backtest_results/backtest_results-yyyy-mm-dd_hh-mm-ss.json
- Optimization
- Create new hyperopt from scratch:
- (.env) user1@user1-computer1:~/freqtrade$ freqtrade new-hyperopt --hyperopt BBRSI_OPT_001
- New hyperopt file saved in : user_data/hyperopts/BBRSI_OPT_001.py
- Run hyperopt, with specific config file (--config config_001.json), using custom hyperopt file (--hyperopt BBRSI_OPT_001), and getting the 3 functions "populate_indicators" "populate_buy_trend" & "populate_sell_trend" from strategy BBRSI_001 (--strategy BBRSI), using the loss function "SharpeHyperOptLossDaily" as found in "user_data/hyperopts/", and do 1000 iterations (-e 1000), and optimize all possible parameters (--spaces all), applied to data within specific period (--timerange 20191222-20200821)
- (.env) user1@user1-computer1:~/freqtrade$ freqtrade hyperopt --config config_001.json --hyperopt BBRSI_OPT_001 --strategy BBRSI_001 --hyperopt-loss SharpeHyperOptLossDaily -e 1000 --spaces all --timerange 20191222-20200821
- Hyperopt results saved to: /freqtrade/user_data/hyperopt_results/hyperopt_results-yyyy-mm-dd_hh-mm-ss.pickle
- FOR BEST RESULTS: Run this command 10x. Bcos 1000 iterations x 10 different starting random states produce the best results.
- Low virtual disk space? https://www.youtube.com/watch?v=Fteij2amMos
- Errors?
- "freqtrade - ERROR - No pair in whitelist." ==> API down. See Binance FB. Or https://www.binance.com/en/support/search?type=1&q=upgrade%20maintenance
- Important:
- dietpi@DietPi:~$ sudo reboot now (MUST manually do this after the exchange maintenance period is over, in order to get the bot to resume trading)
- Already downloaded data but still get error "WARNING - No history data for pair: "xxx/BTC", timeframe: 15m...." ? ==> check your timerange
- "ValueError: Expected object or value" ==> redownload the data
- Blackscreen?
- Likely due to heavy operation and limited memory
- VM1 menu: Machine > Session Information > "Performance Monitor" tab > "RAM Usage" could have hit max limit
- Wait for "RAM Usage" to drop below max limit and the blackscreen should be replaced by the usual lockscreen (then enter pwd to unlock)
- Consider increasing the machine's base memory limit . How? VirtualBox > select machine (VM1) > Settings > System > "Motherboard" tab > drag to increase "Base Memory")
- IMPORTANT: If blackscreen doesn't go away and u reboot VM and VM has vguest error ([example of error message here]), then: (1) don't create a new VM yet (this is a time-consuming last resort), but try this first: https://itsfoss.com/virtualbox-guest-additions-ubuntu/ (i.e. (1) insert guest addition cd image and run software (2) see the error "" (3) run "sudo apt install build-essential dkms linux-headers-generic" (4) run "sudo rcvboxadd setup" (5) click top right "on/off symbol" button > PowerOff/LogOut > Power Off > Restart (may have to do this step (5) twice until the VGuest Addition works again)
- List the best hyperopt iteration
- After hyperopting several times, u will generate many hyperopt result files
- Use this command to look into each file, and pick the best iteration (highest SR) from among all the files
- See a list of iterations (hyperopt-list), showing only the best i.e. SR keeps improving (--best), and profitable iterations (--profitable), in a specific hyperopt results file (--hyperopt-filename hyperopt_results_2020-12-20_21-35-02.pickle)
- freqtrade hyperopt-list --best --profitable --hyperopt-filename=hyperopt_results_2020-12-20_21-35-02.pickle
- Show a specific hyperopt iteration's details
- Show a hyperopt's details (hyperopt-show), of a specific hyperopt results file (--hyperopt-filename hyperopt_results_2020-12-20_21-35-02.pickle), for a particular iteration (--index 550)
- freqtrade hyperopt-show --hyperopt-filename=hyperopt_results_2021-01-21_00-02-39.pickle --index 550
- View visually
- Create plot (plot-dataframe), using specific config file (--config config_001.json), that shows specific pairs (-p AION/BTC ADA/BTC), for a custom strategy (--strategy BBRSI), using one set of indicators which use price units (--indicators1 bb_lowerband1 bb_lowerband2), using another set of indicators which don't use price units (--indicators2 rsi, fisher_rsi), for a specific period (--timerange=20201015-20201220), using data from a backtest result file (--export-filename user_data/backtest_results/backtest-result-2020-12-21_09-38-07.json),
- (.env) user1@computer1:~/freqtrade$ freqtrade plot-dataframe --config config_001.json -p AION/BTC ADA/BTC --strategy BBRSI_001 --indicators1 bb_lowerband1 bb_lowerband2 --indicators2 rsi fisher_rsi --timerange=20201015-20201220 --export-filename user_data/backtest_results/backtest-result-2020-12-21_09-38-07.json
- Note: each pair generates 1 file. The last 2 lines will show location of the file e.g. /home/user1/freqtrade/user_data/plot/freqtrade-plot-AION_BTC-15m.html
- See plot
- (.env) user1@computer1:~/freqtrade$ firefox /home/user1/freqtrade/user_data/plot/freqtrade-plot-AION_BTC-15m.html
- Navigation:
- To zoom, drag an intentional and complete box, defining the topleft & bottomright corners. Can be done on any area (main plot, volume plot, other plot). Doubleclick to zoom out.
- To view any indicator in the "indicators1" group: they will be on same chart as the pair's candlesticks.
- To view any indicator in the "indicators2" group: Since every indicator could have different scales e.g. 0 to 100, -1.0 to +1.0, etc... some may appear shrunken to the point of being hidden. To see the "shrunken/hidden" indicator, need to hide the visible indicators i.e. click on the visible indicator in the chart's legend. Example: To see fisher_rsi (range is -1.0 to +1.0), need to hide rsi (range is 0 to 100), so, click on "rsi" in the legend to hide "rsi", and then "fisher_rsi" will expand to be more visible.
- Note: If the pair exits due to minimal ROI (and not due to a sell signal), the plot won't show a sell signal.
- Setup Telegram bot
- Sandbox mode
- Setup
- config.json: "dry_run": true,
- Run bot, with specific strategy (-s BBRSI), using specific config file (-c config.json), and read/write into specific database (--db-url sqlite:///user_data/trades004.dryrun.sqlite)
- (.env) user1@computer1:~/freqtrade$ freqtrade trade -s BBRSI -c config.json --db-url sqlite:///user_data/trades001.dryrun.sqlite
- Not getting msgs from telegram? Check if telegram is enabled in config.
- Live mode
- Setup:
- config.json: "dry_run": false,
- Run bot, with specific strategy (-s BBRSI), using specific config file (-c config.json), and read/write into specific database (--db-url sqlite:///user_data/trades004.live.sqlite)
- (.env) user1@computer1:~/freqtrade$ freqtrade trade -s BBRSI -c config.json --db-url sqlite:///user_data/trades004.live.sqlite
- No trades happening?
- Strategy may currently be in a quiet period: If backtest said to expect 40 trades/day, then allow maybe 3 days to see a trade appear. Distribution of trades is not even across all days.
- Strategy mistakenly use future data (backtest won't detect this error). Ensure strategy doesn't use anything that references future data, example:
- Strategy mistakenly used absolute position in dataframe. Example:
- Funds must be in binance.com's SPOT wallet (not other wallets e.g. P2P wallet)
- API keys must be correct or updated
- How to decide which pairs to trade?
- Specific pairs: config.json:
- "pairlists": [
- {"method": "StaticPairList"}
- ],
- "pair_whitelist": [
- "ADA/BTC",
- "ZIL/BTC"
- ],
- "pair_blacklist": [
- "DOGE/BTC",
- "BNB/BTC"
- ]
- Top pairs by volume: config.json:
- Trade pairs which are filtered by volume (method: VolumePairList), sorted by volume (quoteVolume), take the top 20 pairs ("number_assets": 20,), and refresh the list of pairs every 1800 seconds.
- "pairlists": [{
- "method": "VolumePairList",
- "number_assets": 20,
- "sort_key": "quoteVolume",
- "refresh_period": 1800
- }],
- How to close all positions gracefully on Telegram
- /stopbuy (stops buying, continues trying to sell existing open positions)
- Note: use "/reload_config" (to undo the "/stopbuy" command)
- /status table (see remaining open positions)
- /stop (stops freqtrade. Do this once all open positions are closed)
- Note: use "/start" (to undo the "/stop" command)
CONTINUE AT 6:28 @ CHAPTER 53 (Edge Positioning)
