Backtesting Bug / Data mismatch

[Gwyneveree]

Hi there, very interesting project, but I ran into an issue. After running backtests, I examined the detailed trading logs, manually, each trade one by one, how open_rate and close_rate correlate with open_date and close_date (which they obviously should, using the charts), and there are basically three categories that they fall into:

1. both open and close rate exactly align with entry and exit candles' open prices
2. they don't align with specific open or close prices of the candles but fit into the gap between open and close price
3. neither open nor close rate does not align with the actual price on the chart at all, it's outside the open and close of the candle (or nearby candles), but the strategy registers it as a legitimate trade (usually profitable, but in reality this trade would be losing, especially with leverage)

What can be the issue? I didn't use the repo exactly as it is, just the strategy with a simplified config, but I also adapted the repo config for backtesting just in case, same issue.
Tried using it with one asset, multiple assets, use_order_book true/false, but no matter what there is always at least one trade that falls outside the scope of actual prices, with high leverage it would be fatal.
Thanks!

[iterativv]

Hi. You mean for backtests ? NFI is using candle open/close. Everything is done via custom_exit() is not using ROI, trailing and so on. For tests in fact this is handicap, because in live we can exit a trade mid candle, while in tests can move the opposite direction and miss that exit.

Could you bring a specific example ?

[Gwyneveree]

Yes, I was talking about backtesting.
I meant this. Most of the time, if comparing the chart's price with the logs, there is a match, the price either exactly matches the open price or fits in between the open and close, but occasionally (like in the example) there is a mismatch between the logged price (that the strategy used to trade) and the actual price at that moment.
The downloaded historical data is correct, though.
If there is a misunderstanding, please, explain how it works in this case.

[iterativv]

Note, plot the backtest (backtest one, then exactly the same command but plot-dataframe instead of backtesting). The charts is directly from the downloaded data. The date is in UTC, so if your are not in that time zone, you have to offset it.

[iterativv]

With that I mean on the TV (or exchange) chart. Set the daily dataframe, close candle for this is at 12:00 ... if you see different time, you have to offset the from backtest too. Plot it and will be clear.

[Gwyneveree]

That's the thing, both the backtest and the chart are in UTC, that's the issue, all other trades align, but there's always at least one trade like the one in the screenshots that doesn't match. Any suggestions? Again, timeframes are aligned, otherwise all the trades would be wrong.

[iterativv]

Your timezone is in UTC ? The machine you backtest it. Just plot it to see what is going on. Backtest first the pair, then plot & display on browser.

[Gwyneveree]

The blue circle (entry) should be in the same place as the green triangle (like all other trades on the plot, like a triangle inside a circle), but in this trade it's way outside the chart, and the strategy uses the circle (price 0.7855, which doesn't exist) as the place for entry. Both Freqtrade's and chart's timezone is in UTC.

[erickmarx]

To reinforce the issue, I checked on TradingView and there really is a difference between the backtesting value and the real value in UTC time.

It appears to be a legitimate error.

[iterativv]

The blue circle (entry) should be in the same place as the green triangle (like all other trades on the plot, like a triangle inside a circle), but in this trade it's way outside the chart, and the strategy uses the circle (price 0.7855, which doesn't exist) as the place for entry. Both Freqtrade's and chart's timezone is in UTC.

Ah wait, you mean this. Yes, this is the final "open" rate as FT calculated it. After all entries and exits. It's not the original entry.

NFI trades can have partial entries and exit. Now FT for the open is updating it after an entry, but not after a partial exit.

Let's say we buy 1 coin at 100. Then we buy another coin at 90. Same trade. Now the open rate is about 95, so the blue circle is going to appear at 95.

[iterativv]

To reinforce the issue, I checked on TradingView and there really is a difference between the backtesting value and the real value in UTC time.

See above. You have to check the orders. Not that "open rate". It's like for live open trades with multiple entries, the open rate is not the same as the open rate of the first entry then.

Posted above an image of the trade json export. There is "open_rate" for the whole trade. Then there are the "orders", with "safe_price" the entry for each order. On the plot you only see the "open_rate" and the "close_rate" (that this is the same as the safe_price of the last entry - it can be forced if the backtest ends with the trade open). On the plot you don't see the entries of the rebuys/grinds and the exits of those. when there are such.

[Gwyneveree]

Could you please explain in more detail?
From my understanding, the bot counted the blue circle price as a legitimate entry that it bought and sold, just not with actual, real prices. I understand the possibility of partial entries and exits, but how could it count a 0.7855 trade when it's nowhere near the actual price/candles?
When looking at all the other trades in that backtest, it's always a triangle inside a circle (makes sense), but when running other combinations of coins, occasionally the same thing happens, and it can happen at the beginning, or the middle, not necessarily at the end of the run.
In your example, you say "Let's say we buy 1 coin at 100. Then we buy another coin at 90. Same trade." How can it be the same trade? In my example, Freqtrade counted that 0.7855 trade as a separate trade entirely.

[iterativv]

Check the json you posted above as image. There is the "open_rate", that is the blue circle, and it's on the candle AFTER the triangle, not same candle.

Now you have "orders" there. The first one with "safe_price" 0.8104 ... this is the first entry, just after the triangle. But the plot and "open_order" it doesn't show this.

Then we average down (DCA call it how you like), "safe_price" is 0.7849 ... now the "open_rate" is not 0.8104 anymore, it's the average of the both entries (depends on the amounts also) and there is the blue circle, stayed on the original candle, but moved to reflect the new "open_rate". That's how FT is handling this.

Finally there is a final order, "sell", with "safe_price" 0.8015 this is the final exit also.

A trade can have multiple orders, exits and entries. Those don't show on the plot (you can either request such a feature or contribute to FT plot, maybe, if you want it).

[iterativv]

Also, don't include "roi", that is invalid for NFI. Because the multiple entries and exits - for many trades - the "open_rate" is not valid, for the whole trade. NFI is using custom calculations, from all orders, to calculate current profits, including all the fees.

[Gwyneveree]

Alright, thanks for the clarification, appreciate it.
A few small questions:
When you say "don't include roi", do you mean don't include in the config file, or do you mean just ignore it in the log?
Which leverage do you recommend for the strategy?
Right now my config is set up to spot, how do I properly set it up for trading with leverage? Like, what are the necessary parameters to include and things like that?
How much power does the strategy need (like processor/RAM/etc.)? Does the amount of pairs or method (StaticPairList/Volume/etc.) affect the performance?
Thanks again.

[iterativv]

Yes, remove ROI settings from your config. It's not going to be accurate. If there is no partial exit for the trade, then yes. But NFI can have partial exits.

Leverage leave the defaults, it's 3x

See configs dir here. There is exampleconfig, also recommended and for spot/futures trading examples. Also blacklists and volume pairlists.

A pi 5 is going to work well (pi 4 is going to struggle some). A 10-15 years old CPU can do it. If VPS, it depends if they are throttling it too.