Python Backtesting library for trading strategies

Prevent YahooFinanceData from throwing No such file or directory: since Yahoo's response type has changed

For context: https://community.backtrader.com/topic/2363/errno-2-no-such-file-or-directory/7

Alpaca is a new stock broker in US with Web API and WebSocket streaming with free commission.

It would be a great value to add in backtrader to support the API.

Here are some links for the information.

https://alpaca.markets/ https://docs.alpaca.markets https://github.com/alpacahq/alpaca-trade-api-python

Fixed .ix is deprecated.

Message is below

/site-packages/backtrader/feeds/pandafeed.py:221: DeprecationWarning: 
.ix is deprecated. Please use
.loc for label based indexing or
.iloc for positional indexing

See the documentation here:
http://pandas.pydata.org/pandas-docs/stable/indexing.html#deprecate_ix
  line[0] = self.p.dataname.ix[self._idx, colindex]
/site-packages/pandas/core/indexing.py:992: DeprecationWarning: 
.ix is deprecated. Please use
.loc for label based indexing or
.iloc for positional indexing

See the documentation here:
http://pandas.pydata.org/pandas-docs/stable/indexing.html#deprecate_ix
  return getattr(section, self.name)[new_key]

I have always seen the Sharpe ratio given in an annualized form rather than a raw form. If I am using daily returns and my risk free return is annual, I want to know how much in excess of the risk free return I am generating. Hence, multiplying the average excess return by 365 and the standard deviation by sqrt(365) gets the above calculation.

Issue: Backtrader by default uses TKinter (TkAgg) backend for plotting. But, MacOS does not like TKinter. So when cerebro.plot() is used, it causes a bad crash causing the system to force logout.

Fix: Added platform dependent code. Now on MacOS, it uses MacOS (MacOSX) backend.

Side Issue(s):

1. I tried to run the code with cerebro.plot(use='MacOSX'), still the system was crashing. Looks like the "use" param in plot() function of backtrader / cerebro is not implemented (correct me if wrong). If that issue can be fixed, it would be much better.
2. Also jupyter lab gives the below error :
   • Tried installing https://github.com/matplotlib/jupyter-matplotlib but no use.

References:

1. https://community.backtrader.com/topic/1597/cerebro-plot-lead-to-system-crash
2. https://community.backtrader.com/topic/393/python-crashes-when-i-try-the-sample-code
3. https://stackoverflow.com/questions/57943180/matplotlib-crashes-in-plt-show-on-macos-with-tkagg-backend

WriterFile did not work under optimization, since sys.stdout can't be serialized. I made None default to stdout. Personally, I don't feel like this is correct, since None should result in the write not writing anywhere, but that's not really useful. Open to suggestions, perhaps there's a Python trick I'm not aware of.

Also, I have not run the tests, since I'm not familiar with how to do this.

@backtrader @verybadsoldier First of all, thanks a lot for your amazing work on these projects!

IMHO, it makes for the best open source stack in this field based on my admittedly incomplete research, but still, thank you very much for making your work available to everyone.

Have you considered combining the two? I was getting up and running on IB recently and found myself making changes to both projects in order to make my backtesting/live trading experience better.

@backtrader as far as I can see this drop in should be fully backwards compatible with 1 minor change, namely specifying backend when using plot function as I've made the bokeh backend default (it's much better for rapid iteration/prototyping and doesn't have any issues when using virtualenv, which some people might be doing if they are using pipenv, unlike matplotlib), e.g. to use matplotlib in plot:

cerebro.plot(backend='matplotlib')

@verybadsoldier I've kept all of your improvements with optimization server as-is, but made them available via:

cerebro.visualize(style='bar')

as per your plot_result function it will start up an optimization server if OptResults are passed in and just a plain file otherwise.

cerebro.plot() with bokeh backend will also handle OptResults now (open new file for each OptResult).

The idea is that cerebro.plot() is a fully static interface and guaranteed to return where as cerebro.visualize is a dynamic one (starts a blocking server if necessary).

both functions can take a results argument that allows end-user to filter/pre-process Cerebro results before plotting them, e.g.

results = cerebro.run() and cerebro.plot(results) or cerebro.visualize(results) if results are not passed in then it defaults to using strategies from cerebro as is the case currently.

I think I removed all Python 3 specific code so it should be fully compatible with @backtrader supported python list (as far as I can tell all the dependencies are python2 compatible). If you find something that's python 3 specific, I'll be happy to fix it up.

There's a few other minor improvements here and there:

• Added ability to plot or not plot analyzers (and customize their titles) when using bokeh backend
• Extended the writer interface a bit so it's possible to do stuff with it when csv=False and also extend WriterBase to do other things.
• Fixed influxdb feed loading so it fetches the right time series (grouped correctly) from the source database.
• Added IBdownloader tool so it's possible to fetch data directly from a running IB gateway (useful for minute/tick data), plenty of sources for free daily data of course.

Finally, is there any reason to have both tools and scripts in setup.py? It makes auto-completing in bash a bit of pain since there are now 2 things that basically do the same thing in the command line prompt. I've kept only the tools version so there's only 1 btrun btrewrite etc now (primarily to keep the command-line auto-completion clutter free).

Let me know if it's a good idea to share this and if both of you think it's a good idea to combine the 2 projects. I kind of barged in without asking but I felt that I needed to make these changes to make my own experience better and now that I have maybe other people will benefit too.

IB doesn't send any data when RTVolume generic tick data is requested for indexes as indexes don't trade. They also don't have bid/ask data. As a result current backtrader code doesn't work with indexes.

Fixed this by using approach similar to CASH securities: created RTVolume objects using 'last' prices reported by IB through tickPrice callback.

When running big optimizations, child processes consume memory until they crash with broken pipe errors. Limiting the number of tasks assigned to each child process forces them to exit and release memory.

See also https://docs.python.org/3/library/multiprocessing.html?#multiprocessing.pool.Pool https://stackoverflow.com/a/21613370/3706242

• OLS_Slope_InterceptN Removing self.p.prepend_constant in order to avoid invalid result parameters order. With prepend = True intercept is the first parameter and slope the second.

• CointN Updating parameter name to trend

durning backtrader plot operation I have encounter the following error:

  File "/home/jack/backtrader/backtrader/cerebro.py", line 991, in plot
    start=start, end=end, use=use)
  File "/home/jack/backtrader/backtrader/plot/plot.py", line 189, in plot
    self.plotind(None, ptop, subinds=self.dplotsover[ptop])
  File "/home/jack/backtrader/backtrader/plot/plot.py", line 457, in plotind
    plottedline = pltmethod(xdata, lplotarray, **plotkwargs)
  File "/usr/local/lib/python3.5/dist-packages/matplotlib/__init__.py", line 1898, in inner
    return func(ax, *args, **kwargs)
  File "/usr/local/lib/python3.5/dist-packages/matplotlib/axes/_axes.py", line 1406, in plot
    for line in self._get_lines(*args, **kwargs):
  File "/usr/local/lib/python3.5/dist-packages/matplotlib/axes/_base.py", line 407, in _grab_next_args
    for seg in self._plot_args(remaining, kwargs):
  File "/usr/local/lib/python3.5/dist-packages/matplotlib/axes/_base.py", line 385, in _plot_args
    x, y = self._xy_from_xy(x, y)
  File "/usr/local/lib/python3.5/dist-packages/matplotlib/axes/_base.py", line 244, in _xy_from_xy
    "have shapes {} and {}".format(x.shape, y.shape))
ValueError: x and y must have same first dimension, but have shapes (8621,) and (8620,)

The proposed solution make sure/force the dimension of both axis are of same length

The Iterable class was moved from collections to collections.abc since Python 3.10. With https://github.com/mementum/backtrader/pull/466, https://github.com/mementum/backtrader/pull/465 pull requests.

,This fixes the backtest broker displaying NaN for the portfolio value returned by get_value/getvalue, even if there are no positions in the data feeds where the data.close[0] value is NaN.

Practical Example: One wants to backtest multiple data feeds of different assets over various times. Backtrader is prefilled with all assets, but cannot invest in stocks that haven't IPOed yet, for example. Nevertheless, the data feed is already loaded. This problem was also discussed here

Fix:

• add python standard math dependency for isnan()
• check for NaN when computing the portfolio value.
• continue the value calculation loop if the closing price of the asset is NaN, no matter the position. If there is one, its worthless anyways.

Potential loopholes of fix:

• None is not yet checked for
• Infinity is not yet checked for.

Datafeed sourced from IB API receives timezone info from contractdetails. This tz info was not passed to num2date() method in lines 275 and 276. This results in an offset in the point value returned by self._gettmpoint(tm). This offset distorts the resampling and produces bars at irregular intervals. Above modifications takes care of the same.

Note: Modified code was tested on Minute 1 timeframe resampled to 20, 25 & 75 minutes using ibdata and data form local csv file.

This can happen when joining data from file and live data (backfilling). In this case, if the last bar of the file data is of the same time as the bar in the live data - both of them will be added.

Detailed example: File data contains the daily bars of: [21/11/17, 21/11/18]. We're sending request for historical data between the dates: 21/11/17 - 21/11/23, and using backfilling from the file. When loading the data - the 11/17 and 11/18 will be taken from the file, and since the data point of the 11/18 from the server does not fulfill the condition dt < self.lines.datetime[-1] it will be taken as well. End result: data contains 2 data points of the 11/18.