Forecasting directional movements of stock prices for intraday trading using LSTM and random forest

Overview

Forecasting directional movements of stock-prices for intraday trading using LSTM and random-forest

https://arxiv.org/abs/2004.10178
Pushpendu Ghosh, Ariel Neufeld, Jajati K Sahoo

We employ both random forests on the one hand and LSTM networks (more precisely CuDNNLSTM) on the other hand as training methodology to analyze their effectiveness in forecasting out-of-sample directional movements of constituent stocks of the S&P 500, for intraday trading, from January 1993 till December 2018.

Requirements

pip install scikit-learn==0.20.4
pip install tensorflow==1.14.0

Plots

We plot three important metrics to quantify the effectiveness of our model: Intraday-240,3-LSTM.py and Intraday-240,3-RF.py, in the period January 1993 till December 2018.
Intraday LSTM: Intraday-240,3-LSTM.py
Intraday RF: Intraday-240,3-RF.py
Next Day LSTM, krauss18: NextDay-240,1-LSTM.py [1]
Next Day RF, krauss17: NextDay-240,1-RF.py [2]

Cumulative Money growth (after transaction cost)

Average daily returns (after transaction cost)

Average (Annualized) Sharpe ratio (after transaction cost)

Appendix

Feature Importance

This figure analyzes the Sharpe Ratio achieved when single features are used instead of our 3-features for the same intraday trading strategy. It hence analyzes which single feature is important.
The result suggest that or (returns from close price to next day open price) has the highest importance. This is justifiable by the fact that it is the only feature which considers the latest available data (the trading day's open price) at the time of making the trading decision. We also see that our 3-features setting achieves the highest Sharpe Ratio and hence outperforms each single feature.

LSTM hyperparameter tuning

We see that the amount of 25 cells for our chosen LSTM architecture to be at least as good as other amounts between 5 and 100. We have chosen 25 cells also for the ease of comparison with Fischer & Krauss (2018)

Benchmark against other LSTM architectures

We consider various new LSTM architectures. Note that the other LSTM architectures involve much more parameters than the one we chose for our empirical study and do not achieve better results in terms of Sharpe Ratio. Moreover, we also compare our LSTM architecture with GRU, which is a relatively simpler variation of LSTM, and see that it generates reasonably good but still slightly lower Sharpe ratio than our chosen LSTM architecture.

References to the LSTM models:

  1. Single Layer GRU: https://arxiv.org/abs/1412.3555
  2. Stacked LSTM: https://www.sciencedirect.com/science/article/pii/S1877050920304865
  3. Stacked Residual LSTM: https://arxiv.org/abs/1610.03098

Acknowledgements

The first author gratefully acknowledges the NTU-India Connect Research Internship Programme which allowed him to carry out part of this research project while visiting the Nanyang Technological University, Singapore.
The second author gratefully acknowledges financial support by his Nanyang Assistant Professorship Grant (NAP Grant) Machine Learning based Algorithms in Finance and Insurance.

References

[1] Fischer, Thomas, and Christopher Krauss. "Deep learning with long short-term memory networks for financial market predictions." European Journal of Operational Research 270.2 (2018): 654-669.
[2] Krauss, Christopher, Xuan Anh Do, and Nicolas Huck. "Deep neural networks, gradient-boosted trees, random forests: Statistical arbitrage on the S&P 500." European Journal of Operational Research 259.2 (2017): 689-702.

You might also like...
Stock-history-display - something like a easy yearly review for your stock performance
Stock-history-display - something like a easy yearly review for your stock performance

Stock History Display Available on Heroku: https://stock-history-display.herokua

Multivariate Time Series Forecasting with efficient Transformers. Code for the paper
Multivariate Time Series Forecasting with efficient Transformers. Code for the paper "Long-Range Transformers for Dynamic Spatiotemporal Forecasting."

Spacetimeformer Multivariate Forecasting This repository contains the code for the paper, "Long-Range Transformers for Dynamic Spatiotemporal Forecast

Event-forecasting - Event Forecasting Algorithms With Python

event-forecasting Event Forecasting Algorithms Theory Correlating events in comp

Attentional Focus Modulates Automatic Finger‑tapping Movements

"Attentional Focus Modulates Automatic Finger‑tapping Movements", in Scientific Reports

This project uses reinforcement learning on stock market and agent tries to learn trading. The goal is to check if the agent can learn to read tape. The project is dedicated to hero in life great Jesse Livermore.

Reinforcement-trading This project uses Reinforcement learning on stock market and agent tries to learn trading. The goal is to check if the agent can

A Deep Reinforcement Learning Framework for Stock Market Trading

DQN-Trading This is a framework based on deep reinforcement learning for stock market trading. This project is the implementation code for the two pap

Semi-supervised Semantic Segmentation with Directional Context-aware Consistency (CVPR 2021)
Semi-supervised Semantic Segmentation with Directional Context-aware Consistency (CVPR 2021)

Semi-supervised Semantic Segmentation with Directional Context-aware Consistency (CAC) Xin Lai*, Zhuotao Tian*, Li Jiang, Shu Liu, Hengshuang Zhao, Li

Semi-supervised Semantic Segmentation with Directional Context-aware Consistency (CVPR 2021)
Semi-supervised Semantic Segmentation with Directional Context-aware Consistency (CVPR 2021)

Semi-supervised Semantic Segmentation with Directional Context-aware Consistency (CAC) Xin Lai*, Zhuotao Tian*, Li Jiang, Shu Liu, Hengshuang Zhao, Li

This is a simple backtesting framework to help you test your crypto currency trading. It includes a way to download and store historical crypto data and to execute a trading strategy.

You can use this simple crypto backtesting script to ensure your trading strategy is successful Minimal setup required and works well with static TP a

Comments
  • KeyError: 'S783' when run  NextDay-240,1-LSTM.py

    KeyError: 'S783' when run NextDay-240,1-LSTM.py

    (tensorflow) (base) elife@ubuntu:~/work/Trading/Stock-market-forecasting$ python NextDay-240,1-LSTM.py /usr/lib/python2.7/dist-packages/h5py/init.py:36: FutureWarning: Conversion of the second argument of issubdtype from float to np.floating is deprecated. In future, it will be treated as np.float64 == np.dtype(float).type. from ._conv import register_converters as _register_converters WARNING:tensorflow:From NextDay-240,1-LSTM.py:10: The name tf.keras.layers.CuDNNLSTM is deprecated. Please use tf.compat.v1.keras.layers.CuDNNLSTM instead.

    WARNING:tensorflow:From NextDay-240,1-LSTM.py:22: The name tf.set_random_seed is deprecated. Please use tf.compat.v1.set_random_seed instead.


    1993

    Traceback (most recent call last): File "NextDay-240,1-LSTM.py", line 181, in st_train_data,st_test_data = create_stock_data(df,st) File "NextDay-240,1-LSTM.py", line 137, in create_stock_data daily_change = df[st].pct_change() File "/home/elife/tensorflow/local/lib/python2.7/site-packages/pandas/core/frame.py", line 2688, in getitem return self._getitem_column(key) File "/home/elife/tensorflow/local/lib/python2.7/site-packages/pandas/core/frame.py", line 2695, in _getitem_column return self._get_item_cache(key) File "/home/elife/tensorflow/local/lib/python2.7/site-packages/pandas/core/generic.py", line 2489, in _get_item_cache values = self._data.get(item) File "/home/elife/tensorflow/local/lib/python2.7/site-packages/pandas/core/internals.py", line 4115, in get loc = self.items.get_loc(item) File "/home/elife/tensorflow/local/lib/python2.7/site-packages/pandas/core/indexes/base.py", line 3080, in get_loc return self._engine.get_loc(self._maybe_cast_indexer(key)) File "pandas/_libs/index.pyx", line 140, in pandas._libs.index.IndexEngine.get_loc File "pandas/_libs/index.pyx", line 162, in pandas._libs.index.IndexEngine.get_loc File "pandas/_libs/hashtable_class_helper.pxi", line 1492, in pandas._libs.hashtable.PyObjectHashTable.get_item File "pandas/_libs/hashtable_class_helper.pxi", line 1500, in pandas._libs.hashtable.PyObjectHashTable.get_item KeyError: 'S783' (tensorflow) (base) elife@ubuntu:~/work/Trading/Stock-market-forecasting$

    opened by elife33 6
  •  ValueError: Failed to convert a NumPy array to a Tensor (Unsupported object type float).

    ValueError: Failed to convert a NumPy array to a Tensor (Unsupported object type float).

    Hi, when I tried to porting this project into python notebook for Google colab, I subjected to this issue below:

    1993

    (251416, 244) (121079, 244) 97.39196372032166 Model: "model_2"


    Layer (type) Output Shape Param #

    input_3 (InputLayer) [(None, 240, 1)] 0


    lstm_2 (LSTM) (None, 25) 2700


    dropout (Dropout) (None, 25) 0


    dense_2 (Dense) (None, 2) 52

    Total params: 2,752 Trainable params: 2,752 Non-trainable params: 0


    WARNING:tensorflow:period argument is deprecated. Please use save_freq to specify the frequency in number of batches seen.

    ValueError Traceback (most recent call last) in () 253 print(train_data.shape,test_data.shape,time.time()-start) 254 --> 255 model,predictions = trainer(train_data,test_data) 256 returns = simulate(test_data,predictions) 257 returns.to_csv(result_folder+'/avg_daily_rets-'+str(test_year)+'.csv')

    12 frames in trainer(train_data, test_data, model_type) 148 validation_split=0.2, 149 callbacks=callbacks, --> 150 batch_size=512 151 ) 152

    /usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/training.py in _method_wrapper(self, *args, **kwargs) 64 def _method_wrapper(self, *args, **kwargs): 65 if not self._in_multi_worker_mode(): # pylint: disable=protected-access ---> 66 return method(self, *args, **kwargs) 67 68 # Running inside run_distribute_coordinator already.

    /usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/training.py in fit(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, validation_batch_size, validation_freq, max_queue_size, workers, use_multiprocessing) 795 data_adapter.train_validation_split((x, y, sample_weight), 796 validation_split=validation_split, --> 797 shuffle=False)) 798 799 with self.distribute_strategy.scope(), \

    /usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/data_adapter.py in train_validation_split(arrays, validation_split, shuffle) 1336 1337 train_arrays = nest.map_structure( -> 1338 functools.partial(_split, indices=train_indices), arrays) 1339 val_arrays = nest.map_structure( 1340 functools.partial(_split, indices=val_indices), arrays)

    /usr/local/lib/python3.6/dist-packages/tensorflow/python/util/nest.py in map_structure(func, *structure, **kwargs) 615 616 return pack_sequence_as( --> 617 structure[0], [func(*x) for x in entries], 618 expand_composites=expand_composites) 619

    /usr/local/lib/python3.6/dist-packages/tensorflow/python/util/nest.py in (.0) 615 616 return pack_sequence_as( --> 617 structure[0], [func(*x) for x in entries], 618 expand_composites=expand_composites) 619

    /usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/data_adapter.py in _split(t, indices) 1332 if t is None: 1333 return t -> 1334 t = ops.convert_to_tensor_v2(t) 1335 return array_ops.gather_v2(t, indices) 1336

    /usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/ops.py in convert_to_tensor_v2(value, dtype, dtype_hint, name) 1281 name=name, 1282 preferred_dtype=dtype_hint, -> 1283 as_ref=False) 1284 1285

    /usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/ops.py in convert_to_tensor(value, dtype, name, as_ref, preferred_dtype, dtype_hint, ctx, accepted_result_types) 1339 1340 if ret is None: -> 1341 ret = conversion_func(value, dtype=dtype, name=name, as_ref=as_ref) 1342 1343 if ret is NotImplemented:

    /usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/tensor_conversion_registry.py in _default_conversion_function(failed resolving arguments) 50 def _default_conversion_function(value, dtype, name, as_ref): 51 del as_ref # Unused. ---> 52 return constant_op.constant(value, dtype, name=name) 53 54

    /usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/constant_op.py in constant(value, dtype, shape, name) 260 """ 261 return _constant_impl(value, dtype, shape, name, verify_shape=False, --> 262 allow_broadcast=True) 263 264

    /usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/constant_op.py in _constant_impl(value, dtype, shape, name, verify_shape, allow_broadcast) 268 ctx = context.context() 269 if ctx.executing_eagerly(): --> 270 t = convert_to_eager_tensor(value, ctx, dtype) 271 if shape is None: 272 return t

    /usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/constant_op.py in convert_to_eager_tensor(value, ctx, dtype) 94 dtype = dtypes.as_dtype(dtype).as_datatype_enum 95 ctx.ensure_initialized() ---> 96 return ops.EagerTensor(value, ctx.device_name, dtype) 97 98

    ValueError: Failed to convert a NumPy array to a Tensor (Unsupported object type float).

    opened by Zhang-Jack 5
  • tensorflow.python.framework.errors_impl.InvalidArgumentError: No OpKernel was registered to support Op 'CudnnRNN'

    tensorflow.python.framework.errors_impl.InvalidArgumentError: No OpKernel was registered to support Op 'CudnnRNN'

    My environment Windows 10 conda environment with scikit-learn==0.20.4 & tensorflow==1.14.0

    python -Wignore "Intraday-240,1-LSTM.py"

    WARNING:tensorflow:From Intraday-240,1-LSTM.py:11: The name tf.keras.layers.CuDNNLSTM is deprecated. Please use tf.compat.v1.keras.layers.CuDNNLSTM instead.


    1993

    (251416, 244) (121079, 244) 109.14551615715027 WARNING:tensorflow:From C:\ProgramData\Anaconda3\envs\myenv1\lib\site-packages\tensorflow\python\ops\init_ops.py:1251: calling VarianceScaling.init (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version. Instructions for updating: Call initializer instance with the dtype argument instead of passing it to the constructor Model: "model"


    Layer (type) Output Shape Param #

    input_1 (InputLayer) [(None, 240, 1)] 0


    cu_dnnlstm (CuDNNLSTM) (None, 25) 2800


    dropout (Dropout) (None, 25) 0


    dense (Dense) (None, 2) 52

    Total params: 2,852 Trainable params: 2,852 Non-trainable params: 0


    WARNING:tensorflow:period argument is deprecated. Please use save_freq to specify the frequency in number of samples seen. Train on 201132 samples, validate on 50284 samples 2021-09-02 12:44:05.181180: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 Traceback (most recent call last): File "C:\ProgramData\Anaconda3\envs\myenv1\lib\site-packages\tensorflow\python\client\session.py", line 1356, in _do_call return fn(*args) File "C:\ProgramData\Anaconda3\envs\myenv1\lib\site-packages\tensorflow\python\client\session.py", line 1339, in _run_fn self._extend_graph() File "C:\ProgramData\Anaconda3\envs\myenv1\lib\site-packages\tensorflow\python\client\session.py", line 1374, in _extend_graph tf_session.ExtendSession(self._session) tensorflow.python.framework.errors_impl.InvalidArgumentError: No OpKernel was registered to support Op 'CudnnRNN' used by {{node cu_dnnlstm/CudnnRNN}}with these attrs: [seed=0, dropout=0, T=DT_FLOAT, input_mode="linear_input", direction="unidirectional", rnn_mode="lstm", seed2=0, is_training=true] Registered devices: [CPU] Registered kernels:

         [[cu_dnnlstm/CudnnRNN]]
    

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last): File "Intraday-240,1-LSTM.py", line 194, in model,predictions = trainer(train_data,test_data) File "Intraday-240,1-LSTM.py", line 89, in trainer batch_size=512 File "C:\ProgramData\Anaconda3\envs\myenv1\lib\site-packages\tensorflow\python\keras\engine\training.py", line 780, in fit steps_name='steps_per_epoch') File "C:\ProgramData\Anaconda3\envs\myenv1\lib\site-packages\tensorflow\python\keras\engine\training_arrays.py", line 250, in model_iteration model.reset_metrics() File "C:\ProgramData\Anaconda3\envs\myenv1\lib\site-packages\tensorflow\python\keras\engine\training.py", line 1084, in reset_metrics m.reset_states() File "C:\ProgramData\Anaconda3\envs\myenv1\lib\site-packages\tensorflow\python\keras\metrics.py", line 199, in reset_states K.batch_set_value([(v, 0) for v in self.variables]) File "C:\ProgramData\Anaconda3\envs\myenv1\lib\site-packages\tensorflow\python\keras\backend.py", line 3071, in batch_set_value get_session().run(assign_ops, feed_dict=feed_dict) File "C:\ProgramData\Anaconda3\envs\myenv1\lib\site-packages\tensorflow\python\keras\backend.py", line 462, in get_session _initialize_variables(session) File "C:\ProgramData\Anaconda3\envs\myenv1\lib\site-packages\tensorflow\python\keras\backend.py", line 879, in _initialize_variables [variables_module.is_variable_initialized(v) for v in candidate_vars]) File "C:\ProgramData\Anaconda3\envs\myenv1\lib\site-packages\tensorflow\python\client\session.py", line 950, in run run_metadata_ptr) File "C:\ProgramData\Anaconda3\envs\myenv1\lib\site-packages\tensorflow\python\client\session.py", line 1173, in _run feed_dict_tensor, options, run_metadata) File "C:\ProgramData\Anaconda3\envs\myenv1\lib\site-packages\tensorflow\python\client\session.py", line 1350, in _do_run run_metadata) File "C:\ProgramData\Anaconda3\envs\myenv1\lib\site-packages\tensorflow\python\client\session.py", line 1370, in _do_call raise type(e)(node_def, op, message) tensorflow.python.framework.errors_impl.InvalidArgumentError: No OpKernel was registered to support Op 'CudnnRNN' used by node cu_dnnlstm/CudnnRNN (defined at Intraday-240,1-LSTM.py:44) with these attrs: [seed=0, dropout=0, T=DT_FLOAT, input_mode="linear_input", direction="unidirectional", rnn_mode="lstm", seed2=0, is_training=true] Registered devices: [CPU] Registered kernels:

         [[cu_dnnlstm/CudnnRNN]]
    
    opened by mh-github 2
  • Hi Newbie With Tech Background

    Hi Newbie With Tech Background

    This looks so amazing. I have gotten it to run with the test data but something seems to be missing that I can't figure out. If it's not a huge pain, Is it possible to walk me through how to use this on the SPY ETF?

    Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 22069 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 3090, pci bus id: 0000:01:00.0, compute capability: 8.6) Epoch 1/1000 2021-09-12 13:34:29.486149: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cublas64_100.dll 2021-09-12 13:35:53.619664: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudnn64_7.dll 2021-09-12 13:52:40.007267: W tensorflow/core/framework/op_kernel.cc:1651] OP_REQUIRES failed at cudnn_rnn_ops.cc:1500 : Internal: No algorithm worked! Traceback (most recent call last): File "NextDay-240,1-LSTM.py", line 197, in predictions = trainer(train_data,test_data,model_type) File "NextDay-240,1-LSTM.py", line 94, in trainer batch_size=512 File "C:\Users\George\anaconda3\envs\predict\lib\site-packages\tensorflow_core\python\keras\engine\training.py", line 727, in fit use_multiprocessing=use_multiprocessing) File "C:\Users\George\anaconda3\envs\predict\lib\site-packages\tensorflow_core\python\keras\engine\training_arrays.py", line 675, in fit steps_name='steps_per_epoch') File "C:\Users\George\anaconda3\envs\predict\lib\site-packages\tensorflow_core\python\keras\engine\training_arrays.py", line 394, in model_iteration batch_outs = f(ins_batch) File "C:\Users\George\anaconda3\envs\predict\lib\site-packages\tensorflow_core\python\keras\backend.py", line 3476, in call run_metadata=self.run_metadata) File "C:\Users\George\anaconda3\envs\predict\lib\site-packages\tensorflow_core\python\client\session.py", line 1472, in call run_metadata_ptr) tensorflow.python.framework.errors_impl.InternalError: 2 root error(s) found. (0) Internal: No algorithm worked! [[{{node cu_dnnlstm/CudnnRNNV2}}]] [[loss/mul/_61]] (1) Internal: No algorithm worked! [[{{node cu_dnnlstm/CudnnRNNV2}}]] 0 successful operations. 0 derived errors ignored.

    opened by ggardiakos 0
Owner
Pushpendu Ghosh
Pushpendu Ghosh
Use deep learning, genetic programming and other methods to predict stock and market movements

StockPredictions Use classic tricks, neural networks, deep learning, genetic programming and other methods to predict stock and market movements. Both

Linda MacPhee-Cobb 386 Jan 3, 2023
Patient-Survival - Using Python, I developed a Machine Learning model using classification techniques such as Random Forest and SVM classifiers to predict a patient's survival status that have undergone breast cancer surgery.

Patient-Survival - Using Python, I developed a Machine Learning model using classification techniques such as Random Forest and SVM classifiers to predict a patient's survival status that have undergone breast cancer surgery.

Nafis Ahmed 1 Dec 28, 2021
Introducing neural networks to predict stock prices

IntroNeuralNetworks in Python: A Template Project IntroNeuralNetworks is a project that introduces neural networks and illustrates an example of how o

Vivek Palaniappan 637 Jan 4, 2023
Introducing neural networks to predict stock prices

IntroNeuralNetworks in Python: A Template Project IntroNeuralNetworks is a project that introduces neural networks and illustrates an example of how o

Vivek Palaniappan 637 Jan 4, 2023
A resource for learning about deep learning techniques from regression to LSTM and Reinforcement Learning using financial data and the fitness functions of algorithmic trading

A tour through tensorflow with financial data I present several models ranging in complexity from simple regression to LSTM and policy networks. The s

null 195 Dec 7, 2022
Forecasting for knowable future events using Bayesian informative priors (forecasting with judgmental-adjustment).

What is judgyprophet? judgyprophet is a Bayesian forecasting algorithm based on Prophet, that enables forecasting while using information known by the

AstraZeneca 56 Oct 26, 2022
Attention-based CNN-LSTM and XGBoost hybrid model for stock prediction

Attention-based CNN-LSTM and XGBoost hybrid model for stock prediction Requirements The code has been tested running under Python 3.7.4, with the foll

zshicode 84 Jan 1, 2023
GazeScroller - Using Facial Movements to perform Hands-free Gesture on the system

GazeScroller Using Facial Movements to perform Hands-free Gesture on the system

null 2 Jan 5, 2022
Technical Indicators implemented in Python only using Numpy-Pandas as Magic - Very Very Fast! Very tiny! Stock Market Financial Technical Analysis Python library . Quant Trading automation or cryptocoin exchange

MyTT Technical Indicators implemented in Python only using Numpy-Pandas as Magic - Very Very Fast! to Stock Market Financial Technical Analysis Python

dev 34 Dec 27, 2022