1. I think your predictions are delayed for one step and if you zoom in your graphic you'll see this
2. Predictions itself are just a data of the previous step with some lag

i don't think it's correct model or maybe it's just a oddities of drawing....

please fix me if i wrong

I believe this may be an issue with keras / tensorflow updating, and not a 'code' problem.

absl-py==0.11.0 astunparse==1.6.3 cachetools==4.2.1 certifi==2020.12.5 chardet==4.0.0 cycler==0.10.0 flatbuffers==1.12 gast==0.3.3 google-auth==1.27.0 google-auth-oauthlib==0.4.2 google-pasta==0.2.0 grpcio==1.32.0 h5py==2.10.0 idna==3.1 joblib==1.0.1 Keras==2.4.3 Keras-Preprocessing==1.1.2 kiwisolver==1.3.1 Markdown==3.3.4 matplotlib==3.3.4 mplfinance==0.12.7a7 numpy==1.19.5 oauthlib==3.1.0 opt-einsum==3.3.0 pandas==1.2.2 Pillow==8.1.0 protobuf==3.15.3 pyasn1==0.4.8 pyasn1-modules==0.2.8 pyparsing==2.4.7 python-dateutil==2.8.1 pytz==2021.1 PyYAML==5.4.1 requests==2.25.1 requests-oauthlib==1.3.0 rsa==4.7.2 scikit-learn==0.24.1 scipy==1.6.1 six==1.15.0 tensorboard==2.4.1 tensorboard-plugin-wit==1.8.0 tensorflow==2.4.1 tensorflow-estimator==2.4.0 termcolor==1.1.0 threadpoolctl==2.1.0 typing-extensions==3.7.4.3 urllib3==1.26.3 Werkzeug==1.0.1 wrapt==1.12.1

No errors of any kind, please advise.

First of all thank you for sharing your code. Im just looking for this type code (predict future price). Now im testing but i got wrong predicted values. All predicted values are increased. Why?

There is a big mistake in your code. In the StockPrediction.py , model.add('linear') is not right. We all know that the activation function have to be not linear, the linear activation will make the perdiction not right.

Hello ,

I am facing some issues with Preprocessing. When I a running the section with preprocessing this is what I get:

AttributeError: module 'sklearn.preprocessing' has no attribute 'new_dataset'

Here is the code of yours. Am I missing any steps?

#Edit Author: Ray

IMPORTING IMPORTANT LIBRARIES

import pandas as pd import matplotlib.pyplot as plt import numpy as np import math from sklearn.preprocessing import MinMaxScaler from sklearn.metrics import mean_squared_error from keras.models import Sequential from keras.layers import Dense, Activation from keras.layers import LSTM from sklearn import preprocessing # how to import preprocessing import sklearn.preprocessing import numpy as np

FOR REPRODUCIBILITY

np.random.seed(7)

IMPORTING DATASET

dataset = pd.read_csv('C:/Users/ray/Documents/Python Scripts/LSTM-Stock-prediction-master/apple_share_price.csv', usecols=[1,2,3,4]) dataset = dataset.reindex(index = dataset.index[::-1])

CREATING OWN INDEX FOR FLEXIBILITY

obs = np.arange(1, len(dataset) + 1, 1)

TAKING DIFFERENT INDICATORS FOR PREDICTION

OHLC_avg = dataset.mean(axis = 1) HLC_avg = dataset[['High', 'Low', 'Close']].mean(axis = 1) close_val = dataset[['Close']]

PLOTTING ALL INDICATORS IN ONE PLOT

plt.plot(obs, OHLC_avg, 'r', label = 'OHLC avg') plt.plot(obs, HLC_avg, 'b', label = 'HLC avg') plt.plot(obs, close_val, 'g', label = 'Closing price') plt.legend(loc = 'upper right') plt.show()

PREPARATION OF TIME SERIES DATASET

OHLC_avg = np.reshape(OHLC_avg.values, (len(OHLC_avg),1)) # 1664 scaler = MinMaxScaler(feature_range=(0, 1)) OHLC_avg = scaler.fit_transform(OHLC_avg)

TRAIN-TEST SPLIT

train_OHLC = int(len(OHLC_avg) * 0.75) test_OHLC = len(OHLC_avg) - train_OHLC train_OHLC, test_OHLC = OHLC_avg[0:train_OHLC,:], OHLC_avg[train_OHLC:len(OHLC_avg),:]

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ step_size = 1

FUNCTION TO CREATE 1D DATA INTO TIME SERIES DATASET

def new_dataset(dataset, step_size): trainX, trainY = [], [] for i in range(len(dataset)-step_size-1): a = dataset[i:(i+step_size), 0] trainX.append(a) trainY.append(dataset[i + step_size, 0]) return np.array(trainX), np.array(trainY)

TIME-SERIES DATASET (FOR TIME T, VALUES FOR TIME T+1)

trainX, trainY = sklearn.preprocessing.new_dataset(train_OHLC, 1) testX, testY = sklearn.preprocessing.new_dataset(test_OHLC, 1) +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

RESHAPING TRAIN AND TEST DATA

trainX = np.reshape(train_OHLC, (train_OHLC.shape[0], 1, train_OHLC.shape[1])) testX = np.reshape(testX, (testX.shape[0], 1, testX.shape[1])) step_size = 1

LSTM MODEL

model = Sequential() model.add(LSTM(32, input_shape=(1, step_size), return_sequences = True)) model.add(LSTM(16)) model.add(Dense(1)) model.add(Activation('linear'))

MODEL COMPILING AND TRAINING

model.compile(loss='mean_squared_error', optimizer='adagrad') # Try SGD, adam, adagrad and compare!!! model.fit(trainX, trainY, epochs=5, batch_size=1, verbose=2)

PREDICTION

trainPredict = model.predict(trainX) testPredict = model.predict(testX)

DE-NORMALIZING FOR PLOTTING

trainPredict = scaler.inverse_transform(trainPredict) trainY = scaler.inverse_transform([trainY]) testPredict = scaler.inverse_transform(testPredict) testY = scaler.inverse_transform([testY])

TRAINING RMSE

trainScore = math.sqrt(mean_squared_error(trainY[0], trainPredict[:,0])) print('Train RMSE: %.2f' % (trainScore))

TEST RMSE

testScore = math.sqrt(mean_squared_error(testY[0], testPredict[:,0])) print('Test RMSE: %.2f' % (testScore))

CREATING SIMILAR DATASET TO PLOT TRAINING PREDICTIONS

trainPredictPlot = np.empty_like(OHLC_avg) trainPredictPlot[:, :] = np.nan trainPredictPlot[step_size:len(trainPredict)+step_size, :] = trainPredict

CREATING SIMILAR DATASSET TO PLOT TEST PREDICTIONS

testPredictPlot = np.empty_like(OHLC_avg) testPredictPlot[:, :] = np.nan testPredictPlot[len(trainPredict)+(step_size*2)+1:len(OHLC_avg)-1, :] = testPredict

DE-NORMALIZING MAIN DATASET

OHLC_avg = scaler.inverse_transform(OHLC_avg)

PLOT OF MAIN OHLC VALUES, TRAIN PREDICTIONS AND TEST PREDICTIONS

plt.plot(OHLC_avg, 'g', label = 'original dataset') plt.plot(trainPredictPlot, 'r', label = 'training set') plt.plot(testPredictPlot, 'b', label = 'predicted stock price/test set') plt.legend(loc = 'upper right') plt.xlabel('Time in Days') plt.ylabel('OHLC Value of Apple Stocks') plt.show()

PREDICT FUTURE VALUES

last_val = testPredict[-1] last_val_scaled = last_val/last_val next_val = model.predict(np.reshape(last_val_scaled, (1,1,1))) print "Last Day Value:", np.asscalar(last_val) print "Next Day Value:", np.asscalar(last_val*next_val)

print np.append(last_val, next_val)

Hi,

nice post but the results are wrong.

You are not predicting some days ahead but only one day ahead at a time. You are not taking your prediction as input for your next prediciton, but you are taking the actual value.

This results in a lag of the actual signal, all your network has to do is produce a similar value to the last input of the price.

If you would take your prediction as the input for the next prediction you would see that the results are quite bad…

I see lot’s of LSTM price prediction examples but they all seem to be wrong and I don’t think it is possible to predict accuratly the next prices.

Nice work! I have a suggest: You are testing the model with RMSE this way:

testScore = math.sqrt(mean_squared_error(testY[0], testPredict[:,0]))

which is not a good metric to quantify the accuracy neither the performance of the model. Much more important that RMSE is the ability of the model to predict stock movements. So I would like to see some metrics taking into account how much ups and downs are efficiently predicted. I suggest to implement recall and prediction measurements.

This is awesome and helpful. I have been playing with it and have RMSE down on test data under 1. If I am reading the output right, it displays the previous days average you are using and then the next days prediction? Is there a way to look ahead more steps?

Also, you are taking the OHLC and turning it into a 1 value input. Is there a way to use this model to add more inputs such as features on top of the OHLC value?

I tried the initial code, but there were some problems in the CSV file. I tried with another CSV file from yahoo. I also found an anomaly that the ohlc_avg was in millions, so I removed 'volume' column from the table for making the average. And I did remove 'dataset = dataset.reindex(index = dataset.index[::-1])' because due to it, the price was falling, whereas it should have increased with time.

I tried the code in this notebook, you can review it too. https://colab.research.google.com/drive/1v6FF9zHMYgBy8QD1kuSL3q3ln-pD7lQw?usp=sharing

And thanks to you for sharing this code, so that I could learn LSTM. You can contact me for any clarifications at [email protected] or +918617781030(WhatsApp & Call)

Hello there, I would like to know the logic behind the future prediction of the code snippets. last_val = testPredict[-1] last_val_scaled = last_val/last_val next_val = model.predict(np.reshape(last_val_scaled, (1,1,1))) print ("Last Day Value:", np.asscalar(last_val)) print ("Next Day Value:", np.asscalar(last_val*next_val))

Hi. Thank you for this project.. Although its written in python 2.7. but there is a package named preprocessing (USED FOR TEXT PREPRPOCESSING) and cannot be install in python2. what can be suitable alternate function instead of using preprocessing.new_data()? preferably in scikit learn. Thank you

TTM Squeeze Pro {"m":"create_study","p":["cs_YcJ513Cvsuhb","st4","st1","sds_1","Script@tv-scripting-101!",{"text":"CJj4Ug3Yp5M7ZVNBdQXoRQ==_pHhbuDEwXDaCrSt+VDSNI9LL/zg2VhikNtpD6S8x8lri/FQ/Ko1DMDAYAC25z10cjq79NfyXj9qs+jiQMVCHm5rcQX6lirXbH9n+4RiT4An1nXIxb0nx6F++RV3uMAQa/9s2nygsfsh4KL1sKlc3e6ZcaXRqjSAemX74kYVPP/XSw2gEGYwAGXgVrb9NoWDi1it9eWk9scCPqvmsrgMMvNVVLmBOBhSShrCbA8B6NtIuuSiVw2sJpIvJHzqJuf6dBCo+7oIrIjX2mMhdWYfY+qL+xtu4p56QMFkSiI5RNtMnU/a8y09DfDA6W5+q/la23KcUu1awoRHH3leen8apCGTvMDtLFrJQ3m3cUYttpzu7YL/tuNlGzrnrhUzepe3BHjOjz5PFQK6MO0JyHSlvzUjLEa5JsUpWDBYgWF1UJkGMhu0yT2n87omqlm8zfp5hxhYl3BiCwUuVvfzHj56QmaGE5PaZBmhaci0DsPkUsHIh4atk7lXs7HMlz2sAxfOjWogEZo/cfWckWoQUEyLsEgAvB5hYrnGG9yofTVvKE90O3v5slNSo7lE/Df9iJUveUIzFQezCLUZTEGiUR0n8KVNwehGGRdTRnljSbbLLt7Y+wtYn2GQHEhMNzmufjP0LcZpySZfl4o2yIrOV1u1p7Ypg6PHK+ULBJccxQUmQq39hb6JPuAH4JTCtNBbWRR3PwCPtfK2p2dTn/h6Hsb542S1x1OtFr427I2Pi/cWxOEzuOTwHgsFHpSF4pawUyJ7bIlfEawpa5PwU3wfJX9/+xcllFfnvporcVTsa/GqI0CmDU/CiaFFIrFkVKZ0lgbdwbpKD6znXGllglWEt5kDBKw3P1F0vD75hhQoQOEzjsbeuGsOaNXuYtet8wSnJguhSCUIbcrH8yoN9frYEYe/GQsxAvPx9CSbRQSbvZEys28FOOHQNflD3oORqEx1HbER1xLpJ90VNVKo7OF43ul54Yne8jkcXz+7x3n5Q45ZQsfejoUPZYHIHRcEPTSFCReQDfPa/iM+JnW39Lq0mYJxQSQomf4L71c8GuWoEqGpZlFKkQYFWmg==","pineId":"PUB;7e9cf40f672c4ab88ac70c580a327870","pineVersion":"4.0","pineFeatures":{"v":"{"indicator":1,"plot":1,"ta":1,"math":1,"alert":1}","f":true,"t":"text"},"in_0":{"v":20,"f":true,"t":"integer"},"in_1":{"v":2,"f":true,"t":"float"},"in_2":{"v":1,"f":true,"t":"float"},"in_3":{"v":1.5,"f":true,"t":"float"},"in_4":{"v":2,"f":true,"t":"float"},"in_5":{"v":true,"f":true,"t":"bool"},"in_6":{"v":true,"f":true,"t":"bool"}}]}