I'm experimenting with using the wtte model on teh CMAPPS dataset. I note you recommend using batch size = 1:

Loss was calculated as mean over timesteps. With batch-size 1 this means that each individual training sequence was given equal weight regardless of length.

However this is resulting in very slow training (around 200 secs per epoch with GPU).

Are there any workarounds for this? I've tried modifying the loss function so it's divided by the number of timesteps of interest (which I'd hoped would normalise it), but this doesn't produce good results.

Hello @ragulpr !

Great work on this, very compelling. I have 2 higher-level questions (to potentially be followed up by other questions once I'm sure I understand the concept).

1. We only use groups with that have the event we're interested in modeling, correct? For instance, if I'm building a churn model, I could use the event I'm modeling to be something like clicking the 'Cancel' button on the web page, so my training set should only include groups that have done that event at least once right, or not?

2. We pick a time frame according to our resolution and within each sequence, we censor the data beyond that time period? So if I'm interested in people that will churn within the next 14 days, I set max_time=14 (assuming my data is in the day resolution) and at each sequence/step, everything > 14 days ahead gets censored (features and target turned to 0) right?

Thanks in advance for the insights. Looking forward to trying this out!

Hi Egil,

Thank you so much for making your code available! This is really great stuff.

So in trying to understand better how it all works I tried using the tensorflow.log-extracted data (as in your data_pipeline notebook) as inputs to the network (same config as in your simple_example). Unfortunately I got all nan's as losses:

Model summary:

 init_alpha:  -785.866918162
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
gru_1 (GRU)                  (None, 101, 1)            18        
_________________________________________________________________
dense_1 (Dense)              (None, 101, 2)            4         
_________________________________________________________________
activation_1 (Activation)    (None, 101, 2)            0         
=================================================================
Total params: 22.0
Trainable params: 22.0
Non-trainable params: 0.0  

Results of running model.fit:

Train on 72 samples, validate on 24 samples
Epoch 1/75
2s - loss: nan - val_loss: nan
....

I was wondering if you've tried doing the same experiment and if so, whether it worked for you? Thanks so much!

Hey there! We've been dealing with the churn prediction problem using the typical approach. Now, we'd would love to give a try to this less hacky approach!

First, I'd love to share a bit more of what we have. I think our evented data is a bit different from the required on this approach since is not made by evenly spaced events. Each user will have it's own timeseries data with events at different (and continuous) times. Looking at the data by time we can see something like the following table.

| Time | Event Type | User | |------|------------|------| | 1 | 1 | A | | 2 | 3 | A | | 3 | 1 | B | | 4 | 4 | B | | 5 | 3 | A | | 6 | 1 | C | | 7 | 1 | D | | 8 | 5 | B | | 9 | 5 | D | | 10 | 4 | A | | 11 | 3 | C | | 12 | 5 | C | | 13 | 0 | B | | 14 | 2 | D | | 15 | 4 | C | | 16 | 4 | C | | 17 | 0 | D | | 18 | 3 | A | | 19 | 2 | A | | 20 | 2 | A |

For example, looking at the table and assuming that the event 0 means churn, we know that users B and D churned.

My question then is: What would be the input of the Keras model if we want to use the WTTE-RNN approach?

I'm also a bit confused about the training, since we're predicting sequences, should we fed the network the entire sequence of churned users at prediction time? Since the sequence will be right censored at prediction time, I'm not sure what's the correct approach here!

Sorry if this are very basic questions! Also, thanks for writing such a great and comprehensible article!

Hello,

Thank you for your wonderful work on this model! Very interesting. I'm working to save and then load the model for future evaluations.

Considering Keras' guide on saving/loading here:

https://keras.io/getting-started/faq/#how-can-i-save-a-keras-model

I've tried this code:

from keras.models import load_model
model.save('my_model.h5')
model_loaded = load_model('my_model.h5')

But I get the following error:

Traceback (most recent call last):
  File "/home/python/envs/python3.6/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 2881, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-124-9911263901fe>", line 1, in <module>
    model2 = load_model('my_model.h5')
  File "/home/python/envs/python3.6/lib/python3.6/site-packages/keras/models.py", line 243, in load_model
    model = model_from_config(model_config, custom_objects=custom_objects)
  File "/home/python/envs/python3.6/lib/python3.6/site-packages/keras/models.py", line 317, in model_from_config
    return layer_module.deserialize(config, custom_objects=custom_objects)
  File "/home/python/envs/python3.6/lib/python3.6/site-packages/keras/layers/__init__.py", line 55, in deserialize
    printable_module_name='layer')
  File "/home/python/envs/python3.6/lib/python3.6/site-packages/keras/utils/generic_utils.py", line 143, in deserialize_keras_object
    list(custom_objects.items())))
  File "/home/python/envs/python3.6/lib/python3.6/site-packages/keras/models.py", line 1353, in from_config
    model.add(layer)
  File "/home/python/envs/python3.6/lib/python3.6/site-packages/keras/models.py", line 492, in add
    output_tensor = layer(self.outputs[0])
  File "/home/python/envs/python3.6/lib/python3.6/site-packages/keras/engine/topology.py", line 617, in __call__
    output = self.call(inputs, **kwargs)
  File "/home/python/envs/python3.6/lib/python3.6/site-packages/keras/layers/core.py", line 663, in call
    return self.function(inputs, **arguments)
  File "/home/python/envs/python3.6/lib/python3.6/site-packages/wtte/wtte.py", line 85, in output_lambda
    a, b = _keras_unstack_hack(x)
NameError: name '_keras_unstack_hack' is not defined

Which I find strange because the function _keras_unstack_hack clearly exists in the wtte.py.

I'm running Ubuntu 16.04.3 and Python 3.6, with Keras v 2.1.3.

Hi Egil,

Thanks so much for releasing the end-to-end wtte-rnn code in data_pipeline! Very cool stuff.

I had a question about validating the performance though. To check how well the model predicts TTE, I did the following:

predicted_t = model.predict(x_test)
predicted_t[:,:,1]=predicted_t[:,:,1]+predicted_t[:,:,0]*0# lazy re-add NAN-mask
print(predicted_t.shape)

pred_df = tr.padded_to_df(predicted_t,column_names=["alpha","beta"],dtypes=[float,float],ids=pd.unique(df.id))
pred_df['pred_tte'] = pred_df.apply(lambda g: g.alpha*math.pow(math.log1p(0.5),(1/g.beta)),axis=1)
pred_df['actual_tte'] = y_test[:,:,0].flatten()

where

 x_test      = left_pad_to_right_pad(right_pad_to_left_pad(x)[:,(n_timesteps-n_timesteps_to_hide):,:])
 y_test      = left_pad_to_right_pad(right_pad_to_left_pad(y)[:,(n_timesteps-n_timesteps_to_hide):,:])
 events_test = left_pad_to_right_pad(right_pad_to_left_pad(events)[:,(n_timesteps-n_timesteps_to_hide):])

 y_test[:,:,0] = tr.padded_events_to_tte(events_test,discrete_time=discrete_time,t_elapsed=padded_t)
 y_test[:,:,1] = tr.padded_events_to_not_censored(events_test,discrete_time)

What I got looked like this:

>  pred_df

Out[16]:
        id	t	alpha	       beta	    pred_tte	actual_tte
0	1	0	0.148557	0.743166	0.044092	10
1	1	1	18.626453	0.687964	5.014936	9 
2	1	2	21.242054	0.726595	6.132385	8
3	1	3	29.170420	0.734831	8.539321	7
4	1	4	30.190809	0.744385	8.978482	6
...	...	...	...	...	...	...
5233	802	49	4.187856	0.667144	1.082288	1
5234	802	50	5.580938	0.632970	1.340699	0
5235	802	51	2.631150	0.609310	0.598024	3
5236	802	52	4.732635	0.670265	1.230809	2
5237	802	53	5.632733	0.642269	1.381371	1

So if you plot predicted TTE vs. actual they don't agree much, not even directionally. Clearly I am missing something. Is this not a valid way to compare predicted vs. actual TTE?

Thank you! Natalia

(both for personal use and practicing using PyTorch,) I'm planning to port WTTE-RNN to PyTorch. What are the things (if any) that could be sort of a guide to port this? also, (If I need it) is porting this permitted? Sorry for mediocre English usage

HI, I'm fairly new to this area, and I just wanted a sanity check to see if it makes sense to pre-filter a dataset based on number of events. For example, remove all users with less than k events in the observation period.

I can see this making sense with k=1 since we tend to drop the first event for all sequences anyway (https://github.com/ragulpr/wtte-rnn/issues/37#issuecomment-354388046). Of course this might depend on the dataset, and I plan to play around with it. However, I just wanted to know if it was maybe common practice to drop records like this. Or do we favor keeping all users so we can learn user-features correlated with single event->churn

Thanks

In your blog post you have a nice visualization and good predictive results for Predicting the destruction of jet-engines.
Could you attach python code or jupyter notebook to examples directory to reproduce visualizations and prediction results?

I'm running tensorflow 1.0.1 and tried the ipython notebook in the example folder. But it break at several points. It seems code is broken because of incompatibility of tensorflow versions. Which version of tensorflow have you used to developed your code? And is it possible to migrate this to Tensorflow 1.0?

Thanks for the great work by the way. It's useful.

Thanks for the great blog post, got a detailed understanding of why/what of WTTE-RNN. I've recently started working with left censored data (observed labels are higher than the true labels for censored instances) and planned to implement a similar methodology. My initial enthusiasm was quickly killed by looking at the loss function ->

I still went ahead and implemented this loss function but am running into numerical instability issues with t/alpha or beta/alpha approaching values close to zero (after clipping these values at 0 to be able to compute log).

Just wondering if you thought about left censored data and have any recommendations to leverage this methodology.

Hello! Thank you for the materials!

Please, can you help with advise?

I am trying to solve churn prediction task. I have 12 timestamp (month) time-series for 500000 customers with some features. I have churn event for some customers, and some of customers don't have event among whole history. Is it correct, if i do follow definition of targets: For customers, who have an event: event: [0,0,0,0,0,0,0,1,0,0,0,0] tte: [7,6,5,4,3,2,1,0,4,3,2,1] u: [1,1,1,1,1,1,1,1,0,0,0,0] For customers, who don't have an event: event: [0,0,0,0,0,0,0,0,0,0,0,0] tte:[12,11,10,9,8,7,6,5,4,3,2,1] u:[1,1,1,1,1,1,1,1,1,1,1,1]

Did i make a mistake by definition the target in this way?

How I should, if it is wrong? Thank you!

great code thanks may you clarify : will it work for multivariate time series prediction both regression and classification 1 where all values are continues values weight height age target 1 56 160 34 1.2 2 77 170 54 3.5 3 87 167 43 0.7 4 55 198 72 0.5 5 88 176 32 2.3

2 or even will it work for multivariate time series where values are mixture of continues and categorical values for example 2 dimensions have continues values and 3 dimensions are categorical values

color        weight     gender  height  age  target 

1 black 56 m 160 34 yes 2 white 77 f 170 54 no 3 yellow 87 m 167 43 yes 4 white 55 m 198 72 no 5 white 88 f 176 32 yes

First of all I just want to say that your WTTE is really cool. Great blog post and paper. I've been using an adapted version of it for a time-to-event task and wanted to share a trick I've found useful for numerical instability issues in case you or anyone else is interested.

A couple of things to note in my case:

• I'm not using an RNN, since I have sufficient engineered features for the history at a point in time.
• I rewrote it in pytorch, so my code here is in pytorch.
• My case uses the discrete likelihood. I haven't tested anything for the continuous case but I don't see why it wouldn't work there too.

While testing it I had a lot of issues with nan loss and numeric instability during the fit of alpha and beta. I know you've worked a lot on this from reading the other github issues.

I've found that this parameterization for alpha and beta helps a lot:

class WTTE(nn.Module):
    
    def __init__(self, nnet_output_dim):
        super(WTTE, self).__init__()
        
        # this is the neural net whose outputs then are used to find alpha and beta
        self.nnet = InnerNNET()

        self.softplus = nn.Softplus()
        self.tanh = nn.Tanh()

        self.alpha_scaling = nn.Linear(nnet_output_dim, 1)
        self.beta_scaling = nn.Linear(nnet_output_dim, 1)

        # offset and scale parameters
        alpha_offset_init, beta_offset_init = 1.0, 1.0
        alpha_scale_init, beta_scale_init = 1.0, 1.0

        self.alpha_offset = nn.Parameter(tt.from_numpy(np.array([alpha_offset_init])).float(), requires_grad=True)
        self.beta_offset = nn.Parameter(tt.from_numpy(np.array([beta_offset_init])).float(), requires_grad=True)
        
        self.alpha_scale = nn.Parameter(tt.from_numpy(np.array([alpha_scale_init])).float(), requires_grad=True)
        self.beta_scale = nn.Parameter(tt.from_numpy(np.array([beta_scale_init])).float(), requires_grad=True)
        
    
    def forward(self, x):
        
        x = self.nnet(x)
        
        # derive alpha and beta individual scaling factors
        a_scaler = self.alpha_scaling(x)
        b_scaler = self.beta_scaling(x)

        # enforce the scaling factors to be between -1 and 1
        a_scaler = self.tanh(a_scaler)
        b_scaler = self.tanh(b_scaler)
        
        # combine the global offsets and scale factors with individual ones
        alpha = self.alpha_offset + (self.alpha_scale * a_scaler)
        beta = self.beta_offset + (self.beta_scale * b_scaler)

       # put alpha on positive range with exp, beta with softplus
        alpha = tt.exp(alpha)
        beta = self.softplus(beta)

        return alpha, beta

Essentially why this helps is that the tanh activation function enforces the individual/observation scaling factors to always be between -1 and 1, so you don't have to worry about too small or large outputs from your network. The alpha_scale and beta_scale are responsible for setting the range to multiply the -1 to 1 outputs by. The offsets are nice as an intercept or centering mechanism.

If you set the initialization for the offsets and scaling factors to be low numbers (I start them at 1.0, for example), they will slowly creep up to their optimal values during fit. Here is some output from a recent fit of mine to show what I mean:

A off: 1.10000	A scale: 1.10000	B off: 0.90000	B scale: 1.10000	
A off: 1.23279	A scale: 1.03885	B off: 0.90022	B scale: 0.89804	
A off: 1.25786	A scale: 1.06547	B off: 0.90056	B scale: 0.89798	
A off: 1.28466	A scale: 1.09343	B off: 0.90163	B scale: 0.89878	
A off: 1.34988	A scale: 1.16266	B off: 0.90678	B scale: 0.90290	
A off: 1.44370	A scale: 1.25528	B off: 0.93324	B scale: 0.93015	
A off: 1.53040	A scale: 1.32979	B off: 0.98308	B scale: 0.98226		
...[many epochs later]...
A off: 1.92782	A scale: 1.57879	B off: 2.97086	B scale: 2.55308	
A off: 1.93340	A scale: 1.59249	B off: 3.01380	B scale: 2.59065	
A off: 1.94988	A scale: 1.59956	B off: 3.01739	B scale: 2.54407	
A off: 1.94464	A scale: 1.59733	B off: 3.03923	B scale: 2.55807	
A off: 1.95629	A scale: 1.60365	B off: 3.06733	B scale: 2.58807	
A off: 1.95865	A scale: 1.59092	B off: 3.09355	B scale: 2.60830

You could also enforce maximums on alpha and beta easily if you wanted to by adding torch.clamp calls around the outputs, but I have not found this to be necessary.

I have only tested this on my own data and so I can't make any claims that this will solve numerical instability issues for other people, but I figured it may help someone!

Hello,

First of all, thanks for this great contribution.

I've been trying to use it to solve churn issues, but I wanted to save so time and directly ask you: how do I take only one register and use the already trained model to predict its WTTE?

Thanks again!

great code thanks but what to do with multi variate time series when at each time point we have many observatoins when we have categorical and continues data for example t1: red 0.5 123 big t2: green 0.1 34 small t3: red -0.3 56 big etc? by the way why you do not use LSTM