Was running YATSM for p012r031, everything was going fine until the 67th job, when I suddenly got the following error message:

09:26:06:DEBUG:66:config_parser.convert_config:Predicting using "GLMNET_LassoCV" pickle specified from configuration file (/usr3/graduate/valpasq/Documents/yatsm/yatsm/regression/pickles/glmnet_LassoCV_n50.pkl)
Traceback (most recent call last):
  File "/usr3/graduate/valpasq/venv/bin/yatsm", line 8, in <module>
    load_entry_point('yatsm==0.5.5', 'console_scripts', 'yatsm')()
  File "/project/earth/packages/Python-2.7.5/lib/python2.7/site-packages/click/core.py", line 700, in __call__
    return self.main(*args, **kwargs)
  File "/project/earth/packages/Python-2.7.5/lib/python2.7/site-packages/click/core.py", line 680, in main
    rv = self.invoke(ctx)
  File "/project/earth/packages/Python-2.7.5/lib/python2.7/site-packages/click/core.py", line 1027, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/project/earth/packages/Python-2.7.5/lib/python2.7/site-packages/click/core.py", line 873, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/project/earth/packages/Python-2.7.5/lib/python2.7/site-packages/click/core.py", line 508, in invoke
    return callback(*args, **kwargs)
  File "/project/earth/packages/Python-2.7.5/lib/python2.7/site-packages/click/decorators.py", line 16, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr3/graduate/valpasq/Documents/yatsm/yatsm/cli/line.py", line 50, in line
    cfg = parse_config_file(config)
  File "/usr3/graduate/valpasq/Documents/yatsm/yatsm/config_parser.py", line 145, in parse_config_file
    return convert_config(cfg)
  File "/usr3/graduate/valpasq/Documents/yatsm/yatsm/config_parser.py", line 69, in convert_config
    cfg[pred_method]['pickle'])
  File "/usr3/graduate/valpasq/Documents/yatsm/yatsm/config_parser.py", line 150, in _unpickle_predictor
    reg = joblib.load(pickle)
  File "/project/earth/packages/Python-2.7.5/lib/python2.7/site-packages/sklearn/externals/joblib/numpy_pickle.py", line 425, in load
    obj = unpickler.load()
  File "/project/earth/packages/Python-2.7.5/lib/python2.7/pickle.py", line 858, in load
    dispatch[key](self)
  File "/project/earth/packages/Python-2.7.5/lib/python2.7/pickle.py", line 1090, in load_global
    klass = self.find_class(module, name)
  File "/project/earth/packages/Python-2.7.5/lib/python2.7/pickle.py", line 1124, in find_class
    __import__(module)
ImportError: No module named elastic_net_cv

I tried changing to GLMNET_Lasso20, then got this message:

09:47:50:DEBUG:66:config_parser.convert_config:Predicting using "GLMNET_Lasso20" pickle specified from configuration file (/usr3/graduate/valpasq/Documents/yatsm/yatsm/regression/pickles/glmnet_Lasso20.pkl)
Traceback (most recent call last):
  File "/usr3/graduate/valpasq/venv/bin/yatsm", line 8, in <module>
    load_entry_point('yatsm==0.5.5', 'console_scripts', 'yatsm')()
  File "/project/earth/packages/Python-2.7.5/lib/python2.7/site-packages/click/core.py", line 700, in __call__
    return self.main(*args, **kwargs)
  File "/project/earth/packages/Python-2.7.5/lib/python2.7/site-packages/click/core.py", line 680, in main
    rv = self.invoke(ctx)
  File "/project/earth/packages/Python-2.7.5/lib/python2.7/site-packages/click/core.py", line 1027, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/project/earth/packages/Python-2.7.5/lib/python2.7/site-packages/click/core.py", line 873, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/project/earth/packages/Python-2.7.5/lib/python2.7/site-packages/click/core.py", line 508, in invoke
    return callback(*args, **kwargs)
  File "/project/earth/packages/Python-2.7.5/lib/python2.7/site-packages/click/decorators.py", line 16, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr3/graduate/valpasq/Documents/yatsm/yatsm/cli/line.py", line 50, in line
    cfg = parse_config_file(config)
  File "/usr3/graduate/valpasq/Documents/yatsm/yatsm/config_parser.py", line 145, in parse_config_file
    return convert_config(cfg)
  File "/usr3/graduate/valpasq/Documents/yatsm/yatsm/config_parser.py", line 69, in convert_config
    cfg[pred_method]['pickle'])
  File "/usr3/graduate/valpasq/Documents/yatsm/yatsm/config_parser.py", line 150, in _unpickle_predictor
    reg = joblib.load(pickle)
  File "/project/earth/packages/Python-2.7.5/lib/python2.7/site-packages/sklearn/externals/joblib/numpy_pickle.py", line 425, in load
    obj = unpickler.load()
  File "/project/earth/packages/Python-2.7.5/lib/python2.7/pickle.py", line 858, in load
    dispatch[key](self)
  File "/project/earth/packages/Python-2.7.5/lib/python2.7/pickle.py", line 1083, in load_newobj
    obj = cls.__new__(cls, *args)
TypeError: function.__new__(X): X is not a type object (function)

Oddly enough, when I tried the sklearn Lasso20, things seem to be running alright, though I do get a warning about convergence: 10:03:31:DEBUG:66:config_parser.convert_config:Predicting using "Lasso20" pickle specified from configuration file (/usr3/graduate/valpasq/Documents/yatsm/yatsm/regression/pickles/sklearn_Lasso20.pkl) 10:03:31:DEBUG:93:cache.test_cache:Attempt reading in from cache directory?: True 10:03:31:DEBUG:95:cache.test_cache:Attempt writing to cache directory?: True 10:03:31:INFO:81:line.line:Job 0 of 5 - using config file /projectnb/landsat/projects/Massachusetts/p012r031/p012r031_config.yaml 10:03:31:DEBUG:96:line.line:Responsible for lines: [ 0 5 10 ..., 7140 7145 7150] 10:03:31:DEBUG:125:line.line:Already processed line 0 10:03:31:DEBUG:125:line.line:Already processed line 5 10:03:31:DEBUG:125:line.line:Already processed line 10 10:03:31:DEBUG:125:line.line:Already processed line 15 10:03:31:DEBUG:125:line.line:Already processed line 20 10:03:31:DEBUG:125:line.line:Already processed line 25 10:03:31:DEBUG:125:line.line:Already processed line 30 10:03:31:DEBUG:125:line.line:Already processed line 35 10:03:31:DEBUG:125:line.line:Already processed line 40 10:03:31:DEBUG:125:line.line:Already processed line 45 10:03:31:DEBUG:125:line.line:Already processed line 50 10:03:31:DEBUG:125:line.line:Already processed line 55 10:03:31:DEBUG:125:line.line:Already processed line 60 10:03:31:DEBUG:125:line.line:Already processed line 65 10:03:31:DEBUG:128:line.line:Running line 70 10:03:32:DEBUG:158:reader.read_line:Read in Y from cache file /project/earth/packages/Python-2.7.5/lib/python2.7/site-packages/sklearn/linear_model/coordinate_descent.py:444: ConvergenceWarning: Objective did not converge. You might want to increase the number of iterations ConvergenceWarning) 10:07:41:DEBUG:192:line.line: Saving YATSM output to /projectnb/landsat/projects/Massachusetts/p012r031/images/YATSM/yatsm_r70.npz 10:07:41:DEBUG:199:line.line:Line 70 took 249.888620138s to run 10:07:41:DEBUG:128:line.line:Running line 75 10:07:42:DEBUG:158:reader.read_line:Read in Y from cache file

Did something change with the pickles? I do find it really strange that my first 60+ jobs ran fine before I started getting errors, which makes me think this is not something to do with my copy of YATSM (since I didn't do a pull or anything that should change those files). Since the last lines of the errors have to do with File "/project/earth/packages/Python-2.7.5/lib/python2.7/pickle.py" I'm wondering if this maybe has something to do with site packages?

Any insight would be much appreciated--was hoping to run all 5 MA scenes this week,

PS - My log files are a mess, but the first few runs are in /projectnb/landsat/projects/Massachusetts/p012r031/images/, more recent runs in /projectnb/landsat/projects/Massachusetts/p012r031.

WIP to address some of the design flaws, limitations, and unimplemented functionality. Before this is merged, I'd like to have some progress toward including the data reading and preprocessing explicitly.

@wkearn - I have some thoughts I'd love to get your feedback on, mostly having to do with how to represent tasks in the graph that have side-effects. I'll try to write via the code review process...

When I run the cache function in the terminal of Ubuntu I get this error: TypeError: long() argument must be a string or a number, not 'NoneType'. I do not know how to fix it. I have tried different options like change the job number the total jobs, the locations and directories where I have the .csv, the input images (Landsat 7. Envi BSQ), the folder of the output and also the parameter called prediction. But it has not worked. Thanks in advance for your help

p007r059yaml.txt

(yatsm_venv)ideam@ideam-Precision-WorkStation-T7500:~$ yatsm cache /home/ideam/yatsm/7_59/p007r059/p007r059.yaml 1 8 /home/ideam/yatsm_venv/local/lib/python2.7/site-packages/matplotlib/font_manager.py:273: UserWarning: Matplotlib is building the font cache using fc-list. This may take a moment. warnings.warn('Matplotlib is building the font cache using fc-list. This may take a moment.') Traceback (most recent call last): File "/home/ideam/yatsm_venv/bin/yatsm", line 9, in load_entry_point('yatsm==0.6.1', 'console_scripts', 'yatsm')() File "/home/ideam/yatsm_venv/local/lib/python2.7/site-packages/click/core.py", line 716, in call return self.main(_args, *_kwargs) File "/home/ideam/yatsm_venv/local/lib/python2.7/site-packages/click/core.py", line 696, in main rv = self.invoke(ctx) File "/home/ideam/yatsm_venv/local/lib/python2.7/site-packages/click/core.py", line 1060, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/home/ideam/yatsm_venv/local/lib/python2.7/site-packages/click/core.py", line 889, in invoke return ctx.invoke(self.callback, *_ctx.params) File "/home/ideam/yatsm_venv/local/lib/python2.7/site-packages/click/core.py", line 534, in invoke return callback(_args, *_kwargs) File "/home/ideam/yatsm_venv/local/lib/python2.7/site-packages/click/decorators.py", line 17, in new_func return f(get_current_context(), *args, *_kwargs) File "/home/ideam/yatsm/yatsm/cli/cache.py", line 108, in cache Y = io.gdal_reader.read_row(df['filename'], job_line) File "/home/ideam/yatsm/yatsm/io/stack_line_readers.py", line 155, in read_row return self._read_row(row) File "/home/ideam/yatsm/yatsm/io/stack_line_readers.py", line 138, in _read_row data[n_b, i, :] = band.ReadAsArray(0, row, self.n_col, 1) TypeError: long() argument must be a string or a number, not 'NoneType'

Error occurs when attempting to run phenology fitting:

  File "/usr3/graduate/valpasq/Documents/yatsm/yatsm/cli/line.py", line 190, in line
    ltm = pheno.LongTermMeanPhenology(yatsm, **cfg['phenology'])
TypeError: __init__() got an unexpected keyword argument 'year_interval'

Config location: /projectnb/landsat/projects/Massachusetts/p012r031/p012r031_config.yaml

The "test statistic" used in CCDC is the square root of the sum of squared scaled residuals for all bands tested. This "test statistic" isn't normalized by how many bands you're adding, so the critical value needs to depend on how many test indices you have. If you use more indices to test with, then you'll need to increase the critical value by some amount.

Zhu derives the "test statistic" critical value for p=0.01 and k=len(test_indices) using the inverse survival function of the scipy.stats.chi distribution since we're probably summing squares of normally distributed variables (the scaled residuals).

Questions of independence, normality, statistical soundness, etc. aside, my biggest concern is that we don't really care about finding change according to some null hypothesis testing framework value. CCDC is, at best, vaguely statistical and we've never analytically or numerically explored what the distribution of the "test statistic" is under the null hypothesis of no change. However, using the scipy.stats.chi.isf does convey the important message that the critical value depends on how many bands are being tested.

So, the proposed solution either includes:

1. Better documentation explaining this!
2. Convert threshold parameter to p_value and retrieve the threshold using scipy.stats.chi.isf for a given number of test_indices
3. Accept both threshold and p_value with threshold being the default input that overrides p_value if both are specified
   • If the user specifies p_value, then we back out threshold against the chi distribution
   • If threshold is specified, well then there's no difference versus current behavior.
   • If the user specifies both parameters, we keep threshold as being authoritative and warn the user

Thoughts, @bullocke, @valpasq, @parevalo, and @xjtang?

https://github.com/ceholden/yatsm/blob/master/yatsm/cli/line.py#L196

File "/usr3/graduate/valpasq/Documents/yatsm/yatsm/cli/line.py", line 196, in line output.extend(yatsm.record) TypeError: 'NoneType' object is not iterable

Fix (tested): Add if statement to check if record exists:

if yatsm.record is not None: output.extend(yatsm.record)

Tried to run YATSM with commission_alpha=0.5 using config file /projectnb/landsat/projects/Massachusetts/p013r030/p013r030_config.yaml

YATSM:
    algorithm: "CCDCesque"
    prediction: "GLMNET_Lasso20"
    design_matrix: "1 + x + harm(x, 1) + harm(x, 2)"
    reverse: False
    commission_alpha: 0.05

But got the following error:

  File "/usr3/graduate/valpasq/venv/bin/yatsm", line 8, in <module>
    load_entry_point('yatsm==0.5.6-beta', 'console_scripts', 'yatsm')()
  File "/project/earth/packages/Python-2.7.5/lib/python2.7/site-packages/click/core.py", line 700, in __call__
    return self.main(*args, **kwargs)
  File "/project/earth/packages/Python-2.7.5/lib/python2.7/site-packages/click/core.py", line 680, in main
    rv = self.invoke(ctx)
  File "/project/earth/packages/Python-2.7.5/lib/python2.7/site-packages/click/core.py", line 1027, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/project/earth/packages/Python-2.7.5/lib/python2.7/site-packages/click/core.py", line 873, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/project/earth/packages/Python-2.7.5/lib/python2.7/site-packages/click/core.py", line 508, in invoke
    return callback(*args, **kwargs)
  File "/project/earth/packages/Python-2.7.5/lib/python2.7/site-packages/click/decorators.py", line 16, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr3/graduate/valpasq/Documents/yatsm/yatsm/cli/line.py", line 178, in line
    yatsm, cfg['YATSM']['commission_alpha'])
  File "/usr3/graduate/valpasq/Documents/yatsm/yatsm/algorithms/postprocess.py", line 127, in commission_test
    for i_m, _m in enumerate(_models):
TypeError: 'NoneType' object is not iterable

Pulled updated repository yesterday (currently up to date). Tried running YATSM with phenology enabled. A good number of lines process with no issue, but eventually the following error occurs:

  File "/usr3/graduate/valpasq/Documents/yatsm/yatsm/phenology.py", line 309, in fit
    self.q_min, self.q_max)
  File "/usr3/graduate/valpasq/Documents/yatsm/yatsm/phenology.py", line 241, in _fit_record
    pad_start = np.arange(1, yeardoy[:, 1].min() + 1)
  File "/project/earth/packages/Python-2.7.5/lib/python2.7/site-packages/numpy/core/_methods.py", line 29, in _amin
    return umr_minimum(a, axis, None, out, keepdims)
ValueError: zero-size array to reduction operation minimum which has no identity 

yatsm_*.o* logfiles located in /projectnb/landsat/projects/Massachusetts/ Verbose turned on to help ID which lines caused the error.

The recent updates to the config schema changed the reader section to have a name and a config section where before it was name and $name (like GDAL). Should config be a required entry in the schema?

This just screwed me up temporarily as I was using my old (i.e. Friday's) config file and I got an error about the input_file field missing that I had to trace down to this name change, which I just missed as I scanned the new example config. Obviously it's in the new example, so new users who copy that won't hit this.

It seems that if we're going to throw an error when something in the config section (or the section itself) is missing, we should throw it in the config validation step rather than a few steps later when we start trying to load data.

Added simple seasonal symbology option to TS (sequential date) plot for 0.6.x-maintenance version of yatsm pixel.

Current symbology puts emphasis on growing season observations (100% opaque) vs. other seasons (50% transparent).

Seasons defined by month and most applicable to temperate ecoregions in the Northern Hemisphere.

• Winter (leaf-off) = November, December, January, February, March
• Spring = April, May
• Summer (leaf-on) = June, July, August
• Fall = September, October

Dates loosely correspond to this visual:

Source: https://nccwsc.usgs.gov/content/ecological-drought-northeast-united-states-anticipating-changes-iconic-species-landscapes

I'm trying to run yatsm line and getting the following error:

15:15:19:INFO:51:line.line:Job 1 of 400 - using config file /projectnb/landsat/projects/Colombia/images/005057/Results/FIT1/557_FIT1.yaml
0-th dimension must be fixed to 4 but got 6

Traceback (most recent call last):
  File "/usr3/graduate/parevalo/miniconda2/envs/conda_env/bin/yatsm", line 9, in <module>
    load_entry_point('yatsm==0.6.0.dev0', 'console_scripts', 'yatsm')()
  File "/usr3/graduate/parevalo/miniconda2/envs/conda_env/lib/python2.7/site-packages/click/core.py", line 716, in __call__
    return self.main(*args, **kwargs)
  File "/usr3/graduate/parevalo/miniconda2/envs/conda_env/lib/python2.7/site-packages/click/core.py", line 696, in main
    rv = self.invoke(ctx)
  File "/usr3/graduate/parevalo/miniconda2/envs/conda_env/lib/python2.7/site-packages/click/core.py", line 1060, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr3/graduate/parevalo/miniconda2/envs/conda_env/lib/python2.7/site-packages/click/core.py", line 889, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr3/graduate/parevalo/miniconda2/envs/conda_env/lib/python2.7/site-packages/click/core.py", line 534, in invoke
    return callback(*args, **kwargs)
  File "/usr3/graduate/parevalo/miniconda2/envs/conda_env/lib/python2.7/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr3/graduate/parevalo/miniconda2/envs/conda_env/lib/python2.7/site-packages/yatsm/cli/line.py", line 156, in line
    yatsm.fit(_X, _Y, _dates, **algo_cfg.get('fit', {}))
  File "/usr3/graduate/parevalo/miniconda2/envs/conda_env/lib/python2.7/site-packages/yatsm/algorithms/ccdc.py", line 222, in fit
    self.train()
  File "/usr3/graduate/parevalo/miniconda2/envs/conda_env/lib/python2.7/site-packages/yatsm/algorithms/ccdc.py", line 326, in train
    bands=self.test_indices)
  File "/usr3/graduate/parevalo/miniconda2/envs/conda_env/lib/python2.7/site-packages/yatsm/algorithms/yatsm.py", line 195, in fit_models
    model.fit(X, y, **self.estimator_fit)
  File "/usr3/graduate/parevalo/miniconda2/envs/conda_env/lib/python2.7/site-packages/glmnet/elastic_net.py", line 86, in fit
    enet_path(X, y, alpha=self.alpha, **kwargs)
  File "/usr3/graduate/parevalo/miniconda2/envs/conda_env/lib/python2.7/site-packages/glmnet/elastic_net.py", line 349, in enet_path
    = elastic_net(X, y, alpha, **kwargs)
  File "/usr3/graduate/parevalo/miniconda2/envs/conda_env/lib/python2.7/site-packages/glmnet/glmnet.py", line 91, in elastic_net
    nlam=nlam, isd=standardize)
_glmnet.error: failed in converting 6th argument `vp' of _glmnet.elnet to C/Fortran array

• I'm using a conda venv and packages were installed during its creation, using the environment.yaml file and following the instructions provided in the documentation.
• I'm using yatsm 0.6.0dev (very latest version)
• Config files in: /projectnb/landsat/projects/Colombia/images/005057/Results/FIT1
• The script used is Yatsm.sh

I'm not even sure how to try to debug it myself or trace the cause of the problem. Also, I checked your glmnet-python repository and you suggested using the fork by the user "shuras" and tried to install that to see if it fixed the problem but the installation was unsuccessful.

Dear, please I having an error message below when I'm trying to install yatsm on Ubuntu 17.04 Anaconda3. What does it mean? Thanks

Command "/home/user/anaconda3/envs/yatsm/bin/python -u -c "import setuptools, tokenize;file='/tmp/pip-kred0d4t-build/setup.py';f=getattr(tokenize, 'open', open)(file);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, file, 'exec'))" install --record /tmp/pip-qkvpe3vz-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-kred0d4t-build/

CondaValueError: pip returned an error.

P.S. if it's not the right place to post such request please let me know

Resolve changes occurring in models that are not near-simultaneously co-occurring within pixels in a neighborhood region.

Depends a bit on #14 because finding a pixel's neighbors in results will require quite a lot of np.where to index. Using a file format that could store variables in some indexing scheme would speed up the searches substantially (using pytables, h5py, etc.)

Motivation

Right now we're using NumPy saved files that store structural arrays for the results but this might change in the future (see #14), especially to accommodate some visualization utilities that would benefit from having all the results for an image in one container indexed intelligently.

Another annoyance is that each CLI utility uses duplicate code to open/inspect/read/write from/to the result files. Ideally this should be refactored into some common set of functions.

Proposal

Implement a "drivers" for each format (so just NumPy for now) that contains the logic for inspecting/reading/writing to/etc. each format. Eventually this will necessitate updating the configuration files to specify what result storage driver should be used.

The currently implemented iter_records, for example, would still iterate over result records, but would do so in a way that makes sense for the format. For the current NumPy saved files, we'd yield one row worth of records at a time. If we used something that stores results in blocks, maybe it would be chunks of data irregardless of the row:

driver = drivers.register(result_format)

for rec in driver.iter_records(config):
    # do stuff

We usually want to perform a query on the records based on the segment dates, so there could be some higher level API access that would perform a query optimized for the format (NumPy files would just use simple np.where against them but we could use in kernel searches if using pytables):

driver = drivers.register(result_format)

for matching_rec in driver.query_records(config, start='2000-01-01', end='2001-01-01'):
    # do more stuff

Justification

If we refactor out all of the result IO from the CLI scripts, we'll make testing much easier and probably reduce the overall amount of code. Refactoring out just the NumPy format probably won't take too much time and would set us up to easily transition to a better file format.

Feature request: it would be very useful to be able to save the results of different classifications in the same result files for each of the scenes. For example, after modifying the training data iteratively and running three subsequent classifications, each of the resulting classes (and the probability, if enabled) would be appended to the result file.