@maliheha and I are working on this on my fork. These updates aims to separate the parsers for different sections of the log into submodules so things are easier to unit-test and modify as we go.

@mattmilten no need to review for now, we'll let you know when it's in a more complete state.

@ronaldvdv we noticed you're also adding some tests and example data - can we fold this into the refactored code? Just want to make sure we aren't duplicating work and writing tests that may eventually clash with one another. Happy to discuss and coordinate together.

For some set of log files, I'm seeing this error:

UnboundLocalError: local variable 'tree_search_log' referenced before assignment

I believe this relates to this line which fails if the file does not contain the data tested for in the if-statement above.

When extracting statistics about generated cuts, the pattern for matching cut names is too restrictive. It does not match Relax-and-lift which is generated for 912-glass4-0.log. I believe the hyphen is not included in the [\w ] pattern for the cut name.

This is a small PR to add the header parser. Specifically:

• The module header_parser.py is added to the src/grblogtools folder. Reading the current code conveyed to me that there is the underlying assumption that all header log information is captured in one line. I used the same assumption. Please let me know if this is not the right assumption and I am overlooking something.
• The module test_header_parser.py is added. It includes just two tests and running pytest from the parent folder picks up the tests.
• The current grblogtools.py module is not refactored to reflect these changes. The plan is to do the presolve parser in the next PR (hopefully by the end of this week) and then see how the main code can change to include these changes.

• Implemented filling of default parameter values where they are not shown in the logs
• Added script to generate parameter defaults using gurobipy and committed these (avoids having gurobipy as a dependency or a license requirement)
• Implemented a prettyparams keyword arg for get_dataframe which converts some parameters to categorical descriptions

• Adds another timeline dataframe for NoRel incumbent/bound/timing information. Note that this currently just captures NoRel's measure of elapsed time, which starts from the start of NoRel, not the start of the solve.
• Changes the get_dataframe return signature to:

summary, timelines = glt.get_dataframe(["run1/*.log", "run2/*.log"], timelines=True)

where timelines is a dict with three dataframes: nodelog, rootlp, and norel. This seemed cleaner than extending the output tuple again, but am happy to change.

• Adds a simple test and tox config for testing compatible python versions.

Implements https://github.com/Gurobi/grblogtools/issues/27 for nodelog, barrier, and simplex. Tests seem to pass but a pre-commit hook is failing (I think)...

For example, in MIP logs we parse this line but don't collect the work units info

Explored 1466 nodes (267947 simplex iterations) in 12.83 seconds (18.68 work units)

If the glob pattern matches no files (which can easily happen due to a typo, or being in the wrong directory), we get this unhelpful error during post-processing:

>>> import grblogtools as glt
>>> glt.parse("asdjkhaskd")
<grblogtools.api.ParseResult object at 0x1023f75b0>
>>> glt.parse("asdjkhaskd").summary()
Traceback (most recent call last):
  File "/Users/bowly/.pyenv/versions/3.10.3/lib/python3.10/site-packages/pandas/core/indexes/base.py", line 3800, in get_loc
    return self._engine.get_loc(casted_key)
  File "pandas/_libs/index.pyx", line 138, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/index.pyx", line 165, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/hashtable_class_helper.pxi", line 5745, in pandas._libs.hashtable.PyObjectHashTable.get_item
  File "pandas/_libs/hashtable_class_helper.pxi", line 5753, in pandas._libs.hashtable.PyObjectHashTable.get_item
KeyError: 'Version'

We should throw something more useful, like a FileNotFoundError at the parsing step if no files match.

In latest master branch, if a MIP log is incomplete (i.e. cut off with no termination message for whatever reason), we might report optimal status incorrectly. For example:

Variable types: 23522 continuous, 2343 integer (0 binary)

Root barrier log...

Barrier solved model in 50 iterations and 72.71 seconds (53.24 work units)
Optimal objective -1.76339641e+08

Solved with barrier

Root relaxation: objective -1.763396e+08, 104343 iterations, 108.23 seconds (79.42 work units)

    Nodes    |    Current Node    |     Objective Bounds      |     Work
 Expl Unexpl |  Obj  Depth IntInf | Incumbent    BestBd   Gap | It/Node Time

here we get 'OPTIMAL' status from ContinuousParser, but no termination message from NodeLogParser.

grblogtoolsv1 would give an 'incomplete log' warning in this situation (and report unknown status? I'm not sure).

We should check for this with some custom logic for Status and Runtime, something like:

• If the model is continuous, we can get Runtime and Status from ContinuousParser
• If the model is (a) a MIP or (b) a continuous model solved as a MIP, we should ignore Runtime and Status from ContinuousParser
  • (a) We can check using model type in SingleLogParser
  • (b) Look for the message Solving as a MIP in header or presolve
• If TerminationParser reports runtime or status, it should take precedence (this already happens)

Adds the run_label function in the postprocessing module which builds a label based on version and changed parameters.

Example usage:

import grblogtools as glt
from grblogtools.postprocessing import run_label
summary = glt.get_dataframe(["data/*.log"])
run_label(summary)

Output is a Series, the same as current "Log" column for the glass4 examples, but built from the parameter values so does not rely on the file naming convention.

0     912-MIPFocus2-Presolve1
1     912-Cuts2-Heuristics0.1
2     912-Cuts2-Heuristics0.1
3     912-MIPFocus2-Presolve1
4     912-Cuts2-Heuristics0.1
               ...           
59    912-MIPFocus1-Presolve2
60                  912-Cuts1
61          912-Heuristics0.1
62          912-Heuristics0.0
63    912-Cuts2-Heuristics0.0
Length: 64, dtype: object