f90nml - A Fortran namelist parser, generator, and editor

Overview

f90nml - A Fortran namelist parser, generator, and editor

A Python module and command line tool for parsing Fortran namelist files

https://travis-ci.org/marshallward/f90nml.svg?branch=master https://ci.appveyor.com/api/projects/status/bcugyoqxiyyvemy8?svg=true https://coveralls.io/repos/marshallward/f90nml/badge.svg?branch=master

Documentation

The complete documentation for f90nml is available from Read The Docs.

http://f90nml.readthedocs.org/en/latest/

About f90nml

f90nml is a Python module and command line tool that provides a simple interface for the reading, writing, and modifying Fortran namelist files.

A namelist file is parsed and converted into an Namelist object, which behaves like a standard Python dict. Values are converted from Fortran data types to equivalent primitive Python types.

The command line tool f90nml can be used to modify individual values inside of a shell environment. It can also be used to convert the data between namelists and other configuration formats. JSON and YAML formats are currently supported.

Quick usage guide

To read a namelist file sample.nml which contains the following namelists:

&config_nml
   input = 'wind.nc'
   steps = 864
   layout = 8, 16
   visc = 1.0e-4
   use_biharmonic = .false.
/

we would use the following script:

import f90nml
nml = f90nml.read('sample.nml')

which would would point nml to the following dict:

nml = {
    'config_nml': {
        'input': 'wind.nc',
        'steps': 864,
        'layout': [8, 16],
        'visc': 0.0001,
        'use_biharmonic': False
    }
}

File objects can also be used as inputs:

with open('sample.nml') as nml_file:
    nml = f90nml.read(nml_file)

To modify one of the values, say steps, and save the output, just manipulate the nml contents and write to disk using the write function:

nml['config_nml']['steps'] = 432
nml.write('new_sample.nml')

Namelists can also be saved to file objects:

with open('target.nml') as nml_file:
   nml.write(nml_file)

To modify a namelist but preserve its comments and formatting, create a namelist patch and apply it to a target file using the patch function:

patch_nml = {'config_nml': {'visc': 1e-6}}
f90nml.patch('sample.nml', patch_nml, 'new_sample.nml')

Command line interface

A command line tool is provided to manipulate namelist files within the shell:

$ f90nml config.nml -g config_nml -v steps=432
&config_nml
   input = 'wind.nc'
   steps = 432
   layout = 8, 16
   visc = 1.0e-4
   use_biharmonic = .false.
/

See the documentation for details.

Installation

f90nml is available on PyPI and can be installed via pip:

$ pip install f90nml

The latest version of f90nml can be installed from source:

$ git clone https://github.com/marshallward/f90nml.git
$ cd f90nml
$ pip install .

Package distribution

f90nml is not distributed through any official packaging tools, but it is available on Arch Linux via the AUR:

$ git clone https://aur.archlinux.org/python-f90nml.git
$ cd python-f90nml
$ makepkg -sri

Volunteers are welcome to submit and maintain f90nml on other distributions.

Local install

Users without install privileges can append the --user flag to pip from the top f90nml directory:

$ pip install --user .

If pip is not available, then setup.py can still be used:

$ python setup.py install --user

When using setup.py locally, some users have reported that --prefix= may need to be appended to the command:

$ python setup.py install --user --prefix=

YAML support

The command line tool offers support for conversion between namelists and YAML formatted output. If PyYAML is already installed, then no other steps are required. To require YAML support, install the yaml extras package:

$ pip install f90nml[yaml]

To install as a user:

$ pip install --user .[yaml]

Contributing to f90nml

Users are welcome to submit bug reports, feature requests, and code contributions to this project through GitHub. More information is available in the Contributing guidelines.

Comments
  • Add file arguments

    Add file arguments

    Some API questions are how to handle force= and the default output name for patching; I decided to ignore force for files. For the latter issue, I raise ValueError when nml_fname is a file since there is no sane default.

    (personally, if it was my library, I would rip out the default name functionality entirely, and make it so that when there is no output file or path, it will still build and return the patched Namelist object without writing a patched namelist file. But that's just me)

    Addresses #25 (although there is no string API here, at least yet)

    opened by ExpHP 25
  • Order of repeated blocks not preserved when mixed with other named blocks

    Order of repeated blocks not preserved when mixed with other named blocks

    File file contains

    &block00
    i=1
    k=2
    &end
    &block01
    m=0
    n=1
    &end
    &block00
    j=3
    z=4
    &end
    

    The statement f90nml.write(f90nml.read(file),outfile) will result in outfile containing

    &block00
    i=1
    k=2
    /
    &block00
    j=3
    z=4
    /
    &block01
    m=0
    n=1
    /
    

    where order of namelist blocks has not been preserved.

    opened by jpf-x 24
  • Strings starting with + are parsed as a list

    Strings starting with + are parsed as a list

    I have a file formatted this way

    &iostuff
    output_folder = +out
    /
    

    When parsed, ['iostuff']['output_folder'] equals ['+', 'out']. I guess this is done so that if the variable out is defined somewhere else, this list is replaced by the corresponding numerical value. But if it isn't defined, the string '+out' should be returned instead. Maybe a simple solution would be having a way to tell f90nml that some variables are strings and should be considered as such.

    opened by amorison 24
  • Namelist with repeated groups doesn't convert to string

    Namelist with repeated groups doesn't convert to string

    Trying to convert a namelist with repeated groups to a string seems to fail. If I do the following:

    >>> import f90nml
    >>> nml = f90nml.reads("""
    ... &foo
    ...  a = 1
    ... /
    ... &foo
    ...  b = 2
    ... /
    ... """)
    >>> str(nml)
    "Namelist([('foo', [Namelist([('a', 1)]), Namelist([('b', 2)])])])"
    

    I was expecting str(nml) to return "&foo\n a = 1\n/\n&foo\n b = 2\n/\n", as is the case when the group names do not repeat. Interestingly, if I use nml.write('out.nml') the the output file is formatted correctly.

    opened by chunderbunny 21
  • Write bug in v0.20

    Write bug in v0.20

    I'm getting an odd bug when trying the write the following namelist:

     &a
     b%c_d_E(1)%ID = 1,
     b%c_d_E(2)%ID = 2,
     /
    

    Code is:

        import f90nml
        data = f90nml.read('test.nml')
        with open('test_RESAVED.nml','w') as f:
            data.write(f, force=True)
    

    Error message is:

    Traceback (most recent call last):
      File "test.py", line 309, in <module>
        data.write(f, force=True)
      File "C:\Anaconda3\lib\site-packages\f90nml\namelist.py", line 232, in write
        self.write_nmlgrp(grp_name, grp_vars, nml_file)
      File "C:\Anaconda3\lib\site-packages\f90nml\namelist.py", line 253, in write_nmlgrp
        for v_str in self.var_strings(v_name, v_val, v_start=v_start):
      File "C:\Anaconda3\lib\site-packages\f90nml\namelist.py", line 302, in var_strings
        v_strs = self.var_strings(v_title, f_vals, v_start=v_start_new)
      File "C:\Anaconda3\lib\site-packages\f90nml\namelist.py", line 313, in var_strings
        i_s = v_start[::-1][len(v_idx)]
    TypeError: 'NoneType' object is not subscriptable
    

    Oddly, when I change c_d_E to c_d_e, it works fine.

    I just updated to the latest release using pip. I've been able to read variables like this before, so something has changed.

    opened by jacobwilliams 20
  • Add a dependency on the yaml module

    Add a dependency on the yaml module

    Trying to use f90nml after installing on macOS via pip3 install f90nml with the -f yaml flag results in:

    f90nml: error: YAML module could not be found.
    

    I think you should just add an explicit dependency on pyyaml. Or, better yet, if there's a way to add a recommended (installed by default) dependency on pyyaml that someone could opt out of. (I'm not familiar enough with python packaging to know for certain if this is possible one way or the other.)

    opened by zbeekman 19
  • Change to accept NumPy arrays and similar objects

    Change to accept NumPy arrays and similar objects

    Hi there! We've been using f90nml in Pyrokinetics to read/write input files to plasma physics codes, and we ran into an issue where NumPy 0D arrays would propagate unnoticed into a Namelist and cause an exception to be thrown on calling Namelist.write. I've added a small change to Namelist._f90repr to detect 0D arrays and convert them to their equivalent primitive type. Nothing changes if NumPy is not installed.

    Let me know if you'd like me to add further test cases, or if you'd prefer to avoid including these sorts of edge cases in your library.

    opened by LiamPattinson 18
  • Optimizing performance

    Optimizing performance

    I was wondering if there were possibilities for optimizing performance for very large namelist files. I'm in the process of doing some benchmarks for some large files (say around 1500 lines, with multiple namelists and pretty much all variable types). I can only get around 1.7 calls of f90nml.read() per second. I assume that it's the parsing and/or creating of the structures that is the bottleneck (actually reading in the lines should only take a fraction of a second), but I'm going to investigate further.

    One thing I was wondering if parsing of different namelists in the same file could take place in parallel? I don't know if such a thing is possible or not, but if it is that might be something to explore. Maybe this is something I can try and contribute to, rather than just reporting bugs and asking for features!

    enhancement 
    opened by jacobwilliams 17
  • repetition of null value bug

    repetition of null value bug

    Hi Marshall, Thanks for this great tool!

    The bug occurs for the following variable input line: AXFCLN = 3.0, 3.0, 48*, The 48* denotes a repetition of 48 null values Upon .write(), it is being turned into:
    AXFCLN = 3.0, 3.0, ','*,

    One clue is that everything is fine (output = input) if the trailing comma is removed prior to processing.

    Processing attached files as follows: nml=f90nml.read('repetition_of_null_win.nml') f90nml.patch('repetition_of_null_win.nml', nml, 'patched_data_win.nml') and likewise for the unix line-endings version.

    I'm using latest master commit 151438f on Oct 31, 2019 (ver 1.1.2) My sys.version is: 3.7.5 (tags/v3.7.5:5c02a39a0b, Oct 15 2019, 00:11:34) [MSC v.1916 64 bit (AMD64)] I'm running Win10/64/Pro.

    patched_data_unix.txt patched_data_win.txt repetition_of_null_win.txt repetition_of_null_unix.txt

    opened by frankeye 15
  • whitespace between indices for array in namelist cannot be parsed by nag

    whitespace between indices for array in namelist cannot be parsed by nag

    When writing namelists containing nested structures that need to be indexed, e.g.

    idx_nml
       v(1, 1) = 5
       v(2, 1) = 5
    /
    

    the resulting namelist file cannot be parsed by a program compiled with nag (tested here with version 6.1) , if there is a whitespace between the indices. I was wondering, if that whitespace can be omitted?

    opened by gitporst 15
  • f90nml.read() crashes for namelists without a terminating EOL.

    f90nml.read() crashes for namelists without a terminating EOL.

    I'm trying to transform a list of configuration files into a list of f90nml.Namelist objects. StopIteration is being raised by Tokenizer.update_chars().

    For example, let's say I have two nml files, foo.nml and bar.nml, this bug can be triggered by :

    [f90nml.read(s) for s in ['foo.nml','bar.nml']]
    
    opened by neutrinoceros 14
  • patching a repeated section on a namelist returns an unclear message

    patching a repeated section on a namelist returns an unclear message

    Related to esm-tools/esm_tools#843

    Problem description

    When loading a namelist with repeated sections, and then I try to patch one or several repeated sections, I get the following error:

    AttributeError: 'Cogroup' object has no attribute 'update'
    

    I understand it is not possible to update repeated sections, as it is not clear which one f90nml should patch. However, I find the error message misleading.

    Example

    To reproduce the error you can write the following fort.4 namelist:

    &NAMDIM
        NPROMA = -8, 
    /
    
    &NAMPAR0
        LSTATS = true,
        LDETAILED_STATS = false,
        LSYNCSTATS = false,
        MP_TYPE = 2,
        MBX_SIZE = 32000000,
        NPROC = 144,      ! Number of MPI ranks (cores of no OpenMP)
        NOUTPUT = 1,
    /
    
    &NAEPHY
        LEPHYS = true,
        LERADI = true,
        LELAIV = false,
    /
    
    &NAERAD
        NRPROMA = -8, 
        CRTABLEDIR = "./rtables/",  ! Modify this for your installation, note trailing /
    /
    
    &NAMPAR0
    /
    &NAEPHY
    /
    &NAERAD
    /
    

    Then you can use the following python3 script to reproduce the error:

    import f90nml
    
    nml = f90nml.read("fort.4")
    
    changes = { 
        "namdim": {
            "nproma": 0
        }   
    }
    
    nml.patch(changes)
    
    print(nml)
    print("Changing variables in non-repeated sections works!")
    print()
    print("Here comes the problem:")
    
    changes = { 
        "nampar0": {
            "lstats": False
        },  
        "naephy": {
            "leradi": False
        },
        "naerad": {
            "nrproma": 0
        },
    }
    
    nml.patch(changes)
    

    Suggested solution

    Ideally, the user should be reported which variables are problematic in this context. I have sketched a very rough solution with the desired behaviour for these lines: https://github.com/marshallward/f90nml/blob/2a8663aa7dfc06a446fbab6aae20d928f694f3fb/f90nml/namelist.py#L649-L658

    The rough solution:

        def patch(self, nml_patch):
            """Update the namelist from another partial or full namelist.
    
            This is different from the intrinsic `update()` method, which replaces
            a namelist section.  Rather, it updates the values within a section.
            """
    +       conflicting_repeated_secs = [] 
            for sec in nml_patch:
                if sec not in self:
                    self[sec] = Namelist()
    -           self[sec].update(nml_patch[sec])
    +           try: 
    +               self[sec].update(nml_patch[sec])
    +           except AttributeError:
    +               conflicting_repeated_secs.append(sec)
    +       if conflicting_repeated_secs:
    +           raise AttributeError(
    +               "The following sections are repeated in the namelist and cannot be "
    +               f"updated: {conflicting_repeated_secs}"
    +           )  
    
    opened by mandresm 1
  • patching of fortran types only works for first occurence

    patching of fortran types only works for first occurence

    Given the namelist file setup.nml

    &setup
     test%var1 = 1
     test%var2 = 2
    /
    

    and the patch

    patch={'setup': {'test':{'var2': 3}}}
    

    the call

    f90nml.patch('setup.nml', patch, 'patched_setup.nml')
    

    does not patch anything.

    However, patching the first occurrence of test% in setup.nml works

    patch={'setup': {'test':{'var1': 3}}}
    
    opened by rainbowsend 1
  • Problem when loading a namefile

    Problem when loading a namefile

    Hi I am trying to load a namefile from the simulation code Ramses (name = namelist.txt). It works most of the time but I have a run when this fails and gives the following message (see below). This may be just that my namefile is wrongly formatted. Is there a way for me to easily read the namelist and see where the formatting may be wrong?

    Apologies for the naive question.

    Thanks for any tips that may help.

    Eric

    ====================================== [within ipython]

    nm = "namelist.txt" f90nml.read(nm)


    AssertionError                            Traceback (most recent call last)
    Input In [10], in <cell line: 1>()
    ----> 1 f90nml.read(nm)
    
    File ~/.local/lib/python3.9/site-packages/f90nml/__init__.py:31, in read(nml_path)
         13 """Parse a Fortran namelist file and return its contents.
         14 
         15 File object usage:
       (...)
         28 >>> nml = parser.read(nml_file)
         29 """
         30 parser = Parser()
    ---> 31 return parser.read(nml_path)
    
    File ~/.local/lib/python3.9/site-packages/f90nml/parser.py:281, in Parser.read(self, nml_fname, nml_patch_in, patch_fname)
        279 nml_file = open(nml_fname, 'r') if nml_is_path else nml_fname
        280 try:
    --> 281     return self._readstream(nml_file, nml_patch)
        282 except StopIteration:
        283     raise ValueError('End-of-file reached before end of namelist.')
    
    File ~/.local/lib/python3.9/site-packages/f90nml/parser.py:388, in Parser._readstream(self, nml_file, nml_patch_in)
        385 # Set the next active variable
        386 if self.token in ('=', '(', '%'):
    --> 388     v_name, v_values = self._parse_variable(
        389         g_vars,
        390         patch_nml=grp_patch
        391     )
        393     if v_name in g_vars:
        394         v_prior_values = g_vars[v_name]
    
    File ~/.local/lib/python3.9/site-packages/f90nml/parser.py:554, in Parser._parse_variable(self, parent, patch_nml)
        551 self._update_tokens()
        552 self._update_tokens()
    --> 554 v_att, v_att_vals = self._parse_variable(
        555     v_parent,
        556     patch_nml=v_patch_nml
        557 )
        559 next_value = Namelist()
        560 next_value[v_att] = v_att_vals
    
    File ~/.local/lib/python3.9/site-packages/f90nml/parser.py:566, in Parser._parse_variable(self, parent, patch_nml)
        561     self._append_value(v_values, next_value, v_idx)
        563 else:
        564     # Construct the variable array
    --> 566     assert self.token == '='
        567     n_vals = None
        569     self._update_tokens()
    
    AssertionError: 
    
    opened by emsellem 3
  • Cannot parse a namelist without the right boundary of the array

    Cannot parse a namelist without the right boundary of the array

    I have a namelist like this ( q is an eight dimensional array, q(8) ) :

    &example
    q(1) = 1, 2, 3
    q(4) = 4, 5, 6
    /
    

    Fortan can parse correctly and get q = 1, 2, 3, 4, 5, 6, but f90nml cannot parse correctly: Namelist([('example', Namelist([('q', [1, None, None, 4])]))])

    opened by coreur 18
  • Parsing error using array assignment in namelist

    Parsing error using array assignment in namelist

    One of the namelists I am trying to parse using f90nml contains an array assignment of the following syntax

    &example
      arr(1:2)%foo =   1.0,   2.0
      arr(1:2)%bar =   3.0,   4.0
    /
    

    If I try to read/parse this namelist

    import f90nml
    f90nml.reads('''
    &example
      arr(1:2)%foo =   1.0,   2.0
      arr(1:2)%bar =   3.0,   4.0
    /
    ''')
    

    I will get the following error

    Traceback (most recent call last):
      File "<pyshell#1>", line 5, in <module>
        ''')
      File "C:\Python\Python37-64\lib\site-packages\f90nml\__init__.py", line 47, in reads
        return parser.reads(nml_string)
      File "C:\Python\Python37-64\lib\site-packages\f90nml\parser.py", line 280, in reads
        return self._readstream(iter(nml_string.splitlines()))
      File "C:\Python\Python37-64\lib\site-packages\f90nml\parser.py", line 359, in _readstream
        patch_nml=grp_patch
      File "C:\Python\Python37-64\lib\site-packages\f90nml\parser.py", line 459, in _parse_variable
        assert v_idx_bounds[0][1] - v_idx_bounds[0][0] == 1
    AssertionError
    
    bug 
    opened by Cory-Kramer 4
  • Assign absolute/relative path to string variable in CLI

    Assign absolute/relative path to string variable in CLI

    It doesn't seem that the CLI supports the use of '/'.

    Using

    f90nml -v DIR='./path/to/dir' -v ODIR='/path/to/dir' input

    will make DIR have the value '.', and ODIR empty. Escaping doesn't work either.

    opened by mateusffreitas 4
Releases(v1.4.1)
Owner
Marshall Ward
Marshall Ward
cmdpxl: a totally practical command-line image editor

cmdpxl: a totally practical command-line image editor

Jieruei Chang 476 Jan 7, 2023
Quickly open any path on your terminal window in your $EDITOR of choice!

Tmux fpp Plugin wrapper around Facebook PathPicker. Quickly open any path on your terminal window in your $EDITOR of choice! Demo Dependencies fpp - F

null 257 Dec 28, 2022
🦎 A NeoVim plugin for highlighting visual selections like in a normal document editor!

?? HighStr.nvim A NeoVim plugin for highlighting visual selections like in a normal document editor! Demo TL;DR HighStr.nvim is a NeoVim plugin writte

Pocco81 222 Jan 3, 2023
Command line parser for common log format (Nginx default).

Command line parser for common log format (Nginx default).

Lucian Marin 138 Dec 19, 2022
Python3 parser for Apple's crash reports

pyCrashReport in intended for analyzing crash reports from Apple devices into a clearer view, without all the thread listing and loaded images, just the actual data you really need to debug the problem ?? .

null 7 Aug 19, 2022
ghfetch is ai customizable CLI GitHub personal README generator.

ghfetch is ai customizable CLI GitHub personal README generator. Inspired by famous fetch such as screenfetch, neofetch and ufetch, the purpose of this tool is to introduce yourself as if you were a machine.

Alessio Celentano 3 Sep 10, 2021
topalias - Linux alias generator from bash/zsh command history with statistics, written on Python.

topalias topalias - Linux alias generator from bash/zsh command history with statistics, written on Python. Features Generate short alias for popular

Sergey Chudakov 38 May 26, 2022
A CLI password generator

passgen - A CLI password generator Usage python3 main.py <arguments> Arguments Argument Short Description --length -l The length of the password to ge

null 1 Nov 13, 2021
A CLI tool for searching and watching videos on youtube with no spyware and MPV and yt-dlp

A CLI tool for searching and watching videos on youtube with no spyware and MPV and yt-dlp

TruncatedDinosour 3 Feb 22, 2022
Python wrapper and CLI utility to render LaTeX markup and equations as SVG using dvisvgm and svgo.

latex2svg Python wrapper and CLI utility to render LaTeX markup and equations as SVG using dvisvgm and svgo. Based on the original work by Tino Wagner

Matthias C. Hormann 4 Feb 18, 2022
Management commands to help backup and restore your project database and media files

Django Database Backup This Django application provides management commands to help backup and restore your project database and media files with vari

null 687 Jan 4, 2023
A python script that enables a raspberry pi sd card through the CLI and automates the process of configuring network details and ssh.

This project is one script (wpa_helper.py) written in python that will allow for the user to automate the proccess of setting up a new boot disk and configuring ssh and network settings for the pi

Theo Kirby 6 Jun 24, 2021
liquidctl – liquid cooler control Cross-platform tool and drivers for liquid coolers and other devices

Cross-platform CLI and Python drivers for AIO liquid coolers and other devices

null 1.7k Jan 8, 2023
Sink is a CLI tool that allows users to synchronize their local folders to their Google Drives. It is similar to the Git CLI and allows fast and reliable syncs with the drive.

Sink is a CLI synchronisation tool that enables a user to synchronise local system files and folders with their Google Drives. It follows a git C

Yash Thakre 16 May 29, 2022
git-partial-submodule is a command-line script for setting up and working with submodules while enabling them to use git's partial clone and sparse checkout features.

Partial Submodules for Git git-partial-submodule is a command-line script for setting up and working with submodules while enabling them to use git's

Nathan Reed 15 Sep 22, 2022
Run an FFmpeg command and see the percentage progress and ETA.

Run an FFmpeg command and see the percentage progress and ETA.

null 25 Dec 22, 2022
pyNPS - A cli Linux and Windows Nopaystation client made with python 3 and wget

Currently, all the work is being done inside the refactoring branch. pyNPS - A cli Linux and Windows Nopaystation client made with python 3 and wget P

Everton Correia 45 Dec 11, 2022
A lightweight Python module and command-line tool for generating NATO APP-6(D) compliant military symbols from both ID codes and natural language names

Python military symbols This is a lightweight Python module, including a command-line script, to generate NATO APP-6(D) compliant military symbol icon

Nick Royer 5 Dec 27, 2022