I'm using this documentation almost all the time by the way: https://lazka.github.io/pgi-docs/

Stats as of 06. August:

+2133 -244 from tests

+493 -187 from the batch mode

+2246 -2030 for the rest

Changes

Features

• faster scanning of directories
• more precise calculation of the remaining time
• aligned some things in the settings dialog
• added filename and line number to --debug logs
• in batch mode, abr, cbr and vbr can be specified like -f mp3 -m abr.
• batch mode logs errors to the console if the cli args are used wrong
• batch mode supports path patterns like -p {artist}/{album}/{title}
• batch mode can select whether to skip, overwrite or increment files when the output already exists
• the invalid files dialog shows more of the path of each file than just the filename and has more beautiful paddings
• batch mode supports arbitrary bitrates that are not available in the ui
• info logs are colored as well with --debug

Bugs

• faac did not respect bitrate settings
• soundfile objects from the discovery step are kept for conversion, hence fixing the mp4mux tag message problem because tags are already there https://github.com/kassoulet/soundconverter/pull/36

Quality

• more unittests. went from 36 to 105 tests and from 79 to 502 assertions.
• went from a pylint rating of 7.49 to 8.83
• fixed confusion between self.basename and basename in get_target_name
• avoiding redundancy of storing stuff in settings and the gio settings at the same time
• reduced confusion between various basename, folder and pattern variables in TargetNameGenerator
• added many docstrings, especially to new functions
• no more confusion between functions like done vs finished, __init__ vs init
• removed unnecessary forwarding of Gio settings parameters, instead querying them exactly when they are needed, (some of them still need to be stored in the converter class though to avoid them getting changed during conversion by interacting with the ui) thus reducing function header sizes and complexity
• batch mode uses Gio settings (without writing them persistently) in order to be more compatible to the rest of the code
• tasks have less inheritance. Converter only inherits from task
• taskqueue doesn't need any inheritance anymore
• no more event emitting, but rather hardcoded calls to needed functions when they are needed, because the converter class doesn't need to cover for various use cases that require different code to be executed on events. Reduces complexity
• moved code that can be generalized out of ui.py, so that it can be potentially used by the batch mode.
• removed unused imports, variables, functions and classes
• I'm pretty sure ftp uris in the tests were wrong and should start with ftp:// (see https://en.wikipedia.org/wiki/Uniform_Resource_Identifier#Generic_syntax)
• sound_files only accept uris as parameters, selected_folder may only be an uri as well, and patterns may not be uris, which means the target name code can focus on a better defined set of rules and work well within that.
• using discover_uri for tagreader and typefinder, which is faster and less complex. Multithreading is used to further increase the performance of the discoverers (which, however, added some complexity back)
• added some tags to c.mp3 for tests
• saving some redundancy of progress queries
• added screenshot of the batch mode to the readme
• moved logic of moving temporary files to final files to the converter class.

I added some some subfolder patterns that i think are useful. And I also properly renamed the old ones which otherwise seems to use the artist tag, but instead they use the album-artist tag

I tried once to convert a file on a device having not enough place, with the "delete original files" option checked. The converted file has not been created because of the missing place, but the original file has been deleted.

I set self.error in Converter.on_error, so the test "if self.delete_original and self.processing and not self.error:" properly fails in Converter.finished.

Please let me know if it is the good way to achieve this.

There were two discoverer tasks, and one of them received only non-audio files, so it was removed from queue.running right away.

No idea why this is suddenly happening, maybe gstreamers behaviour changed a little bit with an update

• threaded conversion in batch mode
• reduced number of prints with -q
• -t option: when -q is provided don't print nothing at all, rather just hide errors and files without tags because otherwise the combination of -t and -q makes no sense at all
• removed info about which gtk version is used and such. Not relevant to the user
• fixed an error from one of my recent commits
• added -c mode, doing what the gtk interface does when loading files. showing which ones cannot be converted.
• the CliProgress class accepts multiple arguments like print to create messages now: show("a", "b")
• nicer console output

example output of -b (-q disables all output):

SoundConverter 3.0.1

checking files and walking dirs in the specified paths...
Queue start: 36 tasks, 4 thread(s).
Queue done in 0.052s (36 tasks)

preparing converters...
overwriting '/funky_files/converted/original/audio 2.ogg'
skipping 'not_music.ogg': invalid soundfile
skipping 'not_music.flac': invalid soundfile
(...)
overwriting '/funky_files/converted/original/foo/audio 4.ogg'
overwriting '/funky_files/converted/original/foo/bar/audio 5.ogg'
overwriting '/funky_files/converted/original/foo/bar/audio 6.ogg'

starting conversion...
Queue start: 6 tasks, 4 thread(s).
1/6: '/funky_files/original/audio 2.flac'
2/6: '/funky_files/original/audio 1.flac'
3/6: '/funky_files/original/foo/audio 3.flac'
4/6: '/funky_files/original/foo/audio 4.flac'
5/6: '/funky_files/original/foo/bar/audio 5.flac'
6/6: '/funky_files/original/foo/bar/audio 6.opus'
Queue done in 12.346s (6 tasks)

example output of -c (-q just prints a naked list of file paths):

SoundConverter 3.0.1
Queue start: 36 tasks, 4 thread(s).
Queue done in 0.052s (36 tasks)

Non-Audio Files:
/funky_files/original/not_music.ogg
/funky_files/original/not_music.flac
(...)
/funky_files/original/pictures/pic (another copy).jpg

1. gtk tool, add folder and broken/non audiofiles

When I put a text file into the sound source directory and renamed it to foobar.flac, no other flac files from other subdirectories were added if this textfile was the first .flac file seen. This was caused, because the TypeFinder only looked at one single sample of .flac files to detect the mime type. Hence the .flac type was not added to the list of accepted file extensions.

This change checks the mime type of all the sound files instead in order to find out which ones are broken. After this fix, all the .flac files are added to the list for conversion except those that don't contain sound (those for which gstreamer throws errors)

This increases the time to read in large folders, but it is the way for integrity, which is the first priority when handling files. Also, reading in the files is still fast compared to the conversion. Because of this longer required time for reading in files, the statusbar now also shows the progress of reading in the files.

An Error message appears to show how many files (only those with audio extensions) failed. The ErrorDialog apperance was also fixed.

old:

new:

After reading this message, the user can decide whether or not to start the conversion.

2. Cleanup Changelog

• added some docstrings
• removed a dead path from .gitignore
• removed 7 year old TODO file in a foreign language. If those TODOs are still relevant, can you please create issues in github in english?
• added building and installation information to readme
• used vulture and removed some unused variables and imports (after checking if it really is unused)
• added missing whitespaces after commas in lists and functions
• formatted docstring whitespaces into a uniform format across all python files
• removed all trailing whitespaces in python files

3. soundconverter.py

• moved imports and such things to the top, dividing it from the mode, command line arg and settings handling. It is considered better practice to have them on the top.
• if from soundconverter.ui import gui_main fails, it would switch to batch mode. However, this would start conversion immediatelly, either with no command line arguments at all or some files, which would convert all the files into their folders without further user configuration. This is sudden unexpected behaviour and therefore I would rather go a different way. The import of gui_main will fail if there is an e.g. syntax error in that file. So the proper way would be not make sure gui_main doesn't throw errors upon importing. I moved this import to the top.
• grouped batch options in --help using an OptionParser group. Also some modifications to the help output for batch parameters.

I was having trouble with the command line interface, mostly due to missing features.

So I set up some directories containing files that should be converted with a subdirectory structure and worked on the soundconverter cli until I was able to convert them into a target directory.

added -i, -r, -Q and -o options:

• -r goes into subdirectories
• -i ignores files that have already been converted
• -o specifies an output directory in which the original file structure is restored
• -Q takes values between 0 and 5 and indicates the quality of the conversion

furthermore:

• removed --help-gst from help since it is not used
• added example usage to --help
• reduced redundancy of quality numbers in ui.py with get_quality in settings.py

in .gitignore was a path "src/soundconverter". This is something old and can be removed, right?

This fixes the following runtime warning(s):

/app/lib/python3.8/site-packages/soundconverter/interface/ui.py:1443: Warning: g_value_type_transformable: assertion 'src_type' failed
  builder.add_from_file(gladefile)

(soundconverter:2): Gtk-WARNING **: 18:11:39.109: ../gtk/gtkliststore.c:834: Unable to convert from (null) to gchararray

• using setup.py instead of make to reduce complexity, similar to https://github.com/bluesabre/menulibre
• moved from .rst to .md readme
• removed unused files
• made readme more beautiful, added screenshot and logo, removed some redundancy, condensed it
• added spec that checks if '#' is getting quoted https://github.com/kassoulet/soundconverter/pull/31
• fixed permission denied when %(genre)s/%(title)s results in a path in root '/...' after os.path.join
• established cleaner folder structure
• using githubs default python .gitignore

note, the branch name mentions ffmpeg but nothing in that direction has been done yet in this PR. I'm thinking about making a base class that gstreamer.py and future ffmpeg.py inherit, so that both can be used interchangeably. Some people have issues with gstreamer, so this might make soundconverter work for them as long as one of the two works. A dropdown or radio button will then be added to the settings menu to select gstreamer or ffmpeg.

• uses the logging module now, since that is what many python modules use - makes the code possibly more familiar - and it allows to log to files and such if that is required at some point.
• correctly forwards --gst- arguments. For example soundconverter --gst-debug-level=3. See https://gstreamer.freedesktop.org/documentation/application-development/appendix/checklist-element.html?gi-language=python#debugging and gst-launch-1.0 --help-gst
• does not modify the users setting during make test using a memory backend (https://github.com/kassoulet/soundconverter/pull/30#issuecomment-485700568). The way how the code gets the Gio.Settings object had to be changed a bit for this.
• moved from '%s' % 'string' to the newer '{}'.format('string') formatting almost everywhere
• clarified differences between argv and gio settings
• put audio-format related code from settings into separate file
• splitted test file into multiple files
• fixed changing the number of jobs required a restart
• fixed deprecation warnings for Gtk.Dialog
• rearranged readme a bit
• limit-jobs is treated as boolean now instead of int

example showing gstreamers debug messages and our new colored debug logs. The lines saying "WARN" in yellow are gstreamers debug messages.

gstreamer really logs a lot of things with increasing verbosity (up to 9). I removed the debug logs for the message types at some point because they were actually a bit too much, so the screenshot is a bit outdated.

The basefolder is not quoted as it should be. If the converted file is within a subdirectory containing for example a # sign, soundconverter would hang.

More precise remaining time calculation by replicating how soundfiles are distributed among processes. It also calculates a correction factor based on historic calculations and actual outcomes. The leftover problems are, that the decoding speed of the source files depends on the format. If it werent for the uncertain encoding speed due to various settings, the only way to avoid that would be to generate a lookup table of type -> bitrate -> relative speed, query the closest value and multiplicate that on the second return value of get_progress in converter.py. Even worse, the conversion speed for files slows down over time, and I need to fit a linear function to it and do some integration and inversion stuff, and I can't do that quickly and need to properly verify what the hell I'm doing since it has been a few years since I learned that.

The progressbar doesn't reflect how many seconds of data have been converted anymore, but rather how much of the time has passed. As long as all processes are working, this is almost identical. This avoids having an almost full progressbar when a single process is still working on a large file that takes a long time to finish. Since this calculation happens only once per second, it is wrapped into a class that interpolates smoothly to the next value.

Created multiple files from ui.py, because that file was very long and contained multiple classes