Summary

The 'tools' system from 351ELEC (es_systems.cfg) contains a unicode character for a 'wrench'. In the web UI, the wrench is displayed as a 'missing glyph' (box).

edit: Thanks to @benphelps for simplifying my way more complex solution. The fix is just to use CSS to have anything in the body fallback to FontAwesome for missing glyphs. The end result is the wrench in tools displays properly.

Summary

Implementing full support for 'folder based' roms is a large amount of work. This change should be thought of as 'fixing bugs related to folder based roms' (as opposed to 'full support of folder based roms')

Limitations of games which are folders:

• Size of folder will not be displayed
• Download link will not be allowed/displayed
• Upload will not be allowed (a folder must be selected for upload - this is technically not a change - just a statement of what's supported)

Editing of metadata, etc, should work as expected

Images

• Added laserdisc logo

Summary

System logos were pulled in ad hoc during development of the Web UI. Soon - we will be switching to the es-theme-art-boot-next theme (https://github.com/anthonycaccese/es-theme-art-book-next) across all 351ELEC devices. Ideally (at least default) ES system logos should be synced up with the web UI.

This PR adds all system logos from artbook-next. It uses a small conversion process due to some issues described below in loading directly from the artbook next installation within 351ELEC.

Issue Using Artbook SVG logos directly

Ideally, the web UI could actually load the logos directly from the artbook installation within 351ELEC and not duplicate svgs, however, there appears to be an issue. For some reason (I don't claim to be an svg expert), many of the existing artbook logos don't work properly when using in the browser with the web ui. Ex:

This is what it should look like:

Conversion Process

Manual Fix In inkscape - you can manually fix by: File->Document Properties->Resize Page To Content -> press Resize Page to drawing or selection and then save the svg file.

Automated Fix - Instead of trying to convert one by one, I came up with a very rough automated conversion process to bulk convert from artbook next using inkscape on linux. This is mostly for my reference or if anyone else wants to bulk fix these images for webui consumption.

1. Tell inkspace to FitCanvasToDrawing for all images (https://github.com/anthonycaccese/es-theme-art-book-next/tree/main/_inc/images/systems/logos) you wish to convert. for img in $(ls *.svg) ; do inkscape --verb=FitCanvasToDrawing --verb=FileSave --verb=FileQuit $img ; done

2. Use availble systems in es_systems to copy just the needed .svg files: cat /storage/.emulationstation/es_systems.cfg | grep -i "<name>" | sed "s/<name>//g" | sed "s|</name>|.svg|g" | awk '{$1=$1};1' | xargs -I '{}' mv '{}' ./all/

Issues Fixed

• tools and scummvm systems show 404
• scummvm and tools have images in a non-standard location and also gives 404
• Dates on certain tools scripts have dates which issue a 500
• Roms in gamelist.xml that don't exist on the file system are shown (and they really shouldn't show up)
• Rom not found with non-ascii file names (causing 'duplicates' before previous issue with hiding files not found in gamelist fixed)
• 'Editing' a rom that's not in gamedata gives 500. entry wouldn't work as it didn't create a new UUID. (fix is to do the same as 'upload' and create a new XML element with UUID)

Root Cause and Logic to Address Issues

• Tools and scummvm have non-standard locations for their ‘roms_folder’.
  • Incorrect assumption in code: The main roms folder (/storage/roms) + system name can be used to find the roms directory for a given system
  • Fix logic: look up system folder in es_systems.cfg. NOTE: a simplification possible by using this lookup - can use 'name' of system to identify it (no need for 'folder')
• Tools and scummvm have non-standard locations for their ‘images’
  • Incorrect assumption in code: The system roms folder + “images/”+ image name can be used to find an image
  • Fix logic: use path from gamedata.xml and don’t derive images path
    • Also fix upload to create 'images' folder if it does not exist (needed for scummvm) - Fix image logic to not issue 500 error on file size if file doesn't exit (gamedata.xml references missing file). I just ran into this fixing image upload and seemed like a good idea.
• Tools provides dates like ‘2020’ which are not parsed
  • Fix logic: if a date cannot be parsed for formatting, just display it.
• Rom not found with non-ascii file. It appears that the text cleanup processing with ftfy fix_text was causing the utf-8 normalization differences with the rom name. This post-processing didn't matter for the cleaned up text per say (it looks fine), but when using the file it appeared to not exist unless it's an exact binary match.
• Fix logic avoid calling the find_normalized methods on the rom name.

Fix Overview

Most of the above fixes are simple to implement. However, getting the system roms directory from es_systems.cfg is not always obvious as it is often needed on a per game basis in contexts where es_systems.cfg has not been parsed. Parsing it for each game (for example) would be a huge overhead.

The proposed solution is to lazily cache the systems.cfg object in memory and use it across all APIs. This is the most controversial part of this PR as there is no existing pattern to follow for this in the code. A few thoughts as to why this is ok:

• the file is read only and the in memory object is not updated so it should be ok to use concurrently.
• In the event the server were to be multithreaded, each thread would just cache es_systems.cfg so it should work without additional logic.

NOTE on memory: Caching the es_systems.cfg object does potentially use a bit more memory, but it should be relatively small. If this were to be an issue, parsing and keeping a dictionary with only the system paths would be a good option.

Testing

• Ensure tools system does not display a 404
  • Ensure each script inside tools shows up w/o a 404/500.
  • Ensure that the 'images' for tools show up properly in UI
  • Ensure that scripts from tools can be launched with 'launch' button.
  • Ensure that 'Release Date' shows up on 351Files as 2021
• Ensure that screenshots system does not display a 404 (might make sense to hide/enhance in the future)
• Ensure that scummvm system does not display a 404
  • Ensure scummvm game can be launched via launch button.
• Ensure existing systems which have been scraped work properly
  • Ensure launch works
  • Ensure download rom works
  • Ensure download save state works
  • Ensure screen cap of save state works
  • Ensure edit rom works
  • Ensure upload rom works

Quick memory usage check:

Nothing fancy - just check total memory used reported by top before and after killing python server.

• Without fixes after clicking around in systems/games/edit/etc: 210.4 - 194.4 = 16MB
• With fixes after clicking around into systems/games/edit/etc, etc: 209.5 - 194.0 = 15.5MB
• I kind of expect that memory usage isn't actually better and my testing was just not exact enough, but it at least shows that memory usage isn't way more or anything.

Will enable building as part of the 351ELEC build system in the future.

Feel free to just copy/paste/edit this and add yourself (I don't need to be credited for this 'contribution' - I just was trying to test a working build)

The only thing that matters to 351ELEC build system is that you put any dependencies in: install_requires (ftfy)