These scripts look for non-printable unicode characters in all text files in a source tree

Siddhesh Poyarekar

Last update: Aug 30, 2022

Related tags

General Utilities find-unicode-control

Overview

find-unicode-control

These scripts look for non-printable unicode characters in all text files in a source tree. find_unicode_control.py should work with python2 as well as python3. It uses python-magic if available to determine file type, or simply spawns the file --mime-type command. They should be functionally the same and find_unicode_control.py could eventually get disposed.

usage: find_unicode_control.py [-h] [-p {all,bidi}] [-v] [-c CONFIG] path [path ...]

Look for Unicode control characters

positional arguments:
  path                  Sources to analyze

optional arguments:
  -h, --help            show this help message and exit
  -p {all,bidi}, --nonprint {all,bidi}
                        Look for either all non-printable unicode characters or bidirectional control characters.
  -v, --verbose         Verbose mode.
  -d, --detailed        Print line numbers where characters occur.
  -t, --notests         Exclude tests (basically test.* as a component of path).
  -c CONFIG, --config CONFIG
                        Configuration file to read settings from.

If unicode BIDI control characters or non-printable characters are found in a file, it will print output as follows:

$ python3 find_unicode_control.py -p bidi *.c
commenting-out.c: bidirectional control characters: {'\u202e', '\u2066', '\u2069'}
early-return.c: bidirectional control characters: {'\u2067'}
stretched-string.c: bidirectional control characters: {'\u202e', '\u2066', '\u2069'}

Using the -d flag, the output is more detailed, showing line numbers in files, but this mode is also slower:

find_unicode_control.py -p bidi -d .
./commenting-out.c:4 bidirectional control characters: ['\u202e', '\u2066', '\u2069', '\u2066']
./commenting-out.c:6 bidirectional control characters: ['\u202e', '\u2066']
./early-return.c:4 bidirectional control characters: ['\u2067']
./stretched-string.c:6 bidirectional control characters: ['\u202e', '\u2066', '\u2069', '\u2066']

The optimal workflow would be to do a quick scan through a source tree and if any issues are found, do a detailed scan on only those files.

Configuration file

If files need to be excluded from the scan, make a configuration file and define a scan_exclude variable to a list of regular expressions that match the files or paths to exclude. Alternatively, add a scan_exclude_mime list with the list of mime types to ignore; this can again be a regular expression. Here is an example configuration that glibc uses:

scan_exclude = [
        # Iconv test data
        r'/iconvdata/testdata/',
        # Test case data
        r'libio/tst-widetext.input$',
        # Test script.  This is to silence the warning:
        # 'utf-8' codec can't decode byte 0xe9 in position 2118: invalid continuation byte
        # since the script tests mixed encoding characters.
        r'localedata/tst-langinfo.sh$']

Notes

This script was quickly hacked together to scan repositories with mostly LTR, unicode content. If you have RTL content (either in comments, literals or even identifiers in code), it will give false warnings that you need to weed out. For now you need to exclude such RTL code using scan_exclude but a long term wish list (if this remains relevant, hopefully more sophisticated RTL diagnostics will make it obsolete!) is to handle RTL a bit more intelligently.

Comments

Script fails if there are non-printable unicode characters

The script fails if there are non-printable unicode characters.

[There is no non-printable unicode characters]

$ cat unicode_control_good_1.cs                                                    
string access_level = "user";
if (access_level != "user") //Check if admin
{    
  Console.WriteLine("You are an admin."); 
}

$ python3 find_unicode_control.py unicode_control_good_1.cs 
find_unicode_control.py:16: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
  import sys, os, argparse, re, unicodedata, subprocess, imp

$ echo $?
0

[There are non-printable unicode characters]

$ cat unicode_control_bad_1.cs 
string access_level = "user";
if (access_level != "user"⁦) //Check if admin 
{    
  Console.WriteLine("You are an admin."); 
}

$ python3 find_unicode_control.py unicode_control_bad_1.cs 
find_unicode_control.py:16: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
  import sys, os, argparse, re, unicodedata, subprocess, imp
unicode_control_bad_1.cs: unicode control characters: {'\u2066'}

$ echo $?
1

opened by massongit 1

$scan_exclude is a bit too aggressive with path contains {.hg,.git}$

scan_exclude is a bit too aggressive with path contains {.hg,.git}

For example:

$ mkdir firefox.hg 
$ touch firefox.hg/foo.cpp
$ find_unicode_control.py -v firefox.hg/foo.cpp
firefox.hg/foo.cpp: Reading file
firefox.hg/foo.cpp: SKIPPED

because it contains .hg in the path

opened by sylvestre 1

Fix settings file loading with Python 2

Using importlib.util.spec_from_file_location to load the external settings file doesn't work with Python < 3.4.0. Falling back to imp.load_source fixes the issue.

opened by nexjhealth 1
Adds batch script for testing branches and tags in a git repository

I was using the find-unicode script to manually check multiple branches and tags of a number of git repositories. Therefore, I preferred to write a batch script to make the process easier.

Wondered if it would be useful to others. Probably could be proved by better bash coders but it did the job for me.

opened by phantomjinx 1
Don't abort when os.stat fails

Currently the script aborts execution if os.stat() fails for some reason, for instance, if it tries to read a symlink that points to an inexistent file.

opened by jwendell 1