Guesslang detects the programming language of a given source code

Y. SOMDA

Last update: Dec 29, 2022

Related tags

Code Analysis python programming-language machine-learning deep-neural-networks deep-learning tensorflow classification

Overview

Guesslang

Guesslang detects the programming language of a given source code:

echo '
package main
import "fmt"

func main() {
    fmt.Println("My mascot is a gopher and Google loves me. Who am I?")
}

' | guesslang

# ⟶ Programming language: Go

Guesslang supports 54 programming languages:

Languages
`Assembly`	`Batchfile`	`C`	`C#`	`C++`
`Clojure`	`CMake`	`COBOL`	`CoffeeScript`	`CSS`
`CSV`	`Dart`	`DM`	`Dockerfile`	`Elixir`
`Erlang`	`Fortran`	`Go`	`Groovy`	`Haskell`
`HTML`	`INI`	`Java`	`JavaScript`	`JSON`
`Julia`	`Kotlin`	`Lisp`	`Lua`	`Makefile`
`Markdown`	`Matlab`	`Objective-C`	`OCaml`	`Pascal`
`Perl`	`PHP`	`PowerShell`	`Prolog`	`Python`
`R`	`Ruby`	`Rust`	`Scala`	`Shell`
`SQL`	`Swift`	`TeX`	`TOML`	`TypeScript`
`Verilog`	`Visual Basic`	`XML`	`YAML`

With a guessing accuracy higher than 90%.

Apps powered by Guesslang

Chameledit

Chameledit is a simple web-editor that automatically highlights your code.

Pasta

Pasta is a Slack bot that pretty pastes source code.

Watch the demo here

GG

GG is a silly guessing game.

Documentation

Guesslang documentation is available at https://guesslang.readthedocs.io/en/latest/
Guesslang language detection explained here https://guesslang.readthedocs.io/en/latest/contents.html#how-does-guesslang-guess
Guesslang is based on Tensorflow machine learning framework

Installation

Python 3.7+ is required
Install the latest stable version:

pip3 install guesslang

or install Guesslang from source code:

pip3 install .

Windows specific

To run Tensorflow on Microsoft Windows you need to install Visual C++ runtime libraries, available on Microsoft website

Guesslang command line

Show all available options

guesslang --help

Detect the programming language of /etc/bashrc configuration file:

guesslang /etc/bashrc

# ⟶ Programming language: Shell

Detect the programming language of a given text:

echo '
/** Turn command line arguments to uppercase */
object Main {
  def main(args: Array[String]) {
    val res = for (a <- args) yield a.toUpperCase
    println("Arguments: " + res.toString)
  }
}
' | guesslang

# ⟶ Programming language: Scala

Show the detection probabilities for a given source code:

= pivot] return qsort(less) + [pivot] + qsort(more) if __name__ == '__main__': items = [1, 4, 2, 7, 9, 3] print(f'Sorted: {qsort(items)}') " | guesslang --probabilities # Language name Probability # Python 74.80% # Haskell 6.73% # CoffeeScript 5.32% # Groovy 1.95% # Markdown 0.93% # ... ">

echo "
def qsort(items):
    if not items:
        return []
    else:
        pivot = items[0]
        less = [x for x in items if x <  pivot]
        more = [x for x in items[1:] if x >= pivot]
        return qsort(less) + [pivot] + qsort(more)


if __name__ == '__main__':
    items = [1, 4, 2, 7, 9, 3]
    print(f'Sorted: {qsort(items)}')

" | guesslang --probabilities

# Language name       Probability
#  Python               74.80%
#  Haskell               6.73%
#  CoffeeScript          5.32%
#  Groovy                1.95%
#  Markdown              0.93%
#  ...

Guesslang Python package

Guesslang can be used as a Python package. Package documentation available here

[]; qsort([Pivot|T]) -> qsort([X || X <- T, X < Pivot]) ++ [Pivot] ++ qsort([X || X <- T, X >= Pivot]). """) print(name) # ⟶ Erlang ">

from guesslang import Guess


guess = Guess()

name = guess.language_name("""
    % Quick sort

    -module (recursion).
    -export ([qsort/1]).

    qsort([]) -> [];
    qsort([Pivot|T]) ->
          qsort([X || X <- T, X < Pivot])
          ++ [Pivot] ++
          qsort([X || X <- T, X >= Pivot]).
""")

print(name)  # ⟶ Erlang

License and credits

Guesslang documentation
Gesslang training dataset created with GuesslangTools
Guesslang developped with Tensorflow
Guesslang icon created with AndroidAssetStudio and Eduardo Tunni's Warnes font
Example source codes used here retrieved from Rosetta Code
Guesslang — Copyright (c) 2021 Y. SOMDA, MIT License

Comments

Guesslang in VS Code

Hi there,

My name is Isidor and I work on VS Code. We have the following problem:

Novice user create a new untitled files and start typing and they have no clue that they have to set the language mode to get all the language smartness that VS Code provides. Thus we were thinking to use some smart language detection so we could automatically set the language for the user.

I was doing a bit of online research and I came across this project - looks very cool!

Is it possible to somehow have this work as a node module instead of python? Since then we could consume it easily in VS Code and things might just work. Even cooler would be if it worked in the browser.

Let me know if you are interested we can also setup a meeting where I could explain our use case in more detail. Thanks!
help wanted

opened by isidorn 19
ModuleNotFoundError: No module named 'guesslang'

Hello Team,

I have installed Tensorflow and guesslang

C:\Users\bhuvaneshwari>pip3 show tensorflow Name: tensorflow Version: 2.5.0 Summary: TensorFlow is an open source machine learning framework for everyone. Home-page: https://www.tensorflow.org/ Author: Google Inc. Author-email: [email protected] License: Apache 2.0 Location: c:\users\bhuvaneshwari\appdata\local\programs\python\python39\lib\site-packages Requires: keras-preprocessing, termcolor, flatbuffers, tensorboard, gast, tensorflow-estimator, google-pasta, numpy, wheel, six, wrapt, keras-nightly, absl-py, typing-extensions, astunparse, protobuf, h5py, opt-einsum, grpcio Required-by: guesslang

C:\Users\bhuvaneshwari>pip3 show tensorflow Name: tensorflow Version: 2.5.0 Summary: TensorFlow is an open source machine learning framework for everyone. Home-page: https://www.tensorflow.org/ Author: Google Inc. Author-email: [email protected] License: Apache 2.0 Location: c:\users\bhuvaneshwari\appdata\local\programs\python\python39\lib\site-packages Requires: keras-preprocessing, termcolor, flatbuffers, tensorboard, gast, tensorflow-estimator, google-pasta, numpy, wheel, six, wrapt, keras-nightly, absl-py, typing-extensions, astunparse, protobuf, h5py, opt-einsum, grpcio Required-by: guesslang

When i try guesslang as a CLI , i get 'guesslang' is not recognized as an internal or external command,

When i try as a package in python program i get ModuleNotFoundError: No module named 'guesslang'

Thanks in advance

opened by bhuvi11 14
Support 24 more languages, including JSON, Kotlin, XML, YAML etc...
Support the following languages:

Assembly

CSV

Dart

Fortran

Groovy

INI

JSON

Julia

Kotlin

Pascal (currently broken :warning:)

TOML

TypeScript

VBA

XML

YAML

Prediction accuracy is 92.59% but the training and test dataset were not well balanced due to lack of files for some languages. And there were errors in the Pascal dataset.
work in progress
opened by yoeo 9

Update pypi version and dependences

Hi, actually i'm in debian with python 3.7, and i try install from pip but sadlt didn't works

root@La:/home/pipe/a# pip3 install guesslang
Collecting guesslang
  Using cached https://files.pythonhosted.org/packages/d5/e2/0416000ea8c42665ca1c3a9b527b585e9a9610b4a6494d728358f25a3a46/guesslang-0.9.3-py3-none-any.whl
Collecting numpy (from guesslang)
  Downloading https://files.pythonhosted.org/packages/d6/c6/58e517e8b1fb192725cfa23c01c2e60e4e6699314ee9684a1c5f5c9b27e1/numpy-1.18.5-cp37-cp37m-manylinux1_x86_64.whl (20.1MB)
    100% |████████████████████████████████| 20.1MB 16kB/s 
Collecting tensorflow==1.7.0rc1 (from guesslang)
  Could not find a version that satisfies the requirement tensorflow==1.7.0rc1 (from guesslang) (from versions: 1.13.0rc1, 1.13.0rc2, 1.13.1, 1.13.2, 1.14.0rc0, 1.14.0rc1, 1.14.0, 2.0.0a0, 2.0.0b0, 2.0.0b1)
No matching distribution found for tensorflow==1.7.0rc1 (from guesslang)

Thx.

opened by latot 7

Uninstallable due to outdated tensorflow specification

pip install guesslang

Collecting guesslang
  Using cached https://files.pythonhosted.org/packages/d5/e2/0416000ea8c42665ca1c3a9b527b585e9a9610b4a6494d728358f25a3a46/guesslang-0.9.3-py3-none-any.whl
Collecting numpy (from guesslang)
  Using cached https://files.pythonhosted.org/packages/91/e7/6c780e612d245cca62bc3ba8e263038f7c144a96a54f877f3714a0e8427e/numpy-1.16.2-cp37-cp37m-manylinux1_x86_64.whl
Collecting tensorflow==1.7.0rc1 (from guesslang)
  Could not find a version that satisfies the requirement tensorflow==1.7.0rc1 (from guesslang) (from versions: 1.13.0rc1, 1.13.0rc2, 1.13.1)
No matching distribution found for tensorflow==1.7.0rc1 (from guesslang)

pip 19.0.3

opened by AndydeCleyre 6

The class "tf.contrib.learn.DNNLinearCombinedClassifier" has been deprecated in TensorFlow

Thanks for your package.

The api of the deep learning model used for this package has been deprecated, reported in the TensorFlow website.

https://www.tensorflow.org/api_docs/python/tf/contrib/learn/DNNLinearCombinedClassifier
enhancement

opened by Zidane-Han 6

Conversion to ONNX and TFJS fails.

Has anyone been able to convert the default model to either ONNX or TFJS ? I have tried but mostly it seems to be an issue of unsupported operators. I am attaching the output that I get for each of them.

Command line for TFJS: tensorflowjs_converter --input_format=tf_saved_model --output_format=tfjs_graph_model <path-to-default-saved-model> <path-to-new-folder>

Output for the above:

2022-10-10 15:18:45.017186: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2022-10-10 15:18:45.161669: E tensorflow/stream_executor/cuda/cuda_blas.cc:2981] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2022-10-10 15:18:45.657095: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory
2022-10-10 15:18:45.657174: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory
2022-10-10 15:18:45.657185: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
2022-10-10 15:18:55.993704: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcusolver.so.11'; dlerror: libcusolver.so.11: cannot open shared object file: No such file or directory
2022-10-10 15:18:55.994301: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudnn.so.8'; dlerror: libcudnn.so.8: cannot open shared object file: No such file or directory
2022-10-10 15:18:55.994318: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1934] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...
2022-10-10 15:18:55.994979: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
WARNING:tensorflow:Issue encountered when serializing global_step.
Type is unsupported, or the types of the items don't match field type in CollectionDef. Note this is a warning and probably safe to ignore.
This operation is not supported when eager execution is enabled.
WARNING:tensorflow:Issue encountered when serializing global_step.
Type is unsupported, or the types of the items don't match field type in CollectionDef. Note this is a warning and probably safe to ignore.
This operation is not supported when eager execution is enabled.
WARNING:tensorflow:Issue encountered when serializing variables.
Type is unsupported, or the types of the items don't match field type in CollectionDef. Note this is a warning and probably safe to ignore.
This operation is not supported when eager execution is enabled.
WARNING:tensorflow:Issue encountered when serializing variables.
Type is unsupported, or the types of the items don't match field type in CollectionDef. Note this is a warning and probably safe to ignore.
This operation is not supported when eager execution is enabled.
WARNING:tensorflow:Issue encountered when serializing trainable_variables.
Type is unsupported, or the types of the items don't match field type in CollectionDef. Note this is a warning and probably safe to ignore.
This operation is not supported when eager execution is enabled.
WARNING:tensorflow:Issue encountered when serializing trainable_variables.
Type is unsupported, or the types of the items don't match field type in CollectionDef. Note this is a warning and probably safe to ignore.
This operation is not supported when eager execution is enabled.
2022-10-10 15:18:56.598822: I tensorflow/core/grappler/devices.cc:66] Number of eligible GPUs (core count >= 8, compute capability >= 0.0): 8
2022-10-10 15:18:56.599119: I tensorflow/core/grappler/clusters/single_machine.cc:358] Starting new session
2022-10-10 15:18:57.721903: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1934] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...
2022-10-10 15:18:57.757114: E tensorflow/core/grappler/grappler_item_builder.cc:670] Init node head/predictions/class_string_lookup/table_init/LookupTableImportV2 doesn't exist in graph
WARNING:tensorflow:From /home/hari/miniconda3/envs/srclangdetect/lib/python3.8/site-packages/tensorflowjs/converters/tf_saved_model_conversion_v2.py:398: load (from tensorflow.python.saved_model.loader_impl) is deprecated and will be removed in a future version.
Instructions for updating:
This function will only be available through the v1 compatibility library as tf.compat.v1.saved_model.loader.load or tf.compat.v1.saved_model.load. There will be a new function for importing SavedModels in Tensorflow 2.0.
WARNING:tensorflow:From /home/hari/miniconda3/envs/srclangdetect/lib/python3.8/site-packages/tensorflowjs/converters/tf_saved_model_conversion_v2.py:398: load (from tensorflow.python.saved_model.loader_impl) is deprecated and will be removed in a future version.
Instructions for updating:
This function will only be available through the v1 compatibility library as tf.compat.v1.saved_model.loader.load or tf.compat.v1.saved_model.load. There will be a new function for importing SavedModels in Tensorflow 2.0.
2022-10-10 15:18:57.904875: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:354] MLIR V1 optimization pass is not enabled
2022-10-10 15:18:57.917561: W tensorflow/core/common_runtime/forward_type_inference.cc:332] Type inference failed. This indicates an invalid graph that escaped type checking. Error message: INVALID_ARGUMENT: expected compatible input types, but input 1:
type_id: TFT_OPTIONAL
args {
  type_id: TFT_PRODUCT
  args {
    type_id: TFT_TENSOR
    args {
      type_id: TFT_INT64
    }
  }
}
 is neither a subtype nor a supertype of the combined inputs preceding it:
type_id: TFT_OPTIONAL
args {
  type_id: TFT_PRODUCT
  args {
    type_id: TFT_TENSOR
    args {
      type_id: TFT_INT32
    }
  }
}

	while inferring type of node 'dnn/zero_fraction/cond/output/_44'
WARNING:tensorflow:From /home/hari/miniconda3/envs/srclangdetect/lib/python3.8/site-packages/tensorflowjs/converters/tf_saved_model_conversion_v2.py:402: convert_variables_to_constants (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.compat.v1.graph_util.convert_variables_to_constants`
WARNING:tensorflow:From /home/hari/miniconda3/envs/srclangdetect/lib/python3.8/site-packages/tensorflowjs/converters/tf_saved_model_conversion_v2.py:402: convert_variables_to_constants (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.compat.v1.graph_util.convert_variables_to_constants`
WARNING:tensorflow:From /home/hari/miniconda3/envs/srclangdetect/lib/python3.8/site-packages/tensorflow/python/framework/convert_to_constants.py:936: extract_sub_graph (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.compat.v1.graph_util.extract_sub_graph`
WARNING:tensorflow:From /home/hari/miniconda3/envs/srclangdetect/lib/python3.8/site-packages/tensorflow/python/framework/convert_to_constants.py:936: extract_sub_graph (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.compat.v1.graph_util.extract_sub_graph`
Traceback (most recent call last):
  File "/home/hari/miniconda3/envs/srclangdetect/bin/tensorflowjs_converter", line 8, in <module>
    sys.exit(pip_main())
  File "/home/hari/miniconda3/envs/srclangdetect/lib/python3.8/site-packages/tensorflowjs/converters/converter.py", line 827, in pip_main
    main([' '.join(sys.argv[1:])])
  File "/home/hari/miniconda3/envs/srclangdetect/lib/python3.8/site-packages/tensorflowjs/converters/converter.py", line 831, in main
    convert(argv[0].split(' '))
  File "/home/hari/miniconda3/envs/srclangdetect/lib/python3.8/site-packages/tensorflowjs/converters/converter.py", line 817, in convert
    _dispatch_converter(input_format, output_format, args, quantization_dtype_map,
  File "/home/hari/miniconda3/envs/srclangdetect/lib/python3.8/site-packages/tensorflowjs/converters/converter.py", line 528, in _dispatch_converter
    tf_saved_model_conversion_v2.convert_tf_saved_model(
  File "/home/hari/miniconda3/envs/srclangdetect/lib/python3.8/site-packages/tensorflowjs/converters/tf_saved_model_conversion_v2.py", line 809, in convert_tf_saved_model
    _convert_tf_saved_model(output_dir, saved_model_dir=saved_model_dir,
  File "/home/hari/miniconda3/envs/srclangdetect/lib/python3.8/site-packages/tensorflowjs/converters/tf_saved_model_conversion_v2.py", line 699, in _convert_tf_saved_model
    optimize_graph(frozen_graph, signature,
  File "/home/hari/miniconda3/envs/srclangdetect/lib/python3.8/site-packages/tensorflowjs/converters/tf_saved_model_conversion_v2.py", line 157, in optimize_graph
    raise ValueError('Unsupported Ops in the model before optimization\n' +
ValueError: Unsupported Ops in the model before optimization
OptionalNone, ReadVariableOp, OptionalFromValue

Command line for ONNX: python -m tf2onnx.convert --saved-model <path-to-model-directory> --output guesslang.onnx --opset 11 --verbose

Output for the above:

2022-10-10 15:22:09.947615: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2022-10-10 15:22:10.093875: E tensorflow/stream_executor/cuda/cuda_blas.cc:2981] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2022-10-10 15:22:10.584761: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory
2022-10-10 15:22:10.584838: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory
2022-10-10 15:22:10.584845: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
/home/hari/miniconda3/envs/srclangdetect/lib/python3.8/runpy.py:127: RuntimeWarning: 'tf2onnx.convert' found in sys.modules after import of package 'tf2onnx', but prior to execution of 'tf2onnx.convert'; this may result in unpredictable behaviour
  warn(RuntimeWarning(msg))
2022-10-10 15:22:20.974883: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcusolver.so.11'; dlerror: libcusolver.so.11: cannot open shared object file: No such file or directory
2022-10-10 15:22:20.975415: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudnn.so.8'; dlerror: libcudnn.so.8: cannot open shared object file: No such file or directory
2022-10-10 15:22:20.975432: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1934] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...
2022-10-10 15:22:20.976057: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2022-10-10 15:22:21,027 - WARNING - tf2onnx.tf_loader: '--tag' not specified for saved_model. Using --tag serve
2022-10-10 15:22:21,588 - INFO - tf2onnx.tf_loader: Signatures found in model: [serving_default,classification,predict].
2022-10-10 15:22:21,588 - WARNING - tf2onnx.tf_loader: '--signature_def' not specified, using first signature: serving_default
2022-10-10 15:22:21,588 - INFO - tf2onnx.tf_loader: Output names: ['classes', 'scores']
WARNING:tensorflow:Issue encountered when serializing global_step.
Type is unsupported, or the types of the items don't match field type in CollectionDef. Note this is a warning and probably safe to ignore.
This operation is not supported when eager execution is enabled.
2022-10-10 15:22:21,601 - WARNING - tensorflow: Issue encountered when serializing global_step.
Type is unsupported, or the types of the items don't match field type in CollectionDef. Note this is a warning and probably safe to ignore.
This operation is not supported when eager execution is enabled.
WARNING:tensorflow:Issue encountered when serializing variables.
Type is unsupported, or the types of the items don't match field type in CollectionDef. Note this is a warning and probably safe to ignore.
This operation is not supported when eager execution is enabled.
2022-10-10 15:22:21,601 - WARNING - tensorflow: Issue encountered when serializing variables.
Type is unsupported, or the types of the items don't match field type in CollectionDef. Note this is a warning and probably safe to ignore.
This operation is not supported when eager execution is enabled.
WARNING:tensorflow:Issue encountered when serializing trainable_variables.
Type is unsupported, or the types of the items don't match field type in CollectionDef. Note this is a warning and probably safe to ignore.
This operation is not supported when eager execution is enabled.
2022-10-10 15:22:21,602 - WARNING - tensorflow: Issue encountered when serializing trainable_variables.
Type is unsupported, or the types of the items don't match field type in CollectionDef. Note this is a warning and probably safe to ignore.
This operation is not supported when eager execution is enabled.
2022-10-10 15:22:21.620369: I tensorflow/core/grappler/devices.cc:66] Number of eligible GPUs (core count >= 8, compute capability >= 0.0): 8
2022-10-10 15:22:21.620657: I tensorflow/core/grappler/clusters/single_machine.cc:358] Starting new session
2022-10-10 15:22:22.735793: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1934] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...
2022-10-10 15:22:22.771439: E tensorflow/core/grappler/grappler_item_builder.cc:670] Init node head/predictions/class_string_lookup/table_init/LookupTableImportV2 doesn't exist in graph
Traceback (most recent call last):
  File "/home/hari/miniconda3/envs/srclangdetect/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/hari/miniconda3/envs/srclangdetect/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/hari/miniconda3/envs/srclangdetect/lib/python3.8/site-packages/tf2onnx/convert.py", line 706, in <module>
    main()
  File "/home/hari/miniconda3/envs/srclangdetect/lib/python3.8/site-packages/tf2onnx/convert.py", line 238, in main
    graph_def, inputs, outputs, initialized_tables, tensors_to_rename = tf_loader.from_saved_model(
  File "/home/hari/miniconda3/envs/srclangdetect/lib/python3.8/site-packages/tf2onnx/tf_loader.py", line 614, in from_saved_model
    _from_saved_model_v2(model_path, input_names, output_names,
  File "/home/hari/miniconda3/envs/srclangdetect/lib/python3.8/site-packages/tf2onnx/tf_loader.py", line 598, in _from_saved_model_v2
    frozen_graph, initialized_tables = from_trackable(imported, concrete_func, inputs, outputs, large_model)
  File "/home/hari/miniconda3/envs/srclangdetect/lib/python3.8/site-packages/tf2onnx/tf_loader.py", line 225, in from_trackable
    raise e
  File "/home/hari/miniconda3/envs/srclangdetect/lib/python3.8/site-packages/tf2onnx/tf_loader.py", line 221, in from_trackable
    frozen_graph = from_function(concrete_func, inputs, outputs, large_model)
  File "/home/hari/miniconda3/envs/srclangdetect/lib/python3.8/site-packages/tf2onnx/tf_loader.py", line 280, in from_function
    raise e
  File "/home/hari/miniconda3/envs/srclangdetect/lib/python3.8/site-packages/tf2onnx/tf_loader.py", line 273, in from_function
    frozen_func = convert_variables_to_constants_v2(func, lower_control_flow=False, aggressive_inlining=True)
  File "/home/hari/miniconda3/envs/srclangdetect/lib/python3.8/site-packages/tensorflow/python/framework/convert_to_constants.py", line 1154, in convert_variables_to_constants_v2
    converter_data = _FunctionConverterDataInEager(
  File "/home/hari/miniconda3/envs/srclangdetect/lib/python3.8/site-packages/tensorflow/python/framework/convert_to_constants.py", line 817, in __init__
    graph_def = _run_inline_graph_optimization(func, lower_control_flow,
  File "/home/hari/miniconda3/envs/srclangdetect/lib/python3.8/site-packages/tensorflow/python/framework/convert_to_constants.py", line 1052, in _run_inline_graph_optimization
    return tf_optimizer.OptimizeGraph(config, meta_graph)
  File "/home/hari/miniconda3/envs/srclangdetect/lib/python3.8/site-packages/tensorflow/python/grappler/tf_optimizer.py", line 65, in OptimizeGraph
    out_graph = tf_opt.TF_OptimizeGraph(cluster.tf_cluster,
ValueError: Failed to import metagraph, check error log for more info.

Environment:

absl-py==1.2.0
astunparse==1.6.3
cachetools==5.2.0
certifi @ file:///croot/certifi_1665076670883/work/certifi
charset-normalizer==2.1.1
chex==0.1.5
coloredlogs==15.0.1
commonmark==0.9.1
contourpy==1.0.5
cycler==0.11.0
dm-tree==0.1.7
etils==0.8.0
flatbuffers==22.9.24
flax==0.6.1
fonttools==4.37.4
gast==0.4.0
google-auth==2.12.0
google-auth-oauthlib==0.4.6
google-pasta==0.2.0
grpcio==1.34.1
guesslang==2.2.1
h5py==3.1.0
humanfriendly==10.0
idna==3.4
importlib-metadata==5.0.0
importlib-resources==5.10.0
jax==0.3.21
jaxlib==0.3.20
keras==2.10.0
keras-nightly==2.5.0.dev2021032900
Keras-Preprocessing==1.1.2
kiwisolver==1.4.4
libclang==14.0.6
Markdown==3.4.1
MarkupSafe==2.1.1
matplotlib==3.6.1
mpmath==1.2.1
msgpack==1.0.4
numpy==1.23.3
oauthlib==3.2.1
onnx==1.12.0
onnxruntime==1.12.1
opt-einsum==3.3.0
optax==0.1.3
packaging==20.9
Pillow==9.2.0
protobuf==3.19.6
pyasn1==0.4.8
pyasn1-modules==0.2.8
Pygments==2.13.0
pyparsing==3.0.9
python-dateutil==2.8.2
PyYAML==6.0
requests==2.28.1
requests-oauthlib==1.3.1
rich==12.6.0
rsa==4.9
scipy==1.9.2
six==1.15.0
sympy==1.11.1
tensorboard==2.10.1
tensorboard-data-server==0.6.1
tensorboard-plugin-wit==1.8.1
tensorflow==2.10.0
tensorflow-estimator==2.10.0
tensorflow-hub==0.12.0
tensorflow-io-gcs-filesystem==0.27.0
tensorflowjs==3.21.0
termcolor==1.1.0
tf2onnx==1.12.1
toolz==0.12.0
typing_extensions==4.4.0
urllib3==1.26.12
Werkzeug==2.2.2
wrapt==1.12.1
zipp==3.9.0

opened by harishankar-gopalan 3

Question - Can I use it in my node web app?

I really like this DL module and I don't know anything about machine learning. But I want to use this in my app to guess the language based on user input.

Now, this app uses Next.js and I wanted to ask that is there a way to use it in a nodejs based app?

opened by max-programming 3
Bump tensorflow from 2.2.0 to 2.3.1
Bumps tensorflow from 2.2.0 to 2.3.1.

Release notes

Sourced from tensorflow's releases.

TensorFlow 2.3.1

Release 2.3.1

Bug Fixes and Other Changes

Fixes an undefined behavior causing a segfault in tf.raw_ops.Switch (CVE-2020-15190)

Fixes three vulnerabilities in conversion to DLPack format (CVE-2020-15191, CVE-2020-15192, CVE-2020-15193)

Fixes two vulnerabilities in SparseFillEmptyRowsGrad (CVE-2020-15194, CVE-2020-15195)

Fixes several vulnerabilities in RaggedCountSparseOutput and SparseCountSparseOutput operations (CVE-2020-15196, CVE-2020-15197, CVE-2020-15198, CVE-2020-15199, CVE-2020-15200, CVE-2020-15201)

Fixes an integer truncation vulnerability in code using the work sharder API (CVE-2020-15202)

Fixes a format string vulnerability in tf.strings.as_string (CVE-2020-15203)

Fixes segfault raised by calling session-only ops in eager mode (CVE-2020-15204)

Fixes data leak and potential ASLR violation from tf.raw_ops.StringNGrams (CVE-2020-15205)

Fixes segfaults caused by incomplete SavedModel validation (CVE-2020-15206)

Fixes a data corruption due to a bug in negative indexing support in TFLite (CVE-2020-15207)

Fixes a data corruption due to dimension mismatch in TFLite (CVE-2020-15208)

Fixes several vulnerabilities in TFLite saved model format (CVE-2020-15209, CVE-2020-15210, CVE-2020-15211)

Fixes several vulnerabilities in TFLite implementation of segment sum (CVE-2020-15212, CVE-2020-15213, CVE-2020-15214)

Updates sqlite3 to 3.33.00 to handle CVE-2020-15358.

Fixes deprecated usage of collections API

Removes scipy dependency from setup.py since TensorFlow does not need it to install the pip package

TensorFlow 2.3.0

Release 2.3.0

Major Features and Improvements

tf.data adds two new mechanisms to solve input pipeline bottlenecks and save resources:

snapshot

tf.data service.

In addition checkout the detailed guide for analyzing input pipeline performance with TF Profiler.

tf.distribute.TPUStrategy is now a stable API and no longer considered experimental for TensorFlow. (earlier tf.distribute.experimental.TPUStrategy).

TF Profiler introduces two new tools: a memory profiler to visualize your model’s memory usage over time and a python tracer which allows you to trace python function calls in your model. Usability improvements include better diagnostic messages and profile options to customize the host and device trace verbosity level.

Introduces experimental support for Keras Preprocessing Layers API (tf.keras.layers.experimental.preprocessing.*) to handle data preprocessing operations, with support for composite tensor inputs. Please see below for additional details on these layers.

TFLite now properly supports dynamic shapes during conversion and inference. We’ve also added opt-in support on Android and iOS for XNNPACK, a highly optimized set of CPU kernels, as well as opt-in support for executing quantized models on the GPU.

Libtensorflow packages are available in GCS starting this release. We have also started to release a nightly version of these packages.

The experimental Python API tf.debugging.experimental.enable_dump_debug_info() now allows you to instrument a TensorFlow program and dump debugging information to a directory on the file system. The directory can be read and visualized by a new interactive dashboard in TensorBoard 2.3 called Debugger V2, which reveals the details of the TensorFlow program including graph structures, history of op executions at the Python (eager) and intra-graph levels, the runtime dtype, shape, and numerical composistion of tensors, as well as their code locations.

Breaking Changes

Increases the minimum bazel version required to build TF to 3.1.0.

tf.data

Makes the following (breaking) changes to the tf.data.

C++ API: - IteratorBase::RestoreInternal, IteratorBase::SaveInternal, and DatasetBase::CheckExternalState become pure-virtual and subclasses are now expected to provide an implementation.

The deprecated DatasetBase::IsStateful method is removed in favor of DatasetBase::CheckExternalState.

Deprecated overrides of DatasetBase::MakeIterator and MakeIteratorFromInputElement are removed.

... (truncated)

Changelog

Sourced from tensorflow's changelog.

Release 2.3.1

Bug Fixes and Other Changes

Fixes an undefined behavior causing a segfault in tf.raw_ops.Switch (CVE-2020-15190)

Fixes three vulnerabilities in conversion to DLPack format (CVE-2020-15191, CVE-2020-15192, CVE-2020-15193)

Fixes two vulnerabilities in SparseFillEmptyRowsGrad (CVE-2020-15194, CVE-2020-15195)

Fixes several vulnerabilities in RaggedCountSparseOutput and SparseCountSparseOutput operations (CVE-2020-15196, CVE-2020-15197, CVE-2020-15198, CVE-2020-15199, CVE-2020-15200, CVE-2020-15201)

Fixes an integer truncation vulnerability in code using the work sharder API (CVE-2020-15202)

Fixes a format string vulnerability in tf.strings.as_string (CVE-2020-15203)

Fixes segfault raised by calling session-only ops in eager mode (CVE-2020-15204)

Fixes data leak and potential ASLR violation from tf.raw_ops.StringNGrams (CVE-2020-15205)

Fixes segfaults caused by incomplete SavedModel validation (CVE-2020-15206)

Fixes a data corruption due to a bug in negative indexing support in TFLite (CVE-2020-15207)

Fixes a data corruption due to dimension mismatch in TFLite (CVE-2020-15208)

Fixes several vulnerabilities in TFLite saved model format (CVE-2020-15209, CVE-2020-15210, CVE-2020-15211)

Fixes several vulnerabilities in TFLite implementation of segment sum (CVE-2020-15212, CVE-2020-15213, CVE-2020-15214)

Updates sqlite3 to 3.33.00 to handle CVE-2020-15358.

Fixes deprecated usage of collections API

Removes scipy dependency from setup.py since TensorFlow does not need it to install the pip package

Release 2.2.1

... (truncated)

Commits

fcc4b96 Merge pull request #43446 from tensorflow-jenkins/version-numbers-2.3.1-16251

4cf2230 Update version numbers to 2.3.1

eee8224 Merge pull request #43441 from tensorflow-jenkins/relnotes-2.3.1-24672

0d41b1d Update RELEASE.md

d99bd63 Insert release notes place-fill

d71d3ce Merge pull request #43414 from tensorflow/mihaimaruseac-patch-1-1

9c91596 Fix missing import

f9f12f6 Merge pull request #43391 from tensorflow/mihaimaruseac-patch-4

3ed271b Solve leftover from merge conflict

9cf3773 Merge pull request #43358 from tensorflow/mm-patch-r2.3

Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

@dependabot rebase will rebase this PR

@dependabot recreate will recreate this PR, overwriting any edits that have been made to it

@dependabot merge will merge this PR after your CI passes on it

@dependabot squash and merge will squash and merge this PR after your CI passes on it

@dependabot cancel merge will cancel a previously requested merge and block automerging

@dependabot reopen will reopen this PR if it is closed

@dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually

@dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

@dependabot use these labels will set the current labels as the default for future PRs for this repo and language

@dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language

@dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language

@dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

You can disable automated security fix PRs for this repo from the Security Alerts page.

dependencies
opened by dependabot[bot] 3
Request: Detection support for additional syntaxes: json, kotlin, nginx, toml, xml, yaml
Thanks so much for reviving your work!

I wasn't sure if you'd prefer a separate issue for each language here, just let me know.

Otherwise:

[x] json

[x] kotlin

[ ] nginx

[x] toml

[x] xml

[x] yaml

enhancement
opened by AndydeCleyre 3
[feature request] return probability from language_name()

language_name() returns the best guess for the language. I would like to know the associated probability such that I can decide if I use the information or not.

opened by nschloe 2
Bump tensorflow from 2.5.0 to 2.9.3
Bumps tensorflow from 2.5.0 to 2.9.3.

Release notes

Sourced from tensorflow's releases.

TensorFlow 2.9.3

Release 2.9.3

This release introduces several vulnerability fixes:

Fixes an overflow in tf.keras.losses.poisson (CVE-2022-41887)

Fixes a heap OOB failure in ThreadUnsafeUnigramCandidateSampler caused by missing validation (CVE-2022-41880)

Fixes a segfault in ndarray_tensor_bridge (CVE-2022-41884)

Fixes an overflow in FusedResizeAndPadConv2D (CVE-2022-41885)

Fixes a overflow in ImageProjectiveTransformV2 (CVE-2022-41886)

Fixes an FPE in tf.image.generate_bounding_box_proposals on GPU (CVE-2022-41888)

Fixes a segfault in pywrap_tfe_src caused by invalid attributes (CVE-2022-41889)

Fixes a CHECK fail in BCast (CVE-2022-41890)

Fixes a segfault in TensorListConcat (CVE-2022-41891)

Fixes a CHECK_EQ fail in TensorListResize (CVE-2022-41893)

Fixes an overflow in CONV_3D_TRANSPOSE on TFLite (CVE-2022-41894)

Fixes a heap OOB in MirrorPadGrad (CVE-2022-41895)

Fixes a crash in Mfcc (CVE-2022-41896)

Fixes a heap OOB in FractionalMaxPoolGrad (CVE-2022-41897)

Fixes a CHECK fail in SparseFillEmptyRowsGrad (CVE-2022-41898)

Fixes a CHECK fail in SdcaOptimizer (CVE-2022-41899)

Fixes a heap OOB in FractionalAvgPool and FractionalMaxPool(CVE-2022-41900)

Fixes a CHECK_EQ in SparseMatrixNNZ (CVE-2022-41901)

Fixes an OOB write in grappler (CVE-2022-41902)

Fixes a overflow in ResizeNearestNeighborGrad (CVE-2022-41907)

Fixes a CHECK fail in PyFunc (CVE-2022-41908)

Fixes a segfault in CompositeTensorVariantToComponents (CVE-2022-41909)

Fixes a invalid char to bool conversion in printing a tensor (CVE-2022-41911)

Fixes a heap overflow in QuantizeAndDequantizeV2 (CVE-2022-41910)

Fixes a CHECK failure in SobolSample via missing validation (CVE-2022-35935)

Fixes a CHECK fail in TensorListScatter and TensorListScatterV2 in eager mode (CVE-2022-35935)

TensorFlow 2.9.2

Release 2.9.2

This releases introduces several vulnerability fixes:

Fixes a CHECK failure in tf.reshape caused by overflows (CVE-2022-35934)

Fixes a CHECK failure in SobolSample caused by missing validation (CVE-2022-35935)

Fixes an OOB read in Gather_nd op in TF Lite (CVE-2022-35937)

Fixes a CHECK failure in TensorListReserve caused by missing validation (CVE-2022-35960)

Fixes an OOB write in Scatter_nd op in TF Lite (CVE-2022-35939)

Fixes an integer overflow in RaggedRangeOp (CVE-2022-35940)

Fixes a CHECK failure in AvgPoolOp (CVE-2022-35941)

Fixes a CHECK failures in UnbatchGradOp (CVE-2022-35952)

Fixes a segfault TFLite converter on per-channel quantized transposed convolutions (CVE-2022-36027)

Fixes a CHECK failures in AvgPool3DGrad (CVE-2022-35959)

Fixes a CHECK failures in FractionalAvgPoolGrad (CVE-2022-35963)

Fixes a segfault in BlockLSTMGradV2 (CVE-2022-35964)

Fixes a segfault in LowerBound and UpperBound (CVE-2022-35965)

... (truncated)

Changelog

Sourced from tensorflow's changelog.

Release 2.9.3

This release introduces several vulnerability fixes:

Fixes an overflow in tf.keras.losses.poisson (CVE-2022-41887)

Fixes a heap OOB failure in ThreadUnsafeUnigramCandidateSampler caused by missing validation (CVE-2022-41880)

Fixes a segfault in ndarray_tensor_bridge (CVE-2022-41884)

Fixes an overflow in FusedResizeAndPadConv2D (CVE-2022-41885)

Fixes a overflow in ImageProjectiveTransformV2 (CVE-2022-41886)

Fixes an FPE in tf.image.generate_bounding_box_proposals on GPU (CVE-2022-41888)

Fixes a segfault in pywrap_tfe_src caused by invalid attributes (CVE-2022-41889)

Fixes a CHECK fail in BCast (CVE-2022-41890)

Fixes a segfault in TensorListConcat (CVE-2022-41891)

Fixes a CHECK_EQ fail in TensorListResize (CVE-2022-41893)

Fixes an overflow in CONV_3D_TRANSPOSE on TFLite (CVE-2022-41894)

Fixes a heap OOB in MirrorPadGrad (CVE-2022-41895)

Fixes a crash in Mfcc (CVE-2022-41896)

Fixes a heap OOB in FractionalMaxPoolGrad (CVE-2022-41897)

Fixes a CHECK fail in SparseFillEmptyRowsGrad (CVE-2022-41898)

Fixes a CHECK fail in SdcaOptimizer (CVE-2022-41899)

Fixes a heap OOB in FractionalAvgPool and FractionalMaxPool(CVE-2022-41900)

Fixes a CHECK_EQ in SparseMatrixNNZ (CVE-2022-41901)

Fixes an OOB write in grappler (CVE-2022-41902)

Fixes a overflow in ResizeNearestNeighborGrad (CVE-2022-41907)

Fixes a CHECK fail in PyFunc (CVE-2022-41908)

Fixes a segfault in CompositeTensorVariantToComponents (CVE-2022-41909)

Fixes a invalid char to bool conversion in printing a tensor (CVE-2022-41911)

Fixes a heap overflow in QuantizeAndDequantizeV2 (CVE-2022-41910)

Fixes a CHECK failure in SobolSample via missing validation (CVE-2022-35935)

Fixes a CHECK fail in TensorListScatter and TensorListScatterV2 in eager mode (CVE-2022-35935)

Release 2.8.4

This release introduces several vulnerability fixes:

Fixes a heap OOB failure in ThreadUnsafeUnigramCandidateSampler caused by missing validation (CVE-2022-41880)

Fixes a segfault in ndarray_tensor_bridge (CVE-2022-41884)

Fixes an overflow in FusedResizeAndPadConv2D (CVE-2022-41885)

Fixes a overflow in ImageProjectiveTransformV2 (CVE-2022-41886)

Fixes an FPE in tf.image.generate_bounding_box_proposals on GPU (CVE-2022-41888)

Fixes a segfault in pywrap_tfe_src caused by invalid attributes (CVE-2022-41889)

Fixes a CHECK fail in BCast (CVE-2022-41890)

Fixes a segfault in TensorListConcat (CVE-2022-41891)

Fixes a CHECK_EQ fail in TensorListResize (CVE-2022-41893)

Fixes an overflow in CONV_3D_TRANSPOSE on TFLite (CVE-2022-41894)

Fixes a heap OOB in MirrorPadGrad (CVE-2022-41895)

Fixes a crash in Mfcc (CVE-2022-41896)

Fixes a heap OOB in FractionalMaxPoolGrad (CVE-2022-41897)

Fixes a CHECK fail in SparseFillEmptyRowsGrad (CVE-2022-41898)

Fixes a CHECK fail in SdcaOptimizer (CVE-2022-41899)

... (truncated)

Commits

a5ed5f3 Merge pull request #58584 from tensorflow/vinila21-patch-2

258f9a1 Update py_func.cc

cd27cfb Merge pull request #58580 from tensorflow-jenkins/version-numbers-2.9.3-24474

3e75385 Update version numbers to 2.9.3

bc72c39 Merge pull request #58482 from tensorflow-jenkins/relnotes-2.9.3-25695

3506c90 Update RELEASE.md

8dcb48e Update RELEASE.md

4f34ec8 Merge pull request #58576 from pak-laura/c2.99f03a9d3bafe902c1e6beb105b2f2417...

6fc67e4 Replace CHECK with returning an InternalError on failing to create python tuple

5dbe90a Merge pull request #58570 from tensorflow/r2.9-7b174a0f2e4

Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

@dependabot rebase will rebase this PR

@dependabot recreate will recreate this PR, overwriting any edits that have been made to it

@dependabot merge will merge this PR after your CI passes on it

@dependabot squash and merge will squash and merge this PR after your CI passes on it

@dependabot cancel merge will cancel a previously requested merge and block automerging

@dependabot reopen will reopen this PR if it is closed

@dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually

@dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

@dependabot use these labels will set the current labels as the default for future PRs for this repo and language

@dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language

@dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language

@dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

You can disable automated security fix PRs for this repo from the Security Alerts page.

dependencies
opened by dependabot[bot] 0

pkg_resources.DistributionNotFound: The 'tensorflow==2.5.0' distribution was not found and is required by guesslang

Hi, with latest python-tensorflow 2.9.1 upgrade, guesslang stop working.

Traceback (most recent call last):
  File "/usr/lib/python3.10/site-packages/pkg_resources/__init__.py", line 573, in _build_master
    ws.require(__requires__)
  File "/usr/lib/python3.10/site-packages/pkg_resources/__init__.py", line 891, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/usr/lib/python3.10/site-packages/pkg_resources/__init__.py", line 782, in resolve
    raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.ContextualVersionConflict: (tensorflow 2.9.1 (/usr/lib/python3.10/site-packages), Requirement.parse('tensorflow==2.5.0'), {'guesslang'})

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/sbin/guesslang", line 33, in <module>
    sys.exit(load_entry_point('guesslang==2.2.1', 'console_scripts', 'guesslang')())
  File "/usr/sbin/guesslang", line 25, in importlib_load_entry_point
    return next(matches).load()
  File "/usr/lib/python3.10/importlib/metadata/__init__.py", line 171, in load
    module = import_module(match.group('module'))
  File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 992, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/usr/lib/python3.10/site-packages/guesslang/__init__.py", line 16, in <module>
    from guesslang.guess import Guess, GuesslangError  # noqa: F401
  File "/usr/lib/python3.10/site-packages/guesslang/guess.py", line 10, in <module>
    from guesslang import model
  File "/usr/lib/python3.10/site-packages/guesslang/model.py", line 11, in <module>
    import tensorflow as tf
  File "/usr/lib/python3.10/site-packages/tensorflow/__init__.py", line 37, in <module>
    from tensorflow.python.tools import module_util as _module_util
  File "/usr/lib/python3.10/site-packages/tensorflow/python/__init__.py", line 45, in <module>
    from tensorflow.python.feature_column import feature_column_lib as feature_column
  File "/usr/lib/python3.10/site-packages/tensorflow/python/feature_column/feature_column_lib.py", line 18, in <module>
    from tensorflow.python.feature_column.feature_column import *
  File "/usr/lib/python3.10/site-packages/tensorflow/python/feature_column/feature_column.py", line 143, in <module>
    from tensorflow.python.layers import base
  File "/usr/lib/python3.10/site-packages/tensorflow/python/layers/base.py", line 16, in <module>
    from tensorflow.python.keras.legacy_tf_layers import base
  File "/usr/lib/python3.10/site-packages/tensorflow/python/keras/__init__.py", line 25, in <module>
    from tensorflow.python.keras import models
  File "/usr/lib/python3.10/site-packages/tensorflow/python/keras/models.py", line 20, in <module>
    from tensorflow.python.keras import metrics as metrics_module
  File "/usr/lib/python3.10/site-packages/tensorflow/python/keras/metrics.py", line 34, in <module>
    from tensorflow.python.keras import activations
  File "/usr/lib/python3.10/site-packages/tensorflow/python/keras/activations.py", line 18, in <module>
    from tensorflow.python.keras.layers import advanced_activations
  File "/usr/lib/python3.10/site-packages/tensorflow/python/keras/layers/__init__.py", line 68, in <module>
    from tensorflow.python.keras.layers.core import Masking
  File "/usr/lib/python3.10/site-packages/tensorflow/python/keras/layers/core.py", line 54, in <module>
    from tensorflow.python.ops import standard_ops
  File "/usr/lib/python3.10/site-packages/tensorflow/python/ops/standard_ops.py", line 108, in <module>
    from tensorflow.python.compiler.tensorrt import trt_convert as trt
  File "/usr/lib/python3.10/site-packages/tensorflow/python/compiler/tensorrt/__init__.py", line 18, in <module>
    from tensorflow.python.compiler.tensorrt import trt_convert as trt
  File "/usr/lib/python3.10/site-packages/tensorflow/python/compiler/tensorrt/trt_convert.py", line 30, in <module>
    from tensorflow.python.compiler.tensorrt import utils as trt_utils
  File "/usr/lib/python3.10/site-packages/tensorflow/python/compiler/tensorrt/utils.py", line 17, in <module>
    from distutils import version
  File "/usr/lib/python3.10/site-packages/setuptools/__init__.py", line 16, in <module>
    import setuptools.version
  File "/usr/lib/python3.10/site-packages/setuptools/version.py", line 1, in <module>
    import pkg_resources
  File "/usr/lib/python3.10/site-packages/pkg_resources/__init__.py", line 3317, in <module>
    def _initialize_master_working_set():
  File "/usr/lib/python3.10/site-packages/pkg_resources/__init__.py", line 3291, in _call_aside
    f(*args, **kwargs)
  File "/usr/lib/python3.10/site-packages/pkg_resources/__init__.py", line 3329, in _initialize_master_working_set
    working_set = WorkingSet._build_master()
  File "/usr/lib/python3.10/site-packages/pkg_resources/__init__.py", line 575, in _build_master
    return cls._build_from_requirements(__requires__)
  File "/usr/lib/python3.10/site-packages/pkg_resources/__init__.py", line 588, in _build_from_requirements
    dists = ws.resolve(reqs, Environment())
  File "/usr/lib/python3.10/site-packages/pkg_resources/__init__.py", line 777, in resolve
    raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'tensorflow==2.5.0' distribution was not found and is required by guesslang

opened by carlosal1015 1

update to tensorflow 2.8.0
You install_requires is taken from requirements.txt where the version number is fixed. This is usually not a good idea, because you prevent Python users to update their installation when guesslang is installed. (Remember Python doesn't install packages locally, per package, but globally.)

A better solution is to provide a version range, usually between major versions, e.g.,

tensorflow >=2, <3

There are quite a few other things that could be improved in an around metadata handling in guesslang.
opened by nschloe 1

dependency error when install this

python version: 3.10 os: manjaro 21

Here is the install output:

Defaulting to user installation because normal site-packages is not writeable
Requirement already satisfied: guesslang in /usr/lib/python3.10/site-packages/guesslang-2.2.2-py3.10.egg (2.2.2)
Collecting guesslang
  Downloading guesslang-2.2.1-py3-none-any.whl (2.5 MB)
     |████████████████████████████████| 2.5 MB 210 kB/s
  Downloading guesslang-2.2.0-py3-none-any.whl (2.5 MB)
     |████████████████████████████████| 2.5 MB 12.8 MB/s
  Downloading guesslang-2.0.3-py3-none-any.whl (2.1 MB)
     |████████████████████████████████| 2.1 MB 12.4 MB/s
  Downloading guesslang-2.0.1-py3-none-any.whl (2.1 MB)
     |████████████████████████████████| 2.1 MB 12.9 MB/s
  Downloading guesslang-2.0.0-py3-none-any.whl (13.0 MB)
     |████████████████████████████████| 13.0 MB 12.0 MB/s
  Downloading guesslang-0.9.3-py3-none-any.whl (3.2 MB)
     |████████████████████████████████| 3.2 MB 11.8 MB/s
Requirement already satisfied: numpy in /home/lizhe/.local/lib/python3.10/site-packages (from guesslang) (1.22.1)
  Downloading guesslang-0.9.1-py3-none-any.whl (3.2 MB)
     |████████████████████████████████| 3.2 MB 12.5 MB/s
Collecting numpy<1.13,>=1.12
  Downloading numpy-1.12.1.zip (4.8 MB)
     |████████████████████████████████| 4.8 MB 13.5 MB/s
  Preparing metadata (setup.py) ... done
ERROR: Cannot install guesslang==0.9.1, guesslang==0.9.3, guesslang==2.0.0, guesslang==2.0.1, guesslang==2.0.3, guesslang==2.2.0, guesslang==2.2.1 and guesslang==2.2.2 because these package versions have conflicting dependencies.

The conflict is caused by:
    guesslang 2.2.2 depends on tensorflow==2.5.0
    guesslang 2.2.1 depends on tensorflow==2.5.0
    guesslang 2.2.0 depends on tensorflow==2.5.0
    guesslang 2.0.3 depends on tensorflow==2.5.0
    guesslang 2.0.1 depends on tensorflow==2.2.0
    guesslang 2.0.0 depends on tensorflow==2.2.0
    guesslang 0.9.3 depends on tensorflow==1.7.0rc1
    guesslang 0.9.1 depends on tensorflow==1.1.0

To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict

ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/user_guide/#fixing-conflicting-dependencies

opened by woshichuanqilz 1

Releases(v2.2.1)

v2.2.1(Aug 1, 2021)

Make Guesslang command line tool fully compatible with Windows. No change made on the machine learning model.
Source code(tar.gz)
Source code(zip)
v2.2.0(Jul 25, 2021)

Support 24 new languages
Source code(tar.gz)
Source code(zip)
v2.0.1(Jul 1, 2020)

Use a character level model instead of a word level model to better detect "minified" CSS and JavaScript code.
Source code(tar.gz)
Source code(zip)
v2.0.0(Jun 29, 2020)
(cf. https://github.com/yoeo/guesslang/releases/tag/v2.0.0a1)

Supports 30 languages (20 languages supported by previous versions). Including Powershell and Batch

Robust and scalable training workflow, using Tensorflow dataset API https://www.tensorflow.org/api_docs/python/tf/data/Dataset

Way simpler yet as performant feature engineering

More concise documentation

Simple check on detection probabilities to avoid classifying plain text as source code

Exposes the detection probabilities with guess.probabilities(source_code) method.

Updated Tensorflow to the latest version 2.2.0

Use up to date Tensorflow canned classifier

guess.language_name(source_code) now identifies empty content

Guesslang dataset creation moved to a new dedicated repository https://github.com/yoeo/guesslangtools

Source code(tar.gz)
Source code(zip)
v2.0.0a1(Jun 14, 2020)
Supports 30 languages (20 languages supported by previous versions). Including Powershell and Batch https://github.com/yoeo/guesslang/issues/2

Robust and scalable training workflow, using Tensorflow dataset API https://www.tensorflow.org/api_docs/python/tf/data/Dataset

Way simpler yet as performant feature engineering

More concise documentation

Simple check on detection probabilities to avoid classifying plain text as source code https://github.com/yoeo/guesslang/issues/16 https://github.com/yoeo/guesslang/issues/15

Exposes the detection probabilities with guess.probabilities(source_code) method. https://github.com/yoeo/guesslang/issues/14

Updated Tensorflow to the latest version 2.2.0 https://github.com/yoeo/guesslang/issues/12

Use up to date Tensorflow canned classifier https://github.com/yoeo/guesslang/issues/9

guess.language_name(source_code) now identifies empty content https://github.com/yoeo/guesslang/issues/6

Guesslang dataset creation moved to a new dedicated repository https://github.com/yoeo/guesslangtools https://github.com/yoeo/guesslang/issues/5

Source code(tar.gz)
Source code(zip)
v0.9.4(Nov 1, 2018)

Support Python static types.
Source code(tar.gz)
Source code(zip)
v0.9.3(Mar 26, 2018)
Multiplatform:

Guesslang can officially be deployed on Linux and Windows platforms.

It may also run on OSX too but that have not been fully tested yet.

Source code(tar.gz)
Source code(zip)
v0.9.2(May 30, 2017)

First public release Try it!
Source code(tar.gz)
Source code(zip)