👄 The most accurate natural language detection library for Java and the JVM, suitable for long and short text alike

As pointed out in #39 and #57 Lingua's great accuracy comes at the cost of high memory usage. This imposes a problem for some projects trying to use Lingua. In this issue I will try to highlight some main areas where performance can be improved, some of this is already covered by #98. Note that some of the proposed changes might decrease execution speed or require some larger refactoring.

Model files

• Instead of storing the model data in JSON format, a binary format could be used matching the in-memory format (see "In-memory models" section). This would have the advantage that:

  • Lookup maps such as Char2DoubleOpenHashMap could be created with the expected size avoiding rehashing of the maps during deserialization.
  • Model file loading is faster.
  • Model file sizes will be slightly smaller when encoding the frequency only once, followed by the number of ngrams which share this frequency, followed by the ngram values.

  Note that even though the fastutil maps are Serializable, using JDK serialization might introduce unnecessary overhead and would make this library dependent on the internal serialization format of the fastutil maps. Instead the data could be written manually to a DataOutputStream.

Model file loading

• Use streaming JSON library. The currently used kotlinx-serialization-json does not seem to support streaming yet. Therefore currently the complete model files are loaded as String before being parsed. This is (likely) slow and requires large amounts of memory. Instead streaming JSON libraries such as https://github.com/square/moshi should be used. Note that this point becomes obsolete if a binary format (as described in the "Model files" section above) is used.

In-memory models

• Object2DoubleOpenHashMap load factor can increased from the default 0.75 to a higher value. This reduces memory usage but might slow down execution.

• Ngrams can be encoded using primitives. Since this project uses only up to fivegrams (5 chars), most of the ngrams (and for some languages even ngrams of all lengths) can be encoded as JVM primitives using bitwise operations, e.g.:

  • Unigrams as Byte or Char
  • Bigrams as Short or Int
  • Trigrams as Int or Long
  • Quadrigrams as Int or Long
  • Fivegrams as Long or in the worst case as String object. Note that at least for fivegrams the binary encoding should probably be offset based, so one char is the base code point and the the remaining bits of the Long encode the offsets of the other chars to the base char. This allows encoding alphabets such as Georgian where each char is > Long.SIZE_BITS / 5.

  This might even increase execution speed since it avoids hashCode() and equals(...) calls when looking up frequencies (speed-up, if any, has to be tested though).

• Reduce frequency accuracy for in-memory models and model files from 64-bit Double to 32-bit. This can have a big impact on memory usage, saving more than 100MB with all models preloaded. However, instead of using a 32-bit Float to store the frequency, a custom 32-bit encoding can (and maybe should) be used since Float 'wastes' some bits for the sign (frequency will never be negative) and the exponent (frequency will never be >= 1.0), though this might decrease language detection speed due to the decoding overhead.

• Remove Korean fivegrams (and quadrigrams?). The Korean language models are quite large, additionally due to the large range of Korean code points a great majority (> 1.000.000 fivegrams (?)) cannot be encoded with the primitive encoding approach outlined above. Chinese and Japanse don't seem to have quadrigram and fivegram models as well, not sure if this is due to how the languages work, but maybe it would be acceptable to drop them for Korean as well; also because detection of Korean seems to be rather unambiguous.

Runtime performance

• Remove Alphabet. The Alphabet class can probably removed, Character.UnicodeScript seems to be an exact substitute and might allow avoiding some indirection, e.g. only lookup UnicodeScript for a Char once and then compare it with expected ones instead of having each Alphabet look up UnicodeScript.
• Avoid creation of Ngram objects. Similar to the primitive encoding described in "In-memory models" above, Ngram objects created as part of splitting up the text can be avoided as well (with a different encoding). A Kotlin inline class can be used to still get type safety and have some convenience functions. Primitive encoding can only support trigrams reliably without too much overhead / too complicated encoding, but that is probably fine because since d0f7a7c211abb03885cc89febae9d77fbf640342 at most trigrams will be used for longer texts.
• Instead of accessing lazy frequency lookup in every iteration, it might be faster to access it once at the beginning and then directly use it instead (though this could also be premature optimization).

Conclusion

With some / all of these suggestions applied memory usage can be reduced and execution speed can be increased without affecting accuracy. However, some of the suggestions might be premature optimization, and they only work for 16-bit Char but not for supplementary code points (> 16-bit) (but the current implementation, mainly Ngram creation, seems to have that limitation as well).

I have implemented some of these optimizations and some other minor improvements in https://github.com/Marcono1234/lingua/tree/experimental/performance. However, these changes are pretty experimental: The Git history is not very nice to look at; in some commits I fixed bugs I introduced before or reverted changes again. Additionally the unit tests and model file writing are broken. Some of the changes might also be premature optimization. Though maybe it is interesting nonetheless, it appears the memory usage with all languages being preloaded went down to about ~~640MB~~ (Edit: 920MB, made a mistake in the binary encoding) on AdoptOpenJDK 11.

I changed the runtime memory model, the original JSON is translated to a dense map. This reduces memory requirements at cost of speed (frequencies lookup should be slower). Frequencies are stored as Float instead of Double, this introduces an 0.001% error on calculation, and tests are updated accordingly.

fastutil dependency has been removed.

All changes are performed in internal classes, so this request is compatible with the 1.1 version and I hope that the merge will be considered soon.

I'm using lingua 1.1.0 in a Java web application for language detection. The application is set up to load all models on the first request:

LanguageDetectorBuilder.fromAllLanguages().withPreloadedLanguageModels().build();

When I undeploy the web application from the application server, the models stay in memory. I took a heap dump using Eclipse Memory Anaylzer. The dump shows that there are still instances of the classes related to coroutines (e.g. kotlinx.coroutines.scheduling.CoroutineScheduler$WorkerState, kotlinx.coroutines.scheduling.WorkQueue, kotlinx.coroutines.scheduling.CoroutineScheduler) after undeploying the application. The coroutines still seem to reference the models.

I've built a reproducer using only Servlet API that seems to show similar behaviour on Tomcat. Tomcat shows warnings like this:

WARNUNG: The web application [lingua-reproducer] appears to have started a thread named [DefaultDispatcher-worker-9] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:338) kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.park(CoroutineScheduler.kt:795) kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.tryPark(CoroutineScheduler.kt:740) kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:711) kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:665)

Is there are way to ensure that the threads created by Lingua terminate when the application is undeployed?

I compared version 0.6.1 vs version 1.0.0 on two private test sets

In the following table (see below) I reported results for both the version and their difference for all languages of my benchmark. Scores are the ratio of corrected classified segments with respect to a gold reference. Actually, the sets contains real-world text from the web and from technical domains, and they are not manually checked. So it is possible, that they contain some wrongly classified sentences.

Nevertheless, v1.0.0 seems worse for than v0.6.1 for many languages.

One of the big difference relates with text including string in different languages: For instance, Chinese segment having both Chinese and English, like the following ones, which are detected as English and Portuguese, instead of Chinese

Snapchat 并不 是唯一 一家 触 及 这些 文化 底 线 的公 司 
Gomes ： 我們 的目 標 是 提 升 搜 尋 服 務 的 品 質

or Greek segment with few western strings, like the following ones, which are detected as Danish and Italian, instead of Greek

Rasmussen μετά τη συνάντησή τους στο ΥΠΕΘΑ
γεγονός που, πέραν της σημασίας των σχετικών πολιτικών επαφών, εμπεριέχει και ιδιαίτερη συμβολική αξία, καθώς η επίσκεψη πραγματοποιήθηκε δύο μόλις έτη μετά την επίσκεψη του πρώην ιταλού Προέδρου, κ. Azeglio Ciampi, στην Αθήνα, στις 15-17.

or Arabic segment with few English strings, like the following ones, which are detected as English and Tagalog, instead of Arabic

أداة Google Scholar وضعت أبحاثًا متاحةً للجميع البحث عنها سهل والوصول إليها أسهل.
يارد "YARID" - اللاجئون الأفارقة الشباب للتنمية المتكاملة- بدأت كمحادثة داخل المُجتمع الكونغو

Several other examples, can be found even between languages with more similar alphabets.

It seems that v1.0.0 relies too much on Western alphabets to identify the language, without considering the amount of such Western characters.

set  lng Lingua Lingua100 diffs_vs_Lingua
setA ar  0.930   0.902  diff: -0.028
setA az  0.807   0.784  diff: -0.023
setA be  0.861   0.816  diff: -0.045
setA bg  0.801   0.734  diff: -0.067
setA bs  0.412   0.408  diff: -0.004
setA ca  0.760   0.762  diff: 0.002
setA cs  0.792   0.785  diff: -0.007
setA da  0.760   0.752  diff: -0.008
setA de  0.848   0.848  diff: 0
setA el  0.947   0.932  diff: -0.015
setA es  0.804   0.853  diff: 0.049
setA et  0.856   0.853  diff: -0.003
setA fi  0.865   0.864  diff: -0.001
setA fr  0.868   0.882  diff: 0.014
setA he  0.972   0.961  diff: -0.011
setA hi  0.790   0.733  diff: -0.057
setA hr  0.628   0.623  diff: -0.005
setA hu  0.858   0.848  diff: -0.01
setA hy  0.827   0.801  diff: -0.026
setA id  0.665   0.665  diff: 0
setA is  0.863   0.831  diff: -0.032
setA it  0.866   0.865  diff: -0.001
setA ja  0.758   0.752  diff: -0.006
setA ka  0.802   0.787  diff: -0.015
setA ko  0.887   0.827  diff: -0.06
setA lt  0.839   0.828  diff: -0.011
setA lv  0.882   0.869  diff: -0.013
setA mk  0.786   0.723  diff: -0.063
setA ms  0.801   0.809  diff: 0.008
setA nb  0.735   0.733  diff: -0.002
setA nl  0.799   0.835  diff: 0.036
setA nn  0.768   0.768  diff: 0
setA pl  0.879   0.881  diff: 0.002
setA pt  0.862   0.858  diff: -0.004
setA ro  0.765   0.751  diff: -0.014
setA ru  0.820   0.773  diff: -0.047
setA sk  0.783   0.766  diff: -0.017
setA sl  0.714   0.708  diff: -0.006
setA sq  0.829   0.826  diff: -0.003
setA sr  0.417   0.302  diff: -0.115
setA sv  0.833   0.830  diff: -0.003
setA th  0.940   0.927  diff: -0.013
setA tl  0.747   0.748  diff: 0.001
setA tr  0.901   0.895  diff: -0.006
setA uk  0.877   0.848  diff: -0.029
setA vi  0.920   0.877  diff: -0.043
setA zh  0.941   0.858  diff: -0.083

setB ar   0.996   0.988  diff: -0.008
setB bg   0.957   0.947  diff: -0.01
setB bs   0.495   0.494  diff: -0.001
setB ca   0.946   0.953  diff: 0.007
setB cs   0.993   0.992  diff: -0.001
setB da   0.947   0.946  diff: -0.001
setB de   0.996   0.996  diff: 0
setB el   0.996   0.992  diff: -0.004
setB en   0.964   0.966  diff: 0.002
setB es   0.897   0.920  diff: 0.023
setB et   0.978   0.974  diff: -0.004
setB fi   0.998   0.998  diff: 0
setB fr   0.962   0.971  diff: 0.009
setB he   1.000   0.999  diff: -0.001
setB hr   0.858   0.868  diff: 0.01
setB hu   0.988   0.988  diff: 0
setB id   0.765   0.765  diff: 0
setB is   0.979   0.971  diff: -0.008
setB it   0.939   0.937  diff: -0.002
setB ja   0.986   0.986  diff: 0
setB ko   0.998   0.998  diff: 0
setB lt   0.992   0.990  diff: -0.002
setB lv   0.990   0.983  diff: -0.007
setB mk   0.927   0.930  diff: 0.003
setB ms   0.927   0.927  diff: 0
setB nb   0.927   0.928  diff: 0.001
setB nl   0.921   0.949  diff: 0.028
setB nn   0.942   0.946  diff: 0.004
setB pl   0.993   0.992  diff: -0.001
setB pt   0.952   0.948  diff: -0.004
setB ro   0.964   0.958  diff: -0.006
setB ru   0.997   0.911  diff: -0.086
setB sk   0.977   0.975  diff: -0.002
setB sl   0.943   0.942  diff: -0.001
setB sq   0.983   0.983  diff: 0
setB sv   0.973   0.971  diff: -0.002
setB th   0.996   0.996  diff: 0
setB tr   0.993   0.990  diff: -0.003
setB uk   0.943   0.964  diff: 0.021
setB vi   0.994   0.954  diff: -0.04
setB zh  0.992   0.955  diff: -0.037

Hi,

while testing the library with some texts I encountered some ambiguous detection results. As far as I understand, the detectLanguageOf method always returns a language as soon as it has at least some possibility. However, there exist texts where this behaviour is probably not desired.

Imagine a text which leads to similar possibilities for two languages, with the first one just a little bit more likely. It would be nice to be able to detect such cases, or at least to ensure a certain distance between the possibilities of the most and the second most likely language (otherwise the method may return UNKNOWN). In our use-case we would prefer to have more detection as unknown rather than (a lot of) false-positives.

The following code snippet illustrates my idea:

@JvmOverloads
fun detectLanguageOf(text: String, requiredRelativeDistance: Double = 0.95): Language {
    
    [...]

    return getMostLikelyLanguage(allProbabilities, unigramCountsOfInputText, requiredRelativeDistance)
}
    
internal fun getMostLikelyLanguage(
    probabilities: List<Map<Language, Double>>,
    unigramCountsOfInputText: Map<Language, Int>,
    requiredRelativeDistance: Double = 0.95
): Language {
    
    [...]

    return when {
        filteredProbabilities.none() -> UNKNOWN
        filteredProbabilities.singleOrNull() != null -> filteredProbabilities.first().key
        else -> {
            val candidate = filteredProbabilities.maxBy { it.value }!!
            val second = filteredProbabilities.filter { it.key != candidate.key }.maxBy { it.value }!!
            if (second.value * requiredRelativeDistance < candidate.value) {
                candidate.key
            } else {
                UNKNOWN
            }
        }
    }
}

Feel free to copy the code if you want. I don't know whether this is a good approach for the problem or if there are better ways to do that. However, it would be really nice to have a solution for this in some way.

Thanks in advance!

I had a plan to use 'lingua' in a multi threaded Java environment, but, if I got it right, 'LanguageDetector' instance is not thread safe, i.e. if several threads will use it simultaneously, they may corrupt each other work. Am I right? New 'LanguageDetector' instance seems to be very expensive.

I am trying to run language on using this scrit

            final LanguageDetector detector = LanguageDetectorBuilder.fromLanguages(ENGLISH, FRENCH, GERMAN, SPANISH, JAPANESE, CHINESE,ITALIAN, PORTUGUESE,ARABIC,RUSSIAN,DUTCH,KOREAN,SWEDISH,HINDI,POLISH).build();

	    long start=System.currentTimeMillis();  


	    final Language detectedLanguage = detector.detectLanguageOf("Zum Vergleich kann es auch nützlich sein, diese Rankings neben einigen etwas älteren Forschungsergebnissen zu sehen. Im Jahr 2013, Common Sense Advisory zur Verfügung gestellt , eine empirische Studie basiert auf einer Wallet World Online (WOW) - definiert als ‚die gesamte wirtschaftliche Chance, sowohl online als auch offline, berechnet durch einen Anteil eines Landes BIP zu allen wichtigen Blöcken dieser Gesellschaft assoziieren. ' Hier ist, was uns ihre Studie gezeigt hat.");
//	    System.out.println(detectedLanguage.toString());
	    long end=System.currentTimeMillis();  
	    System.out.println("Time: "+ (end - start));

it's taking 700millisecong. which is very slow. which can not be used for 10000+ files.. is there any approach to get results with 1-10milliseconds?

or any function like isEnglish(). which will be true only for English..

I need to run this lib in a memory constrained environment: less than 200Mb for the unzipped package. How can I exclude rare language profiles from the library?

An alternative: can the memory size be significantly decreased by minifying the json files used for each language?

Note: I am using the maven build of the lingua.

As discussed in #110 , there seems to be a memory leak when using Lingua in a Java web application. I've uploaded an example application at https://github.com/deutsche-nationalbibliothek/lingua-reproducer-memory-leak . The README contains some information about the problem. Please let me know if you need more information.

Hi,

I am trying to use Lingua inside a plain Java maven project and using following maven dependency for that:

<dependency>
                <groupId>com.github.pemistahl</groupId>
                <artifactId>lingua</artifactId>
                <version>1.0.3</version>
</dependency>

Code Sample:

import com.github.pemistahl.lingua.api.Language;
import com.github.pemistahl.lingua.api.LanguageDetector;
import com.github.pemistahl.lingua.api.LanguageDetectorBuilder;

import static com.github.pemistahl.lingua.api.Language.*;

public class Test {
    public static void detect(){
        LanguageDetector detector = LanguageDetectorBuilder.fromLanguages(ENGLISH, FRENCH, GERMAN, SPANISH).build();
        Language language = detector.detectLanguageOf("languages are awesome");
        System.out.println(language);
    }

    public static void main(String[] args) {
        detect();
    }
}

While running I am getting following exception:

Exception in thread "main" java.lang.NoClassDefFoundError: kotlin/KotlinNothingValueException
	at kotlinx.serialization.SerializersKt.serializer(Unknown Source)
	at com.github.pemistahl.lingua.internal.TrainingDataLanguageModel$Companion.fromJson(TrainingDataLanguageModel.kt:150)
	at com.github.pemistahl.lingua.api.LanguageDetector.loadLanguageModel$lingua(LanguageDetector.kt:401)
	at com.github.pemistahl.lingua.api.LanguageDetector$loadLanguageModels$1.invoke(LanguageDetector.kt:407)
	at com.github.pemistahl.lingua.api.LanguageDetector$loadLanguageModels$1.invoke(LanguageDetector.kt:79)
	at kotlin.SynchronizedLazyImpl.getValue(LazyJVM.kt:74)
	at com.github.pemistahl.lingua.api.LanguageDetector.lookUpNgramProbability$lingua(LanguageDetector.kt:390)
	at com.github.pemistahl.lingua.api.LanguageDetector.computeSumOfNgramProbabilities$lingua(LanguageDetector.kt:366)
	at com.github.pemistahl.lingua.api.LanguageDetector.computeLanguageProbabilities$lingua(LanguageDetector.kt:353)
	at com.github.pemistahl.lingua.api.LanguageDetector.computeLanguageConfidenceValues(LanguageDetector.kt:162)
	at com.github.pemistahl.lingua.api.LanguageDetector.detectLanguageOf(LanguageDetector.kt:102)
	at com.tomtom.ssv.apt.ingestion.service.LanguageDetector.detect(LanguageDetector.java:67)
	at com.tomtom.ssv.apt.ingestion.service.LanguageDetector.main(LanguageDetector.java:72)
Caused by: java.lang.ClassNotFoundException: kotlin.KotlinNothingValueException
	at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	... 13 more

Google's Compact Language Detectors (CLD) are good libraries that are used in Chrome browser and in many other projects. While being written in C++ they have wrappers for Java (cld2, cld3) and Python (cld2, cld3). While 2nd version is n-gram based, 3rd version uses Neural Networks.

Please compare their performance on your test set, both accuracy and speed wise.

LanguageDetector detector = LanguageDetectorBuilder.fromLanguages(ENGLISH,CHINESE , THAI, VIETNAMESE).build();
SortedMap<Language, Double> languageDoubleSortedMap = detector.computeLanguageConfidenceValues("ี่มีประสิทธิภาพหลอดไฟพลังงานแสงอาทิตย์กลางแจ้งเซ็นเซอร์ตรวจจับการเคลื่อนไหวสวนกันน้ำ LED พลังงานแสงอาทิตย์โคมไฟสปอร์ตไลท์สำหรับ Garden เส้นทางถนนแบ็คดรอปเป่าลม Led Light");
System.out.println(languageDoubleSortedMap);

The following information is printed : {ENGLISH=1.0, VIETNAMESE=0.5658177137374878} I think it's Thai, but I can recognize English, even Vietnamese, and Thai doesn't version is : 1.2.2

Great tool - thank you! Suggestion: The possibility to add OTHER as a language. Lets say I want to find English and French in a multi-language set. I want to add English and French to LanguageDetectorBuilder.from_languages, but if the probability is low, I don't want everything to be marked as English or French, but something else -> Other.

Currently, the language models are parsed from json files and loaded into simple maps at runtime. Even though accessing the maps is pretty fast, they consume a significant amount of memory. The goal is to investigate whether there are more suitable data structures available that require less storage space in memory, something like NumPy for Python. Perhaps it is even possible to store those data structures in some kind of binary format on disk which can be loaded faster than the current json files.

Promising candidates could be:

• EJML
• Colt
• la4j
• Apache Commons Math

Related to #119

Currently Lingua uses ForkJoinPool.commonPool() for model loading and language detection. However, maybe it would be useful to allow users to specify their own Executor, for example with LanguageDetectorBuilder.withExecutor(Executor) (the default could still be commonPool()). This would have the following advantages:

• could customize worker thread count, or even run single-threaded, e.g. executor = r -> r.run()
• could customize worker threads:
  • custom name to make performance monitoring easier
  • custom priority

It would not be possible anymore to use invokeAll then, but a helper function such as the following one might add the missing functionality:

private fun <E> executeTasks(tasks: List<Callable<E>>): List<E> {
    val futures = tasks.map { FutureTask(it) }
    futures.forEach(executor::execute)
    return futures.map(Future<E>::get)
}

(Note that I have not extensively checked how well this performs compared to invokeAll, and whether exception collection from the Futures could be improved. Probably this implementation is flawed because called would wait on get() call without participating in the work.) Alternatively CompletableFuture could be used; but then care must be taken to not use ForkJoinPool.commonPool() when its parallelism is 1, otherwise performance might be pretty bad due to JDK-8213115.

This would require some changes to the documentation which currently explicitly refers to ForkJoinPool.commonPool().

What do you think?

Hello!

So I've been trying out the lingua library and it's awesome. Was wondering if it's possible to add other classification metrics such as Precision, Recall, Specificity and F1 in the comparisons between tika, Optimaize and the other java language detection libraries for more transparency?

Thanks!