Markup is an online annotation tool that can be used to transform unstructured documents into structured formats for NLP and ML tasks, such as named-entity recognition. Markup learns as you annotate in order to predict and suggest complex annotations. Markup also provides integrated access to existing and custom ontologies, enabling the prediction and suggestion of ontology mappings based on the text you're annotating.

Samuel Dobbie

Last update: Dec 18, 2022

Related tags

Text Processing machine-learning natural-language-processing sequence-to-sequence active-learning annotation-tool text-annotation

Overview

What is Markup?

Markup is an online annotation tool that can be used to transform unstructured documents into structured formats for NLP and ML tasks, such as named-entity recognition. Markup learns as you annotate in order to predict and suggest complex annotations. Markup also provides integrated access to existing and custom ontologies, enabling the prediction and suggestion of ontology mappings based on the text you're annotating.

Usage

A full-feature version of Markup is available both via website and local installation.

Online

The online version of Markup can be found here.

Local Server

Docker

Run docker run -d -p 8000:8000 samueldobbie/markup and visit http://localhost:8000.

Manual Installation

Clone or download the repository.
Run python setup.py using 64-bit Python3.
Visit http://localhost:8000.

For futher sessions, the local server can be started directly by running python manage.py runserver localhost:8000.

Documentation

Documentation to help with setting up and using Markup can be found here.

Features

Ability to navigate between and annotate multiple documents in a single session.
Predictive annotation suggestions (incl. attributes) using underlying active learning and sequence-to-sequence models.
Integrated access to pre-loaded and user-defined ontologies, enabling predictive mappings and direct querying.
Built-in configuration file creator.
Built-in synthetic data generator and custom model trainer (local version only due to high computational expense).
Dynamic attribute display.
Any number of overlaying annotations, enabling the capture of complex data.
Full-feature tool available via local installation and website.
Dark mode.

Future Plans

Add user accounts.
Add ability for users to join a team and share ontologies, documents, guidelines, annotations, etc.
Accessible version for colour-blind users.
Add ability to perform text and image classification.
Add ability to annotate images.

Known Bugs / Issues

Annotations may be offset when annotating across newlines in CRLF (Windows) text documents. The offset is purely visual; the exported indicies will be correct.
When using the website version of Markup, certain features may freeze while annotations are being predicted.

Comments

You need to provide valid config file. ERROR

I am attempting to create a new page, similar to annotate, called analyse. Initially, I simply duplicated the annotate file and renamed all relevant annotate variables to analyse. Now when I upload a folder and try to start a session I receive the error shown below which asks for me to provide a valid config file. I am unsure why this issue arises as following the functions called when clicking the button everything for analyse is the same as annotate. So I am uncertain if the issue lies in another function which actually uploads the files selected on the setup page.

Any help with this issue would be greatly appreciated and please let me know if you need me to elaborate any more.

Thank you, William

opened by williamtraynor 4
updating config file on website causes blank screen when clicking session start

After going into settings in a session, updating the config file and then clikcing start session, the browser begins to load the session but gets stuck at a white screen. @21Monica02
bug

opened by arronlacey 3
Conf File CRLF vs LF

Issue with end of line character for .conf files. Cannot attach files as .conf so are as .conf.txt. Only LF will work. annotationCRLF.txt annotationLF.conf.txt Image of console error also provided - parseCategories() in Helper.ts looks like the issue
bug

opened by huwstrafford 1
Bump tensorflow from 2.5.2 to 2.7.2
Bumps tensorflow from 2.5.2 to 2.7.2.

Release notes

Sourced from tensorflow's releases.

TensorFlow 2.7.2

Release 2.7.2

This releases introduces several vulnerability fixes:

Fixes a code injection in saved_model_cli (CVE-2022-29216)

Fixes a missing validation which causes TensorSummaryV2 to crash (CVE-2022-29193)

Fixes a missing validation which crashes QuantizeAndDequantizeV4Grad (CVE-2022-29192)

Fixes a missing validation which causes denial of service via DeleteSessionTensor (CVE-2022-29194)

Fixes a missing validation which causes denial of service via GetSessionTensor (CVE-2022-29191)

Fixes a missing validation which causes denial of service via StagePeek (CVE-2022-29195)

Fixes a missing validation which causes denial of service via UnsortedSegmentJoin (CVE-2022-29197)

Fixes a missing validation which causes denial of service via LoadAndRemapMatrix (CVE-2022-29199)

Fixes a missing validation which causes denial of service via SparseTensorToCSRSparseMatrix (CVE-2022-29198)

Fixes a missing validation which causes denial of service via LSTMBlockCell (CVE-2022-29200)

Fixes a missing validation which causes denial of service via Conv3DBackpropFilterV2 (CVE-2022-29196)

Fixes a CHECK failure in depthwise ops via overflows (CVE-2021-41197)

Fixes issues arising from undefined behavior stemming from users supplying invalid resource handles (CVE-2022-29207)

Fixes a segfault due to missing support for quantized types (CVE-2022-29205)

Fixes a missing validation which results in undefined behavior in SparseTensorDenseAdd (CVE-2022-29206)

Fixes a missing validation which results in undefined behavior in QuantizedConv2D (CVE-2022-29201)

Fixes an integer overflow in SpaceToBatchND (CVE-2022-29203)

Fixes a segfault and OOB write due to incomplete validation in EditDistance (CVE-2022-29208)

Fixes a missing validation which causes denial of service via Conv3DBackpropFilterV2 (CVE-2022-29204)

Fixes a denial of service in tf.ragged.constant due to lack of validation (CVE-2022-29202)

Fixes a segfault when tf.histogram_fixed_width is called with NaN values (CVE-2022-29211)

Fixes a core dump when loading TFLite models with quantization (CVE-2022-29212)

Fixes crashes stemming from incomplete validation in signal ops (CVE-2022-29213)

Fixes a type confusion leading to CHECK-failure based denial of service (CVE-2022-29209)

Updates curl to 7.83.1 to handle (CVE-2022-22576, (CVE-2022-27774, (CVE-2022-27775, (CVE-2022-27776, (CVE-2022-27778, (CVE-2022-27779, (CVE-2022-27780, (CVE-2022-27781, (CVE-2022-27782 and (CVE-2022-30115

Updates zlib to 1.2.12 after 1.2.11 was pulled due to security issue

TensorFlow 2.7.1

Release 2.7.1

This releases introduces several vulnerability fixes:

Fixes a floating point division by 0 when executing convolution operators (CVE-2022-21725)

Fixes a heap OOB read in shape inference for ReverseSequence (CVE-2022-21728)

Fixes a heap OOB access in Dequantize (CVE-2022-21726)

Fixes an integer overflow in shape inference for Dequantize (CVE-2022-21727)

Fixes a heap OOB access in FractionalAvgPoolGrad (CVE-2022-21730)

Fixes an overflow and divide by zero in UnravelIndex (CVE-2022-21729)

Fixes a type confusion in shape inference for ConcatV2 (CVE-2022-21731)

Fixes an OOM in ThreadPoolHandle (CVE-2022-21732)

Fixes an OOM due to integer overflow in StringNGrams (CVE-2022-21733)

Fixes more issues caused by incomplete validation in boosted trees code (CVE-2021-41208)

Fixes an integer overflows in most sparse component-wise ops (CVE-2022-23567)

Fixes an integer overflows in AddManySparseToTensorsMap (CVE-2022-23568)

... (truncated)

Changelog

Sourced from tensorflow's changelog.

Release 2.7.2

This releases introduces several vulnerability fixes:

Fixes a code injection in saved_model_cli (CVE-2022-29216)

Fixes a missing validation which causes TensorSummaryV2 to crash (CVE-2022-29193)

Fixes a missing validation which crashes QuantizeAndDequantizeV4Grad (CVE-2022-29192)

Fixes a missing validation which causes denial of service via DeleteSessionTensor (CVE-2022-29194)

Fixes a missing validation which causes denial of service via GetSessionTensor (CVE-2022-29191)

Fixes a missing validation which causes denial of service via StagePeek (CVE-2022-29195)

Fixes a missing validation which causes denial of service via UnsortedSegmentJoin (CVE-2022-29197)

Fixes a missing validation which causes denial of service via LoadAndRemapMatrix (CVE-2022-29199)

Fixes a missing validation which causes denial of service via SparseTensorToCSRSparseMatrix (CVE-2022-29198)

Fixes a missing validation which causes denial of service via LSTMBlockCell (CVE-2022-29200)

Fixes a missing validation which causes denial of service via Conv3DBackpropFilterV2 (CVE-2022-29196)

Fixes a CHECK failure in depthwise ops via overflows (CVE-2021-41197)

Fixes issues arising from undefined behavior stemming from users supplying invalid resource handles (CVE-2022-29207)

Fixes a segfault due to missing support for quantized types (CVE-2022-29205)

Fixes a missing validation which results in undefined behavior in SparseTensorDenseAdd (CVE-2022-29206)

Fixes a missing validation which results in undefined behavior in QuantizedConv2D (CVE-2022-29201)

Fixes an integer overflow in SpaceToBatchND (CVE-2022-29203)

Fixes a segfault and OOB write due to incomplete validation in EditDistance (CVE-2022-29208)

Fixes a missing validation which causes denial of service via Conv3DBackpropFilterV2 (CVE-2022-29204)

Fixes a denial of service in tf.ragged.constant due to lack of validation (CVE-2022-29202)

Fixes a segfault when tf.histogram_fixed_width is called with NaN values (CVE-2022-29211)

Fixes a core dump when loading TFLite models with quantization (CVE-2022-29212)

Fixes crashes stemming from incomplete validation in signal ops (CVE-2022-29213)

Fixes a type confusion leading to CHECK-failure based denial of service (CVE-2022-29209)

Updates curl to 7.83.1 to handle (CVE-2022-22576, (CVE-2022-27774, (CVE-2022-27775, (CVE-2022-27776, (CVE-2022-27778, (CVE-2022-27779, (CVE-2022-27780, (CVE-2022-27781, (CVE-2022-27782 and (CVE-2022-30115

Updates zlib to 1.2.12 after 1.2.11 was pulled due to security issue

Release 2.6.4

This releases introduces several vulnerability fixes:

Fixes a code injection in saved_model_cli (CVE-2022-29216)

Fixes a missing validation which causes TensorSummaryV2 to crash (CVE-2022-29193)

Fixes a missing validation which crashes QuantizeAndDequantizeV4Grad (CVE-2022-29192)

Fixes a missing validation which causes denial of service via DeleteSessionTensor (CVE-2022-29194)

Fixes a missing validation which causes denial of service via GetSessionTensor (CVE-2022-29191)

Fixes a missing validation which causes denial of service via StagePeek (CVE-2022-29195)

Fixes a missing validation which causes denial of service via UnsortedSegmentJoin (CVE-2022-29197)

Fixes a missing validation which causes denial of service via LoadAndRemapMatrix (CVE-2022-29199)

Fixes a missing validation which causes denial of service via SparseTensorToCSRSparseMatrix (CVE-2022-29198)

Fixes a missing validation which causes denial of service via LSTMBlockCell (CVE-2022-29200)

Fixes a missing validation which causes denial of service via Conv3DBackpropFilterV2 (CVE-2022-29196)

Fixes a CHECK failure in depthwise ops via overflows (CVE-2021-41197)

Fixes issues arising from undefined behavior stemming from users supplying invalid resource handles (CVE-2022-29207)

Fixes a segfault due to missing support for quantized types (CVE-2022-29205)

Fixes a missing validation which results in undefined behavior in SparseTensorDenseAdd (CVE-2022-29206)

... (truncated)

Commits

dd7b8a3 Merge pull request #56034 from tensorflow-jenkins/relnotes-2.7.2-15779

1e7d6ea Update RELEASE.md

5085135 Merge pull request #56069 from tensorflow/mm-cp-52488e5072f6fe44411d70c6af09e...

adafb45 Merge pull request #56060 from yongtang:curl-7.83.1

01cb1b8 Merge pull request #56038 from tensorflow-jenkins/version-numbers-2.7.2-4733

8c90c2f Update version numbers to 2.7.2

43f3cdc Update RELEASE.md

98b0a48 Insert release notes place-fill

dfa5cf3 Merge pull request #56028 from tensorflow/disable-tests-on-r2.7

501a65c Disable timing out tests

Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

@dependabot rebase will rebase this PR

@dependabot recreate will recreate this PR, overwriting any edits that have been made to it

@dependabot merge will merge this PR after your CI passes on it

@dependabot squash and merge will squash and merge this PR after your CI passes on it

@dependabot cancel merge will cancel a previously requested merge and block automerging

@dependabot reopen will reopen this PR if it is closed

@dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually

@dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

@dependabot use these labels will set the current labels as the default for future PRs for this repo and language

@dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language

@dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language

@dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

You can disable automated security fix PRs for this repo from the Security Alerts page.

dependencies
opened by dependabot[bot] 1
Bump tensorflow from 2.5.2 to 2.6.4
Bumps tensorflow from 2.5.2 to 2.6.4.

Release notes

Sourced from tensorflow's releases.

TensorFlow 2.6.4

Release 2.6.4

This releases introduces several vulnerability fixes:

Fixes a code injection in saved_model_cli (CVE-2022-29216)

Fixes a missing validation which causes TensorSummaryV2 to crash (CVE-2022-29193)

Fixes a missing validation which crashes QuantizeAndDequantizeV4Grad (CVE-2022-29192)

Fixes a missing validation which causes denial of service via DeleteSessionTensor (CVE-2022-29194)

Fixes a missing validation which causes denial of service via GetSessionTensor (CVE-2022-29191)

Fixes a missing validation which causes denial of service via StagePeek (CVE-2022-29195)

Fixes a missing validation which causes denial of service via UnsortedSegmentJoin (CVE-2022-29197)

Fixes a missing validation which causes denial of service via LoadAndRemapMatrix (CVE-2022-29199)

Fixes a missing validation which causes denial of service via SparseTensorToCSRSparseMatrix (CVE-2022-29198)

Fixes a missing validation which causes denial of service via LSTMBlockCell (CVE-2022-29200)

Fixes a missing validation which causes denial of service via Conv3DBackpropFilterV2 (CVE-2022-29196)

Fixes a CHECK failure in depthwise ops via overflows (CVE-2021-41197)

Fixes issues arising from undefined behavior stemming from users supplying invalid resource handles (CVE-2022-29207)

Fixes a segfault due to missing support for quantized types (CVE-2022-29205)

Fixes a missing validation which results in undefined behavior in SparseTensorDenseAdd (CVE-2022-29206)

Fixes a missing validation which results in undefined behavior in QuantizedConv2D (CVE-2022-29201)

Fixes an integer overflow in SpaceToBatchND (CVE-2022-29203)

Fixes a segfault and OOB write due to incomplete validation in EditDistance (CVE-2022-29208)

Fixes a missing validation which causes denial of service via Conv3DBackpropFilterV2 (CVE-2022-29204)

Fixes a denial of service in tf.ragged.constant due to lack of validation (CVE-2022-29202)

Fixes a segfault when tf.histogram_fixed_width is called with NaN values (CVE-2022-29211)

Fixes a core dump when loading TFLite models with quantization (CVE-2022-29212)

Fixes crashes stemming from incomplete validation in signal ops (CVE-2022-29213)

Fixes a type confusion leading to CHECK-failure based denial of service (CVE-2022-29209)

Updates curl to 7.83.1 to handle (CVE-2022-22576, (CVE-2022-27774, (CVE-2022-27775, (CVE-2022-27776, (CVE-2022-27778, (CVE-2022-27779, (CVE-2022-27780, (CVE-2022-27781, (CVE-2022-27782 and (CVE-2022-30115

Updates zlib to 1.2.12 after 1.2.11 was pulled due to security issue

TensorFlow 2.6.3

Release 2.6.3

This releases introduces several vulnerability fixes:

Fixes a floating point division by 0 when executing convolution operators (CVE-2022-21725)

Fixes a heap OOB read in shape inference for ReverseSequence (CVE-2022-21728)

Fixes a heap OOB access in Dequantize (CVE-2022-21726)

Fixes an integer overflow in shape inference for Dequantize (CVE-2022-21727)

Fixes a heap OOB access in FractionalAvgPoolGrad (CVE-2022-21730)

Fixes an overflow and divide by zero in UnravelIndex (CVE-2022-21729)

Fixes a type confusion in shape inference for ConcatV2 (CVE-2022-21731)

Fixes an OOM in ThreadPoolHandle (CVE-2022-21732)

Fixes an OOM due to integer overflow in StringNGrams (CVE-2022-21733)

Fixes more issues caused by incomplete validation in boosted trees code (CVE-2021-41208)

Fixes an integer overflows in most sparse component-wise ops (CVE-2022-23567)

Fixes an integer overflows in AddManySparseToTensorsMap (CVE-2022-23568)

Fixes a number of CHECK-failures in MapStage (CVE-2022-21734)

... (truncated)

Changelog

Sourced from tensorflow's changelog.

Release 2.6.4

This releases introduces several vulnerability fixes:

Fixes a code injection in saved_model_cli (CVE-2022-29216)

Fixes a missing validation which causes TensorSummaryV2 to crash (CVE-2022-29193)

Fixes a missing validation which crashes QuantizeAndDequantizeV4Grad (CVE-2022-29192)

Fixes a missing validation which causes denial of service via DeleteSessionTensor (CVE-2022-29194)

Fixes a missing validation which causes denial of service via GetSessionTensor (CVE-2022-29191)

Fixes a missing validation which causes denial of service via StagePeek (CVE-2022-29195)

Fixes a missing validation which causes denial of service via UnsortedSegmentJoin (CVE-2022-29197)

Fixes a missing validation which causes denial of service via LoadAndRemapMatrix (CVE-2022-29199)

Fixes a missing validation which causes denial of service via SparseTensorToCSRSparseMatrix (CVE-2022-29198)

Fixes a missing validation which causes denial of service via LSTMBlockCell (CVE-2022-29200)

Fixes a missing validation which causes denial of service via Conv3DBackpropFilterV2 (CVE-2022-29196)

Fixes a CHECK failure in depthwise ops via overflows (CVE-2021-41197)

Fixes issues arising from undefined behavior stemming from users supplying invalid resource handles (CVE-2022-29207)

Fixes a segfault due to missing support for quantized types (CVE-2022-29205)

Fixes a missing validation which results in undefined behavior in SparseTensorDenseAdd (CVE-2022-29206)

Fixes a missing validation which results in undefined behavior in QuantizedConv2D (CVE-2022-29201)

Fixes an integer overflow in SpaceToBatchND (CVE-2022-29203)

Fixes a segfault and OOB write due to incomplete validation in EditDistance (CVE-2022-29208)

Fixes a missing validation which causes denial of service via Conv3DBackpropFilterV2 (CVE-2022-29204)

Fixes a denial of service in tf.ragged.constant due to lack of validation (CVE-2022-29202)

Fixes a segfault when tf.histogram_fixed_width is called with NaN values (CVE-2022-29211)

Fixes a core dump when loading TFLite models with quantization (CVE-2022-29212)

Fixes crashes stemming from incomplete validation in signal ops (CVE-2022-29213)

Fixes a type confusion leading to CHECK-failure based denial of service (CVE-2022-29209)

Updates curl to 7.83.1 to handle (CVE-2022-22576, (CVE-2022-27774, (CVE-2022-27775, (CVE-2022-27776, (CVE-2022-27778, (CVE-2022-27779, (CVE-2022-27780, (CVE-2022-27781, (CVE-2022-27782 and (CVE-2022-30115

Updates zlib to 1.2.12 after 1.2.11 was pulled due to security issue

Release 2.8.0

Major Features and Improvements

tf.lite:

Added TFLite builtin op support for the following TF ops:

tf.raw_ops.Bucketize op on CPU.

tf.where op for data types tf.int32/tf.uint32/tf.int8/tf.uint8/tf.int64.

tf.random.normal op for output data type tf.float32 on CPU.

tf.random.uniform op for output data type tf.float32 on CPU.

tf.random.categorical op for output data type tf.int64 on CPU.

tensorflow.experimental.tensorrt:

conversion_params is now deprecated inside TrtGraphConverterV2 in favor of direct arguments: max_workspace_size_bytes, precision_mode, minimum_segment_size, maximum_cached_engines, use_calibration and

... (truncated)

Commits

33ed2b1 Merge pull request #56102 from tensorflow/mihaimaruseac-patch-1

e1ec480 Fix build due to importlib-metadata/setuptools

63f211c Merge pull request #56033 from tensorflow-jenkins/relnotes-2.6.4-6677

22b8fe4 Update RELEASE.md

ec30684 Merge pull request #56070 from tensorflow/mm-cp-adafb45c781-on-r2.6

38774ed Merge pull request #56060 from yongtang:curl-7.83.1

9ef1604 Merge pull request #56036 from tensorflow-jenkins/version-numbers-2.6.4-9925

a6526a3 Update version numbers to 2.6.4

cb1a481 Update RELEASE.md

4da550f Insert release notes place-fill

Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

@dependabot rebase will rebase this PR

@dependabot recreate will recreate this PR, overwriting any edits that have been made to it

@dependabot merge will merge this PR after your CI passes on it

@dependabot squash and merge will squash and merge this PR after your CI passes on it

@dependabot cancel merge will cancel a previously requested merge and block automerging

@dependabot reopen will reopen this PR if it is closed

@dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually

@dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

@dependabot use these labels will set the current labels as the default for future PRs for this repo and language

@dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language

@dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language

@dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

You can disable automated security fix PRs for this repo from the Security Alerts page.

dependencies
opened by dependabot[bot] 1
Record multiple values for the same attribute

If I have an entity A, and it has an attribute B, could we allow to record multiple values for B. i.e. if B denotes risk factors for heart disease there might be a drop down list that includes smoking, high BMI and we would want to include both of those. Currently we can only pick one and the workaround is to annotate two separate entities, one for each attribute value.
enhancement

opened by arronlacey 1
Bump tensorflow from 2.5.2 to 2.5.3
Bumps tensorflow from 2.5.2 to 2.5.3.

Release notes

Sourced from tensorflow's releases.

TensorFlow 2.5.3

Release 2.5.3

Note: This is the last release in the 2.5 series.

This releases introduces several vulnerability fixes:

Fixes a floating point division by 0 when executing convolution operators (CVE-2022-21725)

Fixes a heap OOB read in shape inference for ReverseSequence (CVE-2022-21728)

Fixes a heap OOB access in Dequantize (CVE-2022-21726)

Fixes an integer overflow in shape inference for Dequantize (CVE-2022-21727)

Fixes a heap OOB access in FractionalAvgPoolGrad (CVE-2022-21730)

Fixes an overflow and divide by zero in UnravelIndex (CVE-2022-21729)

Fixes a type confusion in shape inference for ConcatV2 (CVE-2022-21731)

Fixes an OOM in ThreadPoolHandle (CVE-2022-21732)

Fixes an OOM due to integer overflow in StringNGrams (CVE-2022-21733)

Fixes more issues caused by incomplete validation in boosted trees code (CVE-2021-41208)

Fixes an integer overflows in most sparse component-wise ops (CVE-2022-23567)

Fixes an integer overflows in AddManySparseToTensorsMap (CVE-2022-23568)

Fixes a number of CHECK-failures in MapStage (CVE-2022-21734)

Fixes a division by zero in FractionalMaxPool (CVE-2022-21735)

Fixes a number of CHECK-fails when building invalid/overflowing tensor shapes (CVE-2022-23569)

Fixes an undefined behavior in SparseTensorSliceDataset (CVE-2022-21736)

Fixes an assertion failure based denial of service via faulty bin count operations (CVE-2022-21737)

Fixes a reference binding to null pointer in QuantizedMaxPool (CVE-2022-21739)

Fixes an integer overflow leading to crash in SparseCountSparseOutput (CVE-2022-21738)

Fixes a heap overflow in SparseCountSparseOutput (CVE-2022-21740)

Fixes an FPE in BiasAndClamp in TFLite (CVE-2022-23557)

Fixes an FPE in depthwise convolutions in TFLite (CVE-2022-21741)

Fixes an integer overflow in TFLite array creation (CVE-2022-23558)

Fixes an integer overflow in TFLite (CVE-2022-23559)

Fixes a dangerous OOB write in TFLite (CVE-2022-23561)

Fixes a vulnerability leading to read and write outside of bounds in TFLite (CVE-2022-23560)

Fixes a set of vulnerabilities caused by using insecure temporary files (CVE-2022-23563)

Fixes an integer overflow in Range resulting in undefined behavior and OOM (CVE-2022-23562)

Fixes a vulnerability where missing validation causes tf.sparse.split to crash when axis is a tuple (CVE-2021-41206)

Fixes a CHECK-fail when decoding resource handles from proto (CVE-2022-23564)

Fixes a CHECK-fail with repeated AttrDef (CVE-2022-23565)

Fixes a heap OOB write in Grappler (CVE-2022-23566)

Fixes a CHECK-fail when decoding invalid tensors from proto (CVE-2022-23571)

Fixes an unitialized variable access in AssignOp (CVE-2022-23573)

Fixes an integer overflow in OpLevelCostEstimator::CalculateTensorSize (CVE-2022-23575)

Fixes an integer overflow in OpLevelCostEstimator::CalculateOutputSize (CVE-2022-23576)

Fixes a null dereference in GetInitOp (CVE-2022-23577)

Fixes a memory leak when a graph node is invalid (CVE-2022-23578)

Fixes an abort caused by allocating a vector that is too large (CVE-2022-23580)

Fixes multiple CHECK-failures during Grappler's IsSimplifiableReshape (CVE-2022-23581)

Fixes multiple CHECK-failures during Grappler's SafeToRemoveIdentity (CVE-2022-23579)

Fixes multiple CHECK-failures in TensorByteSize (CVE-2022-23582)

Fixes multiple CHECK-failures in binary ops due to type confusion (CVE-2022-23583)

... (truncated)

Changelog

Sourced from tensorflow's changelog.

Release 2.5.3

This releases introduces several vulnerability fixes:

Fixes a floating point division by 0 when executing convolution operators (CVE-2022-21725)

Fixes a heap OOB read in shape inference for ReverseSequence (CVE-2022-21728)

Fixes a heap OOB access in Dequantize (CVE-2022-21726)

Fixes an integer overflow in shape inference for Dequantize (CVE-2022-21727)

Fixes a heap OOB access in FractionalAvgPoolGrad (CVE-2022-21730)

Fixes an overflow and divide by zero in UnravelIndex (CVE-2022-21729)

Fixes a type confusion in shape inference for ConcatV2 (CVE-2022-21731)

Fixes an OOM in ThreadPoolHandle (CVE-2022-21732)

Fixes an OOM due to integer overflow in StringNGrams (CVE-2022-21733)

Fixes more issues caused by incomplete validation in boosted trees code (CVE-2021-41208)

Fixes an integer overflows in most sparse component-wise ops (CVE-2022-23567)

Fixes an integer overflows in AddManySparseToTensorsMap (CVE-2022-23568)

Fixes a number of CHECK-failures in MapStage (CVE-2022-21734)

Fixes a division by zero in FractionalMaxPool (CVE-2022-21735)

Fixes a number of CHECK-fails when building invalid/overflowing tensor shapes (CVE-2022-23569)

Fixes an undefined behavior in SparseTensorSliceDataset (CVE-2022-21736)

Fixes an assertion failure based denial of service via faulty bin count operations (CVE-2022-21737)

Fixes a reference binding to null pointer in QuantizedMaxPool (CVE-2022-21739)

Fixes an integer overflow leading to crash in SparseCountSparseOutput (CVE-2022-21738)

Fixes a heap overflow in SparseCountSparseOutput (CVE-2022-21740)

Fixes an FPE in BiasAndClamp in TFLite (CVE-2022-23557)

Fixes an FPE in depthwise convolutions in TFLite (CVE-2022-21741)

... (truncated)

Commits

959e9b2 Merge pull request #54213 from tensorflow/fix-sanity-on-r2.5

d05fcbc Fix sanity build

f2526a0 Merge pull request #54205 from tensorflow/disable-flaky-tests-on-r2.5

a5f94df Disable flaky test

7babe52 Merge pull request #54201 from tensorflow/cherrypick-510ae18200d0a4fad797c0bf...

0e5d378 Set Env Variable to override Setuptools new behavior

fdd4195 Merge pull request #54176 from tensorflow-jenkins/relnotes-2.5.3-6805

4083165 Update RELEASE.md

a2bb7f1 Merge pull request #54185 from tensorflow/cherrypick-d437dec4d549fc30f9b85c75...

5777ea3 Update third_party/icu/workspace.bzl

Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

@dependabot rebase will rebase this PR

@dependabot recreate will recreate this PR, overwriting any edits that have been made to it

@dependabot merge will merge this PR after your CI passes on it

@dependabot squash and merge will squash and merge this PR after your CI passes on it

@dependabot cancel merge will cancel a previously requested merge and block automerging

@dependabot reopen will reopen this PR if it is closed

@dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually

@dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

@dependabot use these labels will set the current labels as the default for future PRs for this repo and language

@dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language

@dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language

@dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

You can disable automated security fix PRs for this repo from the Security Alerts page.

dependencies
opened by dependabot[bot] 1
Bump django from 3.1.2 to 3.1.10
Bumps django from 3.1.2 to 3.1.10.

Commits

a2407cd [3.1.x] Bumped version for 3.1.10 release.

afb23f5 [3.1.x] Fixed #32713, Fixed CVE-2021-32052 -- Prevented newlines and tabs fro...

fdbf4a7 [3.1.x] Refs CVE-2021-31542 -- Skipped mock AWS storage test on Windows.

48b39a8 [3.1.x] Added CVE-2021-31542 to security archive.

8012441 [3.1.x] Post-release version bump.

8284fd6 [3.1.x] Bumped version for 3.1.9 release.

25d84d6 [3.1.x] Fixed CVE-2021-31542 -- Tightened path & file name sanitation in file...

6b0c7e6 [3.1.x] Added CVE-2021-28658 to security archive.

5b9ca81 [3.1.x] Post-release version bump.

c4928c9 [3.1.x] Bumped version for 3.1.8 release.

Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

@dependabot rebase will rebase this PR

@dependabot recreate will recreate this PR, overwriting any edits that have been made to it

@dependabot merge will merge this PR after your CI passes on it

@dependabot squash and merge will squash and merge this PR after your CI passes on it

@dependabot cancel merge will cancel a previously requested merge and block automerging

@dependabot reopen will reopen this PR if it is closed

@dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually

@dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

@dependabot use these labels will set the current labels as the default for future PRs for this repo and language

@dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language

@dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language

@dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

You can disable automated security fix PRs for this repo from the Security Alerts page.

dependencies
opened by dependabot[bot] 1
Bump django from 3.1.2 to 3.1.9
Bumps django from 3.1.2 to 3.1.9.

Commits

8284fd6 [3.1.x] Bumped version for 3.1.9 release.

25d84d6 [3.1.x] Fixed CVE-2021-31542 -- Tightened path & file name sanitation in file...

6b0c7e6 [3.1.x] Added CVE-2021-28658 to security archive.

5b9ca81 [3.1.x] Post-release version bump.

c4928c9 [3.1.x] Bumped version for 3.1.8 release.

cca0d98 [3.1.x] Fixed CVE-2021-28658 -- Fixed potential directory-traversal via uploa...

6eb01cb [3.1.x] Fixed #32576 -- Corrected dumpdata docs for passing model names to th...

11d241d [3.1.x] Refs #25735 -- Added tags/exclude_tags arguments to DiscoverRunner docs.

4a10c31 [3.1.x] Added parallel argument to DiscoverRunner docs.

c528c71 [3.1.x] Corrected DiscoverRunner.build_suite() signature.

Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

@dependabot rebase will rebase this PR

@dependabot recreate will recreate this PR, overwriting any edits that have been made to it

@dependabot merge will merge this PR after your CI passes on it

@dependabot squash and merge will squash and merge this PR after your CI passes on it

@dependabot cancel merge will cancel a previously requested merge and block automerging

@dependabot reopen will reopen this PR if it is closed

@dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually

@dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

@dependabot use these labels will set the current labels as the default for future PRs for this repo and language

@dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language

@dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language

@dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

You can disable automated security fix PRs for this repo from the Security Alerts page.

dependencies
opened by dependabot[bot] 1
Bump tensorflow from 2.3.1 to 2.5.0
Bumps tensorflow from 2.3.1 to 2.5.0.

Release notes

Sourced from tensorflow's releases.

TensorFlow 2.5.0

Release 2.5.0

Major Features and Improvements

Support for Python3.9 has been added.

tf.data:

tf.data service now supports strict round-robin reads, which is useful for synchronous training workloads where example sizes vary. With strict round robin reads, users can guarantee that consumers get similar-sized examples in the same step.

tf.data service now supports optional compression. Previously data would always be compressed, but now you can disable compression by passing compression=None to tf.data.experimental.service.distribute(...).

tf.data.Dataset.batch() now supports num_parallel_calls and deterministic arguments. num_parallel_calls is used to indicate that multiple input batches should be computed in parallel. With num_parallel_calls set, deterministic is used to indicate that outputs can be obtained in the non-deterministic order.

Options returned by tf.data.Dataset.options() are no longer mutable.

tf.data input pipelines can now be executed in debug mode, which disables any asynchrony, parallelism, or non-determinism and forces Python execution (as opposed to trace-compiled graph execution) of user-defined functions passed into transformations such as map. The debug mode can be enabled through tf.data.experimental.enable_debug_mode().

tf.lite

Enabled the new MLIR-based quantization backend by default

The new backend is used for 8 bits full integer post-training quantization

The new backend removes the redundant rescales and fixes some bugs (shared weight/bias, extremely small scales, etc)

Set experimental_new_quantizer in tf.lite.TFLiteConverter to False to disable this change

tf.keras

tf.keras.metrics.AUC now support logit predictions.

Enabled a new supported input type in Model.fit, tf.keras.utils.experimental.DatasetCreator, which takes a callable, dataset_fn. DatasetCreator is intended to work across all tf.distribute strategies, and is the only input type supported for Parameter Server strategy.

tf.distribute

tf.distribute.experimental.ParameterServerStrategy now supports training with Keras Model.fit when used with DatasetCreator.

Creating tf.random.Generator under tf.distribute.Strategy scopes is now allowed (except for tf.distribute.experimental.CentralStorageStrategy and tf.distribute.experimental.ParameterServerStrategy). Different replicas will get different random-number streams.

TPU embedding support

Added profile_data_directory to EmbeddingConfigSpec in _tpu_estimator_embedding.py. This allows embedding lookup statistics gathered at runtime to be used in embedding layer partitioning decisions.

PluggableDevice

Third-party devices can now connect to TensorFlow as plug-ins through StreamExecutor C API. and PluggableDevice interface.

Add custom ops and kernels through kernel and op registration C API.

Register custom graph optimization passes with graph optimization C API.

oneAPI Deep Neural Network Library (oneDNN) CPU performance optimizations from Intel-optimized TensorFlow are now available in the official x86-64 Linux and Windows builds.

They are off by default. Enable them by setting the environment variable TF_ENABLE_ONEDNN_OPTS=1.

We do not recommend using them in GPU systems, as they have not been sufficiently tested with GPUs yet.

TensorFlow pip packages are now built with CUDA11.2 and cuDNN 8.1.0

Breaking Changes

The TF_CPP_MIN_VLOG_LEVEL environment variable has been renamed to to TF_CPP_MAX_VLOG_LEVEL which correctly describes its effect.

Bug Fixes and Other Changes

tf.keras:

Preprocessing layers API consistency changes:

StringLookup added output_mode, sparse, and pad_to_max_tokens arguments with same semantics as TextVectorization.

IntegerLookup added output_mode, sparse, and pad_to_max_tokens arguments with same semantics as TextVectorization. Renamed max_values, oov_value and mask_value to max_tokens, oov_token and mask_token to align with StringLookup and TextVectorization.

TextVectorization default for pad_to_max_tokens switched to False.

CategoryEncoding no longer supports adapt, IntegerLookup now supports equivalent functionality. max_tokens argument renamed to num_tokens.

Discretization added num_bins argument for learning bins boundaries through calling adapt on a dataset. Renamed bins argument to bin_boundaries for specifying bins without adapt.

Improvements to model saving/loading:

model.load_weights now accepts paths to saved models.

... (truncated)

Changelog

Sourced from tensorflow's changelog.

Release 2.5.0

Breaking Changes

The TF_CPP_MIN_VLOG_LEVEL environment variable has been renamed to to TF_CPP_MAX_VLOG_LEVEL which correctly describes its effect.

Known Caveats

Major Features and Improvements

TPU embedding support

Added profile_data_directory to EmbeddingConfigSpec in _tpu_estimator_embedding.py. This allows embedding lookup statistics gathered at runtime to be used in embedding layer partitioning decisions.

tf.keras.metrics.AUC now support logit predictions.

Creating tf.random.Generator under tf.distribute.Strategy scopes is now allowed (except for tf.distribute.experimental.CentralStorageStrategy and tf.distribute.experimental.ParameterServerStrategy). Different replicas will get different random-number streams.

tf.data:

tf.data service now supports strict round-robin reads, which is useful for synchronous training workloads where example sizes vary. With strict round robin reads, users can guarantee that consumers get similar-sized examples in the same step.

tf.data service now supports optional compression. Previously data would always be compressed, but now you can disable compression by passing compression=None to tf.data.experimental.service.distribute(...).

tf.data.Dataset.batch() now supports num_parallel_calls and deterministic arguments. num_parallel_calls is used to indicate that multiple input batches should be computed in parallel. With num_parallel_calls set, deterministic is used to indicate that outputs can be obtained in the non-deterministic order.

Options returned by tf.data.Dataset.options() are no longer mutable.

tf.data input pipelines can now be executed in debug mode, which disables any asynchrony, parallelism, or non-determinism and forces Python execution (as opposed to trace-compiled graph execution) of user-defined functions passed into transformations such as map. The debug mode can be enabled through tf.data.experimental.enable_debug_mode().

tf.lite

Enabled the new MLIR-based quantization backend by default

The new backend is used for 8 bits full integer post-training quantization

The new backend removes the redundant rescales and fixes some bugs (shared weight/bias, extremely small scales, etc)

... (truncated)

Commits

a4dfb8d Merge pull request #49124 from tensorflow/mm-cherrypick-tf-data-segfault-fix-...

2107b1d Merge pull request #49116 from tensorflow-jenkins/version-numbers-2.5.0-17609

16b8139 Update snapshot_dataset_op.cc

86a0d86 Merge pull request #49126 from geetachavan1/cherrypicks_X9ZNY

9436ae6 Merge pull request #49128 from geetachavan1/cherrypicks_D73J5

6b2bf99 Validate that a and b are proper sparse tensors

c03ad1a Ensure validation sticks in banded_triangular_solve_op

12a6ead Merge pull request #49120 from geetachavan1/cherrypicks_KJ5M9

b67f5b8 Merge pull request #49118 from geetachavan1/cherrypicks_BIDTR

a13c0ad [tf.data][cherrypick] Fix snapshot segfault when using repeat and prefecth

Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

@dependabot rebase will rebase this PR

@dependabot recreate will recreate this PR, overwriting any edits that have been made to it

@dependabot merge will merge this PR after your CI passes on it

@dependabot squash and merge will squash and merge this PR after your CI passes on it

@dependabot cancel merge will cancel a previously requested merge and block automerging

@dependabot reopen will reopen this PR if it is closed

@dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually

@dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

@dependabot use these labels will set the current labels as the default for future PRs for this repo and language

@dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language

@dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language

@dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

You can disable automated security fix PRs for this repo from the Security Alerts page.

dependencies
opened by dependabot[bot] 1
Bump django from 3.1.2 to 3.1.8
Bumps django from 3.1.2 to 3.1.8.

Commits

c4928c9 [3.1.x] Bumped version for 3.1.8 release.

cca0d98 [3.1.x] Fixed CVE-2021-28658 -- Fixed potential directory-traversal via uploa...

6eb01cb [3.1.x] Fixed #32576 -- Corrected dumpdata docs for passing model names to th...

11d241d [3.1.x] Refs #25735 -- Added tags/exclude_tags arguments to DiscoverRunner docs.

4a10c31 [3.1.x] Added parallel argument to DiscoverRunner docs.

c528c71 [3.1.x] Corrected DiscoverRunner.build_suite() signature.

95ee8fe [3.1.x] Fixed #32560 -- Fixed test runner with --pdb and --buffer on fail/error.

b58b214 [3.1.x] Fixed typo in docs/topics/testing/advanced.txt.

0415ac5 [3.1.x] Fixed #32536 -- Added links to BaseDetailView/BaseListView.get() meth...

7c662b7 [3.1.x] Fixed typo in docs/ref/checks.txt.

Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

@dependabot rebase will rebase this PR

@dependabot recreate will recreate this PR, overwriting any edits that have been made to it

@dependabot merge will merge this PR after your CI passes on it

@dependabot squash and merge will squash and merge this PR after your CI passes on it

@dependabot cancel merge will cancel a previously requested merge and block automerging

@dependabot reopen will reopen this PR if it is closed

@dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually

@dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

@dependabot use these labels will set the current labels as the default for future PRs for this repo and language

@dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language

@dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language

@dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

You can disable automated security fix PRs for this repo from the Security Alerts page.

dependencies
opened by dependabot[bot] 1
Build Document Query functionality
Would be able to search documents for specific annotations from .ann files i.e. "find all documents where someone has focal epilepsy or CUI 39192XXXXX" etc

Could also search non-annotated letters with zero-shot / some model to retrieve letters and accept/reject to improve
opened by arronlacey 0
Add more demo corpuses
We could add different domain types for a wider audience

ConLL conference competitions have fully annotated brat format - would look great on startup

Can use different domains to show different functionality i.e. UMLS for health
opened by arronlacey 0