API & Webapp to answer questions about COVID-19. Using NLP (Question Answering) and trusted data sources.

Overview

cover-photo

This open source project serves two purposes.

  1. Collection and evaluation of a Question Answering dataset to improve existing QA/search methods - COVID-QA
  2. Question matching capabilities: Provide trustworthy answers to questions about COVID-19 via NLP - outdated

COVID-QA

Update 14th April, 2020: We are open sourcing the first batch of SQuAD style question answering annotations. Thanks to Tony Reina for managing the process and the many professional annotators who spend valuable time looking through Covid related research papers.

FAQ matching

Update 17th June, 2020: As the pandemic is thankfully slowing down and other information sources have catched up, we decided to take our hosted API and UI offline. We will keep the repository here as an inspiration for other projects and to share the COVID-QA dataset.

Problem

  • People have many questions about COVID-19
  • Answers are scattered on different websites
  • Finding the right answers takes a lot of time
  • Trustworthiness of answers is hard to judge
  • Many answers get outdated soon

💡 Idea

  • Aggregate FAQs and texts from trustworthy data sources (WHO, CDC ...)
  • Provide a UI where people can ask questions
  • Use NLP to match incoming questions of users with meaningful answers
  • Users can provide feedback about answers to improve the NLP model and flag outdated or wrong answers
  • Display most common queries without good answers to guide data collection and model improvements

⚙️ Tech

  • Scrapers to collect data
  • Elasticsearch to store texts, FAQs, embeddings
  • NLP Models implemented via Haystack to find answers via a) detecting similar question in FAQs b) detect answers in free texts (extractive QA)
  • React Frontend
Issues
  • Feature : Matched Question and Feedback Option

    Feature : Matched Question and Feedback Option

    Thanks again for creating this PR. Great work!

    Two comments:

    • Right now the answer displayed by the bot doesn't contain the "matched question". However, it might be helpful for the users to see that in order to judge if the answer is really relevant to their question. You find it in the response JSON in the field "question".
    • We now also have the option for user feedback (see API endpoint). So people can rate if the given answer was helpful or not and we will use the data to improve the NLP model. This could be also an helpful addition to the telegram bot.

    Would be great to hear your thoughts on that and maybe address them in a separate PR.

    Originally posted by @tholor in https://github.com/deepset-ai/COVID-QA/pull/58#issuecomment-602513354

    opened by theapache64 11
  • Question : API

    Question : API

    I've been developing a telegram bot using the API. Currently am using https://covid-middleware.deepset.ai/api/bert/question to get the answers.

    curl -X POST \
      https://covid-middleware.deepset.ai/api/bert/question \
      -H 'content-type: application/json' \
      -d '{
    	"question":"community spread?"
    }'
    

    but the swagger doesn't list this API and shows a different one with different structures.

    So, my question is, Which API should I choose to get the answers? @tanaysoni

    opened by theapache64 10
  • Document Retrieval for extractive QA with COVID-QA

    Document Retrieval for extractive QA with COVID-QA

    Thank you so much for sharing your data and tools! I am working with the question-answering dataset for an experiment of my own.

    @Timoeller mentioned in #103 that the documents used in the annotation tool to create the COVID-QA.json dataset "are a subset of CORD-19 papers that annotators deemed related to Covid." I was wondering if these are the same documents as listed in faq_covidbert.csv.

    The reason I ask is that, as a workaround I've created my own retrieval txt file(s) through extracting the answers from COVID-QA.json, but the results are hit or miss. They are particularly off if I break the file up into chunks to improve performance, for instance into a separate txt file for each answer. I'm assuming this is due to lost context. I'm wondering if I should simply be using faq_covidbert as illustrated here, even though I am using extractive-QA.

    The reason I did my method is that I was trying to follow an approach most closely approximating the extractive QA tutorial.

    My ultimate objective is to compare the experience of using extractive QA vs FAQ-style QA, so I presumed that it would be apropos to have a bit of separation in the doc storage dataset.

    Thank you!

    opened by aaronbriel 6
  • Docker build succeeds, docker run fails with elasticsearch error

    Docker build succeeds, docker run fails with elasticsearch error

    I followed the instructions here https://github.com/deepset-ai/COVID-QA/tree/master/backend. Perhaps a port is not configured correctly?

    INFO:     initializing identifier
    WARNING:  PUT http://localhost:9200/document [status:N/A request:0.004s]
    Traceback (most recent call last):
      File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 157, in _new_conn
        (self._dns_host, self.port), self.timeout, **extra_kw
      File "/usr/local/lib/python3.7/site-packages/urllib3/util/connection.py", line 84, in create_connection
        raise err
      File "/usr/local/lib/python3.7/site-packages/urllib3/util/connection.py", line 74, in create_connection
        sock.connect(sa)
    ConnectionRefusedError: [Errno 111] Connection refused
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "/usr/local/lib/python3.7/site-packages/elasticsearch/connection/http_urllib3.py", line 229, in perform_request
        method, url, body, retries=Retry(False), headers=request_headers, **kw
      File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 720, in urlopen
        method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
      File "/usr/local/lib/python3.7/site-packages/urllib3/util/retry.py", line 376, in increment
        raise six.reraise(type(error), error, _stacktrace)
      File "/usr/local/lib/python3.7/site-packages/urllib3/packages/six.py", line 735, in reraise
        raise value
      File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 672, in urlopen
        chunked=chunked,
      File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 387, in _make_request
        conn.request(method, url, **httplib_request_kw)
      File "/usr/local/lib/python3.7/http/client.py", line 1244, in request
        self._send_request(method, url, body, headers, encode_chunked)
      File "/usr/local/lib/python3.7/http/client.py", line 1290, in _send_request
        self.endheaders(body, encode_chunked=encode_chunked)
      File "/usr/local/lib/python3.7/http/client.py", line 1239, in endheaders
        self._send_output(message_body, encode_chunked=encode_chunked)
      File "/usr/local/lib/python3.7/http/client.py", line 1026, in _send_output
        self.send(msg)
      File "/usr/local/lib/python3.7/http/client.py", line 966, in send
        self.connect()
      File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 184, in connect
        conn = self._new_conn()
      File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 169, in _new_conn
        self, "Failed to establish a new connection: %s" % e
    urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f66d3bb9b10>: Failed to establish a new connection: [Errno 111] Connection refused
    WARNING:  PUT http://localhost:9200/document [status:N/A request:0.002s]
    Traceback (most recent call last):
      File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 157, in _new_conn
        (self._dns_host, self.port), self.timeout, **extra_kw
      File "/usr/local/lib/python3.7/site-packages/urllib3/util/connection.py", line 84, in create_connection
        raise err
      File "/usr/local/lib/python3.7/site-packages/urllib3/util/connection.py", line 74, in create_connection
        sock.connect(sa)
    ConnectionRefusedError: [Errno 111] Connection refused
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "/usr/local/lib/python3.7/site-packages/elasticsearch/connection/http_urllib3.py", line 229, in perform_request
        method, url, body, retries=Retry(False), headers=request_headers, **kw
      File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 720, in urlopen
        method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
      File "/usr/local/lib/python3.7/site-packages/urllib3/util/retry.py", line 376, in increment
        raise six.reraise(type(error), error, _stacktrace)
      File "/usr/local/lib/python3.7/site-packages/urllib3/packages/six.py", line 735, in reraise
        raise value
      File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 672, in urlopen
        chunked=chunked,
      File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 387, in _make_request
        conn.request(method, url, **httplib_request_kw)
      File "/usr/local/lib/python3.7/http/client.py", line 1244, in request
        self._send_request(method, url, body, headers, encode_chunked)
      File "/usr/local/lib/python3.7/http/client.py", line 1290, in _send_request
        self.endheaders(body, encode_chunked=encode_chunked)
      File "/usr/local/lib/python3.7/http/client.py", line 1239, in endheaders
        self._send_output(message_body, encode_chunked=encode_chunked)
      File "/usr/local/lib/python3.7/http/client.py", line 1026, in _send_output
        self.send(msg)
      File "/usr/local/lib/python3.7/http/client.py", line 966, in send
        self.connect()
      File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 184, in connect
        conn = self._new_conn()
      File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 169, in _new_conn
        self, "Failed to establish a new connection: %s" % e
    urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f66d3bb9d50>: Failed to establish a new connection: [Errno 111] Connection refused
    WARNING:  PUT http://localhost:9200/document [status:N/A request:0.002s]
    Traceback (most recent call last):
      File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 157, in _new_conn
        (self._dns_host, self.port), self.timeout, **extra_kw
      File "/usr/local/lib/python3.7/site-packages/urllib3/util/connection.py", line 84, in create_connection
        raise err
      File "/usr/local/lib/python3.7/site-packages/urllib3/util/connection.py", line 74, in create_connection
        sock.connect(sa)
    ConnectionRefusedError: [Errno 111] Connection refused
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "/usr/local/lib/python3.7/site-packages/elasticsearch/connection/http_urllib3.py", line 229, in perform_request
        method, url, body, retries=Retry(False), headers=request_headers, **kw
      File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 720, in urlopen
        method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
      File "/usr/local/lib/python3.7/site-packages/urllib3/util/retry.py", line 376, in increment
        raise six.reraise(type(error), error, _stacktrace)
      File "/usr/local/lib/python3.7/site-packages/urllib3/packages/six.py", line 735, in reraise
        raise value
      File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 672, in urlopen
        chunked=chunked,
      File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 387, in _make_request
        conn.request(method, url, **httplib_request_kw)
      File "/usr/local/lib/python3.7/http/client.py", line 1244, in request
        self._send_request(method, url, body, headers, encode_chunked)
      File "/usr/local/lib/python3.7/http/client.py", line 1290, in _send_request
        self.endheaders(body, encode_chunked=encode_chunked)
      File "/usr/local/lib/python3.7/http/client.py", line 1239, in endheaders
        self._send_output(message_body, encode_chunked=encode_chunked)
      File "/usr/local/lib/python3.7/http/client.py", line 1026, in _send_output
        self.send(msg)
      File "/usr/local/lib/python3.7/http/client.py", line 966, in send
        self.connect()
      File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 184, in connect
        conn = self._new_conn()
      File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 169, in _new_conn
        self, "Failed to establish a new connection: %s" % e
    urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f66d3eb08d0>: Failed to establish a new connection: [Errno 111] Connection refused
    WARNING:  PUT http://localhost:9200/document [status:N/A request:0.001s]
    Traceback (most recent call last):
      File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 157, in _new_conn
        (self._dns_host, self.port), self.timeout, **extra_kw
      File "/usr/local/lib/python3.7/site-packages/urllib3/util/connection.py", line 84, in create_connection
        raise err
      File "/usr/local/lib/python3.7/site-packages/urllib3/util/connection.py", line 74, in create_connection
        sock.connect(sa)
    ConnectionRefusedError: [Errno 111] Connection refused
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "/usr/local/lib/python3.7/site-packages/elasticsearch/connection/http_urllib3.py", line 229, in perform_request
        method, url, body, retries=Retry(False), headers=request_headers, **kw
      File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 720, in urlopen
        method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
      File "/usr/local/lib/python3.7/site-packages/urllib3/util/retry.py", line 376, in increment
        raise six.reraise(type(error), error, _stacktrace)
      File "/usr/local/lib/python3.7/site-packages/urllib3/packages/six.py", line 735, in reraise
        raise value
      File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 672, in urlopen
        chunked=chunked,
      File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 387, in _make_request
        conn.request(method, url, **httplib_request_kw)
      File "/usr/local/lib/python3.7/http/client.py", line 1244, in request
        self._send_request(method, url, body, headers, encode_chunked)
      File "/usr/local/lib/python3.7/http/client.py", line 1290, in _send_request
        self.endheaders(body, encode_chunked=encode_chunked)
      File "/usr/local/lib/python3.7/http/client.py", line 1239, in endheaders
        self._send_output(message_body, encode_chunked=encode_chunked)
      File "/usr/local/lib/python3.7/http/client.py", line 1026, in _send_output
        self.send(msg)
      File "/usr/local/lib/python3.7/http/client.py", line 966, in send
        self.connect()
      File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 184, in connect
        conn = self._new_conn()
      File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 169, in _new_conn
        self, "Failed to establish a new connection: %s" % e
    urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f66d3bb9cd0>: Failed to establish a new connection: [Errno 111] Connection refused
    Traceback (most recent call last):
      File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 157, in _new_conn
        (self._dns_host, self.port), self.timeout, **extra_kw
      File "/usr/local/lib/python3.7/site-packages/urllib3/util/connection.py", line 84, in create_connection
        raise err
      File "/usr/local/lib/python3.7/site-packages/urllib3/util/connection.py", line 74, in create_connection
        sock.connect(sa)
    ConnectionRefusedError: [Errno 111] Connection refused
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "/usr/local/lib/python3.7/site-packages/elasticsearch/connection/http_urllib3.py", line 229, in perform_request
        method, url, body, retries=Retry(False), headers=request_headers, **kw
      File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 720, in urlopen
        method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
      File "/usr/local/lib/python3.7/site-packages/urllib3/util/retry.py", line 376, in increment
        raise six.reraise(type(error), error, _stacktrace)
      File "/usr/local/lib/python3.7/site-packages/urllib3/packages/six.py", line 735, in reraise
        raise value
      File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 672, in urlopen
        chunked=chunked,
      File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 387, in _make_request
        conn.request(method, url, **httplib_request_kw)
      File "/usr/local/lib/python3.7/http/client.py", line 1244, in request
        self._send_request(method, url, body, headers, encode_chunked)
      File "/usr/local/lib/python3.7/http/client.py", line 1290, in _send_request
        self.endheaders(body, encode_chunked=encode_chunked)
      File "/usr/local/lib/python3.7/http/client.py", line 1239, in endheaders
        self._send_output(message_body, encode_chunked=encode_chunked)
      File "/usr/local/lib/python3.7/http/client.py", line 1026, in _send_output
        self.send(msg)
      File "/usr/local/lib/python3.7/http/client.py", line 966, in send
        self.connect()
      File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 184, in connect
        conn = self._new_conn()
      File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 169, in _new_conn
        self, "Failed to establish a new connection: %s" % e
    urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f66d3bb9cd0>: Failed to establish a new connection: [Errno 111] Connection refused
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "/usr/local/bin/uvicorn", line 8, in <module>
        sys.exit(main())
      File "/usr/local/lib/python3.7/site-packages/click/core.py", line 829, in __call__
        return self.main(*args, **kwargs)
      File "/usr/local/lib/python3.7/site-packages/click/core.py", line 782, in main
        rv = self.invoke(ctx)
      File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
        return ctx.invoke(self.callback, **ctx.params)
      File "/usr/local/lib/python3.7/site-packages/click/core.py", line 610, in invoke
        return callback(*args, **kwargs)
      File "/usr/local/lib/python3.7/site-packages/uvicorn/main.py", line 331, in main
        run(**kwargs)
      File "/usr/local/lib/python3.7/site-packages/uvicorn/main.py", line 354, in run
        server.run()
      File "/usr/local/lib/python3.7/site-packages/uvicorn/main.py", line 382, in run
        loop.run_until_complete(self.serve(sockets=sockets))
      File "uvloop/loop.pyx", line 1456, in uvloop.loop.Loop.run_until_complete
      File "/usr/local/lib/python3.7/site-packages/uvicorn/main.py", line 389, in serve
        config.load()
      File "/usr/local/lib/python3.7/site-packages/uvicorn/config.py", line 288, in load
        self.loaded_app = import_from_string(self.app)
      File "/usr/local/lib/python3.7/site-packages/uvicorn/importer.py", line 20, in import_from_string
        module = importlib.import_module(module_str)
      File "/usr/local/lib/python3.7/importlib/__init__.py", line 127, in import_module
        return _bootstrap._gcd_import(name[level:], package, level)
      File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
      File "<frozen importlib._bootstrap>", line 983, in _find_and_load
      File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
      File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
      File "<frozen importlib._bootstrap_external>", line 728, in exec_module
      File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
      File "./backend/api.py", line 11, in <module>
        from backend.controller.router import router as api_router
      File "./backend/controller/router.py", line 3, in <module>
        from backend.controller import autocomplete, model, feedback
      File "./backend/controller/model.py", line 60, in <module>
        excluded_meta_data=EXCLUDE_META_DATA_FIELDS,
      File "/home/user/src/farm-haystack/haystack/database/elasticsearch.py", line 48, in __init__
        self.client.indices.create(index=index, ignore=400, body=custom_mapping)
      File "/usr/local/lib/python3.7/site-packages/elasticsearch/client/utils.py", line 92, in _wrapped
        return func(*args, params=params, headers=headers, **kwargs)
      File "/usr/local/lib/python3.7/site-packages/elasticsearch/client/indices.py", line 104, in create
        "PUT", _make_path(index), params=params, headers=headers, body=body
      File "/usr/local/lib/python3.7/site-packages/elasticsearch/transport.py", line 362, in perform_request
        timeout=timeout,
      File "/usr/local/lib/python3.7/site-packages/elasticsearch/connection/http_urllib3.py", line 241, in perform_request
        raise ConnectionError("N/A", str(e), e)
    elasticsearch.exceptions.ConnectionError: ConnectionError(<urllib3.connection.HTTPConnection object at 0x7f66d3bb9cd0>: Failed to establish a new connection: [Errno 111] Connection refused) caused by: NewConnectionError(<urllib3.connection.HTTPConnection object at 0x7f66d3bb9cd0>: Failed to establish a new connection: [Errno 111] Connection refused)```
    opened by ghost 6
  • Integrate SIL language identification API

    Integrate SIL language identification API

    This PR integrates more inclusive language identification as compared to cld2/3. To this end, the SIL Language Identification API is used as the default language identification model. This API supports 1035 languages currently including many lower resourced languages, and hopefully by using this language ID COVID-QA can start leveraging a variety of data gathered in lowered resourced languages (e.g., via elasticsearch). Some sources of this info are the SIL COVID resources, the endangered languages project, and this repo.

    Important points

    • I've integrated new environmental variables for the API key, secret and url in config.py. These can be obtained for free at developers.sil.org
    • I took cld2/3 out of the loop because closely related languages (e.g., russian and bulgarian) might be misidentified. cld2/3 doesn't support many languages and is thus biased in terms of the data that was used to train the model.
    • The SIL API returns ISO 639-3 codes only (3 letter language codes)

    How to test

    $ cd covid_nlp/language/
    $ python detect_language.py
    

    Other suggestions/ notes:

    • Right now any non-English language is routed to the FAQ-based matching with elasticsearch (I think). It would be great to upgrade this a bit to determine which languages currently have data. Then for languages without any content, we could route those to something like PanLex resources or SIL posters for low resource languages.
    • I would like to upgrade the SIL API to support language ID for many samples at once. This may help in dynamically determining what content we have in the datastore.

    Supported languages in the API

    Supported via Text Classification: biv, cub, cmo, hvn, kus, yal, pwg, myv, guo, des, leu, eip, cso, zia, kri, mca, kno, zza, maz, bps, qub, rmy, lvs, tab, nld, moa, ssg, maw, pww, sab, udm, zsm, zao, dzo, gnw, bru, kog, cwe, bim, tgo, mlh, blz, ckt, lok, smo, kpq, eng, nnq, kmr, pir, cab, tuo, bvc, xte, txu, pny, klv, jic, khm, mhy, yli, kha, dop, ojb, gvl, meq, cof, qvo, kqe, btd, bwu, nii, arb, xtn, top, lex, lob, cjp, por, ote, tmc, sun, grt, mcd, sja, naw, plw, zas, soq, khq, cek, ozm, kud, ted, bmq, pan, rjs, ktb, nhw, krj, ycn, ita, tir, prg, kpf, qup, msy, emp, ncu, qxo, ell, mzk, tim, yaz, dtb, upv, cou, noa, nhy, adh, cly, saj, fuq, rmo, gla, sim, apd, kpr, ota, kqp, gso, afr, kxc, mbt, wiu, pbb, cor, qul, gwr, twu, qve, arl, bku, alz, mto, bak, guc, lat, kgr, agm, cwt, iws, mip, ctp, khz, kyq, vie, dad, dug, yas, irk, kez, mza, nou, yue, law, kur, atg, mco, acr, lhu, myb, tik, djk, hae, tpp, yuj, mwq, rav, kzj, tuk, pbi, ffm, kmo, ybb, bgz, slk, cbv, gof, bjz, jiv, lln, xrb, cjo, qxn, prk, cot, xed, dgi, nsn, mpm, bzj, kne, cnl, bhw, gyr, akh, ntp, pls, aoz, som, tlf, xsb, eus, mfe, hak, aby, mej, myw, dsb, kru, snw, tpt, cle, nyy, tgp, agd, btt, mf1, quz, swg, sck, dyo, qvc, due, mmo, nca, oss, urt, hrv, btx, ban, pib, iri, sba, kub, lif, npl, icr, mbh, amu, sag, zpt, pss, gle, azz, hag, lzh, acu, ara, hns, zpq, mio, zty, cuc, usa, dan, miq, akb, nyo, cbi, caa, gdn, pms, mpt, wer, teo, ghs, mxt, fin, mjv, kwd, cax, zpl, ntr, ake, nog, tlj, aah, ach, mit, fij, apz, ceb, gde, gdr, mcp, cui, twb, mta, ncj, ino, men, mhi, mir, pez, quy, yre, asm, bdd, zpm, hot, zpi, kao, kyu, mvc, zpz, nzi, stp, srp, dik, guk, hat, zca, opm, aso, way, uig, krs, dig, sbl, glg, ava, avk, mkd, con, jac, mbb, heb, ces, mwv, wob, ddn, fuf, jbu, chr, kms, kwi, soy, qvn, rap, sxn, sgw, rel, ukr, gnd, bgt, thk, nob, dga, mie, orv, kyz, guh, pag, pse, tfr, cul, bhl, xsr, vag, qvw, nst, azg, muv, pad, cco, ese, gcf, pol, akp, sey, bex, vut, pam, lus, gvc, vol, stn, kdc, gym, med, wuv, gng, pui, kle, arz, myx, aak, hif, ian, sig, ign, mvp, xuo, kup, bbr, amf, zai, cya, nia, raw, nyf, ayp, czt, saq, zae, sah, kzf, swe, jam, poi, dob, hnn, mhr, okv, aze, gor, nij, aai, mkl, ron, isl, cpb, mup, nod, sus, knf, laj, nnb, tqo, bfd, cok, alj, pcm, kpw, myk, bbo, uvl, jbo, kia, kat, mux, agn, bjv, tly, mak, ixi, spp, xtd, ifu, urd, bom, bel, ruf, mhl, kek, bts, nhe, duo, mfz, otq, trs, old, bus, dbq, tcc, bba, cat, tee, cfm, bef, nwb, tca, dgz, cnk, crn, dah, chv, kwf, aom, bcl, nfr, fal, tpw, gos, crh, tnr, deu, yuw, oku, hoc, luc, rim, zar, ndy, pbc, udu, daa, miy, mog, obo, aia, knk, sgb, kbh, aoj, gaw, jvn, hsb, ljp, rnl, acc, avt, kbm, sbd, nhi, itv, yle, kbp, mzm, ame, amk, srn, ido, mqj, acm, box, xla, gag, tem, ses, boa, lmk, ker, bov, lew, bul, gbo, bmv, agu, aau, kkj, smt, ziw, ind, ter, hla, xsu, lef, qwh, zpu, xal, adj, gux, rus, ztq, kij, lgg, alp, frd, agr, miz, nin, mfq, gmv, urb, bpr, hye, boj, bua, wnu, naf, tgl, acd, sgz, lsm, yat, ton, for, fuv, wwa, tue, atq, iry, kyc, rai, pab, grn, hus, tav, lao, sda, tat, ilo, ury, nyn, lis, nkf, mtp, mxb, waj, kpz, aeu, krc, rwo, tbo, mai, avn, npy, vid, wba, mox, sne, yaa, hun, ben, mhx, viv, bav, vun, tuf, gur, cmr, cgc, sld, aon, ttc, ura, wap, dyi, gwi, ann, kue, quw, cbc, mfy, mtj, mya, mti, mgo, ppk, tac, est, ngp, pkb, zpo, cap, zab, fuh, tbc, dos, mag, mcu, xmm, cmn, mil, mww, apr, big, cdf, gvf, mda, lad, cnt, ipi, bon, kki, mqb, gum, kab, ctd, cme, ong, taj, usp, tpz, moz, ina, kvn, quf, thv, mlp, hin, sps, eka, bmr, sdm, mop, ubu, bnj, lem, gog, kbr, ahk, enb, gej, mif, uzb, ixl, dtp, yid, mnb, mpg, bss, ccp, muy, kto, avu, tos, dww, car, qvm, neb, csk, yrb, amn, jun, imo, nmz, gbi, maa, snc, lip, jpn, nak, bkv, awb, iba, mqf, tvw, xsm, cym, cuk, guu, mxv, nan, coe, mgh, msm, fra, sue, amr, rom, gai, kcg, mur, ctg, nlc, nch, yss, gfk, bos, myy, mxq, kpv, dnw, lac, shn, taq, tna, thl, kdj, spa, jav, anv, atb, cbs, ken, yam, asg, spl, zaw, gah, tnn, alt, enq, sqi, mib, yad, zyp, lit, sur, ife, ktj, ifk, nsu, abt, hne, rro, zpc, mfi, gun, hil, run, qxh, lia, dts, lee, ltz, mzw, pis, epo, ptp, tzj, chz, nim, pes, tzt, ngu, mor, pao, wmw, dsh, bwq, sny, zaa, ber, yut, keo, faa, kxm, ndz, arq, not, cko, ceg, dgk, gqr, tlb, bxr, kaq, mnf, jmc, mar, muh, inb, knj, tha, prf, mon, nnw, nhu, mfh, bcw, bre, kmd, acn, quc, hig, pah, sri, bfo, ade, bgr, rmc, cjv, auy, amh, war, guq, bud, tby, tlh, hrx, pmf, kwj, awa, mee, vmy, cpa, heh, dwr, cni, gui, kje, sas, srm, wuu, buk, lgl, xav, kyf, lwo, mal, fai, far, lww, oci, blt, pau, hto, cbr, abi, mbc, mim, rej, sml, min, yby, nno, lfn, roo, kor, yva, toc, tnk, knv, bvz, nds, gna, nhx, nuj, kjh, urk, gub, amm, nho, huu, qvz, mva, ile, grc, bao, mfk, sil, cbk, sll, snn, mcq, mek, slv, ksr, qvs, kaz, kqy, bkl, bib, tur, yml, suk, kaa, huv, krl, bmh, kze, csb, ape, ppo, ttr, ndj, hub, tte, ess, zos, nvm

    Supported via rule-based methods (based on unicode blocks and writing system scripts): xsr, ind, cmo, lif, ron, ojb, nod, mww, men, jun, rus, btd, jpn, hil, arb, pol, run, mai, alt, kyu, amh, taj, war, nld, pam, ljp, bud, lus, grt, mak, sun, akb, dzo, bku, urd, bru, tgl, pag, som, kbp, arz, pan, bts, vie, sas, gag, kxm, mjv, taq, fuf, ita, chr, bul, mya, jav, atb, blt, ceb, ccp, bcl, lao, ilo, mar, oss, hnn, btx, rej, ban, lis

    opened by dwhitena 6
  • NOT AN ISSUE  : I've created a Telegram Bot

    NOT AN ISSUE : I've created a Telegram Bot

    I've created a telegram bot to your web interface

    https://t.me/corona_scholar_bot

    Can I add it to the README?

    enhancement 
    opened by theapache64 5
  • Testing git commits, pull request and adding aggregation done by computer science student

    Testing git commits, pull request and adding aggregation done by computer science student

    This was simple test in order to fully understand pushing to GitHub.

    opened by jsquire1 5
  • Model 2 Issue

    Model 2 Issue

    While using model 2, the API returns answer for almost everything but not in English. I belive model 2 should only return English answers.

    Try Define gravity?

    opened by theapache64 5
  • Added unified endpoint to answer a question

    Added unified endpoint to answer a question

    This opens a following todo of routing the question correctly to the right model.

    opened by sfakir 5
  • Create english evaluation dataset for question similarity

    Create english evaluation dataset for question similarity

    We should create a simple, evaluation dataset that can be used to benchmark our models for matching similar questions.

    What should be sufficient for a rough baseline:

    • 100-300 question pairs of similar questions
    • extending that with 50% false pairs
    enhancement NLP / Modeling 
    opened by tholor 4
  • Bump scrapy from 2.0.1 to 2.5.1

    Bump scrapy from 2.0.1 to 2.5.1

    Bumps scrapy from 2.0.1 to 2.5.1.

    Release notes

    Sourced from scrapy's releases.

    2.5.1

    Security bug fix:

    If you use HttpAuthMiddleware (i.e. the http_user and http_pass spider attributes) for HTTP authentication, any request exposes your credentials to the request target.

    To prevent unintended exposure of authentication credentials to unintended domains, you must now additionally set a new, additional spider attribute, http_auth_domain, and point it to the specific domain to which the authentication credentials must be sent.

    If the http_auth_domain spider attribute is not set, the domain of the first request will be considered the HTTP authentication target, and authentication credentials will only be sent in requests targeting that domain.

    If you need to send the same HTTP authentication credentials to multiple domains, you can use w3lib.http.basic_auth_header instead to set the value of the Authorization header of your requests.

    If you really want your spider to send the same HTTP authentication credentials to any domain, set the http_auth_domain spider attribute to None.

    Finally, if you are a user of scrapy-splash, know that this version of Scrapy breaks compatibility with scrapy-splash 0.7.2 and earlier. You will need to upgrade scrapy-splash to a greater version for it to continue to work.

    2.5.0

    • Official Python 3.9 support
    • Experimental HTTP/2 support
    • New get_retry_request() function to retry requests from spider callbacks
    • New headers_received signal that allows stopping downloads early
    • New Response.protocol attribute

    See the full changelog

    2.4.1

    • Fixed feed exports overwrite support

    • Fixed the asyncio event loop handling, which could make code hang

    • Fixed the IPv6-capable DNS resolver CachingHostnameResolver for download handlers that call reactor.resolve

    • Fixed the output of the genspider command showing placeholders instead of the import part of the generated spider module (issue 4874)

    2.4.0

    Hihglights:

    • Python 3.5 support has been dropped.

    • The file_path method of media pipelines can now access the source item.

      This allows you to set a download file path based on item data.

    • The new item_export_kwargs key of the FEEDS setting allows to define keyword parameters to pass to item exporter classes.

    • You can now choose whether feed exports overwrite or append to the output file.

      For example, when using the crawl or runspider commands, you can use the -O option instead of -o to overwrite the output file.

    • Zstd-compressed responses are now supported if zstandard is installed.

    ... (truncated)

    Changelog

    Sourced from scrapy's changelog.

    Scrapy 2.5.1 (2021-10-05)

    • Security bug fix:

      If you use :class:~scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware (i.e. the http_user and http_pass spider attributes) for HTTP authentication, any request exposes your credentials to the request target.

      To prevent unintended exposure of authentication credentials to unintended domains, you must now additionally set a new, additional spider attribute, http_auth_domain, and point it to the specific domain to which the authentication credentials must be sent.

      If the http_auth_domain spider attribute is not set, the domain of the first request will be considered the HTTP authentication target, and authentication credentials will only be sent in requests targeting that domain.

      If you need to send the same HTTP authentication credentials to multiple domains, you can use :func:w3lib.http.basic_auth_header instead to set the value of the Authorization header of your requests.

      If you really want your spider to send the same HTTP authentication credentials to any domain, set the http_auth_domain spider attribute to None.

      Finally, if you are a user of scrapy-splash_, know that this version of Scrapy breaks compatibility with scrapy-splash 0.7.2 and earlier. You will need to upgrade scrapy-splash to a greater version for it to continue to work.

    .. _scrapy-splash: https://github.com/scrapy-plugins/scrapy-splash

    .. _release-2.5.0:

    Scrapy 2.5.0 (2021-04-06)

    Highlights:

    • Official Python 3.9 support

    • Experimental :ref:HTTP/2 support <http2>

    • New :func:~scrapy.downloadermiddlewares.retry.get_retry_request function to retry requests from spider callbacks

    ... (truncated)

    Commits
    • 61130c8 Bump version: 2.5.0 → 2.5.1
    • 98d2173 Pin the libxml2 version in CI as a newer one breaks lxml (#5208)
    • 47fb908 [CI] fail-fast: false (#5200)
    • 6d7179b tests: freeze pylint==2.7.4
    • d06dcb8 tests: force queuelib < 1.6.0
    • d99b1a1 Cover 2.5.1 in the release notes
    • c9485a5 Small documentation fixes.
    • a172844 Add http_auth_domain to HttpAuthMiddleware.
    • 5fd75f8 docs: require sphinx-rtd-theme>=0.5.2 and the latest pip to prevent installin...
    • e63188c Bump version: 2.4.1 → 2.5.0
    • Additional commits viewable in compare view

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    • @dependabot use these labels will set the current labels as the default for future PRs for this repo and language
    • @dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language
    • @dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language
    • @dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

    You can disable automated security fix PRs for this repo from the Security Alerts page.

    dependencies 
    opened by dependabot[bot] 0
  • Preprocessing of context to fit max_length

    Preprocessing of context to fit max_length

    Hi, would you please help me understand how the preprocessing is done for theCovidQA corpus ? Why I ask is because the context in the CovidQA dataset seems to be so much larger than the maximum length set in the code (which is 300+ and BERT max_length is 512 tokens). How is the data processed to fit into the limit ? Couldn't find the code for that in the Git. Please advice. Thank you.

    opened by Geethi2020 1
  • Added state pattern to Query class

    Added state pattern to Query class

    Applied the state pattern to the Query class to make methods on involving the filter attribute behave differently depending on the filter state.

    opened by yasserelsaid 0
  • Applied iterator pattern to Response

    Applied iterator pattern to Response

    Implemented the iterator pattern on classes Response and ResponseToIndividualQuestion.

    This makes it easier to traverse the results collection in the Response and get access to the results in a sequential manner without any need to know its underlying representation.

    opened by yasserelsaid 0
  • Second Behavioral Design Pattern (State Pattern)

    Second Behavioral Design Pattern (State Pattern)

    In tfidf_workflow.py, a workflow of Term Frequency–Inverse Document Frequency (TF-IDF) model processing progress which is a part of covid_nlp, includes from language preprocessing, TF-IDF model training to TF-IDF model evaluating. Integrating all the three progresses into one workflow will simplify the work done by client and clarify the whole process of TF-IDF model training for client. During implementation, if some part was skipped, clients will be notified what the current state and the following state are.

    opened by SwithinTan 0
  • First Behavioral Design Pattern (Observer Design Pattern)

    First Behavioral Design Pattern (Observer Design Pattern)

    Description: Two interfaces Subject, Observer, and its correspondent concrete class ConcreteObserver that extending class Observer were created in a new separate observer.py. The Subject interface declares a set of methods for managing subscribers, including attach, detach and notify to a client object. MSTranslator functions to translate the questions and answers in English into another language and write them in a csv file which will be observed by our Observer. Also, observer Design Pattern follows Open-closed principle, so new subscriber objects can be easily introduced while keep publisher code unchanged. Subscriber lists can be customized freely.

    opened by SwithinTan 0
  • Applying proxy pattern

    Applying proxy pattern

    opened by karanpurba 0
  • Application of Creational Design Pattern

    Application of Creational Design Pattern

    Used the Singleton creational pattern to create a single instance of a data ingestor in place of the data_ingestion.py file. Ingestion has been given static constructor and has been made immutable to protect the data collection and update uniformally.

    opened by edm101 0
  • Applying_Singleton_Pattren

    Applying_Singleton_Pattren

    opened by karanpurba 0
Owner
deepset
Building enterprise search systems powered by latest NLP & open-source.
deepset
Social Distancing Detector using deep learning and capable to run on edge AI devices such as NVIDIA Jetson, Google Coral, and more.

Smart Social Distancing Smart Social Distancing Introduction Getting Started Prerequisites Usage Processor Optional Parameters Configuring AWS credent

Neuralet 99 Oct 21, 2021
Deploy an inference API on AWS (EC2) using FastAPI Docker and Github Actions

Deploy an inference API on AWS (EC2) using FastAPI Docker and Github Actions To learn more about this project: medium blog post The goal of this proje

Ahmed BESBES 26 Sep 13, 2021
Web Version of avatarify to democratize even further

Web-avatarify for image animations This is the code base for this website and its backend. This aims to bring technology closer to everyone, just by a

Carlos Andrés Álvarez Restrepo 59 Sep 9, 2021
Backend, modern REST API for obtaining match and odds data crawled from multiple sites. Using FastAPI, MongoDB as database, Motor as async MongoDB client, Scrapy as crawler and Docker.

Introduction Apiestas is a project composed of a backend powered by the awesome framework FastAPI and a crawler powered by Scrapy. This project has fo

Fran Lozano 43 Oct 9, 2021
Web Inventory tool, takes screenshots of webpages using Pyppeteer (headless Chrome/Chromium) and provides some extra bells & whistles to make life easier.

WitnessMe WitnessMe is primarily a Web Inventory tool inspired by Eyewitness, its also written to be extensible allowing you to create custom function

byt3bl33d3r 525 Oct 22, 2021
signal-cli-rest-api is a wrapper around signal-cli and allows you to interact with it through http requests

signal-cli-rest-api signal-cli-rest-api is a wrapper around signal-cli and allows you to interact with it through http requests. Features register/ver

Sebastian Noel Lübke 25 Oct 18, 2021
a lightweight web framework based on fastapi

start-fastapi Version 2021, based on FastAPI, an easy-to-use web app developed upon Starlette Framework Version 2020 中文文档 Requirements python 3.6+ (fo

HiKari 61 Sep 29, 2021
ReST based network device broker

The Open API Platform for Network Devices netpalm makes it easy to push and pull state from your apps to your network by providing multiple southbound

null 316 Sep 28, 2021
row level security for FastAPI framework

Row Level Permissions for FastAPI While trying out the excellent FastApi framework there was one peace missing for me: an easy, declarative way to def

Holger Frey 187 Oct 17, 2021
volunteer-database

This is the official CSM (Crowd source medical) database The What Now? We created this in light of the COVID-19 pandemic to allow volunteers to work t

null 32 Oct 9, 2021
The base to start an openapi project featuring: SQLModel, Typer, FastAPI, JWT Token Auth, Interactive Shell, Management Commands.

The base to start an openapi project featuring: SQLModel, Typer, FastAPI, JWT Token Auth, Interactive Shell, Management Commands.

Bruno Rocha 56 Oct 17, 2021
python template private service

Template for private python service This is a cookiecutter template for an internal REST API service, written in Python, inspired by layout-golang. Th

UrvanovCompany 12 Nov 24, 2020
Publish Xarray Datasets via a REST API.

Xpublish Publish Xarray Datasets via a REST API. Serverside: Publish a Xarray Dataset through a rest API ds.rest.serve(host="0.0.0.0", port=9000) Clie

xarray-contrib 70 Oct 18, 2021
A Jupyter server based on FastAPI (Experimental)

jupyverse is experimental and should not be used in place of jupyter-server, which is the official Jupyter server.

Jupyter Server 78 Oct 18, 2021
FastAPI Admin Dashboard based on FastAPI and Tortoise ORM.

FastAPI ADMIN 中文文档 Introduction FastAPI-Admin is a admin dashboard based on fastapi and tortoise-orm. FastAPI-Admin provide crud feature out-of-the-bo

long2ice 929 Oct 23, 2021
ASGI middleware for authentication, rate limiting, and building CRUD endpoints.

Piccolo API Utilities for easily exposing Piccolo models as REST endpoints in ASGI apps, such as Starlette and FastAPI. Includes a bunch of useful ASG

null 37 Oct 13, 2021
FastAPI Skeleton App to serve machine learning models production-ready.

FastAPI Model Server Skeleton Serving machine learning models production-ready, fast, easy and secure powered by the great FastAPI by Sebastián Ramíre

null 208 Oct 20, 2021
TODO aplication made with Python's FastAPI framework and Hexagonal Architecture

FastAPI Todolist Description Todolist aplication made with Python's FastAPI framework and Hexagonal Architecture. This is a test repository for the pu

Giovanni Armane 40 Oct 14, 2021
JSON-RPC server based on fastapi

Description JSON-RPC server based on fastapi: https://fastapi.tiangolo.com Motivation Autogenerated OpenAPI and Swagger (thanks to fastapi) for JSON-R

null 125 Oct 14, 2021