Note: Obsei is still in alpha stage hence carefully use it in Production. Also as it is constantly undergoing development hence master branch may contains many breaking changes. Please use released version.
Obsei is an open-source low-code AI powered automation tool. Obsei consist of -
Observer, observes platform like Twitter, Facebook, App Stores, Google reviews, Amazon reviews, News, Website etc and feed that information to,
Analyzer, which perform text analysis like classification, sentiment, translation, PII etc and feed that information to,
Informer, which send it to ticketing system, data store, dataframe etc for further action and analysis.
Future thoughts -
Text, Image, Audio, Documents and Video oriented workflows
Collect data from every possible private and public channels
Add each possible to AI piece which can automate manual cognitive workflows
Introductory demo video
Use cases
Obsei use cases are following, but not limited to -
Social listening
Alerting/Notification when user complaints on social media
Automatic customer issue creation based on sentiment analysis (reduction of MTTD)
Proper tagging of ticket based for example login issue, signup issue, delivery issue etc (reduction of MTTR)
Checking effectiveness of social media marketing campaign
Extraction of deeper insight from feedbacks on various platforms
Research purpose
Many more based on creativity
💡
Companies/Projects using Obsei
Here are some companies/projects (alphabetical order) using Obsei. To add your company/project to the list, please raise a PR or contact us via email.
1Page: Giving a better context in meetings and calls
Observe app reviews from Google play store, Analyze them via performing text classification and then Inform them on console via logger
PlayStore Reviews → Classification → Logger
2
Observe app reviews from Google play store, PreProcess text via various text cleaning function, Analyze them via performing text classification, Inform them to Pandas DataFrame and store resultant CSV to Google Drive
PlayStore Reviews → PreProcessing → Classification → Pandas DataFrame → CSV in Google Drive
3
Observe app reviews from Apple app store, PreProcess text via various text cleaning function, Analyze them via performing text classification, Inform them to Pandas DataFrame and store resultant CSV to Google Drive
AppStore Reviews → PreProcessing → Classification → Pandas DataFrame → CSV in Google Drive
4
Observe news article from Google news, PreProcess text via various text cleaning function, Analyze them via performing text classification while splitting text in small chunks and later computing final inference using given formula
Google News → Text Cleaner → Text Splitter → Classification → Inference Aggregator
Demo
We have a minimal streamlit based UI that you can use to test Obsei.
Watch UI demo video
To test remotely, just open: Obsei Demo Link (Note: Due to rate limit sometime Streamlit demo might not work, hence please use docker image locally.)
To test locally, just run
docker run -d --name obesi-ui -p 8501:8501 obsei/obsei-ui-demo
# You can find the UI at http://localhost:8501
To run Obsei workflow easily using GitHub Actions (no signups and cloud hosting require), refer repo for more information.
Documentation
For detailed installation instructions, usages and example refer documentation.
Install from master branch (if you want to try the latest features):
git clone https://github.com/obsei/obsei.git
cd obsei
pip install --editable .
Install via Conda:
To install latest released version -
conda install -c lalitpagaria obsei
Install from master branch (if you want to try the latest features):
git clone https://github.com/obsei/obsei.git
cd obsei
conda env create -f conda/environment.yml
For GPU based local environment -
git clone https://github.com/obsei/obsei.git
cd obsei
conda env create -f conda/gpu-environment.yml
Step 3: Configure Source/Observer
Twitter
` (day|hour|minute) cred_info=TwitterCredentials( # Enter your twitter consumer key and secret. Get it from https://developer.twitter.com/en/apply-for-access consumer_key="
", consumer_secret="
", bearer_token='
', ) ) # initialize tweets retriever source = TwitterSource() ">
fromobsei.source.twitter_sourceimportTwitterCredentials, TwitterSource, TwitterSourceConfig# initialize twitter source configsource_config=TwitterSourceConfig(
keywords=["issue"], # Keywords, @user or #hashtagslookup_period="1h", # Lookup period from current time, format: `
` (day|hour|minute)
cred_info=TwitterCredentials(
# Enter your twitter consumer key and secret. Get it from https://developer.twitter.com/en/apply-for-accessconsumer_key="
"
,
consumer_secret="
"
,
bearer_token='
'
,
)
)
# initialize tweets retrieversource=TwitterSource()
Facebook
` (day|hour|minute) cred_info=FacebookCredentials( # Enter your facebook app_id, app_secret and long_term_token. Get it from https://developers.facebook.com/apps/ app_id="
", app_secret="
", long_term_token="
", ) ) # initialize facebook post comments retriever source = FacebookSource() ">
fromobsei.source.facebook_sourceimportFacebookCredentials, FacebookSource, FacebookSourceConfig# initialize facebook source configsource_config=FacebookSourceConfig(
page_id="110844591144719", # Facebook page id, for example this one for Obseilookup_period="1h", # Lookup period from current time, format: `
` (day|hour|minute)
cred_info=FacebookCredentials(
# Enter your facebook app_id, app_secret and long_term_token. Get it from https://developers.facebook.com/apps/app_id="
"
,
app_secret="
"
,
long_term_token="
"
,
)
)
# initialize facebook post comments retrieversource=FacebookSource()
fromobsei.source.email_sourceimportEmailConfig, EmailCredInfo, EmailSource# initialize email source configsource_config=EmailConfig(
# List of IMAP servers for most commonly used email providers# https://www.systoolsgroup.com/imap/# Also, if you're using a Gmail account then make sure you allow less secure apps on your account -# https://myaccount.google.com/lesssecureapps?pli=1# Also enable IMAP access -# https://mail.google.com/mail/u/0/#settings/fwdandpopimap_server="imap.gmail.com", # Enter IMAP servercred_info=EmailCredInfo(
# Enter your email account username and passwordusername="
"
,
password="
"
),
lookup_period="1h"# Lookup period from current time, format: `
` (day|hour|minute)
)
# initialize email retrieversource=EmailSource()
Google Maps Reviews Scrapper
", # Enter Google Maps link or place id # For example below is for the "Taj Mahal" queries=["https://www.google.co.in/maps/place/Taj+Mahal/@27.1751496,78.0399535,17z/data=!4m5!3m4!1s0x39747121d702ff6d:0xdd2ae4803f767dde!8m2!3d27.1751448!4d78.0421422"], number_of_reviews=10, ) # initialize Outscrapper Maps review retriever source = OSGoogleMapsReviewsSource() ">
fromobsei.sourceimportOSGoogleMapsReviewsSource, OSGoogleMapsReviewsConfig# initialize Outscrapper Maps review source configsource_config=OSGoogleMapsReviewsConfig(
# Collect API key from https://outscraper.com/api_key="
"
,
# Enter Google Maps link or place id# For example below is for the "Taj Mahal"queries=["https://www.google.co.in/maps/place/Taj+Mahal/@27.1751496,78.0399535,17z/data=!4m5!3m4!1s0x39747121d702ff6d:0xdd2ae4803f767dde!8m2!3d27.1751448!4d78.0421422"],
number_of_reviews=10,
)
# initialize Outscrapper Maps review retrieversource=OSGoogleMapsReviewsSource()
fromobsei.source.appstore_scrapperimportAppStoreScrapperConfig, AppStoreScrapperSource# initialize app store source configsource_config=AppStoreScrapperConfig(
# Need two parameters app_id and country.# `app_id` can be found at the end of the url of app in app store.# For example - https://apps.apple.com/us/app/xcode/id497799835# `310633997` is the app_id for xcode and `us` is country.countries=["us"],
app_id="310633997",
lookup_period="1h"# Lookup period from current time, format: `
` (day|hour|minute)
)
# initialize app store reviews retrieversource=AppStoreScrapperSource()
Play Store Reviews Scrapper
` (day|hour|minute) ) # initialize play store reviews retriever source = PlayStoreScrapperSource() ">
fromobsei.source.playstore_scrapperimportPlayStoreScrapperConfig, PlayStoreScrapperSource# initialize play store source configsource_config=PlayStoreScrapperConfig(
# Need two parameters package_name and country.# `package_name` can be found at the end of the url of app in play store.# For example - https://play.google.com/store/apps/details?id=com.google.android.gm&hl=en&gl=US# `com.google.android.gm` is the package_name for xcode and `us` is country.countries=["us"],
package_name="com.google.android.gm",
lookup_period="1h"# Lookup period from current time, format: `
` (day|hour|minute)
)
# initialize play store reviews retrieversource=PlayStoreScrapperSource()
fromobsei.source.reddit_sourceimportRedditConfig, RedditSource, RedditCredInfo# initialize reddit source configsource_config=RedditConfig(
subreddits=["wallstreetbets"], # List of subreddits# Reddit account username and password# You can also enter reddit client_id and client_secret or refresh_token# Create credential at https://www.reddit.com/prefs/apps# Also refer https://praw.readthedocs.io/en/latest/getting_started/authentication.html# Currently Password Flow, Read Only Mode and Saved Refresh Token Mode are supportedcred_info=RedditCredInfo(
username="
"
,
password="
"
),
lookup_period="1h"# Lookup period from current time, format: `
` (day|hour|minute)
)
# initialize reddit retrieversource=RedditSource()
Reddit Scrapper
Note: Reddit heavily rate limit scrappers, hence use it to fetch small data during long period
fromobsei.source.reddit_scrapperimportRedditScrapperConfig, RedditScrapperSource# initialize reddit scrapper source configsource_config=RedditScrapperConfig(
# Reddit subreddit, search etc rss url. For proper url refer following link -# Refer https://www.reddit.com/r/pathogendavid/comments/tv8m9/pathogendavids_guide_to_rss_and_reddit/url="https://www.reddit.com/r/wallstreetbets/comments/.rss?sort=new",
lookup_period="1h"# Lookup period from current time, format: `
` (day|hour|minute)
)
# initialize reddit retrieversource=RedditScrapperSource()
Google News
fromobsei.source.google_news_sourceimportGoogleNewsConfig, GoogleNewsSource# initialize Google News source configsource_config=GoogleNewsConfig(
query='bitcoin',
max_results=5,
# To fetch full article text enable `fetch_article` flag# By default google news gives title and highlightfetch_article=True,
)
# initialize Google News retrieversource=GoogleNewsSource()
importpandasaspdfromobsei.source.pandas_sourceimportPandasSource, PandasSourceConfig# Initialize your Pandas DataFrame from your sources like csv, excel, sql etc# In following example we are reading csv which have two columns title and textcsv_file="https://raw.githubusercontent.com/deepset-ai/haystack/master/tutorials/small_generator_dataset.csv"dataframe=pd.read_csv(csv_file)
# initialize pandas sink configsink_config=PandasSourceConfig(
dataframe=dataframe,
include_columns=["score"],
text_columns=["name", "degree"],
)
# initialize pandas sinksink=PandasSource()
Some analyzer support GPU and to utilize pass device parameter. List of possible values of device parameter (default value auto):
auto: GPU (cuda:0) will be used if available otherwise CPU will be used
cpu: CPU will be used
cuda:{id} - GPU will be used with provided CUDA device id
Text Classification
Text classification, classify text into user provided categories.
fromobsei.analyzer.classification_analyzerimportClassificationAnalyzerConfig, ZeroShotClassificationAnalyzer# initialize classification analyzer config# It can also detect sentiments if "positive" and "negative" labels are added.analyzer_config=ClassificationAnalyzerConfig(
labels=["service", "delay", "performance"],
)
# initialize classification analyzer# For supported models refer https://huggingface.co/models?filter=zero-shot-classificationtext_analyzer=ZeroShotClassificationAnalyzer(
model_name_or_path="typeform/mobilebert-uncased-mnli",
device="auto"
)
Sentiment Analyzer
Sentiment Analyzer, detect the sentiment of the text. Text classification can also perform sentiment analysis but if you don't want to use heavy-duty NLP model then use less resource hungry dictionary based Vader Sentiment detector.
fromobsei.analyzer.sentiment_analyzerimportVaderSentimentAnalyzer# Vader does not need any configuration settingsanalyzer_config=None# initialize vader sentiment analyzertext_analyzer=VaderSentimentAnalyzer()
NER Analyzer
NER (Named-Entity Recognition) Analyzer, extract information and classify named entities mentioned in text into pre-defined categories such as person names, organizations, locations, medical codes, time expressions, quantities, monetary values, percentages, etc
fromobsei.analyzer.ner_analyzerimportNERAnalyzer# NER analyzer does not need configuration settingsanalyzer_config=None# initialize ner analyzer# For supported models refer https://huggingface.co/models?filter=token-classificationtext_analyzer=NERAnalyzer(
model_name_or_path="elastic/distilbert-base-cased-finetuned-conll03-english",
device="auto"
)
Translator
fromobsei.analyzer.translation_analyzerimportTranslationAnalyzer# Translator does not need analyzer configanalyzer_config=None# initialize translator# For supported models refer https://huggingface.co/models?pipeline_tag=translationanalyzer=TranslationAnalyzer(
model_name_or_path="Helsinki-NLP/opus-mt-hi-en",
device="auto"
)
PII Anonymizer
fromobsei.analyzer.pii_analyzerimportPresidioEngineConfig, PresidioModelConfig, \
PresidioPIIAnalyzer, PresidioPIIAnalyzerConfig# initialize pii analyzer's configanalyzer_config=PresidioPIIAnalyzerConfig(
# Whether to return only pii analysis or anonymize textanalyze_only=False,
# Whether to return detail information about anonymization decisionreturn_decision_process=True
)
# initialize pii analyzeranalyzer=PresidioPIIAnalyzer(
engine_config=PresidioEngineConfig(
# spacy and stanza nlp engines are supported# For more info refer# https://microsoft.github.io/presidio/analyzer/developing_recognizers/#utilize-spacy-or-stanzanlp_engine_name="spacy",
# Update desired spacy model and languagemodels=[PresidioModelConfig(model_name="en_core_web_lg", lang_code="en")]
)
)
Dummy Analyzer
Dummy Analyzer, do nothing it simply used for transforming input (TextPayload) to output (TextPayload) also adding user supplied dummy data.
", # To get channel id refer https://stackoverflow.com/questions/40940327/what-is-the-simplest-way-to-find-a-slack-team-id-and-a-channel-id channel_id="C01LRS6CT9Q" ) # initialize slack sink sink = SlackSink() ">
fromobsei.sink.slack_sinkimportSlackSink, SlackSinkConfig# initialize slack sink configsink_config=SlackSinkConfig(
# Provide slack bot/app token# For more detail refer https://slack.com/intl/en-de/help/articles/215770388-Create-and-regenerate-API-tokensslack_token="
"
,
# To get channel id refer https://stackoverflow.com/questions/40940327/what-is-the-simplest-way-to-find-a-slack-team-id-and-a-channel-idchannel_id="C01LRS6CT9Q"
)
# initialize slack sinksink=SlackSink()
Zendesk
fromobsei.sink.zendesk_sinkimportZendeskSink, ZendeskSinkConfig, ZendeskCredInfo# initialize zendesk sink configsink_config=ZendeskSinkConfig(
# For custom domain refer http://docs.facetoe.com.au/zenpy.html#custom-domains# Mainly you can do this by setting the environment variables:# ZENPY_FORCE_NETLOC# ZENPY_FORCE_SCHEME (default to https)# when set it will force request on:# {scheme}://{netloc}/endpoint# provide zendesk domaindomain="zendesk.com",
# provide subdomain if you have onesubdomain=None,
# Enter zendesk user detailscred_info=ZendeskCredInfo(
email="
"
,
password="
"
)
)
# initialize zendesk sinksink=ZendeskSink()
Jira
fromobsei.sink.jira_sinkimportJiraSink, JiraSinkConfig# For testing purpose you can start jira server locally# Refer https://developer.atlassian.com/server/framework/atlassian-sdk/atlas-run-standalone/# initialize Jira sink configsink_config=JiraSinkConfig(
url="http://localhost:2990/jira", # Jira server url# Jira username & password for user who have permission to create issueusername="
"
,
password="
"
,
# Which type of issue to be created# For more information refer https://support.atlassian.com/jira-cloud-administration/docs/what-are-issue-types/issue_type={"name": "Task"},
# Under which project issue to be created# For more information refer https://support.atlassian.com/jira-software-cloud/docs/what-is-a-jira-software-project/project={"key": "CUS"},
)
# initialize Jira sinksink=JiraSink()
ElasticSearch
fromobsei.sink.elasticsearch_sinkimportElasticSearchSink, ElasticSearchSinkConfig# For testing purpose you can start Elasticsearch server locally via docker# `docker run -d --name elasticsearch -p 9200:9200 -e "discovery.type=single-node" elasticsearch:7.9.2`# initialize Elasticsearch sink configsink_config=ElasticSearchSinkConfig(
# Elasticsearch server hostnamehost="localhost",
# Elasticsearch server portport=9200,
# Index name, it will create if not existindex_name="test",
)
# initialize Elasticsearch sinksink=ElasticSearchSink()
Http
fromobsei.sink.http_sinkimportHttpSink, HttpSinkConfig# For testing purpose you can create mock http server via postman# For more details refer https://learning.postman.com/docs/designing-and-developing-your-api/mocking-data/setting-up-mock/# initialize http sink config (Currently only POST call is supported)sink_config=HttpSinkConfig(
# provide http server urlurl="https://localhost:8080/api/path",
# Here you can add headers you would like to pass with requestheaders={
"Content-type": "application/json"
}
)
# To modify or converting the payload, create convertor class# Refer obsei.sink.dailyget_sink.PayloadConvertor for example# initialize http sinksink=HttpSink()
source will fetch data from selected the source, then feed that to analyzer for processing, whose output we feed into sink to get notified at that sink.
# Uncomment if you want logger# import logging# import sys# logger = logging.getLogger(__name__)# logging.basicConfig(stream=sys.stdout, level=logging.INFO)# This will fetch information from configured source ie twitter, app store etcsource_response_list=source.lookup(source_config)
# Uncomment if you want to log source response# for idx, source_response in enumerate(source_response_list):# logger.info(f"source_response#'{idx}'='{source_response.__dict__}'")# This will execute analyzer (Sentiment, classification etc) on source data with provided analyzer_configanalyzer_response_list=text_analyzer.analyze_input(
source_response_list=source_response_list,
analyzer_config=analyzer_config
)
# Uncomment if you want to log analyzer response# for idx, an_response in enumerate(analyzer_response_list):# logger.info(f"analyzer_response#'{idx}'='{an_response.__dict__}'")# Analyzer output added to segmented_data# Uncomment inorder to log it# for idx, an_response in enumerate(analyzer_response_list):# logger.info(f"analyzed_data#'{idx}'='{an_response.segmented_data.__dict__}'")# This will send analyzed output to configure sink ie Slack, Zendesk etcsink_response_list=sink.send_data(analyzer_response_list, sink_config)
# Uncomment if you want to log sink response# for sink_response in sink_response_list:# if sink_response is not None:# logger.info(f"sink_response='{sink_response}'")
Step 7: Execute workflow Copy code snippets from Step 3 to Step 6 into python file for example example.py and execute following command -
First off, thank you for even considering contributing to this package, every contribution big or small is greatly appreciated. Please refer our Contribution Guideline and Code of Conduct.
We would like to thank DailyGet for continuous support and encouragement. Please check DailyGet out. it is a platform which can easily be configured to solve any business process automation requirements.
@lalitpagaria I have added the batching on ZeroShotAnalyzer to get your feedback on changes that I made and design in general, I used the following code to benchmark performance, seems like batch size of 1 are outperforming larger batches which is counter-intuitive this may change on GPU I have only tested on local CPU
if __name__ == "__main__":
import timeit ## for benchmarking execution
start = timeit.default_timer()
GOOD_TEXT = """If anyone is interested... these are our hosts. I can’t recommend them enough, Abc & Pbc.
The unit is just lovely, you go to sleep & wake up to this incredible place, and you have use of a Weber grill and a ridiculously indulgent hot-tub under the stars"""
BAD_TEXT = """I had the worst experience ever with XYZ in Egypt. Bad Cars, asking to pay in cash, do not have enough fuel, do not open AC, wait far away from my location until the trip is cancelled, call and ask about the destination then cancel, and more. Worst service."""
MIXED_TEXT = """I am mixed"""
TEXTS = [
GOOD_TEXT,
BAD_TEXT,
MIXED_TEXT,
GOOD_TEXT,
BAD_TEXT,
MIXED_TEXT,
GOOD_TEXT,
BAD_TEXT,
MIXED_TEXT,
GOOD_TEXT,
BAD_TEXT,
MIXED_TEXT,
GOOD_TEXT,
BAD_TEXT,
MIXED_TEXT,
GOOD_TEXT,
BAD_TEXT,
MIXED_TEXT,
GOOD_TEXT,
BAD_TEXT,
MIXED_TEXT,
GOOD_TEXT,
BAD_TEXT,
MIXED_TEXT,
GOOD_TEXT,
BAD_TEXT,
MIXED_TEXT,
GOOD_TEXT,
BAD_TEXT,
MIXED_TEXT,
GOOD_TEXT,
BAD_TEXT,
MIXED_TEXT,
GOOD_TEXT,
BAD_TEXT,
MIXED_TEXT,
] * 2
zero_shot_analyzer = ZeroShotClassificationAnalyzer(
model_name_or_path="typeform/mobilebert-uncased-mnli",
)
batch_size = 4
labels = ["facility", "food", "comfortable", "positive", "negative"]
source_responses = [
AnalyzerRequest(processed_text=text, source_name="sample") for text in TEXTS
]
analyzer_responses = zero_shot_analyzer.analyze_input(
source_response_list=source_responses,
analyzer_config=ClassificationAnalyzerConfig(
labels=labels, batch_size=batch_size
),
)
stop = timeit.default_timer()
print("Time: ", stop - start)
print("Batch size: ", batch_size)
print("Total Texts: ", len(TEXTS))
"""for res in analyzer_responses:
print(res.segmented_data)
"""
assert len(analyzer_responses) == len(TEXTS)
for analyzer_response in analyzer_responses:
assert len(analyzer_response.segmented_data) == len(labels)
assert "positive" in analyzer_response.segmented_data
assert "negative" in analyzer_response.segmented_data
Could you review the structure of the code, is it as per your expectation and style guide we have for this repo? I'll then start adding functions to this and keep updating this PR. thanks
Apart from the tokenizer fix , I also observed one more thing, in the documentation of configure Ner Analyzer it is mentioned that "# NER analyzer does not need configuration settings" .
But while predicting the NER tags using NERAnalyzer's analyze_input method, analyzer_config is required (below). Hence the NER tagging fails.
if analyzer_config is None:
raise ValueError("analyzer_config can't be None")
So two things can be done, either we mention config setting for analyzer
or fix the batch size 1 in code,
I have done a quick fix here but I need your comment so I can make a cleaner fix. Either make the ner's analyze_input batch free or define a config and a batch size for ners .
Please suggeset the changes and refactoring
Rn the two things I am undecisive of are the parameters or configurations to Umap , HDBSCAN .
Also I have Keras as an autoencoder, undecisive if I should make it to Pytorch completely.
You will notice I have not included batchify , since we want all the text to process in one go .
When this project was started initial intention was to handle only short text. But now we have added Google News and Crawlers, hence there is need to handle longer text as well.
As we know that most of BERT based model support 512 max tokens (with few exceptions like BigBird). Currently Analyzer ignore (https://github.com/lalitpagaria/obsei/issues/113) excessive text.
Now Idea to introduce TextSplitter to split longer text and feed it to Analyzer. But it introduce another complexity with Analyzer predictions? How to combine inferences by multiple chunks for final prediction. Right now there not proper solution exist to handle this scenarios except try few like voting, averaging or like the one suggested here https://discuss.huggingface.co/t/new-pipeline-for-zero-shot-text-classification/681/84.
For sake of simplicity let's first implement TextSplitter. For this purpose let's take inspiration from Haystack splitter along with adding context like chunk_id, passage_id, etc into meta data.
For inference aggregation later we can add another node for Inference aggregation, we may call it InferenceAggregator. This will aggregate Analyzer result on text chunks to compute final inference.
Is your feature request related to a problem? Please describe.
Currently analyzers are iterating over array and calling pipeline method with single argument. This can be improve upon by calling pipeline with array of data.
Describe the solution you'd like
Divide input array into multiple batches and pass batch array to pipeline. Also, do performance analysis if this improves library latency.
An intent classifier is useful to detect what customer goals like buy, sell, and purchase also useful in conversional flow.
This same things can also be done via zero shot classifier as well but it would better to add separate analyzer to separate it from generic text classification. It will help user to load their own models for this purpose.
AttributeError: At least one non empty parameter required (query, keywords, hashtags, and usernames)
The bug is in the TwitterSource class
def lookup(self, config: TwitterSourceConfig, **kwargs) -> List[TextPayload]: # type: ignore[override]
if (
not config.query
and not config.keywords
and not config.hashtags
and config.usernames
):
it should be
def lookup(self, config: TwitterSourceConfig, **kwargs) -> List[TextPayload]: # type: ignore[override]
if (
not config.query
and not config.keywords
and not config.hashtags
and not config.usernames
):
I'm not sure if this actually is a bug, but today I realized while using lookup_period I was getting this error
TypeError: '<' not supported between instances of 'datetime.datetime' and 'NoneType'
To Reproduce
Steps to reproduce the behavior:https://github.com/amrrs/youtube-comment-sentiment-analysis/blob/main/Download_YouTube_Comments_NLP_Sentiment_Analysis.ipynb
Expected behavior
If it's a bug, suggest me how to fix it.
Or if it's actually that the comments aren't there for those time period then we can show that kind of message. But in my case, there were comments during those period and yet I got that above error.
Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.
Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR:
@dependabot rebase will rebase this PR
@dependabot recreate will recreate this PR, overwriting any edits that have been made to it
@dependabot merge will merge this PR after your CI passes on it
@dependabot squash and merge will squash and merge this PR after your CI passes on it
@dependabot cancel merge will cancel a previously requested merge and block automerging
@dependabot reopen will reopen this PR if it is closed
@dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
@dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
@dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
@dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.
Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR:
@dependabot rebase will rebase this PR
@dependabot recreate will recreate this PR, overwriting any edits that have been made to it
@dependabot merge will merge this PR after your CI passes on it
@dependabot squash and merge will squash and merge this PR after your CI passes on it
@dependabot cancel merge will cancel a previously requested merge and block automerging
@dependabot reopen will reopen this PR if it is closed
@dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
@dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
@dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
@dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
Describe the solution you'd like
A clear and concise description of what you want to happen.
Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.
Additional context
Add any other context or screenshots about the feature request here.
Currently we can create simple workflows consist of three node observer, analyzer and informer each. But as use cases demand for Obsei is growing better to allow user to create more complex workflows to enable user to use any nodes in any fashion with few constraints. There will be following type of nodes in DAG -
Observer: Output only nodes, which will not have standard input capability
Informer: Input only nodes, which will not have standard output capability
Analyzer: AI capability nodes, they can receive standard data as Input and produce standard output data
Misc: Few nodes like delay, joiner/merger, switch-case, splitter, aggregator, etc
This added new component to Obsei. It's main idea to provide simple but configurable step to pre-process text before sending it for model prediction. Currently TextCleaning step is added, which helps user to clean raw text's. It is great contribution by @shahrukhx01
Obsei can now have Pandas DataFrame as Informer to publish Analyzer's data to DataFrame.
🔆 Other Changes
Added contribution guideline and code of conduct (https://github.com/obsei/obsei/commit/e102f8915d99e7241974e1b2360b3d075300338a and https://github.com/obsei/obsei/commit/fcc9a9121efc0b049024b100ce28d8bab521f072)
Adding version tag along with default logging config (https://github.com/obsei/obsei/pull/149)
Ignoring error during cleaning and fixing exception in google news module (https://github.com/obsei/obsei/pull/143)
Adding Logger Informer, so user can easily test out end-to-end pipeline (linked commit https://github.com/lalitpagaria/obsei/commit/ca99da8b72611788e5969565438f02d7d356cfd1)
Adding security policy to repo (linked commit https://github.com/lalitpagaria/obsei/commit/2685fffdd36b57acdeb6a8adfc6f9920dfa7082b)
[Bug/Regression]: Rest interface via docker build was failing (linked commit https://github.com/lalitpagaria/obsei/commit/ef48b36792e1ce0d7118df6f3081f1e3b368c65c)
Reddit and Slack integration (https://github.com/lalitpagaria/obsei/commit/b69b64c439fda4d92fee5be18904a3f347d2ea12 and https://github.com/lalitpagaria/obsei/commit/7a0f6ab846a30c7fda606d4bee42bedc81eaa6c7)
New Analyzers NER and Dummy apart from Sentiment, Classification (https://github.com/lalitpagaria/obsei/commit/e1115595372e83708e081a3c05138abb9116b9e9 and https://github.com/lalitpagaria/obsei/commit/870c1eaea3494e00dcda4b9c94aa0203b85082d4)
This release have breaking changes related to change in package and arguments for Analyzers. Refer https://github.com/lalitpagaria/obsei/commit/e1115595372e83708e081a3c05138abb9116b9e9
Google play store reviews Observer, this need authentication (linked issue https://github.com/lalitpagaria/obsei/issues/4)
Apple app store reviews Observer via scrapping, this do not need authentication (linked issue https://github.com/lalitpagaria/obsei/issues/9)
Google play store reviews Observer via scrapping, this do not need authentication (linked issue https://github.com/lalitpagaria/obsei/issues/10)
Persist Observer current state to persistent storage of user choice like SQLite, MySQL, Postgres etc (linked issue https://github.com/lalitpagaria/obsei/issues/6)
![[BUG]TwitterSourceConfig - AttributeError: At least one non empty parameter required (query, keywords, hashtags, and usernames)](https://avatars.githubusercontent.com/u/95418610?v=4)
![[BUG] TypeError: '<' not supported between instances of 'datetime.datetime' and 'NoneType'](https://avatars.githubusercontent.com/u/5347322?v=4)
3.1k Jan 4, 2023
7 Oct 27, 2022
30 Dec 4, 2022
7.7k Jan 7, 2023
7.5k Dec 31, 2022
31 May 28, 2021
25.5k Jan 1, 2023
3.5k Dec 30, 2022
3k Jan 4, 2023
3.2k Dec 31, 2022
1.7k Dec 27, 2022
941 Jan 3, 2023
7.8k Jan 2, 2023
13 Jul 7, 2022
7 Jul 23, 2022
1 Oct 6, 2022
1 Oct 28, 2021
1 Nov 4, 2021
2 Aug 19, 2022