Last commit seems to break the execution of snscrape https://github.com/JustAnotherArchivist/snscrape/commit/25ee014e2970737e500e2964ab09584b11b19fb0

I get missing 1 required positional argument: 'thumbnailUrl' when using

/home/ambnum/.local/bin/snscrape --with-entity --max-results 1000 --jsonl twitter-search "+@nath_yamb " > "/tmp/information-manipulation-analyzer/@nath_yamb/original.json"

I saw this error when I run my code

---------------------------------------------------------------------------
ScraperException                          Traceback (most recent call last)
<ipython-input-18-000bfba11f8b> in <module>
      1 # Using TwitterSearchScraper to scrape data and append tweets to list
      2 for x in D_username:
----> 3     for i,tweet in enumerate(sntwitter.TwitterSearchScraper('from:{}'.format(x)).get_items()):
      4         if i>maxTweets:
      5             break

/opt/homebrew/lib/python3.9/site-packages/snscrape/modules/twitter.py in get_items(self)
    554                         del paginationParams['tweet_search_mode']
    555 
--> 556                 for obj in self._iter_api_data('https://api.twitter.com/2/search/adaptive.json', params, paginationParams, cursor = self._cursor):
    557                         yield from self._instructions_to_tweets(obj)
    558 

/opt/homebrew/lib/python3.9/site-packages/snscrape/modules/twitter.py in _iter_api_data(self, endpoint, params, paginationParams, cursor, direction)
    248                 while True:
    249                         logger.info(f'Retrieving scroll page {cursor}')
--> 250                         obj = self._get_api_data(endpoint, reqParams)
    251                         yield obj
    252 

/opt/homebrew/lib/python3.9/site-packages/snscrape/modules/twitter.py in _get_api_data(self, endpoint, params)
    217 
    218         def _get_api_data(self, endpoint, params):
--> 219                 self._ensure_guest_token()
    220                 r = self._get(endpoint, params = params, headers = self._apiHeaders, responseOkCallback = self._check_api_response)
    221                 try:

/opt/homebrew/lib/python3.9/site-packages/snscrape/modules/twitter.py in _ensure_guest_token(self, url)
    198                         self._apiHeaders['x-guest-token'] = self._guestToken
    199                         return
--> 200                 raise snscrape.base.ScraperException('Unable to find guest token')
    201 
    202         def _unset_guest_token(self):

ScraperException: Unable to find guest token

I am pretty sure this does not come from my bug. Are there ways to fix it?

Traceback (most recent call last):
  File "/lustre/scratch2/ws/1/s2575425-twitter-sentiment-analysis/./sentiment.py", line 56, in <module>
    for i,tweet in tqdm(enumerate(sntwitter.TwitterSearchScraper('bitcoin since:2000-01-01 until:2022-07-01').get_items())):
  File "/lustre/scratch2/ws/1/s2575425-twitter-sentiment-analysis/.venv/lib/python3.10/site-packages/tqdm/std.py", line 1195, in __iter__
    for obj in iterable:
  File "/lustre/scratch2/ws/1/s2575425-twitter-sentiment-analysis/.venv/lib/python3.10/site-packages/snscrape/modules/twitter.py", line 1454, in get_items
    yield from self._v2_timeline_instructions_to_tweets(obj)
  File "/lustre/scratch2/ws/1/s2575425-twitter-sentiment-analysis/.venv/lib/python3.10/site-packages/snscrape/modules/twitter.py", line 804, in _v2_timeline_instructions_to_tweets
    yield from self._v2_instruction_tweet_entry_to_tweet(entry['entryId'], entry['content'], obj)
  File "/lustre/scratch2/ws/1/s2575425-twitter-sentiment-analysis/.venv/lib/python3.10/site-packages/snscrape/modules/twitter.py", line 827, in _v2_instruction_tweet_entry_to_tweet
    yield self._tweet_to_tweet(tweet, obj)
  File "/lustre/scratch2/ws/1/s2575425-twitter-sentiment-analysis/.venv/lib/python3.10/site-packages/snscrape/modules/twitter.py", line 1266, in _tweet_to_tweet
    user = self._user_to_user(obj['globalObjects']['users'][tweet['user_id_str']])
  File "/lustre/scratch2/ws/1/s2575425-twitter-sentiment-analysis/.venv/lib/python3.10/site-packages/snscrape/modules/twitter.py", line 1367, in _user_to_user
    if 'ext' in user and (label := user['ext']['highlightedLabel']['r']['ok'].get('label')):
KeyError: 'highlightedLabel'

Hello,

I get the above error after scraping around 13000 entries: sntwitter.TwitterSearchScraper('bitcoin since:2000-01-01 until:2022-07-01').get_items()

Sorry, I am new to web crawling. How do I make this program work? I try tosnscrape twitter-user textfiles but it only shows the link of Jason Scott. How do I change the user I want to scrap? Also, if I want to scrap tweet by using keyword, how do I do that?

@xmainguyen I assume you're asking about using the profile scraper from a Python script (instead of the CLI).

import snscrape.modules.twitter.

for tweet in snscrape.modules.twitter.TwitterProfileScraper('username').get_items():
	# Do something with the tweet object, e.g.
	print(tweet.url)

Originally posted by @JustAnotherArchivist in https://github.com/JustAnotherArchivist/snscrape/issues/83#issuecomment-805467618

python 3.8, windows 10 pro

I got the following error:

Error retrieving https://api.twitter.com/2/search/adaptive.json?include_profile_interstitial_type=1&include_blocking=1&include_blocked_by=1&include_followed_by=1&include_want_retweets=1&include_mute_edge=1&include_can_dm=1&include_can_media_tag=1&skip_status=1&cards_platform=Web-12&include_cards=1&include_ext_alt_text=true&include_quote_count=true&include_reply_count=1&tweet_mode=extended&include_entities=true&include_user_entities=true&include_ext_media_color=true&include_ext_media_availability=true&send_error_codes=true&simple_quoted_tweets=true&q=vaccine+since%3A2021-01-01+until%3A2021-05-31&tweet_search_mode=live&count=100&query_source=spelling_expansion_revert_click&cursor=scroll%3AthGAVUV0VFVBaAgKmRi82l5SYWgoC58fy05eomEnEV_5iSBBWAiXoYBE5FV1M1ARWasQMVAAA%3D&pc=1&spelling_corrections=1&ext=mediaStats%2ChighlightedLabel: non-200 status code

4 requests to https://api.twitter.com/2/search/adaptive.json?include_profile_interstitial_type=1&include_blocking=1&include_blocked_by=1&include_followed_by=1&include_want_retweets=1&include_mute_edge=1&include_can_dm=1&include_can_media_tag=1&skip_status=1&cards_platform=Web-12&include_cards=1&include_ext_alt_text=true&include_quote_count=true&include_reply_count=1&tweet_mode=extended&include_entities=true&include_user_entities=true&include_ext_media_color=true&include_ext_media_availability=true&send_error_codes=true&simple_quoted_tweets=true&q=vaccine+since%3A2021-01-01+until%3A2021-05-31&tweet_search_mode=live&count=100&query_source=spelling_expansion_revert_click&cursor=scroll%3AthGAVUV0VFVBaAgKmRi82l5SYWgoC58fy05eomEnEV_5iSBBWAiXoYBE5FV1M1ARWasQMVAAA%3D&pc=1&spelling_corrections=1&ext=mediaStats%2ChighlightedLabel failed, giving up.

ScraperException: 4 requests to https://api.twitter.com/2/search/adaptive.json?include_profile_interstitial_type=1&include_blocking=1&include_blocked_by=1&include_followed_by=1&include_want_retweets=1&include_mute_edge=1&include_can_dm=1&include_can_media_tag=1&skip_status=1&cards_platform=Web-12&include_cards=1&include_ext_alt_text=true&include_quote_count=true&include_reply_count=1&tweet_mode=extended&include_entities=true&include_user_entities=true&include_ext_media_color=true&include_ext_media_availability=true&send_error_codes=true&simple_quoted_tweets=true&q=vaccine+since%3A2021-01-01+until%3A2021-05-31&tweet_search_mode=live&count=100&query_source=spelling_expansion_revert_click&cursor=scroll%3AthGAVUV0VFVBaAgKmRi82l5SYWgoC58fy05eomEnEV_5iSBBWAiXoYBE5FV1M1ARWasQMVAAA%3D&pc=1&spelling_corrections=1&ext=mediaStats%2ChighlightedLabel failed, giving up.

I used the following code:

 import snscrape.modules.twitter as sntwitter
 import pandas as pd


 tweets_list2 = []



 for i, tweet in enumerate(sntwitter.TwitterSearchScraper('vaccine since:2021-01-01 until:2021-05-31').get_items()):
 
    attribute_list = [tweet.date, tweet.id, tweet.user.username, tweet.user.id, tweet.user.displayname, tweet.user.location,  
                      tweet.user.followersCount, tweet.user.friendsCount, tweet.user.statusesCount, 
                      tweet.retweetedTweet, tweet.content, tweet.lang, tweet.mentionedUsers
                      ]    

    tweets_list2.append(attribute_list)

ScraperException: 4 requests to https://api.twitter.com/2/search/adaptive.json?include_profile_interstitial_type=1&include_blocking=1&include_blocked_by=1&include_followed_by=1&include_want_retweets=1&include_mute_edge=1&include_can_dm=1&include_can_media_tag=1&skip_status=1&cards_platform=Web-12&include_cards=1&include_ext_alt_text=true&include_quote_count=true&include_reply_count=1&tweet_mode=extended&include_entities=true&include_user_entities=true&include_ext_media_color=true&include_ext_media_availability=true&send_error_codes=true&simple_quoted_tweets=true&q=from%3AAITCofficial&tweet_search_mode=live&count=100&query_source=spelling_expansion_revert_click&pc=1&spelling_corrections=1&ext=mediaStats%2ChighlightedLabel failed, giving up.

Hi， I don't know how to use issue insert code，sorry！ If in json files,i can use this code output the fullUrl and store photo.

table=[]
with open("trump.json",'r',encoding='utf-8') as load_f:
    n=0
    for line in load_f:
        n+=1
        print(n)
        data = json.loads(line)
        #table.append(data)
        #pd.DataFrame.from_records(table).to_csv('text_files_test_10.csv',encoding='utf-8')
        if data['media']:
            for medium in data['media']:
            	if medium['type'] == 'photo':
             		print(medium['fullUrl'])`

Use other ways：

import snscrape
for tweet in snscrape.modules.twitter.TwitterUserScraper(username=name).get_items():
        media=tweet.media

print result

[Photo(previewUrl='https://pbs.twimg.com/media/Ek_hodXVMAEo9pc?format=jpg&name=small', fullUrl='https://pbs.twimg.com/media/Ek_hodXVMAEo9pc?format=jpg&name=large', type='photo')]

or

[Video(thumbnailUrl='https://pbs.twimg.com/ext_tw_video_thumb/1320348316639532008/pu/img/leM_UOyxiYELn-Nz.jpg', variants=[VideoVariant(contentType='application/x-mpegURL', url='https://video.twimg.com/ext_tw_video/1320348316639532008/pu/pl/_wvWUV5zYvh3ldaX.m3u8?tag=10', bitrate=None), VideoVariant(contentType='video/mp4', url='https://video.twimg.com/ext_tw_video/1320348316639532008/pu/vid/1280x720/6CN65DiSr9JokWRg.mp4?tag=10', bitrate=2176000), VideoVariant(contentType='video/mp4', url='https://video.twimg.com/ext_tw_video/1320348316639532008/pu/vid/480x270/gspLjGW5fYeYRWZG.mp4?tag=10', bitrate=256000), VideoVariant(contentType='video/mp4', url='https://video.twimg.com/ext_tw_video/1320348316639532008/pu/vid/640x360/hvE-BMIGCjj5ZceS.mp4?tag=10', bitrate=832000)], duration=17.595, type='video')]

it seems not json format，I tried to get photourl or videourl，but I don't know how to get ，can you help me？

Hey there,

two weeks ago, I set up a Twitter Scraper that used Docker to run e.g. 10 workers/scrapers simultaneously with the GetOldTweets3 package and continuously changed to a Proxy within the Tor-Network. Assigned single days with a query as tasks to my workers. Modified the GOT3 package, so that it keeps scraping/finishing a day even if a request limit occurred during the day/task, by forcing the scraper to retry with a new Proxy from the Network. Then GOT3 broke down..

Now I am trying to integrate your Scraper into my Docker Framework. Tried to understand the source code and got the idea that I could possibly add the proxy parameter to the requests that it performs myself. But then I got stuck, because I don't really understand what happens with the "TwitterSearchScraper" if it runs into a rate limit. Just wanted to watch the behaviour and let it scrape two days of Bitcoin during the hype end of 2017, must have been ten thousands of tweets, most certainly over 100.000. But it finished the task without running into any request limit like GOT3 did after it scraped approx. 10.000-15.000 tweets in a specific timeframe. Moreover it was blasting fast. So my question: Does it ever run into rate limits? Is a proxy or rotating proxies therefore even needed when I scrape several years of a specific term? How does it behave if it runs into a limit? Does it just stop the scraping then? If so, how could I force it to keep running with a new proxy?

I know that's many questions, but hope you find the time. Keep up the great work.

Thanks!

First off, thanks for the great library, I've been using TwitterSearchScraper and it works great for me.

I'm trying to use TwitterThreadScraper now like so:

for i,tweet in enumerate(sntwitter.TwitterThreadScraper(tweetID=str('1343829375804977153')).get_items()):

However, when I run it I keep getting errors like Tweet does not exist, is not a thread, or does not have ancestors. The given tweet definitely exists (was posted by CNN), so I'm not sure what the problem is here. Looking at the source code, maybe Twitter changed the source page so that the BeautifulSoup search doesn't work anymore? Do you know how it can be fixed?

I was wondering if we could use snscrape to fetch the tweet text for a given twitter status id or a set of status ids. I could not find this in the documentation.

For example: for i, tweet in enumerate(sntwitter.TwitterSearchScraper('from:vistara').get_items()): if i >= 100: break else: tweets_detail.append([tweet.user.username, tweet.date.date(), tweet.content])

output:

Username | Date | Content -- | -- | -- Vistara | 2022-11-03 | @elonmusk Same can be said about morons Vistara | 2022-10-28 | @RealCliveBarker\nMy heart... 🥰 https://t.co/W... Vistara | 2022-10-19 | @Yairir Something is always growing with you... Vistara | 2022-06-13 | @sayollo #CandyCrush has probably eaten as muc... Vistara | 2021-11-12 | @tridentgum Flavor flavor ... | ... | ... Vistara | 2019-08-05 | @hotpockets my imaginary friends! :D :/ :( Vistara | 2019-07-31 | @neilhimself Now I need an audio recording of ... Vistara | 2019-07-31 | @Klondikebar Israel... Vistara | 2019-07-21 | @amithpr How do we get this to go viral? ;)\n... Vistara | 2019-07-21 | @FullFrontalSamB Dickhole. The word you're loo...

It start from the year of 2022. How can i scrape from the latest 2023 ?

Describe the feature

Hi @JustAnotherArchivist

Thank you so much for creating this library. It has really helped a lot of people which I am a beneficiary of and will continue to help. However, I will love to give back to the community by writing documentation on how users can use this library because a lot of resources are not available for it. I have written an article in the past that teaches users how to scrape Twitter content using snscrape which you can see below. https://www.freecodecamp.org/news/python-web-scraping-tutorial/amp/

But I will love to improve this repo by writing docs so users can best navigate it. I await your thoughts on it thank you

Would this fix a problem you're experiencing? If so, specify.

No response

Did you consider other alternatives?

No response

Additional context

No response

Hi Dev, I try to scrap old tweet from the particular user like

record_tweets = []
for i, tweet in enumerate(sntwitter.TwitterSearchScraper(('from:Rajnikanth') since :2019-01-01 untill; 2023-01-01).get_items()):
    data = {
        "user_name": tweet.user.username,
        "content": tweet.content,
        "lang": tweet.lang,
        "Date": tweet.date
    }
    if i > 1000:
        break
    result = record_tweets.append(data)

but i cant able to scrap this Any solution?

Describe the feature

Repeat of https://github.com/JustAnotherArchivist/snscrape/issues/306, but now that viewcount is a public number it would be nice to see :)

Would this fix a problem you're experiencing? If so, specify.

No response

Did you consider other alternatives?

No response

Additional context

No response

Describe the bug

I want to get tweets from certain location so I use this code for tweet in sntwitter.TwitterSearchScraper(f'{query} and place:{place_id}').get_items()

It works well but when it doesnt find anymore results, the for loop doesnt quit, instead it bugs, doesnt advance with the code, making the code stop.

How to reproduce

The for loop shall quit when it doesnt find results anymore, I tried setting a time to make it quit after a certain time, but that doesnt work as it bugs, stopping the code from executing

Expected behavior

The for loop shall quit when it doesnt find results anymore

Screenshots and recordings

No response

OS / Distro

Windows 11

Output from snscrape --version

0.4.3.20220106

Scraper

TwitterSearchScraper

Backtrace

doesnt gives any errors

Dump of locals

No response

How are you using snscrape?

Module

Additional context

No response

Since people seem to keep trying to use snscrape with threads (despite this not being listed as a feature anywhere) and running into problems (seemingly without searching the issues)...

snscrape is currently not thread-safe.

I'd like to evaluate at some point whether it's easy enough to make snscrape thread-safe. One known issue is the Twitter module's guest token manager. Testing thread safety will be an issue, too.

Relevant prior issues: #307 #584 #622

(SEO keywords: threading multithreading)