A universal package of scraper scripts for humans

I would like to contribute an NSE scraper which will help scrape the following:

1. Nifty50 index value
2. Last traded price of a particular stock
3. All nifty50 stock prices

P.S. This repository is a Great initiative✨

Description

Summary:

• Use is to compare type of objects
• Remove unnecessary f-string
• Remove unnecessary generator
• Removed multiple import names
• Add .deepsource.toml

I ran a DeepSource Analysis on my fork of this repository. You can see all the issues raised by DeepSource here.

DeepSource helps you to automatically find and fix issues in your code during code reviews. This tool looks for anti-patterns, bug risks, performance problems, and raises issues. There are plenty of other issues in relation to Bug Discovery and Anti-Patterns which you would be interested to take a look at.

If you do not want to use DeepSource to continuously analyze this repo, I'll remove the .deepsource.toml from this PR and you can merge the rest of the fixes. If you want to setup DeepSource for Continuous Analysis, I can help you set that up.

Description

Hi @DarshanDeshpande 👋

I ran DeepSource Static Code Analysis upon the Project, the results for which are available here.

The Static Code Analysis Tool found potential bugs and anti-patterns in the Code, that can be detrimental at a later point in time with respect to the Project. DeepSource helps you to automatically find and fix issues in your code during code reviews. This tool looks for anti-patterns, bug risks, performance problems, and raises issues.

Some of the notable issues are:

• Missing Arguement in Function Call (here)
• Unnecessary Generator (here)
• f-string used without any expression (here)
• Detected subprocess popen call with shell equals True (here)
• Bad Expect Order (here)

There are plenty of other issues in relation to Bug Discovery and Anti-Patterns which you would be interested to take a look at.

If you would like to integrate DeepSource to autofix some of the common occurring issues, I can help you set that up :)

I would like to contribute a program to scrape reddit posts obtained when a specific topic is searched. Following information will be recorded:

• number of upvotes
• number of comments
• title
• author
• link
• subreddit name
• isSponsored flag

Program will:

• make use of reddit endpoints.
• support explicit proxies
• allow to put a cap on max posts to scrap
• allow to specify sleep interval between requests

This update will include same functionality as previous version but the execution is a lot faster. Here are the results:

POSTS SCRAPED | NORMAL | ASYNC --|--|-- 20 | 123s | 33s 50 | 213s | 63s 100 | 375s | 143s 200 | 738s | 262s

With the file sizes remaining almost constant

Hi,

First of all, thank you for initialize such a wonderful project!

As I am reading this repo, I found it will be convenient if I can see what crawler is implemented from README, instead of looking into codes.

Not sure if this will be a good idea at this time. Feel free to leave your comment or simply close it.

This PR includes:

• Use is instead of ==
  • It is recommended to use identity test ( is ) instead of equality test ( == ) when you need to compare types of two objects.
• Simplify boolean expression
• Remove self
  • The method doesn't use its bound instance. Decorate this method with @staticmethod decorator, so that Python does not have to instantiate a bound method for every instance of this class thereby saving memory and computation.