Made a Free Proxy Rotator library

VPN
1

Hello there,
I made a free proxy rotator library that scrapes and validates proxies from freely available sources and rotates them. Integrates well with requests library. Should be helpful for web scraping. Check it out at https://github.com/sachin-sankar/swiftshadow.

2

Hey, I encountered an error preventing me from importing your library as shown in the README. Here is my console output:

Python 3.11.3 (main, Apr  7 2023, 00:39:07) [Clang 14.0.7 (https://android.googlesource.com/toolchain/llvm-project 4c603efb0 on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from swiftshadow.swiftshadow import Proxy
[ swiftshadow ] ERROR : No cache found. Cache will be created after update
[ swiftshadow ] INFO : Updating proxies from Proxyscrape
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/data/data/com.termux/files/usr/lib/python3.11/site-packages/swiftshadow/swiftshadow.py", line 134, in <module>
    print(a.proxy())
          ^^^^^^^^^
  File "/data/data/com.termux/files/usr/lib/python3.11/site-packages/swiftshadow/swiftshadow.py", line 122, in proxy
    if cache.checkExpiry(self.expiry):
                         ^^^^^^^^^^^
AttributeError: 'Proxy' object has no attribute 'expiry'
>>>
3

I don’t quite understand the implementation. I only see scrapping for ip addresses, rotation of ips and setting as self.current but I don’t see making an actual connection throuh the proxy. Could you care to explain where is the part of actualy using proxy’s ip to make a connection?

4

I was working for such a thing some time ago.

5

Thank you for telling us exactly what this is and what it does as the first sentences of your Readme.

Too many projects skip that part and just assume you already know what they are.

6

nice ! Does this work with browsing with JavaScript?

and how does the proxy list is updated?

7

Thanks for sharing, will check this out, as I did exactly the same for free proxies from one source, although I found it wasn’t as reliable, and thought to purchase paid proxies for scraping.

8

I will look into it right away

9

Added an issue https://github.com/sachin-sankar/swiftshadow/issues/9

10

Well i should have added this to the docs which i will but here is a short code on how to make GET requests using the proxy.

from swiftshadow.swiftshadow import Proxy
from requests import get

swift = Proxy()

request = get('https://ip.me',proxies=swift.proxy())
print(request.text)
11

No it doesn’t work with browsing with js currently but can be done using a extension. The proxy lists are scraped from Providers like Proxyscrape who are very generous to have free lists on their site.

12

I get it now. I just assumed that your library made the connection as well and was baffled when I couldn’t find that part in the code.
Thanks for the explanation. I didn’t know that the requests.get method has the proxies parameter.
I will try out your lib in my scrapping tools for sure. :slight_smile:

13

Thanks also tell your friends abt this shud help them too