I'm loving this, it's perfect for what I need. Maybe a feature to consider in the future is to be able to use multiple digital ocean accounts as there's a limit of 10 for new users?

Thanks for releasing this!

Hello

I launched the container as per the documentation, but I'm still not seeing anything in the UI nor the API, I can see it created the droplets and keep removing/adding new droplets every few minutes

What I'm missing?

export DIGITALOCEAN_ENABLED=True
export DIGITALOCEAN_ACCESS_TOKEN="XXXXX"
export DIGITALOCEAN_MIN_SCALING=2
export DIGITALOCEAN_MAX_SCALING=2
export DIGITALOCEAN_SIZE="s-1vcpu-512mb-10gb"
export DIGITALOCEAN_REGION="fra1"
export AGE_LIMIT="1200"
export USERNAME="XXX"
export PASSWORD='XXXX'

docker run -e USERNAME=$USERNAME \
    -e PASSWORD=$PASSWORD \
    -e DIGITALOCEAN_ENABLED=$DIGITALOCEAN_ENABLED \
    -e DIGITALOCEAN_ACCESS_TOKEN=$DIGITALOCEAN_ACCESS_TOKEN \
    -e DIGITALOCEAN_MIN_SCALING=$DIGITALOCEAN_MIN_SCALING \
    -e DIGITALOCEAN_MAX_SCALING=$DIGITALOCEAN_MAX_SCALING \
    -e DIGITALOCEAN_SIZE=$DIGITALOCEAN_SIZE \
    -e DIGITALOCEAN_REGION=$DIGITALOCEAN_REGION \
    -e AGE_LIMIT=$AGE_LIMIT \
    -it -p 8000:8000 laffin/cloudproxy:latest

Actually i'm working with a project with same name called CloudProxy (https://github.com/NoahCardoza/CloudProxy). It's to pass throw cloudflare challenge. To do that is using a chrome with pupeeter. It's starting a chrome browser passing a proxy via parameter but it doesn't accept user and password on it (https://superuser.com/questions/902620/google-chrome-proxy-settings-with-username-and-password). This is the motivation to fork it and implement an alternative way to authenticate with proxy than using user and password. I used the parameter allowed_ip from tinyproxy.

Implemented:

• Alternatrive authentication using ALLOWED_IP
• Variable environments that are using boolean now are using real boolean instead of String. Instead of using ENABLE_AWS=True now you should write ENABLE_AWS=true. And in code we are using now (if ENABLE_AWS:) instead of (if ENABLE_AWS='True')
• Check if proxy is working we are trying to load google.com throw the proxy instead of check the proxy url directly. For some strange reason proxy server response 403 HTTP forbidden when you visit is directly. Ex: http://10.12.23.3:8899 if it's using allowed_id, but it's really working as proxy. Increased timeout to 6 to let the proxy do its job.
• Added optional parameter AWS_KEY_NAME to pass throw a pairkey to login on the EC2 instance.
• Added optional parameter PROXY_STEALTH=true, default false to set proxy in stealth mode and not sending proxy headers on its requests to the sites.
• Update README.md and docs/aws.md

Since is the first time i do something on python and docker. Probably is not really optimitzed and i guess there is space to enhance. I tested it on DigitalOcean, Hetzner and AWS and it works flawless without problems atm.

I published a docker image: bubexel/cloudprroxy:latest

Thank you for your work on it!

Greetings

Expected Behavior

Running the command:

"docker run -e USERNAME='xxx' -e PASSWORD='xx' -e DIGITALOCEAN_ENABLED=True -e DIGITALOCEAN_ACCESS_TOKEN='xxx' -it -p 8000:8000 laffin/cloudproxy:latest

Username & Password being alphanumeric. Token validated by using:

"doctl auth init -t "xxx"

I get the following error:

File "/usr/local/lib/python3.8/site-packages/digitalocean/baseapi.py", line 233, in get_data raise DataReadError(msg) │ └ 'Unable to authenticate you' └ <class 'digitalocean.DataReadError'>

I think my bug is identical to George Roscoe's. I've never had an issue running this before. I ran this a few weeks ago and it worked completely fine

I've run into an issue that I can't seem to pinpoint so I'm not sure if it's due to CloudProxy (TinyProxy).

I've set up CloudProxy to run in Docker with 15 AWS Spot instances. Then I've written a Python Flask script that fetches the IPs from CloudProxy once every minute, accepts an URL (GET request), and returns the html page fetched through one of these AWS proxies. The reason I'm doing it this way is because my original application that uses the html data doesn't allow me to set the user agent, so I need to go through a proxy that allows this.

This is the fetch line in the Flask application (proxy):

proxies = {"http": proxy, "https": proxy} resp = requests.get(url, headers=headers, proxies=proxies, timeout=5, allow_redirects=True, stream=True)

It can run fine for hours until suddenly all my AWS instances started dying. I went through the CloudProxy code and identified that the restarts was due to the ALIVE checks failing. So I disabled that code and also added some exceptions in my own application. It solved the instances dying, but not the original issue.

It turned out that the code line above (requests.get) suddenly starts throwing the following error:

HTTPConnectionPool(host='X.X.X.X', port=8899): Max retries exceeded with url: http://www.url.com?page=1 (Caused by ProxyError('Cannot connect to proxy.', NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f39cd7d7550>: Failed to establish a new connection: [Errno 113] No route to host')))

I've masked the IP and url for privacy reasons.

So basically, my scripts run for hours with 1-2 requests every second until all the requests suddenly starts spitting out the exception above until it bogged down my entire WiFi. The internet on all of my computers almost stops responding. The only solution is to stop the requests, give it a few minutes and then resume like nothing happened.

After fixing the Spot instances dying, my second idea was that there's some kind of TCP limit in AWS. So I upgraded my instances from Nano to Micro with no apparent improvement. I considered it being a Docker issue but I only fetch ALIVE IPs once every minute so I can't see how that would be limited in any way. I don't see it being a TinyProxy limit since my 1-2 requests are spread out over 15 different AWS instances.

Do you know if there is any AWS limit I'm hitting or have you experienced anything similar with CloudProxy?

This Pull Request adds Google Cloud provider support for cloudproxy and solves #35.

Known issues:

• Proxy removal from cloudproxy-ui does not work when "Remove" button is clicked. (At least didn't work for me)

Future work:

• Given that this code is based on the AWS code, it will be probably a good idea at some point in the future to refactor all the providers' code to reduce the amount of duplicated code and logic.

Expected Behavior

I followed the docs here https://github.com/claffin/cloudproxy and here https://github.com/claffin/cloudproxy/blob/main/docs/aws.md I created environment variables, all alphanumerica, for USERNAME and PASSWORD. I created an IAM role as instructed, and I can see the EC2 instances. WHen using the toy example these are correctly filled in (i.e., instead of being changeme:changeme@ip it is user:password@ip).

Actual Behavior

WHen connecting to the proxy I get the error message: The administrator of this proxy has not configured it to service requests from you.

This is almost certainly from my misunderstanding the docs (as I haven't worked with AWS before). Are we meant to set the username and password somewhere in AWS too? I also tried creating a password for the IAM user and using that, but that isn't allowed to be alphanumeric. I'd also be happy to write some documentation for entry/beginners like myself once I get it up and running

docker run -e USERNAME='xxx' -e PASSWORD='xxx' -e GCP_ENABLED=True -e GCP_PROJECT='xxx' -e GCP_SIZE='e2-micro' -e GCP_ZONE='asia-northeast3-a' -e GCP_SERVICE_ACCOUNT_KEY='xxx' -it -p 8000:8000 laffin/cloudproxy:latest

I tried this command but it always creates instances in the us default zone, I can't switch it to the different zone. Can you fix this ?

Thank you

When deleting proxies filled by one-time spot requests, the instances are terminated, but the spot request itself is not cancelled, leaving the door open to being filled again in the future when not associated with cloudproxy. This PR deletes the spot request if it exists in the delete_proxy() function. Related to #42, but unclear if applicable to persistent spot requests.

Expected Behavior

The proxies are destroyed and remain gone.

Actual Behavior

The proxies are destroyed, then some unknown time later are restarted but without the cloudproxy tag since they are started by something other than cloudproxy. They are fully functional proxies though, just missing the tag.

Steps to Reproduce the Problem

1. Start cloudproxy
2. Increase servers to 30, wait.
3. Decrease servers to 5, wait.

Specifications

• Version: 0.5.2

Solution

My guess is that when you destroy the instances you also have to remove the spot request somehow, but I don't quite understand why.

I think this will partly solve https://github.com/claffin/cloudproxy/issues/3.

The cert.pem and key.pem can be generated with mkcert

Tbh I have no idea what I'm doing, any suggestions would be much appreciated. I tested this locally and I have the docker image running on https now.

Edit: I suppose mkcert is only good for https in local environments. My goal is to deploy Cloudproxy to an AWS ec2 instance and make it accessible via HTTPS, and ensure communication between the cloudproxy server and proxy servers are done via HTTPS as well.

Hello

Thank you very much for this great script, simple and can be a replacement for scraproxy

Is it possible to define multiple regions? for example, I want to have 3-5 regions with digitalocean and cloudproxy will randomly create VMs on them

my second question is there any log file or report that I can use to check how many VMs has been created, the duration, and the period? so I can compare my hosting cost let's say on weekly bases and decide which cloud provider is better for me?

One thing I've always missed in Scrapoxy is ability to support multiple clients. Would be great to see it implemented here.

In Scrapoxy you could set (min, required, max) scaling and it works well as long as there is just one client application trying to use the proxy cloud. But as soon as you want to share same cloud between multiple applications, you run into a problem that they conflict with each other. E.g. when one application has finished crawling, it can't just downscale the cloud as it's still being used by another application etc.

Ideally that requires a centralized logic that manages requests from multiple client applications. It would need to track the most recently requested scaling for each client, and combine them. A very simple logic could be to just take max of all min/required/max parameters across clients and use that as the scaling. That way, the cloud would only downscale when the last client sends the downscale request. You can imagine logic becoming more complex though, e.g. when one client asks to destroy an instance that the other client still uses etc.

As an extra feature, it should ideally handle stale clients - if a client has not communicated with it for a while, it should disregard its requirements, to avoid leaving dangling instances when client unexpectedly disappears.

Report of -e DIGITALOCEAN_MIN_SCALING=0 -e DIGITALOCEAN_MAX_SCALING=0 commands do not work, it always starts at 2.

Originally posted by @sblfc in https://github.com/claffin/cloudproxy/issues/21#issuecomment-836554779

At the moment CloudProxy supports AWS and DigitalOcean, which is enough for my own personal use case. I'm keen to hear if there is interest in other providers being supported, please share here and I will prioritise. Otherwise, new features will be prioritised for now.