Keep Your Scrapers Running Forever: A Beginner’s Guide
Introduction
Are you facing trouble with your scrapers getting blocked or constantly needing verification steps like email confirmation, captchas, or phone validation? Don’t worry; you’re not alone! These issues can arise due to various reasons, such as using poor proxies or making too many API calls …
But fear not! In this article, we’ll show you how to improve your scrapers and keep them running smoothly and efficiently. By following our simple steps, you can significantly increase the number of API requests your scrapers can handle while keeping your accounts active and working flawlessly. No more headaches or disruptions – let’s get your scrapers optimized and ready to excel!
Getting Started: Creating Your Accounts
When it comes to setting up scraper accounts, there are two main approaches – some users prefer to create their accounts from scratch, while others opt to buy ready-made accounts from sellers. If you choose to purchase scrapers, you can skip the account creation phase and move on to the next steps.
For those who decide to create their accounts, we recommend following these simple tips:
1. Choosing Proxies: Proxies are essential for creating accounts, but finding reliable ones can be challenging. It’s crucial to test multiple proxy providers to find the best fit. Mobile or 4G proxies are often considered the most effective, though they might be a bit costly. If you have a good budget, investing in these proxies without extensive testing could be a good idea.
2. Using IG Create Tool: If you’re using SuSocial’s IG Create Accounts tool, you can start the creation process. Follow the guide provided to understand how to use the tool and create the accounts. You’ll need to decide whether to create accounts using email addresses or virtual phone numbers. We suggest opting for email addresses since Instagram usually asks for email verification later on when you start using the accounts.
3. Natural Account Creation: To avoid triggering spam suspicions, make sure to customize your accounts. Use real-looking full names, bios, and profile images. Leaving accounts without these details may raise red flags for the Instagram algorithm, suspecting them as potential spam accounts.
4. When Adding Accounts Add Them With Status Stopped: To prevent the newly created accounts from making API calls immediately after being added to the Social Profile view, we recommend checking the appropriate option in the IG Create Tool.
Getting Ready: Warming Up Your Accounts
Once you’ve created your accounts and added them to the SOCIAL PROFILES view, it’s crucial to follow these tips to warm them up properly:
1. Wait for 24 Hours: Right after creating the accounts, take a pause and let them rest for a full 24 hours before moving on to the next steps.
2. Configure API Limits and Delays: Before switching the accounts from STOPPED to VALID status, it’s essential to set up the API Limits and Delays in the Advanced Profile Settings of your scrapers. These settings control how many API Calls your scrapers can make per hour and per day. To learn more about these options, refer to this guide.
As a note, there is no one-size-fits-all recommendation for these settings because each account has its unique limits on how many API calls it can make within a specific timeframe. To find the best settings for your accounts, it’s essential to experiment with different API Limits and Delays.
We have tested three different settings that generally work well for newly created accounts. These settings are as follows:
Settings 1:
Settings 2:
Settings 3:
Understanding API Call Gradual Increase for Scraper Accounts
API calls play a crucial role in the functioning of scraper accounts. To ensure the smooth operation of these accounts, it’s essential to set up their API call behavior effectively. Here’s how it works:
- Start with Fewer API Calls: Initially, the account is configured to make only a limited number of API calls. This cautious approach helps prevent any sudden spikes in activity that might raise suspicions with Instagram’s spam detection system.
- Gradual Increase Over Time: As the days and hours pass, the number of API calls gradually increases. This incremental growth ensures a steady warming-up process for the scraper accounts. They start gaining more access and activity while avoiding overwhelming Instagram’s systems.
- Hitting the Target Number of API Calls: The gradual increase continues until the account reaches a specific pre-determined number of API calls. This target number is carefully chosen to strike the right balance between efficient data extraction and keeping the account safe from detection.
By implementing this gradual increase strategy, we can effectively “warm up” the scraper accounts. This means preparing them to handle more API calls over time, without triggering any suspicion from Instagram’s algorithms. Warming up the accounts in this manner is a smart and cautious approach, ensuring a longer and more productive lifespan for your scraper accounts.
NOTE: Keep in mind that as your main accounts perform more actions, your scrapers may experience delays more frequently. In such cases, it’s better to add more scrapers instead of increasing API Limits and Delays, as doing so could strain your scrapers and shorten their lifespan. The key is to keep your scrapers alive for as long as possible, especially when you’re finding the best API Limits and Delays that work effectively for your setup.
3 – Addionnal Helpful Settings:
- Partial API Emulation: Consider using this option as it reduces the number of API calls made compared to Full API Calls. Although it might trigger more verifications, it tends to work better. We suggest trying it on some of your scrapers to see the results.
- Do Random Actions Using API: Enable this option to have your account perform actions like Following, Unfollowing, and Liking posts randomly. This helps mimic human behavior, making your account appear like a normal user and reducing the chances of Instagram flagging it as spam. You can also use it to post content like regular posts and stories.
- Delay for Proxy Issues: If your proxies go down and your scrapers show a VALID status, your main accounts might start getting “No More Results Found” errors. This can make you think there’s an issue with your main accounts, but it’s actually caused by the scrapers’ proxies being down. To avoid this confusion, we recommend using the “Delay the Account when the Proxy is Down” option.
Reaching the Finish Line: Making the Most of Your Accounts
The key aim is to maximize the longevity of your scraper accounts. However, as time goes on, these scrapers may encounter verification challenges such as email verification, phone validation, or captchas. Handling these verifications manually can be time-consuming and challenging to monitor consistently.
To simplify the process, it’s essential to set up the tool to handle verifications automatically. To do this, check the options below:
To enable these options to work automatically, you’ll need to make some additional setups.
For Email Auto Verification:
To let the tool handle email verifications automatically, you must set up the Email Validation tab. The following guides will help you understand how to configure this option for different email providers.
How To Enable External App Password For Mail.ru (BK.ru, index.ru, List.ru)
For Phone Auto Verification:
To set up automatic phone verification for your accounts, go to the Connectivity tab and configure a phone verification service. Check out the article below for a step-by-step guide on how to do this.
For Captcha:
To manage accounts with captcha verification, you must set up a captcha service. For a detailed guide on how to do this, please refer to the article below.
Summary
In this guide, we’ve covered essential tips to optimize your scraper accounts, ensuring they handle API calls efficiently. By setting up automatic verifications for email, phone, and captchas, you’ll enjoy seamless scraping without manual interventions.