Best strategy for bulk usage?

[emailsfrominstagram]

I'm looking for the best way to retrieve data in bulk.

I noticed on the Telegram group someone said it's best to limit to 150-200 api requests per day.

It seems the client.user_followers will call user_followers_v1_chunk which will keep calling the api over and over until it gets all the needed users or an exception occurs.

If I want to retrieve, say 10k followers, is the best way to just call client.user_followers(username, amount=10000) and hope I don't get banned? Or should I call client.user_followers(username, amount=1000) 10 times across 10 different users? Or is there another, better way to handle bulk requests?

[adw0rd]

If I want to retrieve, say 10k followers, is the best way to just call client.user_followers(username, amount=10000) and hope I don't get banned?

yes it is the best

Or should I call client.user_followers(username, amount=1000) 10 times across 10 different users?

This is a completely different task, you will get not 10k users of a person, but 1k each from 10 different people

[emailsfrominstagram]

yes it is the best
Thanks @adw0rd for the helpful and super fast reply!

What if I needed 100k, would that still be the best option? At some point instagram will ban the user, right?

So should I be using handle_exception to just switch users on exception and keep going? If so, won't I get into the same issue? Or does the paging between requests work even if the user is switched out for another?

[adw0rd]

Yes, you need to switch the user, paging used to work exactly when changing the user

[emailsfrominstagram]

Yes, you need to switch the user, paging used to work exactly when changing the user

Thanks.

I actually think it would be useful to use an iterator a generator here (instead of iterating over all results and then returning them). Would you be open to a PR against it?

[adw0rd]

@emailsfrominstagram show how it will look like, I don't mind, but users may be confused by the change in the user interface

[emailsfrominstagram]

@emailsfrominstagram show how it will look like, I don't mind, but users may be confused by the change in the user interface

Before I create a PR (which requires me to actually test my code etc), here's a quick example of what it would look like. Does this make sense to you? Instead of functions returning a List they return a Generator which for most consumers (who just iterate over the results) it should not make a diff. Note that we no longer need an amount passed to the user_followers method because we page the results lazily. I'm not sure we need amount passed to the other functions either but I kept it in the other places for now.

Please lmk what you think of this idea and if you like it and I'll actually write the code and test and submit a PR:

The user_followers function would look something like:

    def user_followers(
            self, user_id: str, use_cache: bool = True
    ) -> Generator[(str, UserShort), None, None]:
        """
        Get user's followers

        Parameters
        ----------
        user_id: str
            User id of an instagram account
        use_cache: bool, optional
            Whether or not to use information from cache, default value is True

        Returns
        -------
        Dict[str, UserShort]
            Dict of user_id and User object
        """
        return_count = 0
        user_id = str(user_id)
        users = self._users_followers.get(user_id, {})

        # first return users from cache
        if use_cache and users:
            for user in users.items():
                yield item
        
        # then use the api
        try:
            for user in self.self.user_followers_gql(user_id):
                self._users_followers[user.pk] = user
                yield (user.pk, user)
        except Exception as e:
            if not isinstance(e, ClientError):
                self.logger.exception(e)
            for user in self.user_followers_v1(user_id):
                self._users_followers[user.pk] = user
                yield (user.pk, user)

and user_followers_gql_chunk would look like:

    def user_followers_gql_chunk(self, user_id: str, max_amount: int = 0, end_cursor: str = None) -> Generator[
        Tuple[UserShort, str], None, str
    ]:
        """
        Get user's followers information by Public Graphql API and end_cursor

        Parameters
        ----------
        user_id: str
            User id of an instagram account
        max_amount: int, optional
            Maximum number of media to return, default is 0 - Inf
        end_cursor: str, optional
            The cursor from which it is worth continuing to receive the list of followers

        Returns
        -------
        Tuple[List[UserShort], str]
            List of objects of User type with cursor
        """
        user_id = str(user_id)
        users = []
        variables = {
            "id": user_id,
            "include_reel": True,
            "fetch_mutual": False,
            "first": 12
        }
        self.inject_sessionid_to_public()
        while True:
            if end_cursor:
                variables["after"] = end_cursor
            data = self.public_graphql_request(
                variables, query_hash="5aefa9893005572d237da5068082d8d5"
            )
            if not data["user"] and not users:
                raise UserNotFound(user_id=user_id, **data)
            page_info = json_value(data, "user", "edge_followed_by", "page_info", default={})
            edges = json_value(data, "user", "edge_followed_by", "edges", default=[])
            for edge in edges:
                yield extract_user_short(edge["node"])
            end_cursor = page_info.get("end_cursor")
            if not page_info.get("has_next_page") or not end_cursor:
                break
            if max_amount and len(users) >= max_amount:
                break
        return end_cursor

user_followers_gql simplifies to:

    def user_followers_gql(self, user_id: str, amount: int = 0) -> Generator[UserShort, None, None]:
        """
        Get user's followers information by Public Graphql API

        Parameters
        ----------
        user_id: str
            User id of an instagram account
        amount: int, optional
            Maximum number of media to return, default is 0 - Inf

        Returns
        -------
        List[UserShort]
            List of objects of User type
        """
        yield from self.user_followers_gql_chunk(str(user_id), amount)