Ad-hoc cookies retrieval during runtime?

[vhbui02]

This is your reply from worker_vm#2:

Is it possible to also pass cookies to ComicCrawler?

Yes, I know what cookies Colamanga used to authenticate, so I can populate them manually.

There are several ways to use cookies in ComicCrawler:

1. Define browser and browser_profile in setting.ini. ComicCrawler will import cookies from the browser automatically. (https://github.com/eight04/ComicCrawler?tab=readme-ov-file#%E9%97%9C%E6%96%BC%E9%9C%80%E8%A6%81%E7%99%BB%E5%85%A5%E7%9A%84%E7%B6%B2%E7%AB%99)
2. In the module, set a cookie global variable to define cookies.
3. During runtime, get the session object and configure its cookies property. For example:

   ComicCrawler/comiccrawler/mods/twitter.py

   Line 48 in ab1be03

   def init_api_session():

   . This example configures headers but cookies are similar.

All 3 required populating cookies manually either by going to the website once, open DevTools and copy the cookie or specify browser= in setting.ini (btw 3 is actually the last step of 1). What if I know the API that set the authentication cookies, can I fetch it and retrieve the cookies from response object? It could become the 4th way.

[eight04]

grabber returns a Response object and you can get the cookie from .cookies:
https://github.com/lexiforest/curl_cffi/blob/69932267b52219ea353b1c7a3e69bd2e593d5fd8/curl_cffi/requests/models.py#L49

Note that the session object handles cookies automatically, so the next request will use the cookie set by the previous response (suppose they both use the same session.)

[vhbui02]

A chain of same-session requests will accumulate cookies values right? Does grabhtml and grabimg also set cookies for the session?

[vhbui02]

I see in a module, cookies can be specified in 2 locations: config and cookie. Can I ask about your design decision regarding it?

[eight04]

A chain of same-session requests will accumulate cookies values right?

Yes. It works like browser sessions.

Does grabhtml and grabimg also set cookies for the session?

Yes. they use grabber under the hood.

cookies can be specified in 2 locations: config and cookie.

In config, users can edit the value in setting.ini. In cookie they can't.

design decision

When a site requires login, users used to specify credentials in setting.ini. Nowadays, we can read cookies directly from browsers, so there is no need to set them manually anymore. Therefore you won't see cookie configs in new modules.

Another method to set cookies is to use curl configs. Still, it is more convenient to read from the browser automatically.

[vhbui02]

I see, you keep config from legacy and backward-compatible perspective.

P/s: I was wrong. Turns out downloading the encrypted image of Colamanga doesn't require any cookies value. I wrote my progress here, can you check it.

[vhbui02]

Do you use Discord, sometimes I want to ask something with faster reply time so I can continue working on the problem.