Scheduled task running like cron job?

[victorx98]

Can we (or do we) have a time triggered mechanism like cron job? So we can schedule some of our task to run periodically at certain time?
Thanks

[Thinking80s]

+1

[sanqianyuejia]

+1

[ZHLHZHU]

+1

[xjose97x]

For now I think one solid alternative would be to create a cron job on your computer and call the relevant Dify web API.

[y-omr]

+1

[vacheli]

+1

[0xmalina]

+1

[fungxu]

+1

[shieldone]

+1

[yt-koike]

+1

[yt-koike]

I had a quick search and found that #19409 might be the most relevant issue to this discussion.

[yt-koike]

Or this?

[yt-koike]

I think I came up with a simple solution.

Let's add "Regular self-call" feature to the features (see below) of a workflow. It triggers the workflow every second with a query message saying "!cron!". You can easily check if the incoming query is a regular self-call or not by comparing the query message to that keyword.

If you want a schedule feature like, triggering a workflow at 9:00 am every morning, you can use Current Time node like the workflow below.

I think this is one of the simplest solution and is easy to implement. Still, it's a naive idea and needs reviews. For example, hackers can manualy trigger this regular self-call by setting query "!cron!". To prevent that, we can introduce a new input variable "sys.is_cron" along with "sys.query" and "sys.conversation_id". From a different perspective, self-calling every second might create too many queries to process so we could reduce the frequency to every 10 seconds or every minute then.

Any comments will help! Thank you :)

[alterxyz]

Our team is designing a trigger feature, which might include the scheduled task.

[yt-koike]

Hello again, I'm yt-koike.
I tried developing a plugin to run cron jobs.
Please feel free to use it. https://github.com/yt-koike/dify-cron
[Caution] It's still an experimental feature so be careful when you use it. I don't think it's very harmful tho. Deleting the plugin from your Dify would leave no bugs behind.
It works nicely on self-hosted servers but not so well on cloud.dify.ai.
If you have any ideas to improve this cron plugin, please make an issue or an PR to my repo, thank you!

I am also planning to make a PR to publish it to marketplace. langgenius/dify-plugins#550

[yt-koike]

It might become unnecessary after the official release of scheduled tasks.
#5175 (comment)

[alterxyz]

Hi yt-koike,

Thank you for developing this plugin! Our native trigger feature will take several months to implement, so having your plugin available would be extremely valuable for the community.

I've reviewed your work and have a few important points to share about this excellent plugin:

1. Plugin marketplace submission: To contribute to our plugin marketplace, please submit your .difypkg file (the packaged single file) rather than the raw source code and individual files. You can find detailed guidance in our documentation: https://docs.dify.ai/plugin-dev-en/0322-release-to-dify-marketplace

2. Cloud vs. self-hosted compatibility: After confirming with our development team, this plugin will likely work best on self-hosted versions. Our cloud endpoints run on AWS Lambda, which may cause compatibility issues with your current implementation.

   a. Potential improvement: The current approach using started_app_ids = set() maintains state, which isn't recommended for cloud environments. Converting this to a stateless design could improve cloud compatibility.

I'll also be adding more detailed documentation about our cloud runtime architecture (hybrid AWS Lambda + Dify instance) to help developers optimize their plugins for both environments.

Thanks again for your contribution, and please let me know if you have any questions!

[yt-koike]

Hello @alterxyz
I appreciate your comment enormously!
Sorry for accidentally putting my whole source code on plugin submission PR. I fixed it up.
Additionally, thank you for pointing out the state bit. I've been slightly concerned about that part so I want to introduce stateless design and improve the code later on.

[yt-koike]

Hello, people.
I made a little update on Cron.
Now it works on cloud.dify.ai, supports timezones and has more detailed status.

I struggled to figure out how to make it work on cloud Dify and finally made it by integrating an API client for https://cron-job.org. I technically offloaded the calling function so cloud Dify doesn't need to keep calling itself. I kept the old code mostly untouched so it works on self-hosted servers as before.

I've already released the .difypkg on my repo and made a PR to the dify-plugins repo too.

Please try it if you want to! Thank you :)

[kurokobo]

Almost there :)
#23981