How to create recurring pipelines?

Once you‘ve verified a pipeline works well, mostly likely you’ll want to run that pipeline periodically to keep data fresh, and DevLake's pipeline blueprint feature have got you covered.

  1. Click ‘Create Pipeline Run’ and
  • Toggle the plugins you'd like to run, here we use GitHub and GitExtractor plugin as an example
  • Toggle on Automate Pipeline image

  1. Click ‘Add Blueprint’. Fill in the form and ‘Save Blueprint’.

    • NOTE: That the schedule syntax is standard unix cron syntax, Crontab.guru could be a useful reference
    • IMPORANT: The scheduler is running under UTC timezone. If you prefer data collecting happens at 3am NewYork(UTC-04:00) every day, use Custom Shedule and set it to 0 7 * * *

    image

  2. Click ‘Save Blueprint’.

  3. Click ‘Pipeline Blueprints’, you can view and edit the new blueprint in the blueprint list.

    image