Pipelight

Appearance

Parallel tasks execution ​

Pipelight allows you to launch parallel tasks in one uniq way that can be declined to your needs.

Usage ​

ts
const my_pipeline = {
  name: "test",
  steps: [
    {
      parallel: [
        {
          name: "first",
          commands: [...my_commands]
        },
        {
          name: "second",
          commands: [...my_commands]
        }
      ]
    }
  ]
};
const my_pipeline = {
  name: "test",
  steps: [
    {
      parallel: [
        {
          name: "first",
          commands: [...my_commands]
        },
        {
          name: "second",
          commands: [...my_commands]
        }
      ]
    }
  ]
};

Parallel steps ​

Parallel step execution by declaring a Parallel object which is an array of multiple Steps.

  1. At start, first and second steps will run simultaneously.
ts
const my_pipeline = {
  name: "test",
  steps: [
    {
      parallel: [
        {
          name: "first", 
          commands: [...my_commands] 
        },
        {
          name: "second", 
          commands: [...my_commands] 
        }
      ]
    },
    {
      name: "third",
      commands: [...my_commands]
    }
  ]
};
const my_pipeline = {
  name: "test",
  steps: [
    {
      parallel: [
        {
          name: "first", 
          commands: [...my_commands] 
        },
        {
          name: "second", 
          commands: [...my_commands] 
        }
      ]
    },
    {
      name: "third",
      commands: [...my_commands]
    }
  ]
};
  1. Then, the third step will run once the whole parallel object has resolved. Which means, when the first and second steps have both resolved.
ts
steps: [
  {
    parallel: [
      {
        name: "first",
        commands: [...my_commands]
      },
      {
        name: "second",
        commands: [...my_commands]
      }
    ]
  },
  {
    name: "third", 
    commands: [...my_commands] 
  }
];
steps: [
  {
    parallel: [
      {
        name: "first",
        commands: [...my_commands]
      },
      {
        name: "second",
        commands: [...my_commands]
      }
    ]
  },
  {
    name: "third", 
    commands: [...my_commands] 
  }
];

Parallel pipelines ​

Attached subprocess ​

Trigger pipelines in parallel by creating a pipeline that run them from inside parallel steps. You just need to --attach them to the main pipeline.

The main pipeline will wait until both pipelines execution is completed and then continue its execution.

ts
const my_pipeline = {
name: "test",
steps: [
  {
    parallel: [
      {
        name: "first", 
        commands: ["pipelight run pipeline1 --attach"] 
      },
      {
        name: "second", 
        commands: ["pipelight run pipeline2 --attach"] 
      }
    ]
  }
];
}
const my_pipeline = {
name: "test",
steps: [
  {
    parallel: [
      {
        name: "first", 
        commands: ["pipelight run pipeline1 --attach"] 
      },
      {
        name: "second", 
        commands: ["pipelight run pipeline2 --attach"] 
      }
    ]
  }
];
}

Detached subprocess ​

Running them without the --attach flag will launch them in the default mode which is detached. In detached mode, the main pipeline can not know the subpipelines status but only the process status.

It will therefor instantly return the process status:

  • succeeded if the process is spawned
  • failed if the process couldn't be spawned

You may want to run subpipelines in detached mode if you don't care about the result.

For example:

If a step bulk sends emails and you can not afford to wait until it is finished to resume the pipeline execution. Then you should consider using a detached pipeline. The main pipeline execution will continue as soon as the emailing pipeline is spawned.