Recommendations to make the build pipeline fail fast

Our business project builds use our own toolchain, which in itself is composed of multiple projects.
The main reason for having our ow toolchain is that it enforces strict QA assurance rules. We tune those rules from time to time to be even stricter.
Changes in the components take a long time and manual work to complete, and problems are often only apparent when we actually build a business project with the new version.

You job is to take a look at the toolchain and come up with a set of recommendations to make it faster, more automated, and fail faster.

This job is the basis for other ones, most notably actually implementing the recommendations.

We have complex inter-project dependencies:
-- toolchain is our build container.
--- It depends on zenta-tools through a debian repository which we are updating manually at the moment
---- zenta-tools depends on metamodel in a soft way: the projects depend on a specific version of the metamodel, and some functions are dependent on that
--- toolchain tests itself by executing (hence depending on)
---- cdd a java-based business project, and
---- astinxml, a python-based business project

All the mentioned projects are public on github.

More information on the build pipeline:

All of the builds use docker containers, and build with shippable. You can find a reference to the docker container used in shippable
Our projects all use the toolchain container, the zenta ones not yet.

At the bottom of the pit sits zenta. It is an architecture modeler
Zenta is deployed to a debian repository. We include it into the repository by hand. You will find the exact coordinates in the Dockerfile of toolchain

Next is toolchain. It creates the docker container. This docker container is the basis of all of our business project builds.
It is versioned, and the builds use a specific version.
It uses zenta, zenta-tools, and a couple of other things, including modified versions of one or two projects. The modified versions are built from source.

As a check, toolchain runs the build for cdd and astinxml, a java based and a python based project. If any of those builds fail, it does not deploy.

After that come to our projects, cdd and astinxml, which depend on a specific version of the toolchain. We have a couple of other projects which uses toolchain, these projects are just the ones we test on.
After a change in the toolchain, the build of the business project can fail for two kinds of reasons:
1. some components in toolchain (or its components, zenta, and zenta-tools) have a problem.
2. some deliberate changes in toolchain or its components brought in stricter QA rules, to which the business project is not up to yet.

In a lot of cases we modify zenta-tools or the toolchain to remediate some build problems, or enhance the build.
It often brings in incompatible changes, which means that winterboot and cdd should be modified to still build.
Getting all the changes through the chain of zenta-tools, toolchain, cdd and winterboot takes a lot of time, and most of the time is spent on waiting the builds and seeing them fail.

To build the business projects with the new toolchain, shippable.yml should be modified. Which means that the toolchain should look up a list of projects wanting to be triggered, create a new branch, modify shippable in that branch, and build that branch for each of the projects.

In all of the toolchain-based projects you can issue the tools/testenv command, which puts you in the build container. In the container you initiate the build with the Script command. In zenta it is make testenv and bin/script in zenta-tools some combination thereof.
About the recuiter
Member since May 20, 2018
Mohammed Jassim
from Minnesota, United States

---

---

Looking for help? Checkout our video tutorial
How to search and apply for jobs

How to apply? Do you have more questions about the Job?
See frequently asked questions