Ryzom

h1. Pipeline Service Technical

> +This article is part of the work in progress Pipeline V3. Documentation is written as it is being developed. It does *not* apply to the current build pipeline.+

This document describes the contents of the @*.meta@ folders in the @workspace@ directory.

h2. Status tracking

The pipeline service keeps track of the latest known CRC32 and timestamp information for every file encountered in the database and intermediate build directories. This information is stored in the @*.status@ files. This status metadata is purely informational (in the sense that it will recover when erased or poorly tampered with inbetween builds, and fail the build when the same occurs during a build), and is calculated by the master service for input files that need to be used by a slave, and by the slave service for files that are written immediately after writing. When a slave service receives a task, it will automatically request the master service to ensure the status of all directories and files it depends on are up to date. It will wait until the master service has confirmed before continuing. When a slave service encounters a file it depends on where the timestamps mismatch the timestamps in the meta status, the build fails completely or partially due to data inconsistency. In case this happens either someone changed the file during build or there is a bug. When the timestamps are consistent, the actual CRC32 is used when comparing if a file depending on this file previously used the same file as depencency. You can force a reload of all the @*.status@ files by calling the @updateDatabaseStatus@ command on the master service, or by pressing @Status Update All@, but this is not necessary and takes a long time. It is recommended to run the master service on the machine that has a direct connection to the hard drive in order to speed up the initial build.

The implementation of the status tracking is done, fully tested, and foolproof.

h2. Dependencies tracking

Files can have compile-time dependencies [? and runtime dependencies. The runtime dependencies are reported by build tools merely for informational purposes, and can be used by the user to verify that all assets are present in the build. Runtime dependencies are not required to have been built yet or exist yet at build time. ?] All compile dependencies must be reported by build tools to the slave service. [? These are logged. A file can depend on specific files or on an entire directory, although the latter is reported by configuration. The build tool only needs to report the actually used dependencies faithfully. The slave service gets the CRC32 after receiving this information, from the status file where it must always also pass the timestamp test to ensure the file did not change during the build, and safely assumes this is the actual CRC32 of the file that was used to build relevant file. ?] [? How dependencies on directories work, when new files are added, files disappear, files change. How this is reported from the service to the plugin. A file status tree of all files in dependent directories and all dependent files is stored per build project. The last successful build per project also needs to be stored for some reason. The status tree per project is only stored when the build is successful, the redundant adds, removes and changes caused by an outdated full status tree when the build is not successful does not have a functional impact on the build process, but it will make more time as it will assume more changes than there are. This may be flawed when files are reverted to previous versions. May be needed to assume all files changed when building after an incomplete failed build. ?]

The implementation of the dependencies tracking is work in progress.

h2. Error tracking

[? The service keeps track of every error message about a file that is passed from a build tool. A warning or error flag will be kept on the file until the file is more recent than the latest error message. Depending on the build tools it will either be skipped with warnings or the build will completely fail with errors. Any files depending on a bad file will be marked as such as well. ?]

The implementation of the error tracking has not formally started yet.

h2. Plugin interface

Build tools are attached to the pipeline service trough the plugin interface. A plugin can represent multiple build processes and is responsible for interpreting configuration values and commanding multiple build tools. The interface between the service and plugins must be kept to the minimal level, in such way that a lua script can have exactly the same interface as a C++ plugin to the service, and that only minimal thought is required for implementing plugins. It must be possible for the service to decide which files need to be rebuilt, both input files that output many files, as well as input directories that output one file, and anything inbetween.

The implementation of the plugin interface is work in progress.