Pipeline Service Technical¶
This article is part of the work in progress Pipeline V3. Documentation is written as it is being developed. It does not apply to the current build pipeline.
This document describes the contents of the *.meta folders in the workspace directory.
Status tracking¶
The pipeline service keeps track of the latest known CRC32 and timestamp information for every file encountered in the database and intermediate build directories. This information is stored in the *.status files. This status metadata is purely informational (in the sense that it will recover when erased or poorly tampered with inbetween builds, and fail the build when the same occurs during a build), and is calculated by the master service for input files that need to be used by a slave, and by the slave service for files that are written immediately after writing. When a slave service receives a task, it will automatically request the master service to ensure the status of all directories and files it depends on are up to date. It will wait until the master service has confirmed before continuing. When a slave service encounters a file it depends on where the timestamps mismatch the timestamps in the meta status, the build fails completely or partially due to data inconsistency. In case this happens either someone changed the file during build or there is a bug. When the timestamps are consistent, the actual CRC32 is used when comparing if a file depending on this file previously used the same file as depencency. You can force a reload of all the *.status files by calling the updateDatabaseStatus command on the master service, or by pressing Status Update All, but this is not necessary and takes a long time. It is recommended to run the master service on the machine that has a direct connection to the hard drive in order to speed up the initial build.
The implementation of the status tracking is done, fully tested, and foolproof.
Dependencies tracking¶
Files can have compile-time dependencies [? and runtime dependencies. The runtime dependencies are reported by build tools merely for informational purposes, and can be used by the user to verify that all assets are present in the build. Runtime dependencies are not required to have been built yet or exist yet at build time. ?] All compile dependencies must be reported by build tools to the slave service. [? These are logged. A file can depend on specific files or on an entire directory, although the latter is reported by configuration. The build tool only needs to report the actually used dependencies faithfully. The slave service gets the CRC32 after receiving this information, from the status file where it must always also pass the timestamp test to ensure the file did not change during the build, and safely assumes this is the actual CRC32 of the file that was used to build relevant file. ?] [? How dependencies on directories work, when new files are added, files disappear, files change. How this is reported from the service to the plugin. A file status tree of all files in dependent directories and all dependent files is stored per build project. The last successful build per project also needs to be stored for some reason. The status tree per project is only stored when the build is successful, the redundant adds, removes and changes caused by an outdated full status tree when the build is not successful does not have a functional impact on the build process, but it will make more time as it will assume more changes than there are. This may be flawed when files are reverted to previous versions. May be needed to assume all files changed when building after an incomplete failed build. ?] [? It is necessary that output files that no longer have their input files are appropriately handled (this cannot be detected by the previous pipeline, and is usually neither detected by conventional build pipelines that do not fully track dependencies). ?]
The implementation of the dependencies tracking is work in progress.
Error tracking¶
[? The service keeps track of every error message about a file that is passed from a build tool. A warning or error flag will be kept on the file until the file is more recent than the latest error message. Depending on the build tools it will either be skipped with warnings or the build will completely fail with errors. Any files depending on a bad file will be marked as such as well. ?]
The implementation of the error tracking has not formally started yet.
Plugin interface¶
Build tools are attached to the pipeline service trough the plugin interface. A plugin can represent multiple build processes and is responsible for interpreting configuration values and commanding multiple build tools. The interface between the service and plugins must be kept to the minimal level, in such way that a lua script can have exactly the same interface as a C++ plugin to the service, and that only minimal thought is required for implementing plugins. It must be possible for the service to decide which files need to be rebuilt, both input files that output many files, as well as input directories that output one file, and anything inbetween.
The implementation of the plugin interface is work in progress.
Projects, processes and plugins¶
[? The dependencies between processes withing a project and between projects are configured in the workspace sheets ?] The master service sorts trough the processes and dispatches to a slave whenever a slave is free and finds a process has no pending dependencies. [? What plugins are in relation to processes. ?]
The implementation of the build queue is largely done, so far fully tested, and so far foolproof.
Workspace sheets¶
[? The verification process of workspace sheets and their dfn between services is incomplete. There's a bug on first launch that the checksums don't match. ?] The build pipeline is controlled by georges sheets. [? How to use macro parameters. How to refer to configuration parameters. How to refer to directories. ?]
Misc¶
File paths are communicated between services using their configuration name to enable filepaths being different across physical machines. [? Describe how the internal formatting works. ?] [? Something similar used for the georges sheets. ?]
Files may depend on files that are built within the same process. [? I think the .zonew will cause a domino effect the way it is implemented. ?]
Directory paths in the project configuration sheets that contain subdirectories are never evaluated recursively into the subdirectories by the pipeline service itself. If this is really necessary, it must be handled sanely by the plugin.
When a plugin encounters a file that can only be built by dependencies that no longer exist, it's error state will be set to Pending Deletion.
Only the slave services write to error meta files. Only the master service reads from error meta files. It is recommended that the master runs on the machine that has direct access to the drive. You may run the master service without builtin slave service.
[? It is also marked in the error meta file whenever a file is successfully used to build another file. A file that has a change more recent than the last error information is marked as having an Unknown state. ?]
Process build overview¶
The slave asks the plugin which paths it depends on, paths that are directories will always only refer to the files directly under the path and do not recurse into deeper directories. [? How the non-recursiveness shall be handled for filetree hungry build tools such as the leveldesign sheets, likely bypassing the service dependency change information as the leveldesign build tools do all that is necessary. ?] The slave requests the master to update the status for these paths. The master puts this request in the status update queue. The master reports back to the slave when the checksums are up to date. If there is a process status snapshot, the slave will compare the latest status with the snapshot of the last successful build, and if the last build was successful and there is no change the process does not need to run further (if the last build was unsuccessful but there is no change between the last successful build state and the latest state, it implicitly means something changed between the last build and the latest files, and thus we will depend on the per file dependencies [? also should store at begin of build the changes to the last successful to know which file was reverted ?]). ... ... ... The slave tells the plugin which files in the dependencies were added, removed and modified since the last successful build. The plugin asks the slave if a certain input or output file needs to be rebuilt, this for everything that should be built normally. The slave compares the dependencies from last build with the reported information. [? It may also be possible to estimate solely on the listing of added, modified and removed, if the last build was successful, what needs to be built, depending on the the dependency organization of the specific process ?] ... ... ... [? When build successful... ?] [- Store status snapshot of all used dependencies. - Probably not necessary anymore. -] The slave stores the last successful build time of the process.
Slave: Build Ready State¶
When the master starts the build, it will request all the slaves to enter the build ready state. In case a slave is doing something else, it will not enter this state. The master will keep asking until the slave has switched. Only when in build ready state, the master will send build tasks to the service.