Invalid Manifests Cache on CI/CD about tuist HOT 9 OPEN

After demangle:

Loading and constructing the graph
It might take a while if the cache is empty
The 'swift' command exited with error code 255 and message:
JIT session error: Symbols not found: [ ProjectDescription.FileHeaderTemplate.companyCopyright ]
Failed to materialize symbols: { 
(main, { 
ProjectDescription.Workspace.init(name: String, projects: [Path], schemes: [Scheme], fileHeaderTemplate: FileHeaderTemplate?, additionalFiles: [FileElement], generationOptions: [GenerationOption]), 
symbolic Say ProjectDescription.TestingOptions, 
swift_allocate_value_buffer, 
ProjectDescription.TestingOptions.SetAlgebra, 
swift.Array.endMutation(), 
main, 
symbolic ProjectDescription.Workspace.GenerationOptions.AutogeneratedSchemes.CodeCoverageMode ProjectDescription.TestingOptions.testingOptions SchemeLanguage ProjectDescription.ScreenCaptureFormat, 
Workspace.workspace ProjectDescription, 
got.ProjectDescription.FileHeaderTemplate, 
swift.Array.symbolic TestingOptions, 
symbolic ProjectDescription.FileHeaderTemplate, 
got.ProjectDescription.Workspace.GenerationOptions.AutogeneratedSchemes.CodeCoverageMode, 
swift.Array.allocateUninitialized Array Slice ProjectDescription.Scheme, 
ProjectDescription.TestingOptions.SetAlgebra, 
swift.Array.allocateUninitialized Array Slice ProjectDescription.FileElement, 
swift_FORCE_LOAD_$_swiftDispatch_$_Workspace, 
swift.Array.allocateUninitialized Array Slice ProjectDescription.TestingOptions, 
ProjectDescription.FileHeaderTemplate.optional Metadata, 
swift_FORCE_LOAD_$_swiftDarwin_$_Workspace, 
swift_FORCE_LOAD_$_swiftFoundation_$_Workspace, 
swift.instantiateConcreteTypeFromMangledNameAbstract, 
swift_FORCE_LOAD_$_swiftIOKit_$_Workspace, 
swift_FORCE_LOAD_$_swiftCoreFoundation_$_Workspace, 
swift.instantiateConcreteTypeFromMangledName, 
swift_FORCE_LOAD_$_swiftXPC_$_Workspace, 
swift_FORCE_LOAD_$_swiftObjectiveC_$_Workspace, 
ProjectDescription.Workspace.init(name: String, projects: [Path], schemes: [Scheme], fileHeaderTemplate: FileHeaderTemplate?, additionalFiles: [FileElement], generationOptions: [GenerationOption]), 
symbolic ProjectDescription.Version, 
swift.finalizeUninitializedArray, 
swift.Array.symbolic TestingOptions, 
got.ProjectDescription.Version, 
ProjectDescription.Workspace.GenerationOptions.AutogeneratedSchemes.CodeCoverageMode.TestingOptions SchemeLanguage optional testRegion ScreenCaptureFormat metadata version, 
got.ProjectDescription.TestingOptions, 
ProjectDescription.Version.optional Metadata, 
ProjectDescription.Workspace.init(name: String, projects: [Path], schemes: [Scheme], fileHeaderTemplate: FileHeaderTemplate?, additionalFiles: [FileElement], generationOptions: [GenerationOption]), 
ProjectDescription.TestingOptions.metadata, 
swift.project_value_buffer, 
ProjectDescription.Version.optional Associated Type metadata, 
got.ProjectDescription.SchemeLanguage, 
got.ProjectDescription.ScreenCaptureFormat 
}) 
}

Note:
ProjectDescription.FileHeaderTemplate.companyCopyright is my static-factory property helper

from tuist.

@Ernest0-Production, we'd need to know what changed between the run that was cached and the run that failed. We have some environment information like the Swift version or the content in your files, but based on your description, it seems that there's something that we are missing and that we'd need to incorporate into your logic.
I'd refrain from changing the cache directory because that doesn't mitigate the issue, it only brings it closer to the developer file-system-wise.

For anyone running into this, the following command should get you out of that state:

tuist clean
tuist clean manifests projectDescriptionHelpers # Alternatively

from tuist.

@pepicrft What is known is that the Swift version (5.10) and the Xcode version (15.3) have not changed.

It is known that everything worked on one machine, but not on the other.

I can assume that the non-working machine had a rather old, backwards-incompatible cache. And yes, deleting the manifest cache solved the problem. (as workaround)

Since the definition of the machine for assembly is determined automatically, it is difficult for me to guess and determine what changes led to such an error, but what is known is that this has already happened several times (but quite rarely)

One of the reasons to make the cache closer to the project directory is that otherwise build manifests of all pull requests for all time will accumulate on the CI machine.
This is just as bad as having shared xcode DerivedData.

Another advantage of a local cache directory was the ability to control when it needed to be staged between job runs and when it didn't.
Currently, the tuist clean <entities> command clears the entire cache of all projects, which is not convenient.

from tuist.

@Ernest0-Production, thanks for the clarification. I agree that we should support caching across environments to speed up generation times. However, we did not implement the logic with that use case in mind, and therefore I think we are missing some scenarios when calculating the hash to look up binaries in the cache. If you are interested, here is the place where we calculate the hash. I think we should extend the logic to incorporate the variables that cause the compiler to output a different artifact (e.g. Xcode version, macOS version, system architeture)

from tuist.

@pepicrft Thank you, I looked at this code and it seems to me that the potential problem lies in ProjectDescriptionHelpersHasher.prefixHash

public func prefixHash(helpersDirectory: AbsolutePath) -> String {
    let pathString = helpersDirectory.pathString
    let index = pathString.index(pathString.startIndex, offsetBy: 7)
    return String(helpersDirectory.pathString.md5[..<index])
}

I'm not sure why offsetBy: 7 is needed here, but it looks like black magic.

I can assume that because there might have been many directories for my project on the CI (stored under each pull request), a rare situation occurred when two MD5 strings turned out to be equal because they were truncated to the given index in this function. (i.e., it increases the likelihood of a collision)

from tuist.

@pepicrft By the way, I remember in Tuist 3 it was possible to specify the cache directory via Config.swift.

Why did team* decide to remove this?

from tuist.

I can assume that because there might have been many directories for my project on the CI (stored under each pull request), a rare situation occurred when two MD5 strings turned out to be equal because they were truncated to the given index in this function. (i.e., it increases the likelihood of a collision)

This could indeed be the issue. We can iterate on that logic to prevent collisions from happening. Would you like to take a stab at it?

Why did they decide to remove this?

We noted that users used that as an escape hatch to reduce the likelihood of conflicts like this one instead of filing issues so that we could solve those bugs. If we design the cache well, users shouldn't encounter scenarios where they need to customize the caching directory.

from tuist.

@pepicrft

This could indeed be the issue. We can iterate on that logic to prevent collisions from happening. Would you like to take a stab at it?

I write simple abstract test in playground to detect collision of hashes that computed by current login:

typealias Path = String
typealias Hash = String
var hashesPaths: [Hash: Path] = [:]

while true {
    let pathString = generatePath()
    let hash = computeHash(for: pathString)

    if let collisionPath = hashesPaths[hash] {
        print("Detect collision", collisionPath, pathString)
        break
    } else {
        hashesPaths[hash] = pathString
    }
}

// MARK: - Helpers

// Mimics the path in which the repository is cloned on a CI machine
var prNumber = 0
func generatePath() -> Path {
    defer { prNumber += 1 }
    return "Users/jenkins/workspace/PR-\(prNumber)"
}

// Duplicates logic from ProjectDescriptionHelpersHasher.prefixHash
func computeHash(for pathString: Path) -> Hash {
    let index = pathString.index(pathString.startIndex, offsetBy: 7)

    return String(pathString.md5[..<index])
}

And I receive output:

Detect collision Users/jenkins/workspace/PR-2962 Users/jenkins/workspace/PR-5162

The hash for these two paths is indeed the same:

print(
    computeHash(for: "Users/jenkins/workspace/PR-2962"), // 4be35b3
    computeHash(for: "Users/jenkins/workspace/PR-5162") // 4be35b3
)

That is, the reason is clearly still a cache collision of ProjectDescriptionHelpers directory path

We noted that users used that as an escape hatch to reduce the likelihood of conflicts like this one instead of filing issues so that we could solve those bugs. If we design the cache well, users shouldn't encounter scenarios where they need to customize the caching directory.

Thanks for the explanation! But it still seems that even with hashing working correctly, for deploying an environment on CI it is important to have control over all the artifacts and side effects created during the build process, so that after completion it is cleared up this environment without creating collisions/data races/race conditions.

from tuist.

Oh suddenly I was also able to reproduce this error when I uninstalled and reinstalled homebrew on m1.

from tuist.