Q&A for New Contributors

[saehejkang]

I noticed there is not a lot of documentation on tips/tricks/examples to help ease others into the codebase. As a new contributor, there are many things that I am learning about the codebase as I tackle issues, which is awesome! However, I feel there should be maybe something somewhere that helps answer questions, I am sure many of us have.

I had some development questions while working with the APIs, so I’ll go ahead and start asking them.

• How do I setup the Container API in Swift to actually test code changes? Also, maybe a section on maybe how to setup to use the Container API in new Swift Project?
• Can I use Postman/Insomnia to test GRPC requests to the container API? What would the URL be for requests (if it even exists)?
• Is there a .proto file to reference from? I see there is one being reference from the container-builder-shim repo and it is being used in the Protobuf.makefile?

Feel free to use this thread to ask any questions 😄. Hopefully, in the future, I can create an FAQ document to address common questions and provide more guidance for contributors.

[jglogan]

@saehejkang Let's start with the questions

How do I setup the Container API in Swift to actually test code changes? Also, maybe a section on maybe how to setup to use the Container API in new Swift Project?

Could you provide a concrete scenario for this - a specific example of the code changes you want to test?

The "Container API" or "Container SDK" today would be primarily what you would access via the ContainerClient library. It's an API layer over the XPC calls that a client makes to container-apiserver.

@Mcrich23 did some the work to extract the swift-argument-parser commands into a library separate from the container executable, so that you can now perform CLI operations directly without the need to exec container.

We definitely need developer documentation (and ideally a couple tutorial examples). Part of the problem is that IMO the API isn't quite "SDK-quality" yet so I'm hesitant to write (or generate) reams of documentation for things. "Something small", as you suggest, makes a ton of sense though.

[Mcrich23]

Agreed. I'm just waiting for #717 to get merged and then I will write the plugin docs!

[saehejkang]

Could you provide a concrete scenario for this - a specific example of the code changes you want to test?

I was looking into this issue, #298, and was curious how to reproduce it. My initial thought was to create a new Swift project and in the Sources folder, copy over the ContainerClient package? Then in my main function I can call functions like list and create from the ClientContainer file. Would that be all I need to do as long as the container system and container-apiserver is up and running?

@Mcrich23 did some the work to extract the swift-argument-parser commands into a library separate from the container executable, so that you can now perform CLI operations directly without the need to exec container.

Seems awesome and I will be sure to look out for the plugin docs!

"Something small", as you suggest, makes a ton of sense though.

Hopefully whatever I learn from this thread will allow me to add some docs/examples to help others in the future.

[jglogan]

Can I use Postman/Insomnia to test GRPC requests to the container API? What would the URL be for requests (if it even exists)?

Is there a .proto file to reference from? I see there is one being reference from the container-builder-shim repo and it is being used in the Protobuf.makefile?

Currently there is no gRPC version of what ContainerClient offers. The Builder.grpc.swift and Builder.pb.swift files in the ContainerBuilder directory are for the "Builder API" between ContainerClient and container-builder-shim (see the README for that project). From the developer point of view, this is an implementation detail. When we can bring resources back to it, our intention is to replace all of that with a native Swift builder implementation, and the Golang shim+buildkit stuff would be deprecated and eventually go away.

[saehejkang]

Ahhh, I see the README just got updated and shows a diagram of how things work. Just to be clear, when the container builder start command is ran, that creates the connection to the Builder Container? BuildKit is a separate project correct? And that container is using the the BuildKit API (gRPC) to make requests? Lots to wrap my head around 😅

[jglogan]

The two obvious approaches for providing an remotable API that could support multiple language bindings seem to be:

• Create a gRPC API definition for ContainerClient, generate Swift bindings, and integrate the existing ContainerClient against the generated server code so that subsequently, the gRPC API becomes the spec for the "container API". Swift apps could still link the API directly, but we could also build a server for clients implemented in other languages.
• Build a server that presents a Docker API over ContainerClient (the socktainer project is an early version of this). The huge benefit of this is ecosystem support; the difficulty is the level of effort required to build such a thing to the point where it's generally usable, and the level of effort required to maintain and support it.

[saehejkang]

Create a gRPC API definition for ContainerClient, generate Swift bindings, and integrate the existing ContainerClient against the generated server code

Again, would this be the approach the team would take when time allows for it? I took a look at the socktainer project and it looks like they are doing their best to keep up with changes/updates.

[jglogan]

Another thing needing documentation (@Mcrich23 and I discussed it a while back) is the plugin and extension mechanisms that container has. As with the API, it's usable but not quite "SDK quality" and as of now woefully underdocumented.

[Mcrich23]

But we are super close to it being that quality!

[saehejkang]

Is there a link to the discussion? Curious more about what this will become in the future.

[jglogan]

@saehejkang

• Add Compose #239 (comment) (most of the discussion starts here)
• [Docs]: Proper Plugin Development Documentation #630
• [Request]: SDK: Extension developer's guide #462

[saehejkang]

Moving this here from #843

I had some questions about the current workflow for the container build command as it seems different than other commands. My main question is how exactly config values that are passed in the command are actually read/used? For this discussion, let's follow the --no-cache flag.

BuildCommand.swift

Start out by defining the noCache value as a flag and defaulting it to false. It is then passed to Builder.BuildConfig, as that creates the config that is needed to be passed to the build function. This all makes sense to me.

Builder.swift

The config is read and then converted to headers like so below (other fields removed for readability).

extension CallOptions {
    public init(_ config: Builder.BuildConfig) throws {
        var headers: [(String, String)] = [
            ("build-id", config.buildID),
            ("context", URL(filePath: config.contextDir).path(percentEncoded: false)),
            ("dockerfile", config.dockerfile.base64EncodedString()),
            ("progress", config.terminal != nil ? "tty" : "plain"),
            ("target", config.target),
        ]
        if config.noCache {
            headers.append(("no-cache", ""))
        self.init(
            customMetadata: HPACKHeaders(headers)
        )
    }
}

Then this line is executed with the updated options.

let respStream = self.clientAsync.performBuild(reqStream, callOptions: try CallOptions(config))

Builder.gprc

Regarding the function below, I believe this is sort of GRPC request and that the callOptions value contains the --no-cache flag. This also seems like the client side call, but where is the server side logic for this request? I do not see anything in the repo to actually use the callOptions and say "hey the flag it set to true, we want to use the cache" -> insert logic to use the cache. I am also sure when adding new fields to the callOptions, the request will fail as that new field is not expected in the request body?

  public func performBuild<RequestStream>(
    _ requests: RequestStream,
    callOptions: CallOptions? = nil
  ) -> GRPCAsyncResponseStream<Com_Apple_Container_Build_V1_ServerStream> where RequestStream: AsyncSequence & Sendable, RequestStream.Element == Com_Apple_Container_Build_V1_ClientStream {
    return self.performAsyncBidirectionalStreamingCall(
      path: Com_Apple_Container_Build_V1_BuilderClientMetadata.Methods.performBuild.path,
      requests: requests,
      callOptions: callOptions ?? self.defaultCallOptions,
      interceptors: self.interceptors?.makePerformBuildInterceptors() ?? []
    )
  }

Maybe it all has something to do with this build method?

public static let performBuild = GRPCMethodDescriptor(
   name: "PerformBuild",
   path: "/com.apple.container.build.v1.Builder/PerformBuild",
   type: GRPCCallType.bidirectionalStreaming
)

---

Hopefully my thought process makes sense and you can see where I get lost.

[jglogan]

@saehejkang Have you looked at https://github.com/apple/container-builder-shim?

[jglogan]

Also, give this a try:

container exec -it buildkit ps -ef