Question: How to: `BUILD` COMMAND OPTIONS

Answer:
- `--target` - Target container image (name or ID). It's an alternative way to provide the target information. The standard way to provide the target information is by putting as the last value in the `build` command CLI call.
- `--pull` - Try pulling target if it's not available locally (default: false).
- `--docker-config-path` - Set the docker config path used to fetch registry credentials (used with the `--pull` flag).
- `--registry-account` - Account to be used when pulling images from private registries (used with the `--pull` flag).
- `--registry-secret` - Account secret to be used when pulling images from private registries (used with the `--pull` and `--registry-account` flags).
- `--show-plogs` - Show image pull logs (default: false).

- `--compose-file` - Load container info from selected compose file
- `--target-compose-svc` - Target service from compose file
- `--target-compose-svc-image` - Override the container image name and/or tag when targeting a compose service using the target-compose-svc parameter (format: tag_name or image_name:tag_name)
- `--target-compose-svc-no-ports` - Do not publish ports for target service from compose file
- `--dep-exclude-compose-svc-all` - Do not start any compose services as target dependencies
- `--dep-include-compose-svc` - Include specific compose service as a target dependency (only selected services will be started)
- `--dep-exclude-compose-svc` - Exclude specific service from the compose services that will be started as target dependencies
- `--dep-include-compose-svc-deps` - Include all dependencies for the selected compose service (excluding the service itself) as target dependencies
- `--dep-include-target-compose-svc-deps` - Include all dependencies for the target compose service (excluding the service itself) as target dependencies. This is a shortcut flag to avoid repeating the service name (it's a pretty long flag name though :-))
- `--compose-svc-start-wait` - Number of seconds to wait before starting each compose service
- `--compose-net` - Attach target to the selected compose network(s) otherwise all networks will be attached
- `--compose-env-nohost` - Don't include the env vars from the host to compose
- `--compose-env-file` - Load compose env vars from file (host env vars override the values loaded from this file)
- `--compose-workdir` - Set custom work directory for compose
- `--compose-project-name` - Use custom project name for compose
- `--container-probe-compose-svc` - Container test/probe service from compose file
- `--prestart-compose-svc` - placeholder for now
- `--poststart-compose-svc` - placeholder for now
- `--http-probe` - Enables/disables HTTP probing (ENABLED by default; you have to disable the probe if you don't need it by setting the flag to `false`: `--http-probe=false`)
- `--http-probe-off` - Alternative way to disable HTTP probing
- `--http-probe-cmd` - Additional HTTP probe command [can use this flag multiple times]
- `--http-probe-cmd-file` - File with user defined HTTP probe commands
- `--http-probe-start-wait` - Number of seconds to wait before starting HTTP probing
- `--http-probe-retry-count` - Number of retries for each HTTP probe (default value: 5)
- `--http-probe-retry-wait` - Number of seconds to wait before retrying HTTP probe (doubles when target is not ready; default value: 8)
- `--http-probe-ports` - Explicit list of ports to probe (in the order you want them to be probed; excluded ports are not probed!)
- `--http-probe-full` - Do full HTTP probe for all selected ports (if false, finish after first successful scan; default value: false)
- `--http-probe-exit-on-failure` - Exit when all HTTP probe commands fail (default value: true)
- `--http-probe-crawl` - Enable crawling for the default HTTP probe command (default value: true)
- `--http-crawl-max-depth` - Max depth to use for the HTTP probe crawler (default value: 3)
- `--http-crawl-max-page-count` - Max number of pages to visit for the HTTP probe crawler (default value: 1000)
- `--http-crawl-concurrency` - Number of concurrent workers when crawling an HTTP target (default value: 10)
- `--http-max-concurrent-crawlers` - Number of concurrent crawlers in the HTTP probe (default value: 1)
- `--http-probe-apispec` - Run HTTP probes for API spec where the value represents the target path where the spec is available (supports Swagger 2.x and OpenAPI 3.x) [can use this flag multiple times]
- `--http-probe-apispec-file` - Run HTTP probes for API spec from file (supports Swagger 2.x and OpenAPI 3.x) [can use this flag multiple times]
- `--http-probe-exec` - App to execute when running HTTP probes. [can use this flag multiple times]
- `--http-probe-exec-file` - Apps to execute when running HTTP probes loaded from file.
- `--publish-port` - Map container port to host port analyzing image at runtime to make it easier to integrate external tests (format => port | hostPort:containerPort | hostIP:hostPort:containerPort | hostIP::containerPort )[can use this flag multiple


Question: How to: `DEBUG` COMMAND OPTIONS

Answer:
- `--runtime` - Runtime environment type (values: `docker`, `k8s`; defaults to `docker`)
- `--debug-image` - Debug image to use for the debug side-car container (default value for this flag is `busybox`).
- `--list-debug-images` - List possible debug images to use for the debug side-car container (for the `--debug-image` flag). This list is a ready to use set of debug images. You can use other images too.
- `--target` - Target container name or ID (this can also be provided as the last param in the command line invocation of the `debug` command). Note that the target container must be running. You can use the `docker run` command to start the target container (or the kubernetes equivalent).
- `--namespace` - Namespace to target [k8s runtime] (defaults to `default`)
- `--pod` - Pod to target [k8s runtime]
- `--cmd` - (Optional) custom CMD to use for the debug side-car container (alternatively pass custom CMD params after '--').
- `--entrypoint` - (Optional) custom ENTRYPOINT to use for the debug side-car container.
- `--terminal` - Attach interactive terminal to the debug container (default: true). When the interactive terminal is not enabled the debug container output will be printed out to the screen when the `debug` command exits.
- `--kubeconfig` - Kubeconfig file location [k8s runtime]
- `--workdir` - Custom WORKDIR to use for the debug side-car container.
- `--env` - Environment variable to add to the debug side-car container.
- `--run-as-target-shell` - Attach interactive terminal to the debug container and run shell as if it's running in the target container environment.
- `--list-sessions` - List all debug sessions for the selected target (pod and optionally selected container for k8s or container for other runtimes).
- `--show-session-logs` - Show logs for the selected debug session (using namespace, pod, target container or debug session container name for k8s or debug session container name for other runtimes).
- `--session` - Debug session container name (used for debug sessoin actions).
- `--connect-session` - Connect to existing debug session.
- `--list-namespaces` - List names for available namespaces (use this flag by itself) [k8s runtime].
- `--list-pods` - List names for running pods in the selected namespace (use this flag by itself) [k8s runtime].
- `--list-debuggable-containers` - List container names for active containers that can be debugged (use this flag by itself).
- `--list-debug-images` - List possible debug images to use for the debug side-car container (use this flag by itself).
- `--help` show help (default: false)

See the "Debugging Using the `debug` Command" section for more information about this command.


Question: How to: `RUN` COMMAND OPTIONS

Answer:
Run one or more containers

USAGE: `slim [GLOBAL FLAGS] run [FLAGS] [IMAGE]`

Flags:

- `--target` - Target container image to run. Same as specifying the target container image as the last value for the command. Used mostly for the interactive prompt mode where you need to select flag names.
- `--pull` - Pull the target image before trying to run it.
- `--docker-config-path` - Docker config path (used to fetch registry credentials).
- `--registry-account` - Target registry account used when pulling images from private registries.
- `--registry-secret` - Target registry secret used when pulling images from private registries.
- `--show-plogs` - Show image pull logs.
- `--entrypoint` - Override ENTRYPOINT running the target image.
- `--cmd` - Override CMD running the target image.
- `--live-logs` - Show live logs for the container (can't use with --terminal).
- `--terminal` - Attach interactive terminal to the container.
- `--publish` - Map container port to host port (format => port | hostPort:containerPort | hostIP:hostPort:containerPort | hostIP::containerPort ).
- `--rm` - Remove the container when it exits.
- `--detach` - Start the container and do not wait for it to exit.


Question: How to: `MERGE` COMMAND OPTIONS

Answer:
Merge two container images. Optimized to merge minified images.

Flags:

- `--image` - Image to merge. Flag instance position determines the merge order. The command supports two instances of this flag.

- `--use-last-image-metadata` - Use only the last image metadata for the merged image.

- `--tag` - Custom tags for the output image (multiple instances).


Question: How to: `REGISTRY` COMMAND OPTIONS

Answer:
For the operations that require authentication you can reuse the registry credentials from Docker (do `docker login` first and then use the `--use-docker-credentials` flag with the `registry` command) or you can specify the auth info using the `--account` and `--secret` flags).

Current sub-commands: `pull`, `push`, `image-index-create`, `server`.

There's also a placeholder for `copy`, but it doesn't do anything yet. Great opportunity to contribute ;-)

Shared Command Level Flags:

- `--use-docker-credentials` - Use the registry credentials from the default Docker config file.
- `--account` - Registry credentials account.
- `--secret` - Registry credentials secret.

#### `PULL` SUBCOMMAND OPTIONS

USAGE: `slim [GLOBAL FLAGS] registry [SHARED FLAGS] pull [FLAGS] [IMAGE]`

Flags:

- `--target value` - Target container image (name or ID) [$DSLIM_TARGET]
- `--save-to-docker`- Save pulled image to docker (default: true) [$DSLIM_REG_PULL_SAVE_TO_DOCKER]

#### `PUSH` SUBCOMMAND OPTIONS

USAGE: `slim [GLOBAL FLAGS] registry [SHARED FLAGS] push [FLAGS] [IMAGE]`

Flags:

- `--docker` -- Push local docker image.
- `--tar` -- Push image from a local tar file.
- `--as` -- Tag the selected image with the specified name before pushing.

Note that `slim registry push LOCAL_DOCKER_IMAGE_NAME` is a shortcut for `slim registry push --docker LOCAL_DOCKER_IMAGE_NAME`.

Normally you have to explicitly tag the target image to have a name that's appropriate for the destination registry. The `--as` flag is a convenient way to tag the image while you are pushing it. Here's an example pushing a local Docker `nginx` image to a local registry: `slim registry push --docker nginx --as localhost:5000/nginx`

You can create a local registry using the `server` subcommand. See the `server` sub-command section below for more details.

#### `COPY` SUBCOMMAND OPTIONS

USAGE: `slim registry copy [SRC_IMAGE] [DST_IMAGE]`

NOTE: Just a placeholder for now (TBD)

#### `IMAGE-INDEX-CREATE` SUBCOMMAND OPTIONS

USAGE: `slim registry image-index-create --image-index-name [MULTI-ARCH_IMAGE_TAG] --image-name [IMAGE_ONE] --image-name [IMAGE_TWO]`

Flags:

- `--image-index-name` - Image index name to use.
- `--image-name` - Target image name to include in image index.
- `--as-manifest-list` - Create image index with the manifest list media type instead of the default OCI image index type.
- `--dump-raw-manifest` - Dump raw manifest for the created image index.
- `--insecure-refs` - Allow the referenced images from insecure registry connections.

#### `SERVER` SUBCOMMAND OPTIONS

Starts a server which implements the [OCI API spec](https://github.com/opencontainers/distribution-spec/blob/v1.0.1/spec.md) on port 5000 by default.

USAGE: `slim [GLOBAL FLAGS] registry server [FLAGS]`

Flags:

- `--address` - Registry server address to listen on (default: `0.0.0.0`)
- `--port` - Registry server port (default: 5000)
- `--https` - Use HTTPS.
- `--cert-path` - Cert path for use with HTTPS (for use when not using autocert).
- `--key-path` - Key path for use with HTTPS (for use when not using autocert).
- `--domain` - Domain to use for registry server (to get certs). Only works if the registry is internet accessible (see `autocert` Go docs for more details).
- `--referrers-api` - Enables the [referrers API endpoint](https://github.com/opencontainers/distribution-spec/blob/v1.1.0-rc3/spec.md#enabling-the-referrers-api) (OCI 1.1+). Enabled by default (set to `false` to disable).


Question: How to: `VULNERABILITY` COMMAND OPTIONS

Answer:
USAGE: `slim [GLOBAL FLAGS] vulnerability [SHARED FLAGS] [SUBCOMMAND] [FLAGS]`

Current sub-commands:

* `epss` - Gets EPPS information for the target vulnerabilities or based on the selected vulnerability filters.

Shared Command Level Flags:

- `--cve` - Target vulnerability CVE ID (can specify multiple times to target multiple vulnerabilities).

#### `EPSS` SUBCOMMAND OPTIONS

USAGE: `slim [GLOBAL FLAGS] vulnerability [SHARED FLAGS] epss [FLAGS]`

Flags:

- `--op` - EPSS operation (`lookup` | `list`).
- `--date` - Date for the EPSS information (YYYY-MM-DD format). Works with the `lookup` and `list` operations.
- `--with-history` - Return EPSS results with historical data. Works with the `lookup` and `list` operations.
- `--limit` - Limit the number of returned records.
- `offset` - Offset where to start returning records.
- `filter-cve-id-pattern` - 'CVE ID pattern' ESPP list operation filter.
- `filter-days-since-added` - 'days since added' ESPP list operation filter.
- `filter-score-gt` - 'score is greater than' ESPP list operation filter.
- `filter-score-lt` - 'score is less than' ESPP list operation filter.
- `filter-percentile-gt` - 'percentile is greater than' ESPP list operation filter.
- `filter-percentile-lt` - 'percentile is less than' ESPP list operation filter.
- `filter-order-records` - 'order returned records' ESPP list operation filter ('score-desc' | 'score-asc' | 'percentile-desc' | 'percentile-asc').

Examples:

* `slim --quiet vulnerability --cve CVE-2021-21315 epss`
* `slim --output-format=json vulnerability --cve CVE-2021-21315 epss`
* `slim --quiet --output-format=json vulnerability --cve CVE-2021-21315 --cve CVE-2023-49070 epss`
* `slim --quiet vulnerability --cve CVE-2021-21315 epss --with-history --date 2022-12-13`
* `slim --quiet vulnerability epss --op list --date 2024-01-05`
* `slim --quiet vulnerability epss --op list --filter-cve-id-pattern 2023 --filter-score-gt 0.92 --limit 2 --offset 3`


Question: How to: RUNNING CONTAINERIZED

Answer:
The current version of Slim is able to run in containers. It will try to detect if it's running in a containerized environment, but you can also tell Slim explicitly using the `--in-container` global flag.

You can run Slim in your container directly or you can use the Slim container image in your containerized environment. If you are using the Slim container image make sure you run it configured with the Docker IPC information, so it can communicate with the Docker daemon. The most common way to do it is by mounting the Docker unix socket to the Slim app container. Some containerized environments (like Gitlab and their `dind` service) might not expose the Docker unix socket to you, so you'll need to make sure the environment variables used to communicate with Docker (e.g., `DOCKER_HOST`) are passed to the Slim app container. Note that if those environment variables reference any kind of local host names those names need to be replaced or you need to tell the Slim app about them using the `--etc-hosts-map` flag. If those environment variables reference local files those local files (e.g., files for TLS cert validation) will need to be copied to a temporary container, so that temporary container can be used as a data container to make those files accessible by the Slim app container.

When Slim app runs in a container it will attempt to save its execution state in a separate Docker volume. If the volume doesn't exist it will try to create it (`slim-state`, by default). You can pick a different state volume or disable this behavior completely by using the global `--archive-state` flag. If you do want to persist the Slim app execution state (which includes the `seccomp` and `AppArmor` profiles) without using the state archiving feature you can mount your own volume that maps to the `/bin/.slim-state` directory in the Slim app container.

By default, the Slim app will try to create a Docker volume for its sensor unless one already exists. If this behavior is not supported by your containerized environment you can create a volume separately and pass its name to the Slim app using the `--use-sensor-volume` flag.

Here's a basic example of how to use the containerized version of the Slim app:
`docker run -it --rm -v /var/run/docker.sock:/var/run/docker.sock dslim/slim build your-docker-image-name`

Here's a GitLab example for their `dind` `.gitlab-ci.yml` config file:
`docker run -e DOCKER_HOST=tcp://$(grep docker /etc/hosts | cut -f1):2375 dslim/slim build your-docker-image-name`

Here's a CircleCI example for their `remote docker` `.circleci/config.yml` config file (used after the `setup_remote_docker` step):

```bash
docker create -v /dcert_path --name dcert alpine:latest /bin/true
docker cp $DOCKER_CERT_PATH/. dcert:/dcert_path
docker run --volumes-from dcert -e DOCKER_HOST=$DOCKER_HOST -e DOCKER_TLS_VERIFY=$DOCKER_TLS_VERIFY -e DOCKER_CERT_PATH=/dcert_path dslim/slim build your-docker-image-name
```

Different CI/CD services have different containerized environment designs that impose various restrictions that may impact the ability of the main app to communicate with the sensor app embedded in the temporary container Slim creates. Try adjusting the values for the `--sensor-ipc-mode` and `--sensor-ipc-endpoint` flags. This [`Google Cloud Build`](https://medium.com/google-cloud/integrating-dockerslim-container-minify-step-on-cloud-build-64da29fd58d1) blog post by Márton Kodok is a good reference for both of those flags.


Question: How to: Using `*-file` Flags

Answer: - There are several flags that accept file paths (`--include-path-file`, `--compose-file`, `--http-probe-cmd-file`, etc). You need volume mount the location of the referenced paths or the file paths themselves when you use the containerized version of Slim because the Slim app container won't have accept to the referenced files otherwise.


Question: How to: Integrating Slim in Jenkins

Answer:
#### Prerequisites:
- Spin up a virtual machine(e.g.EC2 Instance, Azure VM, GCE) which has an Ubuntu OS via your desired cloud platform(AWS, Azure, GCP), SSH into the machine, update the machine packages and install docker. An example of this step is highlighted below given you are running an AWS EC2 Instance.
```
sudo apt update -y
```
```
sudo apt install docker -y
```
```
sudo systemctl start docker
```
```
sudo usermod -aG docker ec2-user
```
- Install Jenkins on the virtual machine using docker as stipulated by the [Jenkins Documentation](https://github.com/jenkinsci/docker/blob/master/README.md), this step pulls [Jenkins Image from DockerHub](https://hub.docker.com/r/jenkins/jenkins), runs Jenkins as a container via port 8080 and creates an explicit docker volume on the host machine to retain Jenkins data. Given you are running an AWS EC2 Instance, create a TCP rule with port 8080 in the Instance security group rules which allows only your Internet Protocol(IP) address to access the Jenkins server.
```
docker run -p 8080:8080 -p 50000:50000 -d -v jenkins_home:/var/jenkins_home jenkins/jenkins:lts
```
- Given Jenkins is now running as a containerized environment in the virtual machine, you need to make docker available in the Jenkins container, you can do this by bind mounting the virtual machine docker unix socket onto the jenkins container, note that to carry out this step you need to stop the running jenkins container, you can find the jenkins container ID by using the docker ps command, the commands to execute are highlighted below. This step is essential as it makes docker available in the Jenkins container, and with docker you can pull Slim Image which is to be used in further steps.
```
docker ps
```
```
docker stop [jenkins_container_id]
```
```
docker run -p 8080:8080 -p 50000:50000 -d \
-v jenkins_home:/var/jenkins_home \
-v /var/run/docker.sock:/var/run/docker.sock \
-v $(which docker):/usr/bin/docker jenkins/jenkins:lts
```
- Enable Docker permissions in the new jenkins container, such that Jenkins can perform docker commands and pull the [Slim Official Image](https://hub.docker.com/r/dslim/docker-slim) in the container. To do this, you need to get into the Jenkins container as a root user, you can find the jenkins container ID by using the docker ps command, the commands to execute are highlighted below:
```
docker exec -u 0 -it [jenkins_container_id] bash
```
```
chmod 666 /var/run/docker.sock
```
```
docker pull dslim/slim
```
#### Jenkinsfile Slim Stage
Given you have completed the prerequisite steps above, you can build a docker image and minify the image size using Slim via the snippet stage below which should be highlighted in your Jenkinsfile stages.
```
stage("Build and Slim Docker Image") {
  steps {
      script {
          echo "building and slimming docker image..."
          sh 'docker build -t IMAGE_NAME:$BUILD_NUMBER .'
          sh 'docker run --rm -v /var/run/docker.sock:/var/run/docker.sock dslim/slim \
              build --target IMAGE_NAME:$BUILD_NUMBER --tag IMAGE_NAME:slim-$BUILD_NUMBER \
              exit'
      }
  }
}
```
- The snippet stage above allows for customization, you should replace the image name--IMAGE_NAME with your desired image name, the environment variable tag--$BUILD_NUMBER represents a unique incremental number allocated by Jenkins each time your jenkins pipeline runs.
- The docker build command builds a Docker Image of your application from a Dockerfile.
- The docker run command runs Slim in a non-interactive mode via the docker unix socket, minifies the built(target) image--IMAGE_NAME:$BUILD_NUMBER, and adjusting it to a new slimmed image with the image/tag--IMAGE_NAME:slim-$BUILD_NUMBER.
- You should put the Slim stage before a docker tag/push stage and after a build/test artifact in your Jenkinsfile, an example pipeline is highlighted below for a sample nodejs application; The first stage test and builds an artifact of the application; The second stage builds a docker image and a slimmed version of the docker image; The third stage tags the slimmed docker image with a DockerHub account remote repository and pushes the image to the remote repository.
```
pipeline {
    agent any
    stages {
        stage("building nodejs app") {
            steps{
                script {
                    echo "building nodejs app..."
                    sh 'npm run test'
                    sh 'npm pack'
                }
            }
        }
        stage("Build and Slim Docker Image") {
            steps {
                script {
                    echo "building and slimming docker image..."
                    sh 'docker build -t node_alpine:$BUILD_NUMBER .'
                    sh 'docker run --rm -v /var/run/docker.sock:/var/run/docker.sock dslim/slim \
                        build --target node_alpine:$BUILD_NUMBER --tag node_alpine:slim-$BUILD_NUMBER \
                        exit'
                }
            }
        }
        st


Question: How to: Integrating Slimtoolkit in Github Actions

Answer:
#### Github Action
Integrating SlimToolkit in Github Actions in your CI/CD workflow involves using the [Docker-Slim Github Action](https://github.com/marketplace/actions/docker-slim-github-action), this Action(snippet below) minifies a target docker image--IMAGE_NAME:latest in your workflow, making it smaller and adjusting the new slimmed image as IMAGE_NAME:slim.
```


Question: How to: Build the Docker image first

Answer:
- uses: docker/build-push-action@v4
  with:
    push: false
    tags: IMAGE_NAME:{{github.run_number}}


Question: How to: Slim the Image

Answer:
- uses: kitabisa/docker-slim-action@v1
  env:
    DSLIM_HTTP_PROBE: false
  with:
    target: IMAGE_NAME:{{github.run_number}}
    tag: "slim-{{github.run_number}}"


Question: How to: Docker Hub Login

Answer:
uses: docker/login-action@v2
  with:
    username: ${{ secrets.DOCKERHUB_USERNAME }}
    password: ${{ secrets.DOCKERHUB_TOKEN }}


Question: How to: Push to the registry

Answer:
- run: |
   docker tag IMAGE_NAME:slim-{{github.run_number}} ${{ secrets.DOCKERHUB_USERNAME }}/IMAGE_NAME:slim-{{github.run_number}}
   docker push ${{ secrets.DOCKERHUB_USERNAME }}/IMAGE_NAME:slim-{{github.run_number}}
```
The workflow above indicates four steps:
- A [Docker Build/Push Github Action](https://github.com/docker/build-push-action) for building a docker image with the image name/tag--IMAGE_NAME:{{github.run_number}}, you should give replace IMAGE_NAME with your desired image name. Note that this Action must have a false option to push the built image--given that you need the image slimmed/minified before pushing it to a container registry.
- A Docker-Slim Github Action which minifies the target image--IMAGE_NAME:{{github.run_number}}, this Action has the "slim-{{github.run_number}}" tag and adds this tag to the slimmed/minified docker image such that the image name/tag becomes IMAGE_NAME:slim-{{github.run_number}}.
- A Docker Login Github Action which logs into your DockerHub container regristry account, you should store your DockerHub username and personal access token as secrets in the github repository meant for the workflow. Suppose your container registry is not DockerHub, you can check the [Docker Login Github Action documentation](https://github.com/docker/login-action) for the use case of logging into your desired container registry.
- A docker tag command for naming/tagging the slimmed image with your DockerHub account remote repository name which could be the same name(IMAGE_NAME) as the slimmed image; A docker push command to push the slimmed image to your Dockerhub account remote repository.


Question: How to: DOCKER CONNECT OPTIONS

Answer:
If you don't specify any Docker connect options the Slim app expects to find the Docker Unix socket (`/var/run/docker.sock`) or the following environment variables: `DOCKER_HOST`, `DOCKER_TLS_VERIFY` (optional), `DOCKER_CERT_PATH` (required if `DOCKER_TLS_VERIFY` is set to `"1"`). Note that the `DOCKER_HOST` environment variable can be used to point to a Unix socket address (in case the default Unix socket isn't there). This is useful when you use Docker Desktop and you haven't configured Docker Desktop to create the default Unix socket.

If the Docker environment variables are configured to use TLS and to verify the Docker cert (default behavior), but you want to disable the TLS verification you can override the TLS verification behavior by setting the `--tls-verify` to false:

`slim --tls-verify=false build my/sample-node-app-multi`

You can override all Docker connection options using these flags: `--host`, `--tls`, `--tls-verify`, `--tls-cert-path`. These flags correspond to the standard Docker options (and the environment variables). Note that you can also use the `--host` flag (similar to `DOCKER_HOST`) to point to a Unix socket (e.g., `--host=unix:///var/run/docker.sock`).

If you want to use TLS with verification:

`slim --host=tcp://192.168.99.100:2376 --tls-cert-path=/Users/youruser/.docker/machine/machines/default --tls=true --tls-verify=true build my/sample-node-app-multi`

If you want to use TLS without verification:

`slim --host=tcp://192.168.99.100:2376 --tls-cert-path=/Users/youruser/.docker/machine/machines/default --tls=true --tls-verify=false build my/sample-node-app-multi`

If the Docker environment variables are not set and if you don't specify any Docker connect options Slim will try to use the default unix socket.


Question: How to: DOCKER DESKTOP

Answer:
You may not have the default Unix socket (`/var/run/docker.sock`) configured if you use Docker Desktop. By default, Docker Desktop uses `~/.docker/run/docker.sock` as the Unix socket.

You can either use `--host` or `DOCKER_HOST` to point to the Docker Desktop's Unix socket or you can configure Docker Desktop to create the default/traditional Unix socket (creating the `/var/run/docker.sock` symlink manually is an option too).

To configure Docker Desktop to create the default Unix socket open its UI and go to `Settings -> Advanced` where you need to check the `Enable default Docker socket (Requires password)` option.


Question: How to: HTTP PROBE COMMANDS

Answer:
If the HTTP probe is enabled (note: it is enabled by default) it will default to running `GET /` with HTTP and then HTTPS on every exposed port. You can add additional commands using the `--http-probe-cmd` and `--http-probe-cmd-file` options.

If you want to disable HTTP probing set the `--http-probe` flag to false (e.g., `--http-probe=false`). You can also use the `--http-probe-off` flag to do the same (simply use the flag without any parameters).

The `--http-probe-cmd` option is good when you want to specify a small number of simple commands where you select some or all of these HTTP command options: crawling (defaults to false), protocol, method (defaults to GET), resource (path and query string).

If you only want to use custom HTTP probe command and you don't want the default `GET /` command added to the command list you explicitly provided you'll need to set `--http-probe` to false when you specify your custom HTTP probe command. Note that this inconsistency will be addressed in the future releases to make it less confusing.

Possible field combinations:
* `/path` - runs `GET /path`
* `crawl:/path` - runs `GET /path` and then crawls the pages referenced by the target page
* `post:/path` - runs `POST /path`
* `crawl:get:/path` - runs `GET /path` and then crawls the pages referenced by the target page
* `https:get:/path` runs `GET /path` only on https
* `crawl:http:get:/path` - runs `GET /path` and then crawls the pages referenced by the target page

Here are a couple of examples:

Adds two extra probe commands: `GET /api/info` and `POST /submit` (tries http first, then tries https):
`slim build --show-clogs --http-probe-cmd /api/info --http-probe-cmd POST:/submit my/sample-node-app-multi`

Adds one extra probe command: `POST /submit` (using only http):
`slim build --show-clogs --http-probe-cmd http:POST:/submit my/sample-node-app-multi`

The `--http-probe-cmd-file` option is good when you have a lot of commands and/or you want to select additional HTTP command options.

Available HTTP command options:
* `method` - HTTP method to use
* `resource` - target resource URL
* `port` - port number
* `protocol` - `http`, `https`, `http2`, `http2c` (cleartext version of http2), `ws`, `wss` (secure websocket)
* `headers` - array of strings with column delimited key/value pairs (e.g., "Content-Type: application/json")
* `body` - request body as a string
* `body_file` - request body loaded from the provided file
* `username` - username to use for basic auth
* `password` - password to use for basic auth
* `crawl` - boolean to indicate if you want to crawl the target (to visit all referenced resources)

Here's a probe command file example:

`slim build --show-clogs --http-probe-cmd-file probeCmds.json my/sample-node-app-multi`

Commands in `probeCmds.json`:

```
{
  "commands":
  [
   {
     "resource": "/api/info"
   },
   {
     "method": "POST",
     "resource": "/submit"
   },
   {
     "procotol": "http",
     "resource": "/api/call?arg=one"
   },
   {
     "protocol": "http",
     "method": "POST",
     "resource": "/submit2",
     "body": "key=value"
   },
   {
     "protocol": "http",
     "method": "POST",
     "resource": "/submit3",
     "body_file": "mydata.json",
     "headers": ["Content-Type: application/json"]
   }
  ]
}
```

The HTTP probe command file path can be a relative path (relative to the current working directory) or it can be an absolute path.

For each HTTP probe call Slim will print the call status. Example: `info=http.probe.call status=200 method=GET target=http://127.0.0.1:32899/ attempt=1 error=none`.

You can execute your own external HTTP requests using the `target.port.list` field in the container info message Slim prints when it starts its test container: `slim[build]: info=container name=
id=
target.port.list=[
] target.port.info=[
]`. Example: `slim[build]: info=container name=slimk_42861_20190203084955 id=aa44c43bcf4dd0dae78e2a8b3ac011e7beb6f098a65b09c8bce4a91dc2ff8427 target.port.list=[32899] target.port.info=[9000/tcp => 0.0.0.0:32899]`. With this information you can run `curl` or other HTTP request generating tools: `curl http://localhost:32899`.

The current version also includes an experimental `crawling` capability. To enable it for the default HTTP probe use the `--http-probe-crawl` flag. You can also enable it for the HTTP probe commands in your command file using the `crawl` boolean field.

When `crawling` is enabled the HTTP probe will act like a web crawler following the links it finds in the target endpoint.

Probing based on the Swagger/OpenAPI spec is another experimental capability. This feature introduces two new flags:
* `http-probe-apispec` - value: `
:
`
* `http-probe-apispec-file` - value: `
`

You can use the `--http-probe-exec` and `--http-probe-exec-file` options to


Question: How to: Debugging Using the `debug` Command

Answer:
The current version of the `debug` command is pretty basic and it lacks a number of useful capabilities. It will help you debug containers running in Docker or Kubernetes (use the `--runtime` flag and set it to `k8s` if you need to debug a container in Kubernetes).

By default the `debug` command will provide you with an interactive terminal when it attaches the debugger side-car image to the debugged target container. Future versions will allow you to have different interaction modes with the target.

#### The Debug Images

You can use any container image as a debug image, but there's a list of pre-selected debug images you can choose.

You can list all pre-selected debug images with the `--list-debug-images` and if you are using the interactive prompt mode there'll be an auto-complete dropdown menu for the `--debug-image` flag.

Here's the current list of debug images:

* `cgr.dev/chainguard/slim-toolkit-debug:latest` - a general purpose SlimToolkit debug image created by Chainguard
* `cgr.dev/chainguard/wolfi-base:latest` - a basic lightweight Wolfi image
* `busybox:latest` - a lightweight image with common unix utilities
* `nicolaka/netshoot` - a network trouble-shooting swiss-army container
* `lightruncom/koolkits:node` - a debug image for Node.js applications
* `lightruncom/koolkits:python` - a debug image for Python applications
* `lightruncom/koolkits:golang` - a debug image for Go applications
* `lightruncom/koolkits:jvm` - a debug image for Java applications
* `digitalocean/doks-debug:latest` - a kubernetes troubleshooting debug image
* `public.ecr.aws/zinclabs/debug-ubuntu-base:latest` - an image with common debugging utilities

#### Steps to Debug Your Container (Kubernetes Runtime)

1. Make sure the target environment you want to debug is up (the example k8s manifest creates a pod with the minimal nginx image from Chainguard and it has no shell):
```bash

>> kubectl apply -f examples/k8s_nginx_cgr/manifest.yaml

```
2. Run the debug command:

```bash

>> slim debug --runtime=k8s --pod=example-pod example-container

```
or
```bash

>> slim debug --runtime=k8s --pod=example-pod --target=example-container

```

Now you should have an interactive shell into the debug container started by `slim` and you can type your regular shell commands.

By default the `debug` command will connect the interactive terminal to the debugged container and it will run a shell as if it's running in the target container environment, so you will see the file system of the target container as if you are directly connected (you won't have to go through the `proc` file system). You can change this behavior by using the `--run-as-target-shell` (which is true by default). For example, this call will connect you to the debug container in a more traditional way: `slim debug --runtime=k8s --run-as-target-shell=false example-container`

Also note that if you use the interactive `prompt` mode (when you run `slim` with no command line parameters) you will get auto-complete behavior for a number of flags: `--target`, `--namespace`, `--pod`, `--session`.

Each time you try to debug an image `slim` will have a session that represents it. You'll be able to reconnect to the existing active debug sessions and you'll be able to get logs from all available sessions.

#### Steps to Debug Your Container (Docker Runtime)

1. Start the target container you want to debug:
```bash

>> docker run -it --rm -p 80:80 --name mycontainer nginx

```
2. Run the debug command:

```bash

>> slim debug mycontainer

```
or
```bash

>> slim debug --target=mycontainer

```

Now you should have an interactive shell into the debug container started by `slim` and you can type your regular shell commands.

By default the `debug` command will connect the interactive terminal to the debugged container and it will run a shell as if it's running in the target container environment, so you will see the file system of the target container as if you are directly connected (you won't have to go through the `proc` file system). You can change this behavior by using the `--run-as-target-shell` (which is true by default). For example, this call will connect you to the debug container in a more traditional way: `slim debug --run-as-target-shell=false mycontainer`

Also note that if you use the interactive `prompt` mode (when you run `slim` with no command line parameters) you will get auto-complete behavior for a number of flags: `--target`, `--session`.

Each time you try to debug an image `slim` will have a session that represents it. You'll be able to reconnect to the existing active debug sessions and you'll be able to get logs from all available sessions.


Question: How to: Debugging the "Hard Way" (Docker Runtime)

Answer:
You can create dedicated debugging side-car container images loaded with the tools you need for debugging target containers. This allows you to keep your production container images small. The debugging side-car containers attach to the running target containers.

Assuming you have a running container named `node_app_alpine` you can attach your debugging side-car with a command like this: `docker run --rm -it --pid=container:node_app_alpine --net=container:node_app_alpine --cap-add sys_admin alpine sh`. In this example, the debugging side-car is a regular alpine image. This is exactly what happens with the `node_alpine` app sample (located in the `node_alpine` directory of the `examples` repo) and the `run_debug_sidecar.command` helper script.

If you run the `ps` command in the side-car you'll see the application from the target container:

```


Question: How to: ls -lh /proc/1/root/opt/my/service

Answer:
total 8
drwxr-xr-x    3 root     root        4.0K Sep  2 15:51 node_modules
-rwxr-xr-x    1 root     root         415 Sep  8 00:52 server.js
```

Some of the useful debugging commands include `cat /proc/
/cmdline`, `ls -l /proc/
/cwd`, `cat /proc/1/environ`, `cat /proc/
/limits`, `cat /proc/
/status` and `ls -l /proc/
/fd`.


Question: How to: MINIFYING COMMAND LINE TOOLS

Answer:
Unless the default CMD instruction in your Dockerfile is sufficient you'll have to specify command line parameters when you execute the `build` command in Slim. This can be done with the `--cmd` option.

Other useful command line parameters:

- `--show-clogs` - use it if you want to see the output of your container.
- `--mount` - use it to mount a volume when Slim inspects your image.
- `--entrypoint` - use it if you want to override the ENTRYPOINT instruction when Slim inspects your image.

Note that the `--entrypoint` and `--cmd` options don't override the `ENTRYPOINT` and `CMD` instructions in the final minified image.

Here's a sample `build` command:

`slim build --show-clogs=true --cmd docker-compose.yml --mount $(pwd)/data/:/data/ dslim/container-transform`

It's used to minify the `container-transform` tool. You can get the minified image from [`Docker Hub`](https://hub.docker.com/r/dslim/container-transform.slim/).


Question: How to: QUICK SECCOMP EXAMPLE

Answer:
If you want to auto-generate a Seccomp profile AND minify your image use the `build` command. If you only want to auto-generate a Seccomp profile (along with other interesting image metadata) use the `profile` command.

Step one: run Slim

`slim build your-name/your-app`

Step two: use the generated Seccomp profile

`docker run --security-opt seccomp:
/.images/
/artifacts/your-name-your-app-seccomp.json
your-name/your-app`

Feel free to copy the generated profile :-)

You can use the generated Seccomp profile with your original image or with the minified image.


Question: How to: USING AUTO-GENERATED SECCOMP PROFILES

Answer:
You can use the generated profile with your original image or with the minified image Slim created:

`docker run -it --rm --security-opt seccomp:path_to/my-sample-node-app-seccomp.json -p 8000:8000 my/sample-node-app.slim`


Question: How to: ORIGINAL DEMO VIDEO

Answer:
[![DockerSlim demo](http://img.youtube.com/vi/uKdHnfEbc-E/0.jpg)](https://www.youtube.com/watch?v=uKdHnfEbc-E)

[Demo video on YouTube](https://youtu.be/uKdHnfEbc-E)


Question: How to: DEMO STEPS

Answer:
The demo runs on Mac OS X, but you can build a linux version. Note that these steps are different from the steps in the demo video.

1. Get the Slim app binaries:

* [Mac](https://github.com/slimtoolkit/slim/releases/download/1.40.11/dist_mac.zip),
* [Mac M1](https://github.com/slimtoolkit/slim/releases/download/1.40.11/dist_mac_m1.zip),
* [Linux](https://github.com/slimtoolkit/slim/releases/download/1.40.11/dist_linux.tar.gz),
* [Linux ARM](https://github.com/slimtoolkit/slim/releases/download/1.40.11/dist_linux_arm.tar.gz),
* [Linux ARM64](https://github.com/slimtoolkit/slim/releases/download/1.40.11/dist_linux_arm64.tar.gz)

Unzip them and optionally add their directory to your `PATH` environment variable if you want to use the app from other locations.

The extracted directory contains two binaries (and now it also contains a symlink for the old name):

- `slim` <- the main Slim application binary
- `slim-sensor` <- the sensor application used to collect information from running containers
- `docker-slim` <- the symlink to `slim`, the new main app binary (useful if you are still using the old name in your scripts)

2. Clone the `examples` repo to use the sample apps (note: the examples have been moved to a separate repo). You can skip this step if you have your own app.

`git clone https://github.com/slimtoolkit/examples.git`

3. Create a Docker image for the sample node.js app in `examples/node_ubuntu`. You can skip this step if you have your own app.

`cd examples/node_ubuntu`

`docker build -t my/sample-node-app .`

4. Run the Slim app:

`./slim build my/sample-node-app` <- run it from the location where you extracted the Slim app binaries (or update your `PATH` env var to include the directory where the Slim app binaries are located)

Slim creates a special container based on the target image you provided. It also creates a resource directory where it stores the information it discovers about your image: `
/.images/
`.

By default, the Slim app will run its http probe against the temporary container. If you are minifying a command line tool that doesn't expose any web service interface you'll need to explicitly disable http probing (by setting `--http-probe=false`).

5. Use curl (or other tools) to call the sample app (optional)

`curl http://
:
`

This is an optional step to make sure the target app container is doing something. Depending on the application it's an optional step. For some applications it's required if it loads new application resources dynamically based on the requests it's processing (e.g., Ruby or Python).

You'll see the mapped ports printed to the console when the Slim app starts the target container. You can also get the port number either from the `docker ps` or `docker port
` commands. The current version of DockerSlim doesn't allow you to map exposed network ports (it works like `docker run … -P`).

6. Press
and wait until the Slim app says it's done

By default or when http probing is enabled explicitly the Slim app will continue its execution once the http probe is done running. If you explicitly picked a different `continue-after` option follow the expected steps. For example, for the `enter` `continue-after` option you must press the `enter` button on your keyboard.

If http probing is enabled (when `http-probe` is set) and if `continue-after` is set to `enter` and you press the `enter` key before the built-in HTTP probe is done the probe might produce an EOF error because the Slim app will shut down the target container before all probe commands are done executing. It's ok to ignore it unless you really need the probe to finish.

7. Once Slim is done check that the new minified image is there

`docker images`

You should see `my/sample-node-app.slim` in the list of images. Right now all generated images have `.slim` at the end of its name.

8. Use the minified image

`docker run -it --rm --name="slim_node_app" -p 8000:8000 my/sample-node-app.slim`


Question: How to: Is it safe for production use?

Answer: Yes! Either way, you should test your Docker images.


Question: How to: How can I contribute if I don't know Go?

Answer: You don't need to read the language spec and lots of books :-) Go through the [Tour of Go](https://tour.golang.org/welcome/1) and optionally read [50 Shades of Go](https://golang50shades.com/) and you'll be ready to contribute!


Question: How to: What's the best application for Slim?

Answer: SlimRToolkit will work for any containerized application; however, Slim automates app interactions for applications with an HTTP API. You can use Slim even if your app doesn't have an HTTP API. You'll need to interact with your application manually to make sure Slim can observe your application behavior.


Question: How to: Can I use Slim with dockerized command line tools?

Answer:
Yes. The `--cmd`, `--entrypoint`, and `--mount` options will help you minify your image. The `container-transform` tool is a good example.

Notes:

You can explore the artifacts Slim generates when it's creating a slim image. You'll find those in `
/.images/
/artifacts`. One of the artifacts is a "reverse engineered" Dockerfile for the original image. It'll be called `Dockerfile.reversed`.

If you don't want to create a minified image and only want to "reverse engineer" the Dockerfile you can use the `info` command.


Question: How to: What if my Docker images uses the USER command?

Answer:
The current version of Slim does include support for non-default users (take a look at the non-default user examples (including the ElasticSearch example located in the `3rdparty` directory) in the [`examples`](https://github.com/slimtoolkit/examples) repo. Please open tickets if something doesn't work for you.

Everything should work as-is, but for the special cases where the current behavior don't work as expected you can adjust what Slim does using various `build` command parameters: `--run-target-as-user`, `--keep-perms`, `--path-perms`, `--path-perms-file` (along with the `--include-*` parameters).

The `--run-target-as-user` parameter is enabled by default and it controls if the application in the temporary container is started using the identity from the USER instruction in the container's Dockerfile.

The `--keep-perms` parameter is also enabled by default. It tells Slim to retain the permissions and the ownership information for the files and directories copied to the optimized container image.

The `--path-perms` and `--path-perms-file` parameters are similar to the `--include-path` and `--include-path-file` parameters. They are used to overwrite the permission and the user/group information for the target files and directories. Note that the target files/directories are expected to be in the optimized container image. If you don't know if the target files/directories will be in the optimized container you'll need to use one of the `--include-*` parameters (e.g., `--include-path-file`) to explicitly require those artifacts to be included. You can specify the permissions and the ownership information in the `--include-*` parameters too (so you don't need to have the `--path-*` parameters just to set the permissions).

The `--path-*` and `--include-*` params use the same format to communicate the permission/owernship info: `TARGET_PATH_OR_NAME:PERMS_IN_OCTAL_FORMAT#USER_ID#GROUP_ID`.

You don't have to specify the user and group IDs if you don't want to change them.

Here's an example using these parameters to minify the standard `nginx` image adding extra artifacts and changing their permissions: `slim build --include-path='/opt:770#104#107' --include-path='/bin/uname:710' --path-perms='/tmp:700' nginx`.

This is what you'll see in the optimized container image:

```
drwx------  0 0      0           0 Feb 28 22:15 tmp/
-rwx--x---  0 0      0       31240 Mar 14  2015 bin/uname
drwxrwx---  0 104    107         0 Feb 28 22:13 opt/
```

The `uname` binary isn't used by nginx, so the `--include-path` parameter is used to keep it in the optimized image changing its permissions to `710`.

The `/tmp` directory will be included in the optimized image on its own, so the `--path-perms` parameter is used to change its permissions to `700`.

When you set permissions/user/group on a directory the settings are only applied to that directory and not to the artifacts inside. The future versions will allow you to apply the same settings to everything inside the target directory too.

Also note that for now you have to use numeric user and group IDs. The future versions will allow you to use user and group names too.


Question: How to: Nginx fails in my minified image

Answer: If you see `nginx: [emerg] mkdir() "/var/lib/nginx/body" failed` it means your nginx setup uses a non-standard temporary directory. Nginx will fail if the base directory for its temporary folders doesn't exist (they won't create the missing intermediate directories). Normally it's `/var/lib/nginx`, but if you have a custom config that points to something else you'll need to add an `--include-path` flag as an extra flag when you run the Slim app.


Question: How to: Slim fails with a 'no permission to read from' error

Answer:
This problem shouldn't happen anymore because the exported artifacts are saved in a tar file and the master app doesn't need to access the files directly anymore.

If you run older versions of Slim you can get around this problem by running Slim from a root shell. That way it will have access to all exported files.

Slim copies the relevant image artifacts trying to preserve their permissions. If the permissions are too restrictive the master app might not have sufficient privilege to access these files when it's building the new minified image.


Question: How to: BUILD PROCESS

Answer:
#### Build Options

Pick one of the build options that works best for you.

##### Containerized

Run `make build_in_docker` on linux or `make build_m1_in_docker` on Macs (or `./scripts/docker-builder.run.sh` or click on `./scripts/mac/docker-builder.run.command` on Macs) from the project directory (builds Slim in a Docker container; great if you don't want to install Go on your local machine and if you already have Docker).

##### Native

Run `make build` on linux or `make build_m1` on Macs (or `./scripts/src.build.sh` or click on `./scripts/mac/src.build.command` on Macs) to build Slim natively (requires Go installed locally).

Note:

Try using the latest version of Go building the Slim app. The current version of Go used to build the Slim app is 1.21.

##### Gitpod

If you have a web browser, you can get a fully pre-configured development environment in one click:

[![Open in Gitpod](https://gitpod.io/button/open-in-gitpod.svg)](https://gitpod.io/#https://github.com/slimtoolkit/slim)

##### Additional Tools

- `license-bill-of-materials` - Optional tool to track dependencies and their licenses.
- `golint` - Optional tool for code analysis. See `https://github.com/golang/lint` for more details.

You can install these tools using the `tools.get.sh` shell script in the `scripts` directory.

Notes:

- Make sure you have `golint` if you intend to run the `src.inspect.sh` or `mac.src.inspect.command` scripts.


Question: How to: CONTRIBUTING

Answer: If the project sounds interesting or if you found a bug see [`CONTRIBUTING.md`](CONTRIBUTING.md) and submit a PR or open an issue! Non-code contributions including docs are highly appreciated! Open an issue even if you have a question or something is not clear.


Question: How to: CORE CONCEPTS

Answer:
1. Inspect container metadata (static analysis)
2. Inspect container data (static analysis)
3. Inspect running application (dynamic analysis)
4. Build an application artifact graph
5. Use the collected application data to build small images
6. Use the collected application data to auto-generate various security framework configurations.


Question: How to: DYNAMIC ANALYSIS OPTIONS

Answer:
1. Instrument the container image (and replace the entrypoint/cmd) to collect application activity data
2. Use kernel-level tools that provide visibility into running containers (without instrumenting the containers)
3. Disable relevant namespaces in the target container to gain container visibility (can be done with runC)


Question: How to: CHALLENGES

Answer: Some of the advanced analysis options require a number of Linux kernel features that are not always included. The kernel you get with Docker Machine / Boot2docker is a great example of that.


Question: How to: CODE OF CONDUCT

Answer:
The project follows the [CNCF Code of Conduct](https://github.com/cncf/foundation/blob/main/code-of-conduct.md).

---

**We are a [Cloud Native Computing Foundation](https://cncf.io/) sandbox project.**
---

[![Go Report Card](https://goreportcard.com/badge/github.com/slimtoolkit/slim)](https://goreportcard.com/report/github.com/slimtoolkit/slim)


Question: How to: Why should I use distroless images?

Answer:
Restricting what's in your runtime container to precisely what's necessary for your app is a best practice employed by Google
and other tech giants that have used containers in production for many years.
It improves the signal to noise of scanners (e.g. CVE) and reduces the burden of establishing provenance to just what you need.

Distroless images are _very small_.
The smallest distroless image, `gcr.io/distroless/static-debian12`, is around 2 MiB.
That's about 50% of the size of `alpine` (~5 MiB), and less than 2% of the size of `debian` (124 MiB).


Question: How to: How do I use distroless images?

Answer: These images are built using [bazel](https://bazel.build), but they can also be used through other Docker image build tooling.


Question: How to: What images are available?

Answer:
The following images are currently published and updated by the distroless project (see [SUPPORT_POLICY.md](SUPPORT_POLICY.md) for support timelines)

These images refer to image indexes with references to all supported architectures. Architecture specific images can be directly referenced using an additional architecture suffix on the tag, like `gcr.io/distroless/static-debian12:latest-amd64`

Any other tags are considered deprecated and are no longer updated

#### Debian 12

| Image                                 | Tags                                  | Architecture Suffixes             |
| ------------------------------------- | ------------------------------------- | --------------------------------- |
| gcr.io/distroless/static-debian12     | latest, nonroot, debug, debug-nonroot | amd64, arm64, arm, s390x, ppc64le |
| gcr.io/distroless/base-debian12       | latest, nonroot, debug, debug-nonroot | amd64, arm64, arm, s390x, ppc64le |
| gcr.io/distroless/base-nossl-debian12 | latest, nonroot, debug, debug-nonroot | amd64, arm64, arm, s390x, ppc64le |
| gcr.io/distroless/cc-debian12         | latest, nonroot, debug, debug-nonroot | amd64, arm64, arm, s390x, ppc64le |
| gcr.io/distroless/python3-debian12    | latest, nonroot, debug, debug-nonroot | amd64, arm64                      |
| gcr.io/distroless/java-base-debian12  | latest, nonroot, debug, debug-nonroot | amd64, arm64, s390x, ppc64le      |
| gcr.io/distroless/java17-debian12     | latest, nonroot, debug, debug-nonroot | amd64, arm64, s390x, ppc64le      |
| gcr.io/distroless/java21-debian12     | latest, nonroot, debug, debug-nonroot | amd64, arm64, s390x, ppc64le      |
| gcr.io/distroless/nodejs20-debian12   | latest, nonroot, debug, debug-nonroot | amd64, arm64, arm, s390x, ppc64le |
| gcr.io/distroless/nodejs22-debian12   | latest, nonroot, debug, debug-nonroot | amd64, arm64, arm, s390x, ppc64le |
| gcr.io/distroless/nodejs24-debian12   | latest, nonroot, debug, debug-nonroot | amd64, arm64, s390x, ppc64le      |

#### Debian 13

Debian 13 distroless images use the debian [UsrMerge](https://wiki.debian.org/UsrMerge) scheme. If you use `rules_distroless` to add packages to an image, set `mergedusr = True` in [`apt.install`](https://registry.bazel.build/docs/rules_distroless#apt_install).

| Image                                 | Tags                                  | Architecture Suffixes             |
| ------------------------------------- | ------------------------------------- | --------------------------------- |
| gcr.io/distroless/static-debian13     | latest, nonroot, debug, debug-nonroot | amd64, arm64, arm, s390x, ppc64le |
| gcr.io/distroless/base-debian13       | latest, nonroot, debug, debug-nonroot | amd64, arm64, arm, s390x, ppc64le |
| gcr.io/distroless/base-nossl-debian13 | latest, nonroot, debug, debug-nonroot | amd64, arm64, arm, s390x, ppc64le |
| gcr.io/distroless/cc-debian13         | latest, nonroot, debug, debug-nonroot | amd64, arm64, arm, s390x, ppc64le |
| gcr.io/distroless/java-base-debian13  | latest, nonroot, debug, debug-nonroot | amd64, arm64, s390x, ppc64le      |
| gcr.io/distroless/java21-debian13     | latest, nonroot, debug, debug-nonroot | amd64, arm64, s390x, ppc64le      |
| gcr.io/distroless/java25-debian13     | latest, nonroot, debug, debug-nonroot | amd64, arm64, s390x, ppc64le      |
| gcr.io/distroless/nodejs20-debian13   | latest, nonroot, debug, debug-nonroot | amd64, arm64, arm, s390x, ppc64le |
| gcr.io/distroless/nodejs22-debian13   | latest, nonroot, debug, debug-nonroot | amd64, arm64, arm, s390x, ppc64le |
| gcr.io/distroless/nodejs24-debian13   | latest, nonroot, debug, debug-nonroot | amd64, arm64, s390x, ppc64le      |

#### Debian 13 (preview)

Images that are still being worked on

| Image                                 | Tags                                  | Architecture Suffixes             |
| ------------------------------------- | ------------------------------------- | --------------------------------- |
| gcr.io/distroless/python3-debian13    | latest, nonroot, debug, debug-nonroot | amd64, arm64                      |


Question: How to: Why is distroless still using `gcr.io` instead of `pkg.dev`?

Answer: Distroless's serving infrastructure has moved to artifact registry but we still use the `gcr.io` domain. Users will get the benefits of the newer infrastructure without changing their builds.


Question: How to: How do I verify distroless images?

Answer:
All distroless images are signed by [cosign](https://github.com/sigstore/cosign) with ephemeral keys (keyless) -- this is the only supported mechanism starting November 2023.
We recommend verifying any distroless image you use before building your image. You can verify the keyless signature of any distroless image with:

```sh
cosign verify $IMAGE_NAME --certificate-oidc-issuer https://accounts.google.com --certificate-identity keyless@distroless.iam.gserviceaccount.com
```


Question: How to: Entrypoints

Answer:
Note that distroless images by default do not contain a shell.
That means the Dockerfile `ENTRYPOINT` command, when defined, must be specified in `vector` form, to avoid the container runtime prefixing with a shell.

This works:

```dockerfile
ENTRYPOINT ["myapp"]
```

But this does not work:

```dockerfile
ENTRYPOINT "myapp"
```

For the same reasons, if the entrypoint is set to the empty vector, the CMD command should be specified in `vector` form (see examples below).
Note that by default static, base and cc images have the empty vector entrypoint. Images with an included language runtime have a language specific default (see: [java](java/README.md#usage), [nodejs](nodejs/README.md#usage), [python3](python3/README.md#usage)).


Question: How to: Start by building the application.

Answer:
FROM golang:1.18 as build

WORKDIR /go/src/app
COPY . .

RUN go mod download
RUN CGO_ENABLED=0 go build -o /go/bin/app


Question: How to: Now copy it into our base image.

Answer:
FROM gcr.io/distroless/static-debian12
COPY --from=build /go/bin/app /
CMD ["/app"]
```

You can find other examples here:

- [Java](examples/java/Dockerfile)
- [Python 3](examples/python3/Dockerfile)
- [Go](examples/go/Dockerfile)
- [Node.js](examples/nodejs/Dockerfile)
- [Rust](examples/rust/Dockerfile)

To run any example, go to the directory for the language and run:

```sh
docker build -t myapp .
docker run -t myapp
```

To run the [Node.js Express example app](examples/nodejs/node-express) and expose the container's ports:

```sh
npm install # Install express and its transitive dependencies
docker build -t myexpressapp . # Normal build command
docker run -p 3000:3000 -t myexpressapp
```

This should expose the Express application to your `localhost:3000`


Question: How to: Base Operating System

Answer: Distroless images are based on Debian 12 (bookworm). Images are explicitly tagged with Debian version suffixes (e.g. `-debian12`). Specifying an image without the distribution will currently select `-debian12` images, but that will change in the future to a newer version of Debian. It can be useful to reference the distribution explicitly, to prevent breaking builds when the next Debian version is released.


Question: How to: Operating System Updates for Security Fixes and CVEs

Answer: Distroless tracks the upstream Debian releases, using [Github actions to automatically generate a pull request when there are updates](https://github.com/GoogleContainerTools/distroless/blob/main/.github/workflows/update-deb-package-snapshots.yml).


Question: How to: Debug Images

Answer:
Distroless images are minimal and lack shell access. The `:debug` image set for each language provides a busybox shell to enter.

For example:

```sh
cd examples/python3/
```

edit the `Dockerfile` to change the final image to `:debug`:

```dockerfile
FROM gcr.io/distroless/python3-debian12:debug
COPY . /app
WORKDIR /app
CMD ["hello.py", "/etc"]
```

then build and launch with a shell entrypoint:

```sh
docker build -t my_debug_image .
```

```sh
$ docker run --entrypoint=sh -ti my_debug_image

/app # ls
BUILD       Dockerfile  hello.py
```

> Note: If the image you are using already has a tag, for example `gcr.io/distroless/java17-debian12:nonroot`, use the tag `debug-
` instead, for example `gcr.io/distroless/java17-debian12:debug-nonroot`.

> Note: [ldd](http://man7.org/linux/man-pages/man1/ldd.1.html) is not installed in the base image as it's a shell script, you can copy it in or download it.


Question: How to: Who uses Distroless?

Answer:
- [Kubernetes](https://github.com/kubernetes/enhancements/blob/master/keps/sig-release/1729-rebase-images-to-distroless/README.md), since v1.15
- [Knative](https://knative.dev)
- [Tekton](https://tekton.dev)
- [Teleport](https://goteleport.com)
- [BloodHound](https://github.com/SpecterOps/BloodHound) by [SpecterOps](https://specterops.io/)

If your project uses Distroless, send a PR to add your project here!


Question: How to: Community Discussion

Answer:
- [distroless-users Google Group](https://groups.google.com/forum/#!forum/distroless-users)
- [Kubernetes slack #distroless channel](https://slack.k8s.io/)


Question: How to: Awesome Production Machine Learning

Answer:
This repository contains a curated list of awesome open source libraries that will help you deploy, monitor, version, scale, and secure your production machine learning 🚀

You can keep up to date by watching this github repo to get a summary of the new production ML libraries added every month [via releases](https://github.com/EthicalML/awesome-production-machine-learning/releases) 🤩

Additionally, we provide a [search toolkit](https://huggingface.co/spaces/zhiminy/Awesome-Production-Machine-Learning-Search) that helps you quickly navigate through the toolchain.


Question: How to: Quick links to sections on this page

Answer:
| | | |
|-|-|-|
| [🔧 AutoML](#automl) | [🧮 Computation & Communication Optimisation](#computation-and-communication-optimisation) | [🏷 Data Annotation & Synthesis](#data-annotation-and-synthesis) |
| [🧵 Data Pipeline](#data-pipeline) | [📓 Data Science Notebook](#data-science-notebook) | [💾 Data Storage Optimisation](#data-storage-optimisation) |
| [💸 Data Stream Processing](#data-stream-processing) | [💪 Deployment & Serving](#deployment-and-serving) | [📈 Evaluation & Monitoring](#evaluation-and-monitoring) |
| [🔍 Explainability & Fairness](#explainability-and-fairness) | [🎁 Feature Store](#feature-store) | [🔴 Industry-strength Anomaly Detection](#industry-strength-anomaly-detection) |
| [👁 Industry-strength Computer Vision](#industry-strength-computer-vision) | [🔥 Industry-strength Information Retrieval](#industry-strength-information-retrieval) | [🔠 Industry-strength Natural Language Processing](#industry-strength-nlp) |
| [🙌 Industry-strength Recommender System](#industry-strength-recommender-system) | [🍕 Industry-strength Reinforcement Learning](#industry-strength-reinforcement-learning) | [🤖 Industry-strength Robotics](#industry-strength-robotics) |
| [📊 Industry-strength Visualisation](#industry-strength-visualisation) | [📅 Metadata Management](#metadata-management) | [📜 Model, Data & Experiment Management](#model-data-and-experiment-management) |
| [🔩 Model Storage Optimisation](#model-storage-optimisation) | [🏁 Model Training & Orchestration](#model-training-and-orchestration) | [🔏 Privacy & Safety](#privacy-and-safety) |


Question: How to: Contributing to the list

Answer: Please review our [CONTRIBUTING.md](https://github.com/EthicalML/awesome-production-machine-learning/blob/master/CONTRIBUTING.md) requirements when submitting a PR to help us keep the list clean and up-to-date - thank you to the community for supporting its steady growth 🚀


Question: How to: 10 Min Video Overview

Answer:
This
10 minute video
provides an overview of the motivations for machine learning operations as well as a high level overview on some of the tools in this repo. This
newer video
covers the an updated 2024 version of the state of MLOps.


Question: How to: Want to receive recurrent updates on this repo and other advancements?

Answer:
You can join the
Machine Learning Engineer
newsletter. Join over 70,000 ML professionals and enthusiasts who receive weekly curated articles & tutorials on production Machine Learning.
Also check out the
Awesome Production GenAI
List, where we aim to map a curated list of awesome open source libraries to deploy, monitor, version and scale your generative artificial intelligence applications and systems.


Question: How to: Computation and Communication Optimisation

Answer:
* [Accelerate](https://github.com/huggingface/accelerate) ![](https://img.shields.io/github/stars/huggingface/accelerate.svg?cacheSeconds=86400) - Accelerate abstracts exactly and only the boilerplate code related to multi-GPU/TPU/mixed-precision and leaves the rest of your code unchanged.
* [Adapters](https://github.com/adapter-hub/adapters) ![](https://img.shields.io/github/stars/adapter-hub/adapters.svg?cacheSeconds=86400) - Adapters is a unified library for parameter-efficient and modular transfer learning.
* [BitBLAS](https://github.com/microsoft/BitBLAS) ![](https://img.shields.io/github/stars/microsoft/BitBLAS.svg?cacheSeconds=86400) - BitBLAS is a library to support mixed-precision BLAS operations on GPUs
* [Colossal-AI](https://github.com/hpcaitech/ColossalAI) ![](https://img.shields.io/github/stars/hpcaitech/ColossalAI.svg?cacheSeconds=86400) - A unified deep learning system for big model era, which helps users to efficiently and quickly deploy large AI model training and inference.
* [Composer](https://github.com/mosaicml/composer) ![](https://img.shields.io/github/stars/mosaicml/composer.svg?cacheSeconds=86400) - Composer is a PyTorch library that enables you to train neural networks faster, at lower cost, and to higher accuracy.
* [CuDF](https://github.com/rapidsai/cudf) ![](https://img.shields.io/github/stars/rapidsai/cudf.svg?cacheSeconds=86400) - Built based on the Apache Arrow columnar memory format, cuDF is a GPU DataFrame library for loading, joining, aggregating, filtering, and otherwise manipulating data.
* [CuML](https://github.com/rapidsai/cuml) ![](https://img.shields.io/github/stars/rapidsai/cuml.svg?cacheSeconds=86400) - cuML is a suite of libraries that implement machine learning algorithms and mathematical primitives functions that share compatible APIs with other RAPIDS projects.
* [CuPy](https://github.com/cupy/cupy) ![](https://img.shields.io/github/stars/cupy/cupy.svg?cacheSeconds=86400) - An implementation of NumPy-compatible multi-dimensional array on CUDA. CuPy consists of the core multi-dimensional array class, cupy.ndarray, and many functions on it.
* [DEAP](https://github.com/DEAP/deap) ![](https://img.shields.io/github/stars/DEAP/deap.svg?cacheSeconds=86400) - A novel evolutionary computation framework for rapid prototyping and testing of ideas. It seeks to make algorithms explicit and data structures transparent. It works in perfect harmony with parallelisation mechanisms such as multiprocessing and SCOOP.
* [DeepEP](https://github.com/deepseek-ai/DeepEP) ![](https://img.shields.io/github/stars/deepseek-ai/DeepEP.svg?cacheSeconds=86400) - DeepEP is a communication library tailored for Mixture-of-Experts (MoE) and expert parallelism (EP). It provides high-throughput and low-latency all-to-all GPU kernels, which are also known as MoE dispatch and combine. The library also supports low-precision operations, including FP8.
* [DGL](https://github.com/dmlc/dgl) ![](https://img.shields.io/github/stars/dmlc/dgl.svg?cacheSeconds=86400) - DGL is an easy-to-use, high performance and scalable Python package for deep learning on graphs.
* [DLRover](https://github.com/intelligent-machine-learning/dlrover) ![](https://img.shields.io/github/stars/intelligent-machine-learning/dlrover.svg?cacheSeconds=86400) - DLRover makes the distributed training of large AI models easy, stable, fast and green.
* [Dask](https://github.com/dask/dask) ![](https://img.shields.io/github/stars/dask/dask.svg?cacheSeconds=86400) - Distributed parallel processing framework for Pandas and NumPy computations.
* [DeepSpeed](https://github.com/deepspeedai/DeepSpeed) ![](https://img.shields.io/github/stars/deepspeedai/DeepSpeed.svg?cacheSeconds=86400) - DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.
* [FlagGems](https://github.com/FlagOpen/FlagGems) ![](https://img.shields.io/github/stars/FlagOpen/FlagGems.svg?cacheSeconds=86400) - FlagGems is a high-performance general operator library implemented in OpenAI Triton. It builds on a collection of backend neutral kernels that aims to accelerate LLM training and inference across diverse hardware platforms.
* [Flashlight](https://github.com/flashlight/flashlight) ![](https://img.shields.io/github/stars/flashlight/flashlight.svg?cacheSeconds=86400) - A fast, flexible machine learning library written entirely in C++ from the Facebook AI Research and the creators of Torch, TensorFlow, Eigen and Deep Speech.
* [Flax](https://github.com/google/flax) ![](https://img.shields.io/github/stars/google/flax.svg?cacheSeconds=86400) - A neural network library and ecosystem for JAX designed for flexibility.
* [GPUStack](https://github.com/gpustack/gpustack) ![](https://img.shields.io/github/stars/gpustack/gpustack.svg?cacheSeconds=86400) - GPUStack is an open-source GPU cluster manager for running AI models.
* [Hivemind](https://github.com/learning-at-home/hivemind) ![](https://img.shields.io/github/sta


Question: How to: Data Annotation and Synthesis

Answer:
* [Argilla](https://github.com/argilla-io/argilla) ![](https://img.shields.io/github/stars/argilla-io/argilla.svg?cacheSeconds=86400) - Argilla helps domain experts and data teams to build better NLP datasets in less time.
* [cleanlab](https://github.com/cleanlab/cleanlab) ![](https://img.shields.io/github/stars/cleanlab/cleanlab.svg?cacheSeconds=86400) - Python library for data-centric AI. Can automatically: find mislabeled data, detect outliers, estimate consensus + annotator-quality for multi-annotator datasets, suggest which data is best to (re)label next.
* [COCO Annotator](https://github.com/jsbroks/coco-annotator) ![](https://img.shields.io/github/stars/jsbroks/coco-annotator.svg?cacheSeconds=86400) - Web-based image segmentation tool for object detection, localization and keypoints
* [CVAT](https://github.com/cvat-ai/cvat) ![](https://img.shields.io/github/stars/cvat-ai/cvat.svg?cacheSeconds=86400) - CVAT (Computer Vision Annotation Tool) is OpenCV's web-based annotation tool for both videos and images for computer algorithms.
* [Doccano](https://github.com/doccano/doccano) ![](https://img.shields.io/github/stars/doccano/doccano.svg?cacheSeconds=86400) - Open source text annotation tools for humans, providing functionality for sentiment analysis, named entity recognition, and machine translation.
* [Gretel Synthetics](https://github.com/gretelai/gretel-synthetics) ![](https://img.shields.io/github/stars/gretelai/gretel-synthetics.svg?cacheSeconds=86400) - Gretel Synthetics is a synthetic data generators for structured and unstructured text, featuring differentially private learning.
* [Label Studio](https://github.com/HumanSignal/label-studio) ![](https://img.shields.io/github/stars/HumanSignal/label-studio.svg?cacheSeconds=86400) - Multi-domain data labeling and annotation tool with standardized output format.
* [NeMo Curator](https://github.com/NVIDIA/NeMo-Curator) ![](https://img.shields.io/github/stars/NVIDIA/NeMo-Curator.svg?cacheSeconds=86400) - NeMo Curator is a GPU-accelerated framework for efficient large language model data curation.
* [refinery](https://github.com/code-kern-ai/refinery) ![](https://img.shields.io/github/stars/code-kern-ai/refinery.svg?cacheSeconds=86400) - The data scientist's open-source choice to scale, assess and maintain natural language data.
* [SDV](https://github.com/sdv-dev/SDV) ![](https://img.shields.io/github/stars/sdv-dev/SDV.svg?cacheSeconds=86400) - Synthetic Data Vault (SDV) is a Synthetic Data Generation ecosystem of libraries that allows users to easily learn single-table, multi-table and timeseries datasets to later on generate new Synthetic Data that has the same format and statistical properties as the original dataset.
* [Semantic Segmentation Editor](https://github.com/Hitachi-Automotive-And-Industry-Lab/semantic-segmentation-editor) ![](https://img.shields.io/github/stars/Hitachi-Automotive-And-Industry-Lab/semantic-segmentation-editor.svg?cacheSeconds=86400) - Hitachi's Open source tool for labelling camera and LIDAR data.
* [synthcity](https://github.com/vanderschaarlab/synthcity) ![](https://img.shields.io/github/stars/vanderschaarlab/synthcity.svg?cacheSeconds=86400) - synthcity is a library for generating and evaluating synthetic tabular data.
* [ViPE](https://github.com/nv-tlabs/vipe) ![](https://img.shields.io/github/stars/nv-tlabs/vipe.svg?cacheSeconds=86400) - ViPE is a spatial AI tool for annotating camera poses and dense depth maps from raw videos.
* [YData Synthetic](https://github.com/ydataai/ydata-synthetic) ![](https://img.shields.io/github/stars/ydataai/ydata-synthetic.svg?cacheSeconds=86400) - YData Synthetic is a package to generate synthetic tabular and time-series data leveraging the state of the art generative models.


Question: How to: Data Pipeline

Answer:
* [Apache Airflow](https://github.com/apache/airflow) ![](https://img.shields.io/github/stars/apache/airflow.svg?cacheSeconds=86400) - Data Pipeline framework built in Python, including scheduler, DAG definition and a UI for visualisation.
* [Apache Nifi](https://github.com/apache/nifi) ![](https://img.shields.io/github/stars/apache/nifi.svg?cacheSeconds=86400) - Apache NiFi was made for dataflow. It supports highly configurable directed graphs of data routing, transformation, and system mediation logic.
* [Apache Oozie](https://github.com/apache/oozie) ![](https://img.shields.io/github/stars/apache/oozie.svg?cacheSeconds=86400) - Workflow scheduler for Hadoop jobs.
* [Argo Workflows](https://github.com/argoproj/argo-workflows) ![](https://img.shields.io/github/stars/argoproj/argo-workflows.svg?cacheSeconds=86400) - Argo Workflows is an open source container-native workflow engine for orchestrating parallel jobs on Kubernetes. Argo Workflows is implemented as a Kubernetes CRD (Custom Resource Definition).
* [Couler](https://github.com/couler-proj/couler) ![](https://img.shields.io/github/stars/couler-proj/couler.svg?cacheSeconds=86400) - Unified interface for constructing and managing machine learning workflows on different workflow engines, such as Argo Workflows, Tekton Pipelines, and Apache Airflow.
* [DataTrove](https://github.com/huggingface/datatrove) ![](https://img.shields.io/github/stars/huggingface/datatrove.svg?cacheSeconds=86400) - DataTrove is a library to process, filter and deduplicate text data at a very large scale.
* [Dagster](https://github.com/dagster-io/dagster) ![](https://img.shields.io/github/stars/dagster-io/dagster.svg?cacheSeconds=86400) - A data orchestrator for machine learning, analytics, and ETL.
* [DBT](https://github.com/dbt-labs/dbt-core) ![](https://img.shields.io/github/stars/dbt-labs/dbt-core.svg?cacheSeconds=86400) - ETL tool for running transformations inside data warehouses.
* [Flyte](https://github.com/flyteorg/flyte) ![](https://img.shields.io/github/stars/flyteorg/flyte.svg?cacheSeconds=86400) - Lyft’s Cloud Native Machine Learning and Data Processing Platform - [(Demo)](https://youtu.be/KdUJGSP1h9U?t=1451).
* [Genie](https://github.com/Netflix/genie) ![](https://img.shields.io/github/stars/Netflix/genie.svg?cacheSeconds=86400) - Job orchestration engine to interface and trigger the execution of jobs from Hadoop-based systems.
* [Hamilton](https://github.com/dagworks-inc/hamilton) ![](https://img.shields.io/github/stars/dagworks-inc/hamilton.svg?cacheSeconds=86400) - Hamilton is a micro-orchestration framework for defining dataflows. Runs anywhere python runs (e.g. jupyter, fastAPI, spark, ray, dask). Brings software engineering best practices without you knowing it. Use it to define feature engineering transforms, end-to-end model pipelines, and LLM workflows. It complements macro-orchestration systems (e.g. kedro, luigi, airflow, dbt, etc.) as it replaces the code within those macro tasks. Comes with a self-hostable UI that captures lineage & provenance, execution telemetry & data summaries, and builds a self-populating catalog; usable in development as well as production.
* [Instill VDP](https://github.com/instill-ai/instill-core) ![](https://img.shields.io/github/stars/instill-ai/instill-core.svg?cacheSeconds=86400) - Instill VDP (Versatile Data Pipeline) aims to streamline the data processing pipelines from inception to completion.
* [Instructor](https://github.com/instructor-ai/instructor) ![](https://img.shields.io/github/stars/instructor-ai/instructor.svg?cacheSeconds=86400) - Instructor makes it easy to get structured data like JSON from LLMs like GPT-3.5, GPT-4, GPT-4-Vision, and open-source models.
* [Kedro](https://github.com/kedro-org/kedro) ![](https://img.shields.io/github/stars/kedro-org/kedro.svg?cacheSeconds=86400) - Kedro is a workflow development tool that helps you build data pipelines that are robust, scalable, deployable, reproducible and versioned.
* [Luigi](https://github.com/spotify/luigi) ![](https://img.shields.io/github/stars/spotify/luigi.svg?cacheSeconds=86400) - Luigi is a Python module that helps you build complex pipelines of batch jobs, handling dependency resolution, workflow management, visualisation, etc..
* [Metaflow](https://github.com/Netflix/metaflow) ![](https://img.shields.io/github/stars/Netflix/metaflow.svg?cacheSeconds=86400) - A framework for data scientists to easily build and manage real-life data science projects.
* [Pachyderm](https://github.com/pachyderm/pachyderm) ![](https://img.shields.io/github/stars/pachyderm/pachyderm.svg?cacheSeconds=86400) - Open source distributed processing framework build on Kubernetes focused mainly on dynamic building of production machine learning pipelines - [(Video)](https://www.youtube.com/watch?v=LamKVhe2RSM).
* [Ploomber](https://github.com/ploomber/ploomber) ![](https://img.shields.io/github/stars/ploomber/ploomber.svg?cacheSeconds=86400) - The fastest way to build data pipeli


Question: How to: Data Science Notebook

Answer:
* [Apache Zeppelin](https://github.com/apache/zeppelin) ![](https://img.shields.io/github/stars/apache/zeppelin.svg?cacheSeconds=86400) - Web-based notebook that enables data-driven, interactive data analytics and collaborative documents with SQL, Scala and more.
* [Deepnote](https://github.com/deepnote/deepnote) ![](https://img.shields.io/github/stars/deepnote/deepnote.svg?cacheSeconds=86400) - Deepnote is a drop-in replacement for Jupyter with an AI-first design, sleek UI, new blocks, and native data integrations. Use Python, R, and SQL locally in your favorite IDE, then scale to Deepnote cloud for real-time collaboration, Deepnote agent, and deployable data apps.
* [Jupyter Notebooks](https://github.com/jupyter/notebook) ![](https://img.shields.io/github/stars/jupyter/notebook.svg?cacheSeconds=86400) - Web interface python sandbox environments for reproducible development
* [Marimo](https://github.com/marimo-team/marimo) ![](https://img.shields.io/github/stars/marimo-team/marimo.svg?cacheSeconds=86400) - Reactive Python notebook — run reproducible experiments, execute as a script, deploy as an app, and version with git.
* [Papermill](https://github.com/nteract/papermill) ![](https://img.shields.io/github/stars/nteract/papermill.svg?cacheSeconds=86400) - Papermill is a library for parameterizing notebooks and executing them like Python scripts.
* [Polynote](https://github.com/polynote/polynote) ![](https://img.shields.io/github/stars/polynote/polynote.svg?cacheSeconds=86400) - Polynote is an experimental polyglot notebook environment. Currently, it supports Scala and Python (with or without Spark), SQL, and Vega.
* [RMarkdown](https://github.com/rstudio/rmarkdown) ![](https://img.shields.io/github/stars/rstudio/rmarkdown.svg?cacheSeconds=86400) - The rmarkdown package is a next generation implementation of R Markdown based on Pandoc.
* [Stencila](https://github.com/stencila/stencila) ![](https://img.shields.io/github/stars/stencila/stencila.svg?cacheSeconds=86400) - Stencila is a platform for creating, collaborating on, and sharing data driven content. Content that is transparent and reproducible.
* [Voilà](https://github.com/voila-dashboards/voila) ![](https://img.shields.io/github/stars/voila-dashboards/voila.svg?cacheSeconds=86400) - Voilà turns Jupyter notebooks into standalone web applications that can e.g. be used as dashboards.
* [.NET Interactive](https://github.com/dotnet/interactive) ![](https://img.shields.io/github/stars/dotnet/interactive.svg?cacheSeconds=86400) - .NET Interactive takes the power of .NET and embeds it into your interactive experiences.


Question: How to: Data Storage Optimisation

Answer:
* [AIStore](https://github.com/NVIDIA/aistore) ![](https://img.shields.io/github/stars/NVIDIA/aistore.svg?cacheSeconds=86400) - AIStore is a lightweight object storage system with the capability to linearly scale out with each added storage node and a special focus on petascale deep learning.
* [Alluxio](https://github.com/Alluxio/alluxio) ![](https://img.shields.io/github/stars/Alluxio/alluxio.svg?cacheSeconds=86400) - A virtual distributed storage system that bridges the gab between computation frameworks and storage systems.
* [Apache Arrow](https://github.com/apache/arrow) ![](https://img.shields.io/github/stars/apache/arrow.svg?cacheSeconds=86400) - In-memory columnar representation of data compatible with Pandas, Hadoop-based systems, etc..
* [Apache Druid](https://github.com/apache/druid) ![](https://img.shields.io/github/stars/apache/druid.svg?cacheSeconds=86400) - A high performance real-time analytics database. Check this [article](https://towardsdatascience.com/introduction-to-druid-4bf285b92b5a) for introduction.
* [Apache Hudi](https://github.com/apache/hudi) ![](https://img.shields.io/github/stars/apache/hudi.svg?cacheSeconds=86400) - Hudi is a transactional data lake platform that brings core warehouse and database functionality directly to a data lake. Hudi is great for streaming workloads, and also allows creation of efficient incremental batch pipelines. Supports popular query engines including Spark, Flink, Presto, Trino, Hive, etc. More info [here](https://hudi.apache.org/).
* [Apache Iceberg](https://github.com/apache/iceberg) ![](https://img.shields.io/github/stars/apache/iceberg.svg?cacheSeconds=86400) - Iceberg is an ACID-compliant, high-performance format built for huge analytic tables (containing tens of petabytes of data), and it brings the reliability and simplicity of SQL tables to big data, while making it possible for engines like Spark, Trino, Flink, Presto, Hive and Impala to safely work with the same tables, at the same time. More info [here](https://iceberg.apache.org/).
* [Apache Ignite](https://github.com/apache/ignite) ![](https://img.shields.io/github/stars/apache/ignite.svg?cacheSeconds=86400) - A memory-centric distributed database, caching, and processing platform for transactional, analytical, and streaming workloads delivering in-memory speeds at petabyte scale - [Demo](https://www.youtube.com/watch?v=Xt4PWQ__YPw).
* [Apache Parquet](https://github.com/apache/parquet-java) ![](https://img.shields.io/github/stars/apache/parquet-java.svg?cacheSeconds=86400) - On-disk columnar representation of data compatible with Pandas, Hadoop-based systems, etc..
* [Apache Pinot](https://github.com/apache/pinot) ![](https://img.shields.io/github/stars/apache/pinot.svg?cacheSeconds=86400) - A realtime distributed OLAP datastore. Comparison of the open source OLAP systems for big data: ClickHouse, Druid, and Pinot is found [here](https://medium.com/@leventov/comparison-of-the-open-source-olap-systems-for-big-data-clickhouse-druid-and-pinot-8e042a5ed1c7).
* [Casibase](https://github.com/casibase/casibase) ![](https://img.shields.io/github/stars/casibase/casibase.svg?cacheSeconds=86400) - Casibase is a LangChain-like RAG (Retrieval-Augmented Generation) knowledge database with web UI and Enterprise SSO.
* [Chroma](https://github.com/chroma-core/chroma) ![](https://img.shields.io/github/stars/chroma-core/chroma.svg?cacheSeconds=86400) - Chroma is an open-source embedding database.
* [ClickHouse](https://github.com/ClickHouse/ClickHouse) ![](https://img.shields.io/github/stars/ClickHouse/ClickHouse.svg?cacheSeconds=86400) - ClickHouse is an open source column oriented database management system.
* [Delta Lake](https://github.com/delta-io/delta) ![](https://img.shields.io/github/stars/delta-io/delta.svg?cacheSeconds=86400) - Delta Lake is a storage layer that brings scalable, ACID transactions to Apache Spark and other big-data engines.
* [EdgeDB](https://github.com/geldata/gel) ![](https://img.shields.io/github/stars/geldata/gel.svg?cacheSeconds=86400) - Gel supercharges Postgres with a modern data model, graph queries, Auth & AI solutions, and much more.
* [GPTCache](https://github.com/zilliztech/GPTCache) ![](https://img.shields.io/github/stars/zilliztech/GPTCache.svg?cacheSeconds=86400) - GPTCache is a library for creating semantic cache for large language model queries.
* [InfluxDB](https://github.com/influxdata/influxdb) ![](https://img.shields.io/github/stars/influxdata/influxdb.svg?cacheSeconds=86400) Scalable datastore for metrics, events, and real-time analytics.
* [Milvus](https://github.com/milvus-io/milvus) ![](https://img.shields.io/github/stars/milvus-io/milvus.svg?cacheSeconds=86400) Milvus is a cloud-native, open-source vector database built to manage embedding vectors generated by machine learning models and neural networks.
* [Marqo](https://github.com/marqo-ai/marqo) ![](https://img.shields.io/github/stars/marqo-ai/marqo.svg?cacheSeconds=86400) Marqo is an end-to-end


Question: How to: Data Stream Processing

Answer:
* [Apache Beam](https://github.com/apache/beam) ![](https://img.shields.io/github/stars/apache/beam.svg?cacheSeconds=86400) Apache Beam is a unified programming model for Batch and Streaming.
* [Apache Flink](https://github.com/apache/flink) ![](https://img.shields.io/github/stars/apache/flink.svg?cacheSeconds=86400) - Open source stream processing framework with powerful stream and batch processing capabilities.
* [Apache Kafka](https://github.com/apache/kafka) ![](https://img.shields.io/github/stars/apache/kafka.svg?cacheSeconds=86400) - Kafka client library for building applications and microservices where the input and output are stored in kafka clusters.
* [Apache Samza](https://github.com/apache/samza) ![](https://img.shields.io/github/stars/apache/samza.svg?cacheSeconds=86400) - Distributed stream processing framework. It uses Apache Kafka for messaging, and Apache Hadoop YARN to provide fault tolerance, processor isolation, security, and resource management.
* [Apache Spark](https://github.com/apache/spark) ![](https://img.shields.io/github/stars/apache/spark.svg?cacheSeconds=86400) - Micro-batch processing for streams using the apache spark framework as a backend supporting stateful exactly-once semantics.
* [Bytewax](https://github.com/bytewax/bytewax) ![](https://img.shields.io/github/stars/bytewax/bytewax.svg?cacheSeconds=86400) - Flexible Python-centric stateful stream processing framework built on top of Rust engine.
* [FastStream](https://github.com/airtai/faststream) ![](https://img.shields.io/github/stars/airtai/faststream.svg?cacheSeconds=86400) - A modern broker-agnostic streaming Python framework supporting Apache Kafka, RabbitMQ and NATS protocols, inspired by FastAPI and easily integratable with other web frameworks.
* [MOA](https://github.com/Waikato/moa) ![](https://img.shields.io/github/stars/Waikato/moa.svg?cacheSeconds=86400) - MOA (Massive Online Analysis) is an open source framework for Big Data stream mining.
* [MosaicML Streaming](https://github.com/mosaicml/streaming) ![](https://img.shields.io/github/stars/mosaicml/streaming.svg?cacheSeconds=86400) - Fast, deterministic streaming of large datasets from cloud storage for distributed model training.
* [RisingWave](https://github.com/risingwavelabs/risingwave) ![](https://img.shields.io/github/stars/risingwavelabs/risingwave.svg?cacheSeconds=86400) - A distributed SQL streaming database that unifies stream processing and low-latency serving, ideal for building and serving features for online machine learning.
* [TensorStore](https://github.com/google/tensorstore) ![](https://img.shields.io/github/stars/google/tensorstore.svg?cacheSeconds=86400) - Library for reading and writing large multi-dimensional arrays.


Question: How to: Deployment and Serving

Answer:
* [Agenta](https://github.com/Agenta-AI/agenta) ![](https://img.shields.io/github/stars/Agenta-AI/agenta.svg?cacheSeconds=86400) - Agenta provides end-to-end tools for the entire LLMOps workflow: building (LLM playground, evaluation), deploying (prompt and configuration management), and  (LLM observability and tracing).
* [AirLLM](https://github.com/lyogavin/airllm) ![](https://img.shields.io/github/stars/lyogavin/airllm.svg?cacheSeconds=86400) - AirLLM optimizes inference memory usage, allowing 70B large language models to run inference on a single 4GB GPU card without quantization, distillation and pruning.
* [AITemplate](https://github.com/facebookincubator/AITemplate) ![](https://img.shields.io/github/stars/facebookincubator/AITemplate.svg?cacheSeconds=86400) - AITemplate (AIT) is a Python framework that transforms deep neural networks into CUDA (NVIDIA GPU) / HIP (AMD GPU) C++ code for lightning-fast inference serving.
* [BentoML](https://github.com/bentoml/BentoML) ![](https://img.shields.io/github/stars/bentoml/BentoML.svg?cacheSeconds=86400) - BentoML is an open source framework for high performance ML model serving.
* [BISHENG](https://github.com/dataelement/bisheng) ![](https://img.shields.io/github/stars/dataelement/bisheng.svg?cacheSeconds=86400) - BISHENG is an open LLM application devops platform, focusing on enterprise scenarios.
* [DeepDetect](https://github.com/jolibrain/deepdetect) ![](https://img.shields.io/github/stars/jolibrain/deepdetect.svg?cacheSeconds=86400) - Machine Learning production server for TensorFlow, XGBoost and Cafe models written in C++ and maintained by Jolibrain.
* [Dynamo](https://github.com/ai-dynamo/dynamo) ![](https://img.shields.io/github/stars/ai-dynamo/dynamo.svg?cacheSeconds=86400) - NVIDIA Dynamo is a high-throughput, low-latency inference framework designed for serving generative AI and reasoning models in multi-node distributed environments.
* [exo](https://github.com/exo-explore/exo) ![](https://img.shields.io/github/stars/exo-explore/exo.svg?cacheSeconds=86400) - exo helps you run your AI cluster at home with everyday devices.
* [Genkit](https://github.com/firebase/genkit) ![](https://img.shields.io/github/stars/firebase/genkit.svg?cacheSeconds=86400) - Genkit is an open source framework for building AI-powered apps with familiar code-centric patterns. Genkit makes it easy to develop, integrate, and test AI features with observability and evaluations.
* [Inference](https://github.com/roboflow/inference) ![](https://img.shields.io/github/stars/roboflow/inference.svg?cacheSeconds=86400) - A fast, production-ready inference server for computer vision supporting deployment of many popular model architectures and fine-tuned models. With Inference, you can deploy models such as YOLOv5, YOLOv8, CLIP, SAM, and CogVLM on your own hardware using Docker.
* [Infinity](https://github.com/michaelfeil/infinity) ![](https://img.shields.io/github/stars/michaelfeil/infinity.svg?cacheSeconds=86400) - Infinity is a high-throughput, low-latency REST API for serving text-embeddings, reranking models and clip.
* [IPEX-LLM](https://github.com/intel/ipex-llm) ![](https://img.shields.io/github/stars/intel/ipex-llm.svg?cacheSeconds=86400) - IPEX-LLM is a PyTorch library for running LLM on Intel CPU and GPU (e.g., local PC with iGPU, discrete GPU such as Arc, Flex and Max) with very low latency.
* [LiteLLM](https://github.com/BerriAI/litellm) ![](https://img.shields.io/github/stars/BerriAI/litellm.svg?cacheSeconds=86400) - LiteLLM is a Python SDK, Proxy Server (LLM Gateway) to call 100+ LLM APIs in OpenAI format - Bedrock, Azure, OpenAI, VertexAI, Cohere, Anthropic, Sagemaker, HuggingFace, Replicate, Groq.
* [LitServe](https://github.com/Lightning-AI/LitServe) ![](https://img.shields.io/github/stars/Lightning-AI/LitServe.svg?cacheSeconds=86400) - LitServe is a flexible serving engine for AI models built on FastAPI. It supports custom inference engines for models, agents, multi-modal systems, RAG, and complex ML pipelines.
* [Jina-serve](https://github.com/jina-ai/serve) ![](https://img.shields.io/github/stars/jina-ai/serve.svg?cacheSeconds=86400) - Jina-serve is a framework for building and deploying AI services that communicate via gRPC, HTTP and WebSockets.
* [Kiln](https://github.com/kiln-ai/kiln) ![](https://img.shields.io/github/stars/kiln-ai/kiln.svg?cacheSeconds=86400) - Kiln is an OSS tool for fine-tuning LLM models, synthetic data generation, and collaborating on datasets.
* [KServe](https://github.com/kserve/kserve) ![](https://img.shields.io/github/stars/kserve/kserve.svg?cacheSeconds=86400) - KServe provides a Kubernetes Custom Resource Definition for serving predictive and generative ML.
* [KTransformers](https://github.com/kvcache-ai/ktransformers) ![](https://img.shields.io/github/stars/kvcache-ai/ktransformers.svg?cacheSeconds=86400) - KTransformers is a flexible framework for experiencing cutting-edge LLM inference optimizations.
* [Langtrace](https://github.com/Sca