Containerized Shader Compilers
in Pipeline on Docker, Shaders, Linux Last modified at:
When targetting multiple graphics APIs, a large number of compilers and tools are used. Versioning, deploying and updating all these binaries can be a tedious and complex process, especially when using them with multiple platforms.
In previous posts, I discussed containerizing Microsoft DXC (Linux), Microsoft FXC (Windows under Wine), and signing DXIL post compile (custom tool that loads dxil.dll under Wine).
With the goal of having a shader build service containerized within a Docker container, and scaling it out in a Kubernetes cluster (like GKE), having a single image containing the ability to run all necessary compilers and tools is a critical component.
These are the compilers and tools that are of interest:
- Microsoft DXC
- Compiles HLSL to DXIL or SPIR-V
- Supports shader model 6.X
- Native Linux support (unsigned DXIL)
- DirectX ray tracing
- Microsoft FXC
- Compiles HLSL to DXBC
- Supports shader model 5.X
- Windows binary only
- Glslc (glslang)
- Compiles HLSL or GLSL to SPIR-V
- NV mesh shaders (GLSL only, currently)
- NV Vulkan ray tracing (GLSL only, currently)
- AMD Radeon Graphics Analyzer (RGA)
- Compiler and performance analysis tool
- Generate and analyze GCN ISA disassembly
- Windows and Linux binaries
- Windows version has DirectX support
- Custom DXIL signing tool
- Needed to sign DXIL post-compile or with Linux
- Windows binary (compiled from source)
- SMOL-V
- SPIR-V size reduction
- Vulkan SDK
- spirv-cross
- spirv-as
- spirv-dis
- …
Wine Patches
As the container is Linux (not Windows), any of the Windows binaries must be run using Wine:
- Microsoft FXC
- AMD RGA (Windows version)
- Custom DXIL signing tool
Running FXC and RGA with Wine worked out of the box, but in a previous post, I described some fatal errors when dxc.exe or the custom sign tool would call into dxil.dll: 
I submitted some bug reports to WineHQ and Fabian Maurer graciously debugged and submitted patches to Wine that resolve the dxil.dll issues - (155381, 155382). Wine is currently in code lockdown for a new release, but these patches are expected to reach mainline in a few weeks. For the time being, I applied these patches to a custom fork of Wine for use in this Docker image.
Initial Version
The initial version of the all-inclusive compiler Docker file is shown here - all tools are downloaded, compiled, and installed directly into a single massive image.
FROM ubuntu:bionic
WORKDIR /app/dxc
ENV VULKAN_SDK=1.1.92
RUN apt-get update && \
apt-get install -y \
software-properties-common \
pkg-config \
build-essential \
unzip \
wget \
curl \
git \
cmake \
ninja-build \
python \
flex \
bison \
libpng-dev \
&& wget -qO - http://packages.lunarg.com/lunarg-signing-key-pub.asc | apt-key add - \
&& wget -qO /etc/apt/sources.list.d/lunarg-vulkan-${VULKAN_SDK}-bionic.list \
http://packages.lunarg.com/vulkan/${VULKAN_SDK}/lunarg-vulkan-${VULKAN_SDK}-bionic.list \
&& apt update && apt install -y lunarg-vulkan-sdk
# Download and build DXC
ENV DXC_BRANCH=master
ENV DXC_REPO=https://github.com/Microsoft/DirectXShaderCompiler.git
ENV DXC_COMMIT=cd237f5c3f7e8390fafff122333423afe55bc6c7
RUN git clone --recurse-submodules -b ${DXC_BRANCH} ${DXC_REPO} /app/dxc && \
git checkout ${DXC_COMMIT} && \
git reset --hard
RUN mkdir -p /app/dxc/build && cd /app/dxc/build && \
cmake ../ -GNinja -DCMAKE_BUILD_TYPE=Release $(cat ../utils/cmake-predefined-config-params) && \
ninja
# Download and build shaderc/glsang
ENV SHADERC_BRANCH=master
ENV SHADERC_REPO=https://github.com/google/shaderc.git
ENV SHADERC_COMMIT=53c776f776821bc037b31b8b3b79db2fa54b4ce7
ENV GOOGLE_TEST_BRANCH=master
ENV GOOGLE_TEST_REPO=https://github.com/google/googletest.git
ENV GOOGLE_TEST_COMMIT=c6cb7e033591528a5fe2c63157a0d8ce927740dc
ENV GLSLANG_BRANCH=master
ENV GLSLANG_REPO=https://github.com/google/glslang.git
ENV GLSLANG_COMMIT=667506a5eae80931290c2f424888cc5f52fec5d1
ENV SPV_TOOLS_BRANCH=master
ENV SPV_TOOLS_REPO=https://github.com/KhronosGroup/SPIRV-Tools.git
ENV SPV_TOOLS_COMMIT=c512c6864080ff617afb422a3d04dd902809a6cf
ENV SPV_HEADERS_BRANCH=master
ENV SPV_HEADERS_REPO=https://github.com/KhronosGroup/SPIRV-Headers.git
ENV SPV_HEADERS_COMMIT=17da9f8231f78cf519b4958c2229463a63ead9e2
ENV RE2_BRANCH=master
ENV RE2_REPO=https://github.com/google/re2.git
ENV RE2_COMMIT=fadc34500b67414df452c54d980372242f3d7f57
ENV EFFCEE_BRANCH=master
ENV EFFCEE_REPO=https://github.com/google/effcee.git
ENV EFFCEE_COMMIT=8f0a61dc95e0df18c18e0ac56d83b3fa9d2fe90b
# Download shaderc repository
WORKDIR /app/shaderc
RUN git clone --recurse-submodules -b ${SHADERC_BRANCH} ${SHADERC_REPO} /app/shaderc && \
git checkout ${SHADERC_COMMIT} && git reset --hard
# Download shaderc dependencies
WORKDIR /app/shaderc/third_party
RUN git clone --recurse-submodules -b ${GOOGLE_TEST_BRANCH} ${GOOGLE_TEST_REPO} googletest && \
cd googletest && git checkout ${GOOGLE_TEST_COMMIT} && git reset --hard && cd ..
RUN git clone --recurse-submodules -b ${GLSLANG_BRANCH} ${GLSLANG_REPO} glslang && \
cd glslang && git checkout ${GLSLANG_COMMIT} && git reset --hard && cd ..
RUN git clone --recurse-submodules -b ${SPV_TOOLS_BRANCH} ${SPV_TOOLS_REPO} spirv-tools && \
cd spirv-tools && git checkout ${SPV_TOOLS_COMMIT} && git reset --hard && cd ..
RUN git clone --recurse-submodules -b ${SPV_HEADERS_BRANCH} ${SPV_HEADERS_REPO} spirv-headers && \
cd spirv-headers && git checkout ${SPV_HEADERS_COMMIT} && git reset --hard && cd ..
RUN git clone --recurse-submodules -b ${RE2_BRANCH} ${RE2_REPO} re2 && \
cd re2 && git checkout ${RE2_COMMIT} && git reset --hard && cd ..
RUN git clone --recurse-submodules -b ${EFFCEE_BRANCH} ${EFFCEE_REPO} effcee && \
cd effcee && git checkout ${EFFCEE_COMMIT} && git reset --hard && cd ..
# Build shaderc
WORKDIR /app/shaderc/build
RUN cmake -GNinja \
-DCMAKE_BUILD_TYPE=Release \
-DCMAKE_INSTALL_PREFIX=/usr/local \
.. \
&& ninja install
# Download and build SMOL-V
ENV SMOLV_BRANCH=master
ENV SMOLV_REPO=https://github.com/aras-p/smol-v.git
ENV SMOLV_COMMIT=9a787d1354a9e43c9ea6027cd310ce2a2fd78901
WORKDIR /app/smol-v
RUN git clone --recurse-submodules -b ${SMOLV_BRANCH} ${SMOLV_REPO} /app/smol-v && \
git checkout ${SMOLV_COMMIT} && git reset --hard && \
make -f projects/Makefile -j 4
WORKDIR /app
RUN wget -O signing.zip https://github.com/gwihlidal/dxil-signing/releases/download/0.1.2/dxil-signing-0_1_2.zip
RUN unzip -q signing.zip; exit 0
RUN mv dxil-signing-0_1_2 signing && rm -f signing.zip
# Download and install wine (for running FXC, DXIL signing tool, RGA for Windows)
ENV WINE_BRANCH=dxil
ENV WINE_REPO=https://github.com/gwihlidal/wine.git
ENV WINE_COMMIT=4777a57d8a5fd2c0aa0ba06abb9148f77b9c2ddf
WORKDIR /wine
RUN git clone --recurse-submodules -b ${WINE_BRANCH} ${WINE_REPO} /wine && \
git checkout ${WINE_COMMIT} && \
git reset --hard
RUN ./configure --enable-win64 --with-png --without-freetype && \
make -j8 && \
make install
ENV WINEARCH=win64
ENV WINEDEBUG=fixme-all
RUN winecfg
# Copy FXC binaries into container
WORKDIR /app/fxc
COPY fxc_bin /app/fxc
# Download Linux and Windows binaries of AMD RGA
WORKDIR /app/rga
RUN wget -O rga_linux.tgz https://github.com/GPUOpen-Tools/RGA/releases/download/2.0.1/rga-linux-2.0.1.tgz && \
tar zxf rga_linux.tgz && \
mv rga-2.0.1.* linux && \
rm rga_linux.tgz
RUN wget -O rga_windows.zip https://github.com/GPUOpen-Tools/RGA/releases/download/2.0.1/rga-windows-x64-2.0.1.zip
RUN unzip -q rga_windows.zip; exit 0
RUN mv bin windows && rm -f /app/rga/rga_windows.zip
# Convenient path variables
ENV DXC_PATH="/app/dxc/build/bin/dxc"
ENV FXC_PATH="/app/fxc/fxc.exe"
ENV SIGN_PATH="/app/signing/dxil-signing.exe"
ENV RGA_WIN_PATH="/app/rga/windows/rga.exe"
ENV RGA_NIX_PATH="/app/rga/linux/rga"
ENV GLSLC_PATH="/app/shaderc/build/glslc/glslc"
ENV SMOLV_PATH="/app/smol-v/smolv"
WORKDIR /app
ENTRYPOINT ["/bin/bash"]
The image successfully builds and works as expected, but listing the image shows it is a hefty 5.13GB!
$ docker images
REPOSITORY TAG IMAGE ID SIZE
gwihlidal/docker-shader unoptimized 10e54d5e5be1 5.13GB
But what is attributing to the size? You can use the docker history command to list all layers in a Docker image, which includes all the intermediate layers from the build process.
$ docker history gwihlidal/docker-shader:unoptimized
IMAGE CREATED BY SIZE
0a08bd5f62ac /bin/sh -c #(nop) ENTRYPOINT ["/bin/bash"] 0B
3322f44b8605 2 hours agoo /bin/sh -c #(nop) WORKDIR /app 0B
bd89cff6176d /bin/sh -c #(nop) ENV SMOLV_PATH=/app/smol-… 0B
29182233f7b4 /bin/sh -c #(nop) ENV GLSLC_PATH=/app/shade… 0B
ec23246a56ac /bin/sh -c #(nop) ENV RGA_NIX_PATH=/app/rga… 0B
5fffc9199d59 /bin/sh -c #(nop) ENV RGA_WIN_PATH=/app/rga… 0B
9e310abca099 /bin/sh -c #(nop) ENV SIGN_PATH=/app/signin… 0B
b32801e5b3fb /bin/sh -c #(nop) ENV FXC_PATH=/app/fxc/fxc… 0B
f98f96f64aca /bin/sh -c #(nop) ENV DXC_PATH=/app/dxc/bui… 0B
a94e7441d45c /bin/sh -c mv bin windows && rm -f /app/rga/… 141MB
0c7b20423129 /bin/sh -c unzip -q rga_windows.zip; exit 0 141MB
69d110b75297 /bin/sh -c wget -O rga_windows.zip https://g… 58MB
9222b7587f70 /bin/sh -c wget -O rga_linux.tgz https://git… 217MB
819763acda4b /bin/sh -c #(nop) WORKDIR /app/rga 0B
4d8fd2fdabea /bin/sh -c #(nop) COPY dir:de73f1abfa6840418… 4.64MB
fe50b9bab19c /bin/sh -c #(nop) WORKDIR /app/fxc 0B
9509d060a126 /bin/sh -c winecfg 17MB
f212229e801a /bin/sh -c #(nop) ENV WINEDEBUG=fixme-all 0B
98842b9a0b3e /bin/sh -c #(nop) ENV WINEARCH=win64 0B
c0f23b1fe3c8 /bin/sh -c ./configure --enable-win64 --with… 2.02GB
5a25b2ca97c8 /bin/sh -c git clone --recurse-submodules -b… 765MB
5fa7d84fdc04 /bin/sh -c #(nop) WORKDIR /wine 0B
edcf7080c1f8 /bin/sh -c #(nop) ENV WINE_COMMIT=4777a57d8… 0B
90a2792b5223 /bin/sh -c #(nop) ENV WINE_REPO=https://git… 0B
85b67cdaa3c0 /bin/sh -c #(nop) ENV WINE_BRANCH=dxil 0B
3ec6009ec972 /bin/sh -c mv dxil-signing-0_1_2 signing && … 13.7MB
04f6e5347337 /bin/sh -c unzip -q signing.zip; exit 0 13.7MB
cb93ad3461e3 /bin/sh -c wget -O signing.zip https://githu… 6.29MB
53dbaa57d626 /bin/sh -c #(nop) WORKDIR /app 0B
53ea7074364c /bin/sh -c git clone --recurse-submodules -b… 9.67MB
6a0f973ee1d1 /bin/sh -c #(nop) WORKDIR /app/smol-v 0B
2055ded059af /bin/sh -c #(nop) ENV SMOLV_COMMIT=9a787d13… 0B
0b302e7c271a /bin/sh -c #(nop) ENV SMOLV_REPO=https://gi… 0B
fc210db9dadf /bin/sh -c #(nop) ENV SMOLV_BRANCH=master 0B
dcbda32f8ee8 /bin/sh -c cmake -GNinja -DCMAKE_BUILD_T… 415MB
58c76a77f3a8 /bin/sh -c #(nop) WORKDIR /app/shaderc/build 0B
4f1c8e658e81 /bin/sh -c git clone --recurse-submodules -b… 245kB
1120ceea2a0e /bin/sh -c git clone --recurse-submodules -b… 3.68MB
7574130b55cc /bin/sh -c git clone --recurse-submodules -b… 2.74MB
dbdc8f92523f /bin/sh -c git clone --recurse-submodules -b… 21.2MB
68f5c1bffcea /bin/sh -c git clone --recurse-submodules -b… 70.5MB
eca2ea2ef379 /bin/sh -c git clone --recurse-submodules -b… 11MB
a59b0f628fac /bin/sh -c #(nop) WORKDIR /app/shaderc/third… 0B
ef053de23a57 /bin/sh -c git clone --recurse-submodules -b… 2.45MB
c5093478f95e /bin/sh -c #(nop) WORKDIR /app/shaderc 0B
e02f79324bad /bin/sh -c #(nop) ENV EFFCEE_COMMIT=8f0a61d… 0B
dbeadcb25487 /bin/sh -c #(nop) ENV EFFCEE_REPO=https://g… 0B
41ec6c8abe39 /bin/sh -c #(nop) ENV EFFCEE_BRANCH=master 0B
836b6c4a985c /bin/sh -c #(nop) ENV RE2_COMMIT=fadc34500b… 0B
fe62483b6425 /bin/sh -c #(nop) ENV RE2_REPO=https://gith… 0B
21054004573b /bin/sh -c #(nop) ENV RE2_BRANCH=master 0B
2083aba8217a /bin/sh -c #(nop) ENV SPV_HEADERS_COMMIT=17… 0B
f45a7205f3c0 /bin/sh -c #(nop) ENV SPV_HEADERS_REPO=http… 0B
3cc10b40b93b /bin/sh -c #(nop) ENV SPV_HEADERS_BRANCH=ma… 0B
10362e8ae16b /bin/sh -c #(nop) ENV SPV_TOOLS_COMMIT=c512… 0B
60caf847f487 /bin/sh -c #(nop) ENV SPV_TOOLS_REPO=https:… 0B
acdfa5abc624 /bin/sh -c #(nop) ENV SPV_TOOLS_BRANCH=mast… 0B
e2aad7919b4a /bin/sh -c #(nop) ENV GLSLANG_COMMIT=667506… 0B
42eaf14e520b /bin/sh -c #(nop) ENV GLSLANG_REPO=https://… 0B
7e124fafe40e /bin/sh -c #(nop) ENV GLSLANG_BRANCH=master 0B
0fd16826a92a /bin/sh -c #(nop) ENV GOOGLE_TEST_COMMIT=c6… 0B
7ebe8cc8e884 /bin/sh -c #(nop) ENV GOOGLE_TEST_REPO=http… 0B
66f91486db5c /bin/sh -c #(nop) ENV GOOGLE_TEST_BRANCH=ma… 0B
2454aefa1e92 /bin/sh -c #(nop) ENV SHADERC_COMMIT=53c776… 0B
410bf0ecfcaa /bin/sh -c #(nop) ENV SHADERC_REPO=https://… 0B
0d8aa769246c /bin/sh -c #(nop) ENV SHADERC_BRANCH=master 0B
da5e72628bae /bin/sh -c mkdir -p /app/dxc/build && cd /ap… 214MB
10d07ad72cc0 /bin/sh -c git clone --recurse-submodules -b… 190MB
0419b46a5774 /bin/sh -c #(nop) ENV DXC_COMMIT=cd237f5c3f… 0B
6fcc725f1e24 /bin/sh -c #(nop) ENV DXC_REPO=https://gith… 0B
5661a62bfaef /bin/sh -c #(nop) ENV DXC_BRANCH=master 0B
71ac5a1f73e1 /bin/sh -c apt-get update && apt-get instal… 711MB
2a91fc85bce9 /bin/sh -c #(nop) ENV VULKAN_SDK=1.1.92 0B
6ec4d6e0623a /bin/sh -c #(nop) WORKDIR /app/dxc 0B
93fd78260bd1 /bin/sh -c #(nop) CMD ["/bin/bash"] 0B
<missing> /bin/sh -c mkdir -p /run/systemd && echo 'do… 7B
<missing> /bin/sh -c rm -rf /var/lib/apt/lists/* 0B
<missing> /bin/sh -c set -xe && echo '#!/bin/sh' > /… 745B
<missing> /bin/sh -c #(nop) ADD file:39e5bc157a8be63bb… 86.2MB
Reducing Image Size
A large proportion of the total size comes from all build tools, dependencies, source, and intermediate files for building the compilers. There are some common approaches we can use to shrink the Docker image.
One trick is to the clear the package manager cache. For Ubuntu/Debian systems, this is done by adding the following lines to our apt install chain:
&& apt-get clean \
&& apt-get autoremove
In addition, a package target in Ubuntu has a set of required, recommended, and suggested packages. Installing the required packages is mandatory, since they represent dependencies, but the other two package types can be skipped, since they are not essential to the correct functioning of the package being installed. Because of this, we can add --no-install-recommends to apt install -y in order to disable the recommended and suggested packages during installation, which saves additional space.
Another trick is to reduce the total number of layers in the Docker file. Each line in the Docker file functions as a step in the build process, which takes up space. Multiple RUN commands can be merged into one.
Instead of separate RUN command lines:
RUN git clone --recurse-submodules -b ${GOOGLE_TEST_BRANCH} ${GOOGLE_TEST_REPO} googletest && \
cd googletest && git checkout ${GOOGLE_TEST_COMMIT} && git reset --hard && cd ..
RUN git clone --recurse-submodules -b ${GLSLANG_BRANCH} ${GLSLANG_REPO} glslang && \
cd glslang && git checkout ${GLSLANG_COMMIT} && git reset --hard && cd ..
These lines can be merged together into a single RUN command line:
RUN git clone --recurse-submodules -b ${GOOGLE_TEST_BRANCH} ${GOOGLE_TEST_REPO} googletest && \
cd googletest && git checkout ${GOOGLE_TEST_COMMIT} && git reset --hard && cd .. && \
git clone --recurse-submodules -b ${GLSLANG_BRANCH} ${GLSLANG_REPO} glslang && \
cd glslang && git checkout ${GLSLANG_COMMIT} && git reset --hard && cd ..
If running on Google Cloud Platform or AWS, you can use ubuntu-minimal as a base image (~29mb), which use optimized kernels for cloud hypervisors (50% smaller, and boot 40% faster).
When compiling binaries from source which produce lots of intermediate build files, the most important trick is to split the Docker file into a multi-stage build.
Essentially we can have one image that produces the binaries, and then a clean image can copy just the relevant components from the builder image into the final image.
FROM ubuntu:bionic as builder
# ... do stuff that produces /build/my_binary
# Start from a new image
FROM ubuntu:bionic
# Copy `my_binary` from `builder` stage into final stage
WORKDIR /app
COPY --from=builder /build/my_binary /app/my_binary
You can also try linting a Docker file with FromLatest.io, which can offer additional insight for some improvements. Our optimized image looks good, though! 
Optimized Version
Here is the optimized version after applying these techniques, which includes an explicit build of the Vulkan SDK from source in order to easily update to the latest versions:
[GitHub] - [DockerHub: gwihlidal/docker-shader]
FROM ubuntu:bionic as builder
ENV DXC_BRANCH=master
ENV DXC_REPO=https://github.com/Microsoft/DirectXShaderCompiler.git
ENV DXC_COMMIT=cd237f5c3f7e8390fafff122333423afe55bc6c7
ENV SHADERC_BRANCH=master
ENV SHADERC_REPO=https://github.com/google/shaderc.git
ENV SHADERC_COMMIT=53c776f776821bc037b31b8b3b79db2fa54b4ce7
ENV GOOGLE_TEST_BRANCH=master
ENV GOOGLE_TEST_REPO=https://github.com/google/googletest.git
ENV GOOGLE_TEST_COMMIT=c6cb7e033591528a5fe2c63157a0d8ce927740dc
ENV GLSLANG_BRANCH=master
ENV GLSLANG_REPO=https://github.com/google/glslang.git
ENV GLSLANG_COMMIT=667506a5eae80931290c2f424888cc5f52fec5d1
ENV SPV_TOOLS_BRANCH=master
ENV SPV_TOOLS_REPO=https://github.com/KhronosGroup/SPIRV-Tools.git
ENV SPV_TOOLS_COMMIT=c512c6864080ff617afb422a3d04dd902809a6cf
ENV SPV_HEADERS_BRANCH=master
ENV SPV_HEADERS_REPO=https://github.com/KhronosGroup/SPIRV-Headers.git
ENV SPV_HEADERS_COMMIT=17da9f8231f78cf519b4958c2229463a63ead9e2
ENV RE2_BRANCH=master
ENV RE2_REPO=https://github.com/google/re2.git
ENV RE2_COMMIT=fadc34500b67414df452c54d980372242f3d7f57
ENV EFFCEE_BRANCH=master
ENV EFFCEE_REPO=https://github.com/google/effcee.git
ENV EFFCEE_COMMIT=8f0a61dc95e0df18c18e0ac56d83b3fa9d2fe90b
ENV WINE_BRANCH=dxil
ENV WINE_REPO=https://github.com/gwihlidal/wine.git
ENV WINE_COMMIT=4777a57d8a5fd2c0aa0ba06abb9148f77b9c2ddf
ENV SMOLV_BRANCH=master
ENV SMOLV_REPO=https://github.com/aras-p/smol-v.git
ENV SMOLV_COMMIT=9a787d1354a9e43c9ea6027cd310ce2a2fd78901
ENV VULKAN_SDK=1.1.92.1
# Prevents annoying debconf errors during builds
ARG DEBIAN_FRONTEND="noninteractive"
# Download libraries and tools
RUN apt-get update && \
apt-get install -y \
software-properties-common \
build-essential \
git \
cmake \
ninja-build \
python \
wget \
unzip \
flex \
bison \
libpng-dev \
libwayland-dev \
libx11-dev \
libxrandr-dev \
&& apt autoremove -y \
software-properties-common \
&& apt autoclean \
&& apt clean \
&& apt autoremove
# Download and build DXC
RUN git clone --recurse-submodules -b ${DXC_BRANCH} ${DXC_REPO} /dxc && cd /dxc \
git checkout ${DXC_COMMIT} && \
git reset --hard && \
mkdir -p /dxc/build && cd /dxc/build && \
cmake ../ -GNinja -DCMAKE_BUILD_TYPE=Release $(cat ../utils/cmake-predefined-config-params) && \
ninja
# Download shaderc repository and dependencies
RUN git clone --recurse-submodules -b ${SHADERC_BRANCH} ${SHADERC_REPO} /shaderc && cd /shaderc \
git checkout ${SHADERC_COMMIT} && git reset --hard && \
mkdir -p /shaderc/third_party && cd /shaderc/third_party && \
git clone --recurse-submodules -b ${GOOGLE_TEST_BRANCH} ${GOOGLE_TEST_REPO} googletest && \
cd googletest && git checkout ${GOOGLE_TEST_COMMIT} && git reset --hard && cd .. && \
git clone --recurse-submodules -b ${GLSLANG_BRANCH} ${GLSLANG_REPO} glslang && \
cd glslang && git checkout ${GLSLANG_COMMIT} && git reset --hard && cd .. && \
git clone --recurse-submodules -b ${SPV_TOOLS_BRANCH} ${SPV_TOOLS_REPO} spirv-tools && \
cd spirv-tools && git checkout ${SPV_TOOLS_COMMIT} && git reset --hard && cd .. && \
git clone --recurse-submodules -b ${SPV_HEADERS_BRANCH} ${SPV_HEADERS_REPO} spirv-headers && \
cd spirv-headers && git checkout ${SPV_HEADERS_COMMIT} && git reset --hard && cd .. && \
git clone --recurse-submodules -b ${RE2_BRANCH} ${RE2_REPO} re2 && \
cd re2 && git checkout ${RE2_COMMIT} && git reset --hard && cd .. && \
git clone --recurse-submodules -b ${EFFCEE_BRANCH} ${EFFCEE_REPO} effcee && \
cd effcee && git checkout ${EFFCEE_COMMIT} && git reset --hard && cd ..
# Build shaderc
RUN mkdir -p /shaderc/build && cd /shaderc/build && \
cmake -GNinja \
-DCMAKE_BUILD_TYPE=Release \
-DCMAKE_INSTALL_PREFIX=/usr/local \
.. && \
ninja install
# Download and build SMOL-V
WORKDIR /smol-v
RUN git clone --recurse-submodules -b ${SMOLV_BRANCH} ${SMOLV_REPO} /app/smol-v && cd /app/smol-v && \
git checkout ${SMOLV_COMMIT} && git reset --hard && \
make -f projects/Makefile -j 4
# Download and install Wine (for running FXC, DXIL signing tool, RGA for Windows)
WORKDIR /wine_src
RUN git clone --recurse-submodules -b ${WINE_BRANCH} ${WINE_REPO} /wine_src && \
git checkout ${WINE_COMMIT} && \
git reset --hard && \
./configure --enable-win64 --with-png --without-freetype --without-x --prefix=/wine && \
make -j8 && \
make install
# Download and build Vulkan SDK
WORKDIR /
RUN wget -O vulkan.tgz https://sdk.lunarg.com/sdk/download/${VULKAN_SDK}/linux/vulkansdk-linux-x86_64-${VULKAN_SDK}.tar.gz && \
tar zxf vulkan.tgz && \
mv ${VULKAN_SDK} vulkan && \
rm vulkan.tgz && \
cd /vulkan && \
chmod +x setup-env.sh && \
chmod +x build_tools.sh && \
./setup-env.sh && ./build_tools.sh
# Download and extract signing tool
WORKDIR /
RUN wget -O signing.zip https://github.com/gwihlidal/dxil-signing/releases/download/0.1.2/dxil-signing-0_1_2.zip && \
unzip -q signing.zip; exit 0
RUN mv dxil-signing-0_1_2 signing
# Download and extract Linux and Windows binaries of AMD RGA
WORKDIR /rga
RUN wget -O rga_linux.tgz https://github.com/GPUOpen-Tools/RGA/releases/download/2.0.1/rga-linux-2.0.1.tgz && \
tar zxf rga_linux.tgz && \
mv rga-2.0.1.* linux && \
rm rga_linux.tgz && \
wget -O rga_windows.zip https://github.com/GPUOpen-Tools/RGA/releases/download/2.0.1/rga-windows-x64-2.0.1.zip && \
unzip -q rga_windows.zip; exit 0
# Remove RGA GUI binaries
RUN mv bin windows && \
rm -f /rga/rga_windows.zip && \
rm -f /rga/windows/Qt* && \
rm -f /rga/windows/RadeonGPUAnalyzerGUI.exe && \
rm -fr /rga/windows/iconengines && \
rm -fr /rga/windows/imageformats && \
rm -fr /rga/windows/platforms && \
rm -fr /rga/linux/Qt && \
rm -fr /rga/linux/Documentation && \
rm -f /rga/linux/RadeonGPUAnalyzerGUI-bin && \
rm -f /rga/linux/RadeonGPUAnalyzerGUI
# Start from a new image
FROM ubuntu:bionic
# Install libpng (needed for Wine)
RUN apt update && \
apt install --no-install-recommends -y \
libpng-dev \
&& apt-get clean \
&& apt-get autoremove
# Copy DXC binaries from `builder` stage into final stage
WORKDIR /app/dxc
COPY --from=builder /dxc/build/bin/dxc /app/dxc/bin/dxc
COPY --from=builder /dxc/build/lib/libdxcompiler.so.3.7 /app/dxc/lib/libdxcompiler.so.3.7
RUN ln -s /dxc/lib/libdxcompiler.so.3.7 /app/dxc/lib/libdxcompiler.so
# Copy glslc binary from `builder` stage into final stage
WORKDIR /app/shaderc
COPY --from=builder /shaderc/build/glslc/glslc /app/shaderc/glslc
# Copy SMOL-V binaries from `builder` stage into final stage
WORKDIR /app/smol-v
COPY --from=builder /app/smol-v /app/smol-v
# Copy Vulkan install binaries from `builder` stage into final stage
WORKDIR /app/vulkan
COPY --from=builder /vulkan/x86_64/bin /app/vulkan
# Copy Wine install from `builder` stage into final stage
WORKDIR /app/wine
COPY --from=builder /wine /app/wine
# Copy DXIL signing binaries from `builder` stage into final stage
WORKDIR /app/signing
COPY --from=builder /signing /app/signing
# Copy RGA binaries from `builder` stage into final stage
WORKDIR /app/rga
COPY --from=builder /rga /app/rga
# Copy local FXC binaries into container
WORKDIR /app/fxc
COPY fxc_bin /app/fxc
# Convenient path variables
ENV DXC_PATH="/app/dxc/bin/dxc"
ENV FXC_PATH="/app/fxc/fxc.exe"
ENV SIGN_PATH="/app/signing/dxil-signing.exe"
ENV RGA_WIN_PATH="/app/rga/windows/rga.exe"
ENV RGA_NIX_PATH="/app/rga/linux/rga"
ENV GLSLC_PATH="/app/shaderc/glslc"
ENV SMOLV_PATH="/app/smol-v/smolv"
ENV WINE_PATH="/app/wine/bin/wine64"
ENV VULKAN_PATH="/app/vulkan"
# Configuration of Wine
ENV WINEARCH=win64
ENV WINEDEBUG=fixme-all
RUN /app/wine/bin/winecfg
WORKDIR /app
ENTRYPOINT ["/bin/bash"]
After reworking the Docker file into a multi-stage build, collapsing down multiple layers, and cleaning up temporary files, the optimized size is much better.
$ docker images
REPOSITORY TAG IMAGE ID SIZE
gwihlidal/docker-shader optimized 801b435a1301 1.14GB
Again, we can list the image history to see where the majority of the size is going.
% docker history gwihlidal/docker-shader:optimized
IMAGE CREATED BY SIZE
801b435a1301 /bin/sh -c #(nop) ENTRYPOINT ["/bin/bash"] 0B
33dfbc14f72a /bin/sh -c #(nop) WORKDIR /app 0B
b0b27a78e23f /bin/sh -c /app/wine/bin/winecfg 17MB
7088ed2a829b /bin/sh -c #(nop) ENV WINEDEBUG=fixme-all 0B
efbddc393129 /bin/sh -c #(nop) ENV WINEARCH=win64 0B
d6664f56affc /bin/sh -c #(nop) ENV VULKAN_PATH=/app/vulk… 0B
d4f1543c99d6 /bin/sh -c #(nop) ENV WINE_PATH=/app/wine/b… 0B
524ce0c1987f /bin/sh -c #(nop) ENV SMOLV_PATH=/app/smol-… 0B
b9b7524b18bd /bin/sh -c #(nop) ENV GLSLC_PATH=/app/shade… 0B
68df16240c76 /bin/sh -c #(nop) ENV RGA_NIX_PATH=/app/rga… 0B
55f02a3ac8bc /bin/sh -c #(nop) ENV RGA_WIN_PATH=/app/rga… 0B
78455cc2174c /bin/sh -c #(nop) ENV SIGN_PATH=/app/signin… 0B
dd7b67e869d0 /bin/sh -c #(nop) ENV FXC_PATH=/app/fxc/fxc… 0B
37bd8ebb3d3f /bin/sh -c #(nop) ENV DXC_PATH=/app/dxc/bin… 0B
08d9fee097be /bin/sh -c #(nop) COPY dir:a11dddb9151b9bd79… 4.64MB
2c11e63f6817 /bin/sh -c #(nop) WORKDIR /app/fxc 0B
25afe82f290b /bin/sh -c #(nop) COPY dir:0180b95284eb9b27b… 279MB
fcee61d49f65 /bin/sh -c #(nop) WORKDIR /app/rga 0B
f6cf29857fb1 /bin/sh -c #(nop) COPY dir:e9ad49937e85e3ae0… 13.7MB
6900590649b2 /bin/sh -c #(nop) WORKDIR /app/signing 0B
eb0f03625222 /bin/sh -c #(nop) COPY dir:9d900a0720f8b4751… 469MB
555236245873 /bin/sh -c #(nop) WORKDIR /app/wine 0B
c3024a092124 /bin/sh -c #(nop) COPY dir:4e3abc1afe3949db7… 171MB
acd269c9cbd7 /bin/sh -c #(nop) WORKDIR /app/vulkan 0B
3b00d576c493 /bin/sh -c #(nop) COPY dir:5421ce9f0a82643f7… 9.67MB
6aa11e8b27e2 /bin/sh -c #(nop) WORKDIR /app/smol-v 0B
8bb0bb4c9300 /bin/sh -c #(nop) COPY file:f44b94ec0a07bebe… 6.96MB
eeab5ff2535f /bin/sh -c #(nop) WORKDIR /app/shaderc 0B
280c30a2e7f4 /bin/sh -c ln -s /dxc/lib/libdxcompiler.so.3… 29B
02c8adb3f8a2 /bin/sh -c #(nop) COPY file:02e5ccdde34db51c… 33.6MB
dd1ef5d2f1d2 /bin/sh -c #(nop) COPY file:024bda7e343f7327… 553kB
ea7cfefa9092 /bin/sh -c #(nop) WORKDIR /app/dxc 0B
151cf9b3012b /bin/sh -c apt update && apt install --n… 37.6MB
e9e49a465deb /bin/sh -c #(nop) CMD ["bash"] 0B
<missing> /bin/sh -c #(nop) ADD file:dab9baf938799c515… 55.3MB
Based on this listing, the remaining size is largely inflated by Wine and extra binaries within RGA and the Vulkan SDK.
- For Wine, this is currently a full install, but it is possible to remove everything except the specific components that are needed to run FXC, RGA for Windows, and the DXIL signing tool.
- For RGA, only some of the extra binaries are deleted in this container, but there are likely many more files that could be deleted from the install.
- For Vulkan, this is another case of extra binaries that need to be removed.
These improvements are left as an exercise to the reader, or a later date when I get around to it :)
Here are the final results:
$ docker images
REPOSITORY TAG IMAGE ID SIZE
gwihlidal/docker-shader optimized 801b435a1301 1.14GB
gwihlidal/docker-shader unoptimized 10e54d5e5be1 5.13GB
As mentioned, further size reductions could be performed, but 1.14GB isn’t too bad considering all the packages and tooling that are included in this image.
Example Usage
The optimized Docker image has been published as gwihlidal/docker-shader to DockerHub.
A container instance can be easily launched from this image with the following command:
$ docker run --rm -it gwihlidal/docker-shader
root@b31fc74f94b8:/app#
If you want to access local file system resources without needing to copy them into the image, you can bind mount your current working directory like this:
$ docker run --rm -it -v $(pwd):/shaders -w /shaders gwihlidal/docker-shader
root@517a5031d89f:/shaders#
All files in your working directory will be available within the running container at the /shaders mount point.
A collection of environment variables are exported to simplify invoking each compiler or tool:
- $DXC_PATH
- $FXC_PATH
- $SIGN_PATH
- $RGA_WIN_PATH
- $RGA_NIX_PATH
- $GLSLC_PATH
- $SMOLV_PATH
- $WINE_PATH
- $VULKAN_PATH
Microsoft DXC
Invoking DXC:
root@b31fc74f94b8:/app# $DXC_PATH /help
OVERVIEW: HLSL Compiler
Version:
USAGE: dxc.exe [options] <inputs>
Common Options:
-help Display available options
-nologo Suppress copyright message
-Qunused-arguments Don't emit warning for unused driver arguments
Compilation Options:
-all_resources_bound Enables agressive flattening
-auto-binding-space <value>
Set auto binding space - enables auto resource binding in libraries
-Cc Output color coded assembly listings
-default-linkage <value>
Set default linkage for non-shader functions when compiling or linking to a library target (internal, external)
-denorm <value> select denormal value options (any, preserve, ftz). any is the default.
-D <value> Define macro
-enable-16bit-types Enable 16bit types and disable min precision types. Available in HLSL 2018 and shader model 6.2
-export-shaders-only Only export shaders when compiling a library
-exports <value> Specify exports when compiling a library: export1[[,export1_clone,...]=internal_name][;...]
-E <value> Entry point name
-Fc <file> Output assembly code listing file
-Fd <file> Write debug information to the given file or directory; trail \ to auto-generate and imply Qstrip_priv
-Fe <file> Output warnings and errors to the given file
-Fh <file> Output header file containing object code
-flegacy-macro-expansion
Expand the operands before performing token-pasting operation (fxc behavior)
-flegacy-resource-reservation
Reserve unused explicit register assignments for compatibility with shader model 5.0 and below
-force_rootsig_ver <profile>
force root signature version (rootsig_1_1 if omitted)
-Fo <file> Output object file
-Gec Enable backward compatibility mode
-Ges Enable strict mode
-Gfa Avoid flow control constructs
-Gfp Prefer flow control constructs
-Gis Force IEEE strictness
-HV <value> HLSL version (2016, 2017, 2018). Default is 2018
-H Show header includes and nesting depth
-ignore-line-directives Ignore line directives
-I <value> Add directory to include search path
-Lx Output hexadecimal literals
-Ni Output instruction numbers in assembly listings
-no-warnings Suppress warnings
-not_use_legacy_cbuf_load
Do not use legacy cbuffer load
-No Output instruction byte offsets in assembly listings
-Odump Print the optimizer commands.
-Od Disable optimizations
-pack_optimized Optimize signature packing assuming identical signature provided for each connecting stage
-pack_prefix_stable (default) Pack signatures preserving prefix-stable property - appended elements will not disturb placement of prior elements
-recompile recompile from DXIL container with Debug Info or Debug Info bitcode file
-rootsig-define <value> Read root signature from a #define
-T <profile> Set target profile.
<profile>: ps_6_0, ps_6_1, ps_6_2, ps_6_3, ps_6_4,
vs_6_0, vs_6_1, vs_6_2, vs_6_3, vs_6_4,
cs_6_0, cs_6_1, cs_6_2, cs_6_3, cs_6_4,
gs_6_0, gs_6_1, gs_6_2, gs_6_3, gs_6_4,
ds_6_0, ds_6_1, ds_6_2, ds_6_3, ds_6_4,
hs_6_0, hs_6_1, hs_6_2, hs_6_3, hs_6_4,
lib_6_3, lib_6_4
-Vd Disable validation
-Vi Display details about the include process.
-Vn <name> Use <name> as variable name in header file
-WX Treat warnings as errors
-Zi Enable debug information
-Zpc Pack matrices in column-major order
-Zpr Pack matrices in row-major order
-Zsb Build debug name considering only output binary
-Zss Build debug name considering source information
Optimization Options:
-O0 Optimization Level 0
-O1 Optimization Level 1
-O2 Optimization Level 2
-O3 Optimization Level 3 (Default)
SPIR-V CodeGen Options:
-fspv-debug=<value> Specify whitelist of debug info category (file -> source -> line, tool)
-fspv-extension=<value> Specify SPIR-V extension permitted to use
-fspv-reflect Emit additional SPIR-V instructions to aid reflection
-fspv-target-env=<value>
Specify the target environment: vulkan1.0 (default) or vulkan1.1
-fvk-b-shift <shift> <space>
Specify Vulkan binding number shift for b-type register
-fvk-bind-register <type-number> <space> <binding> <set>
Specify Vulkan descriptor set and binding for a specific register
-fvk-invert-y Negate SV_Position.y before writing to stage output in VS/DS/GS to accommodate Vulkan's coordinate system
-fvk-s-shift <shift> <space>
Specify Vulkan binding number shift for s-type register
-fvk-t-shift <shift> <space>
Specify Vulkan binding number shift for t-type register
-fvk-u-shift <shift> <space>
Specify Vulkan binding number shift for u-type register
-fvk-use-dx-layout Use DirectX memory layout for Vulkan resources
-fvk-use-dx-position-w Reciprocate SV_Position.w after reading from stage input in PS to accommodate the difference between Vulkan and DirectX
-fvk-use-gl-layout Use strict OpenGL std140/std430 memory layout for Vulkan resources
-fvk-use-scalar-layout Use scalar memory layout for Vulkan resources
-Oconfig=<value> Specify a comma-separated list of SPIRV-Tools passes to customize optimization configuration (see http://khr.io/hlsl2spirv#optimization)
-spirv Generate SPIR-V code
Utility Options:
-dumpbin Load a binary file rather than compiling
-extractrootsignature Extract root signature from shader bytecode (must be used with /Fo <file>)
-getprivate <file> Save private data from shader blob
-P <value> Preprocess to file (must be used alone)
-Qstrip_debug Strip debug information from 4_0+ shader bytecode (must be used with /Fo <file>)
-Qstrip_priv Strip private data from shader bytecode (must be used with /Fo <file>)
-Qstrip_reflect Strip reflection data from shader bytecode (must be used with /Fo <file>)
-Qstrip_rootsignature Strip root signature data from shader bytecode (must be used with /Fo <file>)
-setprivate <file> Private data to add to compiled shader blob
-setrootsignature <file>
Attach root signature to shader bytecode
-verifyrootsignature <file>
Verify shader bytecode with root signature
Microsoft FXC
Invoking FXC:
root@b31fc74f94b8:/app# $WINE_PATH $FXC_PATH /?
Microsoft (R) Direct3D Shader Compiler 10.1
Copyright (C) 2013 Microsoft. All rights reserved.
Usage: fxc <options> <files>
/?, /help print this message
/T <profile> target profile
/E <name> entrypoint name
/I <include> additional include path
/Vi display details about the include process
/Od disable optimizations
/Op disable preshaders
/O{0,1,2,3} optimization level 0..3. 1 is default
/WX treat warnings as errors
/Vd disable validation
/Zi enable debugging information
/Zpr pack matrices in row-major order
/Zpc pack matrices in column-major order
/Gpp force partial precision
/Gfa avoid flow control constructs
/Gfp prefer flow control constructs
/Gdp disable effect performance mode
/Ges enable strict mode
/Gec enable backwards compatibility mode
/Gis force IEEE strictness
/Gch compile as a child effect for FX 4.x targets
/Fo <file> output object file
/Fl <file> output a library
/Fc <file> output assembly code listing file
/Fx <file> output assembly code and hex listing file
/Fh <file> output header file containing object code
/Fe <file> output warnings and errors to a specific file
/Fd <file> extract shader PDB and write to given file
/Vn <name> use <name> as variable name in header file
/Cc output color coded assembly listings
/Ni output instruction numbers in assembly listings
/No output instruction byte offset in assembly listings
/Lx output hexadecimal literals
/P <file> preprocess to file (must be used alone)
@<file> options response file
/dumpbin load a binary file rather than compiling
/Qstrip_reflect strip reflection data from 4_0+ shader bytecode
/Qstrip_debug strip debug information from 4_0+ shader bytecode
/Qstrip_priv strip private data from 4_0+ shader bytecode
/Qstrip_rootsignature strip root signature from shader bytecode
/setrootsignature <file> attach root signature to shader bytecode
/extractrootsignature <file> extract root signature from shader bytecode
/verifyrootsignature <file> verify shader bytecode against root signature
/compress compress DX10 shader bytecode from files
/decompress decompress bytecode from first file, output files should
be listed in the order they were in during compression
/shtemplate <file> template shader file for merging/matching resources
/mergeUAVs merge UAV slots of template shader and current shader
/matchUAVs match template shader UAV slots in current shader
/res_may_alias assume that UAVs/SRVs may alias for cs_5_0+
/enable_unbounded_descriptor_tables enables unbounded descriptor tables
/all_resources_bound enable aggressive flattening in SM5.1+
/setprivate <file> private data to add to compiled shader blob
/getprivate <file> save private data from shader blob
/force_rootsig_ver <profile> force root signature version (rootsig_1_1 if omitted)
/D <id>=<text> define macro
/nologo suppress copyright message
<profile>: cs_4_0 cs_4_1 cs_5_0 cs_5_1 ds_5_0 ds_5_1 gs_4_0 gs_4_1 gs_5_0
gs_5_1 hs_5_0 hs_5_1 lib_4_0 lib_4_1 lib_4_0_level_9_1
lib_4_0_level_9_1_vs_only lib_4_0_level_9_1_ps_only lib_4_0_level_9_3
lib_4_0_level_9_3_vs_only lib_4_0_level_9_3_ps_only lib_5_0 ps_2_0
ps_2_a ps_2_b ps_2_sw ps_3_0 ps_3_sw ps_4_0 ps_4_0_level_9_1
ps_4_0_level_9_3 ps_4_0_level_9_0 ps_4_1 ps_5_0 ps_5_1 rootsig_1_0
rootsig_1_1 tx_1_0 vs_1_1 vs_2_0 vs_2_a vs_2_sw vs_3_0 vs_3_sw vs_4_0
vs_4_0_level_9_1 vs_4_0_level_9_3 vs_4_0_level_9_0 vs_4_1 vs_5_0 vs_5_1
DXIL Signing Tool
Invoking DXIL signing:
root@b31fc74f94b8:/app# $WINE_PATH $SIGN_PATH --help
DXIL Signing Utility
Usage: Z:\app\signing\dxil-signing.exe [OPTIONS]
Options:
-h,--help Print this help message and exit
-i,--input TEXT REQUIRED Input unsigned dxil file
-o,--output TEXT REQUIRED Output signed dxil file
GLSLc / Glslang
Invoking GLSLc:
root@b31fc74f94b8:/app# $GLSLC_PATH --help
glslc - Compile shaders into SPIR-V
Usage: glslc [options] file...
An input file of - represents standard input.
Options:
-c Only run preprocess, compile, and assemble steps.
-Dmacro[=defn] Add an implicit macro definition.
-E Outputs only the results of the preprocessing step.
Output defaults to standard output.
-fauto-bind-uniforms
Automatically assign bindings to uniform variables that
don't have an explicit 'binding' layout in the shader
source.
-fauto-map-locations
Automatically assign locations to uniform variables that
don't have an explicit 'location' layout in the shader
source.
-fentry-point=<name>
Specify the entry point name for HLSL compilation, for
all subsequent source files. Default is "main".
-fhlsl_functionality1, -fhlsl-functionality1
Enable extension SPV_GOOGLE_hlsl_functionality1 for HLSL
compilation.
-fhlsl-iomap Use HLSL IO mappings for bindings.
-fhlsl-offsets Use HLSL offset rules for packing members of blocks.
Affects only GLSL. HLSL rules are always used for HLSL.
-flimit=<settings>
Specify resource limits. Each limit is specified by a limit
name followed by an integer value. Tokens should be
separated by whitespace. If the same limit is specified
several times, only the last setting takes effect.
-flimit-file <file>
Set limits as specified in the given file.
-fresource-set-binding [stage] <reg0> <set0> <binding0>
[<reg1> <set1> <binding1>...]
Explicitly sets the descriptor set and binding for
HLSL resources, by register name. Optionally restrict
it to a single stage.
-fcbuffer-binding-base [stage] <value>
Same as -fubo-binding-base.
-fimage-binding-base [stage] <value>
Sets the lowest automatically assigned binding number for
images. Optionally only set it for a single shader stage.
For HLSL, the resource register number is added to this
base.
-fsampler-binding-base [stage] <value>
Sets the lowest automatically assigned binding number for
samplers Optionally only set it for a single shader stage.
For HLSL, the resource register number is added to this
base.
-fssbo-binding-base [stage] <value>
Sets the lowest automatically assigned binding number for
shader storage buffer objects (SSBO). Optionally only set
it for a single shader stage. Only affects GLSL.
-ftexture-binding-base [stage] <value>
Sets the lowest automatically assigned binding number for
textures. Optionally only set it for a single shader stage.
For HLSL, the resource register number is added to this
base.
-fuav-binding-base [stage] <value>
For automatically assigned bindings for unordered access
views (UAV), the register number is added to this base to
determine the binding number. Optionally only set it for
a single shader stage. Only affects HLSL.
-fubo-binding-base [stage] <value>
Sets the lowest automatically assigned binding number for
uniform buffer objects (UBO). Optionally only set it for
a single shader stage.
For HLSL, the resource register number is added to this
base.
-fshader-stage=<stage>
Treat subsequent input files as having stage <stage>.
Valid stages are vertex, vert, fragment, frag, tesscontrol,
tesc, tesseval, tese, geometry, geom, compute, and comp.
-g Generate source-level debug information.
Currently this option has no effect.
--help Display available options.
-I <value> Add directory to include search path.
-mfmt=<format> Output SPIR-V binary code using the selected format. This
option may be specified only when the compilation output is
in SPIR-V binary code form. Available options include bin, c
and num. By default the binary output format is bin.
-M Generate make dependencies. Implies -E and -w.
-MM An alias for -M.
-MD Generate make dependencies and compile.
-MF <file> Write dependency output to the given file.
-MT <target> Specify the target of the rule emitted by dependency
generation.
-O Optimize the generated SPIR-V code for better performance.
-Os Optimize the generated SPIR-V code for smaller size.
-O0 Disable optimization.
-o <file> Write output to <file>.
A file name of '-' represents standard output.
-std=<value> Version and profile for GLSL input files. Possible values
are concatenations of version and profile, e.g. 310es,
450core, etc. Ignored for HLSL files.
-S Only run preprocess and compilation steps.
--show-limits Display available limit names and their default values.
--target-env=<environment>
Set the target client environment, and the semantics
of warnings and errors. An optional suffix can specify
the client version. Values are:
vulkan1.0 # The default
vulkan1.1
vulkan # Same as vulkan1.0
opengl4.5
opengl # Same as opengl4.5
--version Display compiler version information.
-w Suppresses all warning messages.
-Werror Treat all warnings as errors.
-x <language> Treat subsequent input files as having type <language>.
Valid languages are: glsl, hlsl.
For files ending in .hlsl the default is hlsl.
Otherwise the default is glsl.
AMD Radeon Graphics Analyzer
Invoking RGA (Linux version):
root@b31fc74f94b8:/app# $RGA_NIX_PATH -h
Radeon GPU Analyzer Version: 2.0.1305.69
Radeon GPU Analyzer is an analysis tool for OpenCL, OpenGL and Vulkan
To view help for ROCm OpenCL: -h -s rocm-cl
To view help for legacy OpenCL: -h -s cl
To view help for OpenGL: -h -s opengl
To view help for Vulkan (GLSL): -h -s vulkan
To view help for Vulkan (SPIR-V binary input): -h -s vulkan-spv
To view help for Vulkan (SPIR-V textual input): -h -s vulkan-spv-txt
Invoking RGA (Windows version):
root@b31fc74f94b8:/app# $WINE_PATH $RGA_WIN_PATH -h
Radeon GPU Analyzer Version: 2.0.1305.268
Radeon GPU Analyzer is an analysis tool for OpenCL, DirectX, OpenGL and Vulkan
To view help for ROCm OpenCL: -h -s rocm-cl
To view help for legacy OpenCL: -h -s cl
To view help for OpenGL: -h -s opengl
To view help for Vulkan (GLSL): -h -s vulkan
To view help for Vulkan (SPIR-V binary input): -h -s vulkan-spv
To view help for Vulkan (SPIR-V textual input): -h -s vulkan-spv-txt
To view help for DirectX: -h -s hlsl
To view help for AMDIL: -h -s amdil
Vulkan Tools
Invoking SPIR-V cross compiler:
root@b31fc74f94b8:/app# $VULKAN_PATH/spirv-cross -h
Usage: spirv-cross
[--output <output path>]
[SPIR-V file]
[--es]
[--no-es]
[--version <GLSL version>]
[--dump-resources]
[--help]
[--force-temporary]
[--vulkan-semantics]
[--flatten-ubo]
[--fixup-clipspace]
[--flip-vert-y]
[--iterations iter]
[--cpp]
[--cpp-interface-name <name>]
[--msl]
[--msl-version <MMmmpp>]
[--msl-swizzle-texture-samples]
[--msl-ios]
[--hlsl]
[--reflect]
[--shader-model]
[--hlsl-enable-compat]
[--separate-shader-objects]
[--pls-in format input-name]
[--pls-out format output-name]
[--remap source_name target_name components]
[--extension ext]
[--entry name]
[--stage <stage (vert, frag, geom, tesc, tese comp)>]
[--remove-unused-variables]
[--flatten-multidimensional-arrays]
[--no-420pack-extension]
[--remap-variable-type <variable_name> <new_variable_type>]
[--rename-interface-variable <in|out> <location> <new_variable_name>]
[--set-hlsl-vertex-input-semantic <location> <semantic>]
[--rename-entry-point <old> <new> <stage>]
[--combined-samplers-inherit-bindings]
[--no-support-nonzero-baseinstance]
Invoking SPIR-V disassembler:
root@b31fc74f94b8:/app# $VULKAN_PATH/spirv-dis -h
/app/vulkan/spirv-dis - Disassemble a SPIR-V binary module
Usage: /app/vulkan/spirv-dis [options] [<filename>]
The SPIR-V binary is read from <filename>. If no file is specified,
or if the filename is "-", then the binary is read from standard input.
Options:
-h, --help Print this help.
--version Display disassembler version information.
-o <filename> Set the output filename.
Output goes to standard output if this option is
not specified, or if the filename is "-".
--color Force color output. The default when printing to a terminal.
Overrides a previous --no-color option.
--no-color Don't print in color. Overrides a previous --color option.
The default when output goes to something other than a
terminal (e.g. a file, a pipe, or a shell redirection).
--no-indent Don't indent instructions.
--no-header Don't output the header as leading comments.
--raw-id Show raw Id values instead of friendly names.
--offsets Show byte offsets for each instruction.
Compile as a Service
In addition to running the image directly (which is great for tests or quick iteration), it wouldn’t be ideal to individually start and stop a container for each shader instance that needs compilation, if there are a large number of shaders. A more effective design is to have a service running within a container that accepts compilation requests, invokes the necessary executables, and responsed to the caller with the appropriate data.
This post won’t go into details behind the shader build service, but here is an example Docker file that inherits from the gwihlidal/docker-shader image, installs the Rust language, compiles a release build of the service (including a trick for fast iteration), and specifies the service binary as the default entry point.
FROM gwihlidal/docker-shader
# Prevents annoying debconf errors during builds
ARG DEBIAN_FRONTEND="noninteractive"
RUN apt update \
&& apt install --no-install-recommends -y \
build-essential \
pkg-config \
curl \
&& apt autoremove -y \
&& apt autoclean \
&& apt clean \
&& apt autoremove
# Install Rust
RUN curl https://sh.rustup.rs -sSf | sh -s -- -y
ENV PATH="/root/.cargo/bin:${PATH}"
WORKDIR /app
# Avoid having to install/build all dependencies when iterating
# by copying the Cargo files and making dummy main source files
COPY Cargo.* ./
RUN mkdir -p src && echo "fn main() {}" > src/main.rs && \
echo "fn main() {}" > src/main.rs && \
cargo build --release
# We need to touch our real source files or
# else Docker will use the cached ones.
COPY src src
RUN touch src/main.rs && \
cargo build --release --color never
ENTRYPOINT ["./target/release/main"]
With a containerized service, we can now realize the goal of scaling out the shader compilation as a load balancing service in Kubernetes!
Here is an example (basic) Kubernetes manifest for the service replicas and load balancer (GKE):
apiVersion: apps/v1
kind: Deployment
metadata:
name: shader-build
labels:
name: "shader-build"
keel.sh/policy: force
keel.sh/trigger: poll
annotations:
keel.sh/pollSchedule: "@every 10m"
spec:
selector:
matchLabels:
app: shader-build
replicas: 4
template:
metadata:
labels:
app: shader-build
spec:
containers:
- name: shader-build
image: gwihlidal/shader-build-test
resources:
requests:
memory: 4Gi
cpu: 4
ports:
- containerPort: 63999
---
apiVersion: v1
kind: Service
metadata:
name: shader-build
labels:
app: shader-build
spec:
type: LoadBalancer
ports:
- port: 80
targetPort: 63999
protocol: TCP
name: http2
selector:
app: shader-build
It works!
