[Build Premake5] How to tell it to use system libraries

[chadchoi]

instead of building its own libraries from the source code it included in contrib.

brechtsanders/winlibs_mingw#244

[nickclark2016]

If you're on Linux, you need to run the build script like:

PLATFORM=x64 CONFIG=release PREMAKE_OPTS="--lib-src=system --cc=gcc" ./Bootstrap.sh

As for Windows, I don't think we've built that in at the moment. @samsinsane I know you were the last one to touch the build matrices, do you know what would be required to get system libraries up and running for mingw? I think that may be easier than the MSVC builds.

[samsinsane]

do you know what would be required to get system libraries up and running for mingw?

This is what is required for any system:

1. Know the names of the dev packages to install them
2. The compiler needs to know where the system packages are (we just say "link this library")
3. The library names specified to the compiler will be curl, lua5.3, z and zip, so if there's any deviation then it won't work.

As for MinGW specifically, I have no idea what's required to get it working. I know of MinGW, but that's kind of where my knowledge about it ends.

[chadchoi]

@brechtsanders

[Jarod42]

packages should normally be:

• mingw-w64-x86_64-curl
• mingw-w64-x86_64-lua53
• mingw-w64-x86_64-libzip
• mingw-w64-x86_64-zlib

[brechtsanders]

I'm trying under MSYS2 shell (all dependencies are installed).

PLATFORM=x64 CONFIG=release PREMAKE_OPTS="--lib-src=system --cc=gcc" ./Bootstrap.sh

results in:

/bin/sh: line 1: cc: command not found

So I tried

CC=gcc PLATFORM=x64 CONFIG=release PREMAKE_OPTS="--lib-src=system" ./Bootstrap.sh

which results in:

../../contrib/curl/lib/escape.c:56:7: error: conflicting types for 'curl_easy_escape'; have
'char *(struct Curl_easy *, const char *, int)'
   56 | char *curl_easy_escape(struct Curl_easy *data, const char *string,
      |       ^~~~~~~~~~~~~~~~
In file included from ../../contrib/curl/lib/escape.c:30:
C:\Prog\winlibs14.2.0msvcrt\custombuilt64\include/curl/curl.h:2665:19: note: previous declaration of 'curl_easy_escape' with type 'char *(CURL *, const char *, int)' {aka 'char *(void
*, const char *, int)'}
 2665 | CURL_EXTERN char *curl_easy_escape(CURL *handle,
      |                   ^~~~~~~~~~~~~~~~
../../contrib/curl/lib/escape.c:179:7: error: conflicting types for 'curl_easy_unescape'; have 'char *(struct Curl_easy *, const char *, int,  int *)'
  179 | char *curl_easy_unescape(struct Curl_easy *data, const char *string,
      |       ^~~~~~~~~~~~~~~~~~
C:\Prog\winlibs14.2.0msvcrt\custombuilt64\include/curl/curl.h:2685:19: note: previous declaration of 'curl_easy_unescape' with type 'char *(CURL *, const char *, int,  int *)' {aka 'char *(void *, const char *, int,  int *)'}
 2685 | CURL_EXTERN char *curl_easy_unescape(CURL *handle,
      |                   ^~~~~~~~~~~~~~~~~~

[Jarod42]

PREMAKE_OPTS="--lib-src=system --cc=gcc" ./Bootstrap.sh
results in:
/bin/sh: line 1: cc: command not found

It seems you don't have the latest source, there was a bug than --cc=gcc results in default CC==cc which is not present in MinGW.
Fixed by #2240

On my side, I'm trying to improve mingw CI to add the matrix for dependency (https://github.com/Jarod42/premake-core/tree/CI_mingw_depsrc).
Success for "system" build, but got some error currently with explicit contrib mode... (investigating).

[samsinsane]

@Jarod42 might be due to this line:

./build/bootstrap/premake_bootstrap --arch=x86 --os=windows --to=build/bootstrap --cc=mingw --lib-src=contrib --cc=gcc gmake2

The original builds used --cc=mingw but you've added in --cc=gcc. When the toolset is mingw the premake5.lua will calls links { "crypt32", "bcrypt" } which is what your error is about.

[Jarod42]

Indeed, that what that...
[OT] mingw as toolset seems strange, mingw can use both gcc and clang... and our mingw toolset copies gcc one.

[nickclark2016]

@chadchoi Can you confirm if the latest sources work as depicted in the CI file?

[chadchoi]

@chadchoi Can you confirm if the latest sources work as depicted in the CI file?

@brechtsanders

[Jarod42]

To resume:

For 64 bits:

pacman -Suy --noconfirm mingw-w64-x86_64-curl mingw-w64-x86_64-lua53 mingw-w64-x86_64-libzip mingw-w64-x86_64-zlib
PLATFORM=x64 CONFIG=release PREMAKE_OPTS="--lib-src=system" ./Bootstrap.sh

For 32 bits, in case you prefer:

pacman -Suy --noconfirm mingw-w64-i686-curl mingw-w64-i686-lua53 mingw-w64-i686-libzip mingw-w64-i686-zlib
PLATFORM=x86 CONFIG=release PREMAKE_OPTS="--lib-src=system" ./Bootstrap.sh

[brechtsanders]

I'm using an environment where I build all dependencies from source (rather than MSYS2's pacman).

When I ran:

PLATFORM=x64 CONFIG=release PREMAKE_OPTS="--lib-src=system" ./Bootstrap.sh

I got:

/bin/sh: line 1: cc: command not found
make: *** [Bootstrap.mak:90: mingw] Error 127

But when I ran:

CC=gcc PLATFORM=x64 CONFIG=release PREMAKE_OPTS="--lib-src=system" ./Bootstrap.sh

I got:

In file included from ../../src/host/resource.rc:4:
../../src/host/premake.h:13:10: fatal error: lua5.3/lua.h: No such file or directory
   13 | #include <lua5.3/lua.h>
      |          ^~~~~~~~~~~~~~
compilation terminated.
C:\Prog\winlibs14.2.0msvcrt\mingw64\bin\windres.exe: preprocessing failed.
make[2]: *** [Premake5.make:415: obj/x64/Release/Premake5/resource.res] Error 1
make[2]: *** Waiting for unfinished jobs....
buffered_io.c
In file included from ../../src/host/buffered_io.c:9:
../../src/host/premake.h:13:10: fatal error: lua5.3/lua.h: No such file or directory
   13 | #include <lua5.3/lua.h>
      |          ^~~~~~~~~~~~~~
compilation terminated.
make[2]: *** [Premake5.make:271: obj/x64/Release/Premake5/buffered_io.o] Error 1
make[1]: *** [Makefile:36: Premake5] Error 2
make[1]: Leaving directory '/R/winlibs14.2.0msvcrt64/premake-core-master/build/bootstrap'
make: *** [Bootstrap.mak:93: mingw] Error 2

Which makes sense because that's not where lua.h lives (I have lua 5.4.7).
So I did the following patch:

sed -i.bak -e "s?\(#include <\)lua5.3/?\1?" src/host/premake.h src/host/curl_utils.h

Then the result was:

../../src/host/debug_prompt.c: In function 'debug_prompt':
../../src/host/debug_prompt.c:38:57: error: expected ')' before 'LUA_QL'
   38 |                                         "error calling " LUA_QL("print") " (%s)",
      |                                                         ^~~~~~~
      |                                                         )
../../src/host/debug_prompt.c:37:68: note: to match this '('
   37 |                                 l_message(progname, lua_pushfstring(L,
      |                                                                    ^
os_chdir.c
make[2]: *** [Premake5.make:280: obj/x64/Release/Premake5/debug_prompt.o] Error 1
make[2]: *** Waiting for unfinished jobs....
os_chmod.c
make[1]: *** [Makefile:36: Premake5] Error 2
make[1]: Leaving directory '/R/winlibs14.2.0msvcrt64/premake-core-master/build/bootstrap'
make: *** [Bootstrap.mak:93: mingw] Error 2

To get past that (as it's only for displaying an error) I did this dirty fix:

patch -ulbf src/host/debug_prompt.c << EOF
@@ -37,3 +37,3 @@
                                l_message(progname, lua_pushfstring(L,
-                                       "error calling " LUA_QL("print") " (%s)",
+                                       "error calling print (%s)",
                                        lua_tostring(L, -1))
EOF

Finally, there was a linker error:

C:/Prog/winlibs14.2.0msvcrt/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/14.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: cannot find -llua5.3: No such file or directory
collect2.exe: error: ld returned 1 exit status
make[2]: *** [Premake5.make:216: ../../bin/release/premake5.exe] Error 1
make[1]: *** [Makefile:36: Premake5] Error 2
make[1]: Leaving directory '/R/winlibs14.2.0msvcrt64/premake-core-master/build/bootstrap'
make: *** [Bootstrap.mak:93: mingw] Error 2

which I fixed with:

sed -i.bak -e "s/\(links.*lua\)[0-9.]*/\1/" premake5.lua

but this should really be properly configured with pkg-config.

Anyway, this did give me a premake5.exe file that runs.

Then I tries 32-bit, but this gave the following errors:

src/host/os_linkdir.c: In function 'do_linkdir':
src/host/os_linkdir.c:41:31: error: implicit declaration of function 'CreateSymbolicLinkW' [-Wimplicit-function-declaration]
   41 |                 BOOLEAN res = CreateSymbolicLinkW(wDstPath, relSrcPath, SYMBOLIC_LINK_FLAG_DIRECTORY | SYMBOLIC_LINK_FLAG_ALLOW_UNPRIVILEGED_CREATE);
      |                               ^~~~~~~~~~~~~~~~~~~
src/host/os_linkdir.c:41:73: error: 'SYMBOLIC_LINK_FLAG_DIRECTORY' undeclared (first use in
this function)
   41 |                 BOOLEAN res = CreateSymbolicLinkW(wDstPath, relSrcPath, SYMBOLIC_LINK_FLAG_DIRECTORY | SYMBOLIC_LINK_FLAG_ALLOW_UNPRIVILEGED_CREATE);
      |                                                                         ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
src/host/os_linkdir.c:41:73: note: each undeclared identifier is reported only once for each function it appears in
src/host/os_linkdir.c:41:104: error: 'SYMBOLIC_LINK_FLAG_ALLOW_UNPRIVILEGED_CREATE' undeclared (first use in this function)
   41 |                 BOOLEAN res = CreateSymbolicLinkW(wDstPath, relSrcPath, SYMBOLIC_LINK_FLAG_DIRECTORY | SYMBOLIC_LINK_FLAG_ALLOW_UNPRIVILEGED_CREATE);
      |
                   ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
src/host/os_linkfile.c: In function 'do_linkfile':
src/host/os_linkfile.c:40:31: error: implicit declaration of function 'CreateSymbolicLinkW'
[-Wimplicit-function-declaration]
   40 |                 BOOLEAN res = CreateSymbolicLinkW(wDstPath, relSrcPath, SYMBOLIC_LINK_FLAG_ALLOW_UNPRIVILEGED_CREATE);
      |                               ^~~~~~~~~~~~~~~~~~~
src/host/os_linkfile.c:40:73: error: 'SYMBOLIC_LINK_FLAG_ALLOW_UNPRIVILEGED_CREATE' undeclared (first use in this function)
   40 |                 BOOLEAN res = CreateSymbolicLinkW(wDstPath, relSrcPath, SYMBOLIC_LINK_FLAG_ALLOW_UNPRIVILEGED_CREATE);
      |                                                                         ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
src/host/os_linkfile.c:40:73: note: each undeclared identifier is reported only once for each function it appears in
make: *** [Bootstrap.mak:90: mingw] Error 1

The reason for the indefined CreateSymbolicLinkW() is that my 32-bit GCC build targets older Windows versions.
I could get around the error with:

sed -i.bak -e '1i #if !defined(_WIN32_WINNT) || _WIN32_WINNT < 0x0600\n#undef _WIN32_WINNT\n#define _WIN32_WINNT 0x0600\n#endif' src/host/os_linkdir.c src/host/os_linkfile.c

But maybe (if possible) the symbolic link stuff should just be skipped when building for older Windows versions.

[brechtsanders]

Additional question: What should PLATFORM be set to for Windows on ARM (aarch64)?

[nickclark2016]

For the Lua version, I think the answer is to install a matching version of Lua or patch like you did and hope for the best.
For the Windows version, my personal opinion is if Microsoft has stopped supporting the OS version, I have no strong desire to support it either. Windows 10 is ancient at this point. What version of Windows are you aiming to support?
Finally, for aarch64, I don't think any of us have tried to stand up a Windows aarch64 build. I personally don't have a windows arm machine lying around to test things on. I also don't believe GH has brought arm-based runners to the free tier of runners yet, which makes it hard for us. @samsinsane is getting the ping for the ARM jobs, since he's been dealing with the CI the most recently.

[Codinablack]

I have successfully built and deployed my project with the target arch being ARM and ARM64, while using premake as the build system.

One thing I learned along the way that wasn't apparent in the beginning is that platform doesn't need to be set unless you plan to target a platform other than the host you are compiling on, which came as a surprise to me.

I also ran into issues with finding and using system libs first for some edge case linux scenarios, but not once had a problem on windows (and I'm using the ancient windows 10)

I will share here my premake5.lua in case it is of any help to someone

workspace "Black-Tek-Server"
   configurations { "Debug", "Release"}
   platforms { "64", "ARM64", "ARM" }
   location ""
   editorintegration "On"

   project        "Black-Tek-Server"
      kind        "ConsoleApp"
      language    "C++"
      cppdialect  "C++20"
      targetdir   "%{wks.location}"
      objdir      "build/%{cfg.buildcfg}/obj"
      location    ""
      files { "src/**.cpp", "src/**.h" }
      flags {"MultiProcessorCompile"}
      enableunitybuild "On"
      intrinsics   "On"
      editandcontinue "Off"

      newoption {
         trigger     = "lua",
         description = "Specific Lua library to use. For example lua5.4. Useful if the packaged lua does not provide a symbolic liblua.so",
         value       = "libname",
         category    = "BlackTek", -- Group options together
         default     = "lua",
         allowed     = {
            {"lua", "Default"},
            {"lua5.4", "Use Lua 5.4"},
            {"lua5.3", "Use Lua 5.3"},
         }
      }

      newoption {
         trigger     = "custom-includes",
         description = "A comma separated list of custom include paths.",
         value       = "include paths",
         category    = "BlackTek", -- Group options together
      }

      newoption {
         trigger     = "custom-libs",
         description = "A comma separated list of custom library paths.",
         value       = "library paths",
         category    = "BlackTek", -- Group options together
      }

      newoption {
         trigger     = "verbose",
         description = "Show warnings during compilation.",
         category    = "BlackTek" -- Group options together
      }

      if _OPTIONS["custom-includes"] then
         includedirs { string.explode(_OPTIONS["custom-includes"], ",") }
      end

      if _OPTIONS["custom-libs"] then
         libdirs { string.explode(_OPTIONS["custom-libs"], ",") }
      end

      filter "configurations:Debug"
         defines { "DEBUG" }
         runtime "Debug"
         symbols "On"
         optimize "Debug"
         flags {"NoIncrementalLink"}
      filter {}

      filter "configurations:Release"
         defines { "NDEBUG" }
         runtime "Release"
         symbols "Off"
         optimize "Full"
      filter {}

      filter "platforms:64"
         architecture "x86_64"
      filter {}

      filter "platforms:ARM64"
         architecture "ARM64"
      filter {}

      filter "platforms:ARM"
         architecture "ARM"
      filter {}

      filter "system:not windows"
         buildoptions { "-Wall", "-Wextra", "-pedantic", "-pipe", "-fvisibility=hidden", "-Wno-unused-local-typedefs" }
         linkoptions{"-flto=auto"}
         flags {}
      filter {}

      filter "system:windows"
         openmp "On"
         characterset "MBCS"
         linkoptions {"/IGNORE:4099"}
         buildoptions {"/bigobj"}
         vsprops { VcpkgEnableManifest = "true" }
         symbolspath '$(OutDir)$(TargetName).pdb'
      filter {}

      filter "architecture:amd64"
	     vectorextensions "AVX"
      filter{}

      filter {"system:linux", "options:verbose"}
         linkoptions { "-v" }
         warnings "Extra"
      filter {}

      filter { "system:linux", "not options:verbose" }
         warnings "Off"
      filter {}

      filter { "system:linux", "architecture:ARM" }
         -- Paths to vcpkg installed dependencies
         libdirs { "vcpkg_installed/arm-linux/lib", "/usr/arm-linux-gnueabihf" }
         includedirs { "vcpkg_installed/arm-linux/include", "/usr/arm-linux-gnueabihf" }
      filter{}

      filter { "system:linux", "architecture:ARM64" }
         -- Paths to vcpkg installed dependencies
         libdirs { "vcpkg_installed/arm64-linux/lib", "/usr/arm-linux-gnueabi" }
         includedirs { "vcpkg_installed/arm64-linux/include", "/usr/arm-linux-gnueabi" }
      filter{}

      filter { "system:linux", "architecture:amd64" }
         -- Paths to vcpkg installed dependencies
         libdirs { "vcpkg_installed/x64-linux/lib" }
         includedirs { "vcpkg_installed/x64-linux/include" }
      filter{}

      filter "system:linux"
         -- Common Linux paths
         libdirs { "/usr/lib" }
         includedirs { "/usr/include", "/usr/include/lua5.*" }
         links { "pugixml", _OPTIONS["lua"], "fmt", "mariadb", "cryptopp", "boost_iostreams", "zstd", "z", "curl", "ssl", "crypto" }
      filter{}

      filter "toolset:gcc"
         buildoptions { "-fno-strict-aliasing" }
      filter {}

      filter "toolset:clang"
         buildoptions { "-Wimplicit-fallthrough", "-Wmove" }
      filter {}

      filter { "system:macosx", "action:gmake" }
         buildoptions { "-fvisibility=hidden" }   
      filter {}

[samsinsane]

It sounds like all questions have been answered and issues resolved. I'm going to close this off as it has greatly deviated from the original question. If there are any other questions or issues, please feel free to open a new issue.