fix msvc linker bug for huge command line arguments

[astrelsky]

Fixes #226 and pypa/setuptools#4177

This is one of two issues that is blocking flash-attention using ROCM CK on Windows. The other issue is in CK itself and not a pypa problem.

This is just to make it easier for the flash-attention maintainers to find when looking through their comments.
Dao-AILab/flash-attention#2400

[0xDELUXA]

It’s a bit concerning to see that the last commit was 2 months ago - is the project still actively maintained?

[astrelsky]

@abravalheri @zooba may you get someone to review this please? I only opened this here instead of in setuptools since it was requested to report the issue here in distutils.

The problem is not file path length, it is command line length. So if you're linking thousands of files it doesn't matter how short your paths are, you're going to run out of space.

[isuruf]

This is not correct if the arguments need to be escaped.

For eg:

>>> import subprocess
>>> ld_args=["1 2"]*5500                           
>>> len(' '.join(ld_args))                    
21999
>>> subprocess.run(["python", "-c", *ld_args])     
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\hostedtoolcache\windows\Python\3.12.10\x64\Lib\subprocess.py", line 548, in run
    with Popen(*popenargs, **kwargs) as process:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\hostedtoolcache\windows\Python\3.12.10\x64\Lib\subprocess.py", line 1026, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "C:\hostedtoolcache\windows\Python\3.12.10\x64\Lib\subprocess.py", line 1538, in _execute_child
    hp, ht, pid, tid = _winapi.CreateProcess(executable, args,
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [WinError 206] The filename or extension is too long

[astrelsky]

I guestimated the size to allow compensation for things like this. At what point should it be moved to a file then?

Ignore my initial response if you saw it (now deleted), I misunderstood.

[isuruf]

something like

Suggested change

	if len(' '.join(ld_args)) > 30000:
	if len(shlex.join(ld_args)) > 30000:

[astrelsky]

Actually, I still fail to see your point. Handling this in a file is something specific to the msvc toolchain. The example you show just uses python in a subprocess?

[astrelsky]

something like

Suggested change

	if len(' '.join(ld_args)) > 30000:
	if len(shlex.join(ld_args)) > 30000:

shlex is for unix shells only.

[astrelsky]

Actually, I still fail to see your point. Handling this in a file is something specific to the msvc toolchain. The example you show just uses python in a subprocess?

Ok. I understand now, sorry I'm viewing this on my phone and it's not helping. I will return in about 30 minutes when I get off work.

There is an internal function in subprocess I can use for proper escaping, will that be acceptable to use?

[0xDELUXA]

@isuruf Thanks for the review!

Could someone with merge rights take a quick look as well?