python: (2.7.6) problem determining filesystem encoding

[Lewis Rosenthal]

This is tangentially related to getting SSL support for yum repos, so I'll open this here. Frankly, python is so complex that we should probably break it out into its own project, here, but I'll leave that to you, Yuri. ;-)

I needed to install M2Crypto as I am having difficulty accessing my repo mirror via SSL (different ticket - to be created). M2Crypto seems to want to be installed via setuptools. Installing setuptools should be fairly straightforward:

wget --no-check-certificate https://bootstrap.pypa.io/ez_setup.py -O - | python

however, once the installation starts, the fun begins:

Extracting in c:/var/temp/tmpcqr6rs
Now working in c:/var/temp/tmpcqr6rs/setuptools-5.6
Installing Setuptools
running install
running bdist_egg
running egg_info
writing requirements to setuptools.egg-info/requires.txt
writing setuptools.egg-info/PKG-INFO
writing top-level names to setuptools.egg-info/top_level.txt
writing dependency_links to setuptools.egg-info/dependency_links.txt
writing entry points to setuptools.egg-info/entry_points.txt
Traceback (most recent call last):
  File "setup.py", line 219, in <module>
    dist = setuptools.setup(**setup_params)
  File "J:/usr/lib/python2.7/distutils/core.py", line 152, in setup
    dist.run_commands()
  File "J:/usr/lib/python2.7/distutils/dist.py", line 953, in run_commands
    self.run_command(cmd)
  File "J:/usr/lib/python2.7/distutils/dist.py", line 972, in run_command
    cmd_obj.run()
  File "C:/var/temp/tmpcqr6rs/setuptools-5.6/setuptools/command/install.py", line 67, in run
    self.do_egg_install()
  File "C:/var/temp/tmpcqr6rs/setuptools-5.6/setuptools/command/install.py", line 109, in do_egg_install
    self.run_command('bdist_egg')
  File "J:/usr/lib/python2.7/distutils/cmd.py", line 326, in run_command
    self.distribution.run_command(command)
  File "J:/usr/lib/python2.7/distutils/dist.py", line 972, in run_command
    cmd_obj.run()
  File "C:/var/temp/tmpcqr6rs/setuptools-5.6/setuptools/command/bdist_egg.py", line 152, in run
    self.run_command("egg_info")
  File "J:/usr/lib/python2.7/distutils/cmd.py", line 326, in run_command
    self.distribution.run_command(command)
  File "J:/usr/lib/python2.7/distutils/dist.py", line 972, in run_command
    cmd_obj.run()
  File "C:/var/temp/tmpcqr6rs/setuptools-5.6/setuptools/command/egg_info.py", line 168, in run
    self.find_sources()
  File "C:/var/temp/tmpcqr6rs/setuptools-5.6/setuptools/command/egg_info.py", line 193, in find_sources
    mm.run()
  File "C:/var/temp/tmpcqr6rs/setuptools-5.6/setuptools/command/egg_info.py", line 277, in run
    self.add_defaults()
  File "C:/var/temp/tmpcqr6rs/setuptools-5.6/setuptools/command/egg_info.py", line 306, in add_defaults
    sdist.add_defaults(self)
  File "C:/var/temp/tmpcqr6rs/setuptools-5.6/setuptools/command/sdist.py", line 151, in add_defaults
    self.filelist.append(fn)
  File "C:/var/temp/tmpcqr6rs/setuptools-5.6/setuptools/command/egg_info.py", line 220, in append
    if self._safe_path(path):
  File "C:/var/temp/tmpcqr6rs/setuptools-5.6/setuptools/command/egg_info.py", line 240, in _safe_path
    u_path = unicode_utils.filesys_decode(path)
  File "C:/var/temp/tmpcqr6rs/setuptools-5.6/setuptools/unicode_utils.py", line 31, in filesys_decode
    return path.decode(enc)
TypeError: decode() argument 1 must be string, not None
Something went wrong during the installation.
See the error message above.

Looking at the section in question in unicode_utils.py, he see:

def filesys_decode(path):
    """
    Ensure that the given path is decoded,
    NONE when no expected encoding works
    """

    fs_enc = sys.getfilesystemencoding()
    if isinstance(path, decoded_string):
        return path

    for enc in (fs_enc, "utf-8"):
        try:
            return path.decode(enc)
        except UnicodeDecodeError:
            continue

Result: cannot install setuptools.

I've tried this under a variety of shells, too, but to no avail. Filesystem is JFS.

[Yuri Dario]

Since encoding is None, .decode fails... This workaround works for me:

for enc in (fs_enc, "utf-8"):

try:

return path #.decode(enc)

except UnicodeDecodeError?:

continue

and instead of easy setup, I followed the standard way of installing python packages

python setup.py install

I'm leaving this ticket open since we need a fix for path.decode() function.

[Lewis Rosenthal]

Confirming that your edit works for me, Yuri; thanks. Unfortunately, I don't know enough python to be of much use, so I truly appreciate the pointer. However, as you say, we should keep this open as this is merely a workaround and not a fix to the underlying problem.

[Silvan Scherrer]

I'm not sure if this is still the case, as setuptools changed a lot since this ticket was opened

[Lewis Rosenthal]

Well, we still fail during the install, but in a different place:

Traceback (most recent call last):
  File "setup.py", line 181, in <module>
    dist = setuptools.setup(**setup_params)
  File "N:/USR/lib/python2.7/distutils/core.py", line 152, in setup
    dist.run_commands()
  File "N:/USR/lib/python2.7/distutils/dist.py", line 953, in run_commands
    self.run_command(cmd)
  File "N:/USR/lib/python2.7/distutils/dist.py", line 972, in run_command
    cmd_obj.run()
  File "N:/var/temp/tmpfmn52e/setuptools-33.1.1/setuptools/command/install.py", line 67, in run
    self.do_egg_install()
  File "N:/var/temp/tmpfmn52e/setuptools-33.1.1/setuptools/command/install.py", line 109, in do_egg_install
    self.run_command('bdist_egg')
  File "N:/USR/lib/python2.7/distutils/cmd.py", line 326, in run_command
    self.distribution.run_command(command)
  File "N:/USR/lib/python2.7/distutils/dist.py", line 972, in run_command
    cmd_obj.run()
  File "N:/var/temp/tmpfmn52e/setuptools-33.1.1/setuptools/command/bdist_egg.py", line 161, in run
    cmd = self.call_command('install_lib', warn_dir=0)
  File "N:/var/temp/tmpfmn52e/setuptools-33.1.1/setuptools/command/bdist_egg.py", line 147, in call_command
    self.run_command(cmdname)
  File "N:/USR/lib/python2.7/distutils/cmd.py", line 326, in run_command
    self.distribution.run_command(command)
  File "N:/USR/lib/python2.7/distutils/dist.py", line 972, in run_command
    cmd_obj.run()
  File "N:/var/temp/tmpfmn52e/setuptools-33.1.1/setuptools/command/install_lib.py", line 12, in run
    outfiles = self.install()
  File "N:/USR/lib/python2.7/distutils/command/install_lib.py", line 115, in install
    outfiles = self.copy_tree(self.build_dir, self.install_dir, preserve_symlinks=1)
  File "N:/var/temp/tmpfmn52e/setuptools-33.1.1/setuptools/command/install_lib.py", line 90, in copy_tree
    assert preserve_mode and preserve_times and not preserve_symlinks
AssertionError
Something went wrong during the installation.
See the error message above.

install_lib.py beginning at line 86 looks like this:

def copy_tree(
        self, infile, outfile,
        preserve_mode=1, preserve_times=1, preserve_symlinks=0, level=1
):
    assert preserve_mode and preserve_times and not preserve_symlinks
    exclude = self.get_exclusions()

On the surface, I don't see anything wrong with the assertion which would cause it to be false.

[Silvan Scherrer]

I wonder what setuptool you use. As have version 34.4.1 installed. And I have exactly this fixed in this version.

[Silvan Scherrer]

And 34 is available as rpm since ages iirc

[Silvan Scherrer]

any news when using the rpm setuptools version 34?

[Lewis Rosenthal]

The original issue (problem determining filesystem encoding) was resolved for the testcase by using Yuri's workaround. I can confirm that installing setuptools from RPM works; that was never the issue. I was installing setuptools from ez_setup.py. Perhaps the build of setuptools downloaded using that method is incompatible with our environment.

This issue started as an inability to install pip (and wheel) necessary to install M2Crypto. Now that setuptools works, I should open specific tickets for what is failing during the installation of pip.

[dmik]

I've just faced a very similar issue here: ​https://github.com/bitwiseworks/qtwebengine-chromium-os2/issues/3#issuecomment-628684752. There, Python own SAX code also assumes that sys.getfilesystemencoding() never returns None while the docs clearly say otherwise.

Since the original issue was just worked around instead of fixing, I'm reopening this.

Given that this is the second failing case, I'm going to make sys.getfilesystemencoding() return what sys.getdefaultencoding() returns instead of None. This will fix both cases w/o any workarounds needed. Note that ​`sys.getdefaultencoding()` is not supposed to return None so we should be on the safe side here.

[dmik]

Doing what I suggested above fixes both issues but I'm now getting problems rebuilding Python. Seems that our sqlite3 package has changed since we built the Python RPM last time and we no more provide sqlite3.lib or such and the Python EMX compiler setup script can't find sqlite3_dll.a. Without the sqlite3 module in Python, yum can't function (at all).

I need to fix more bits in Python it seems.

[dmik]

Closing this in favor of ​https://github.com/bitwiseworks/python-os2/issues/1.