Opened 5 years ago

Closed 4 months ago

#90 closed defect (fixed)

python: (2.7.6) problem determining filesystem encoding

Reported by: Lewisr Owned by: ydario
Priority: Feedback Pending Milestone:
Component: python Version:
Severity: highest Keywords:
Cc:

Description

This is tangentially related to getting SSL support for yum repos, so I'll open this here. Frankly, python is so complex that we should probably break it out into its own project, here, but I'll leave that to you, Yuri. ;-)

I needed to install M2Crypto as I am having difficulty accessing my repo mirror via SSL (different ticket - to be created). M2Crypto seems to want to be installed via setuptools. Installing setuptools should be fairly straightforward:

wget --no-check-certificate https://bootstrap.pypa.io/ez_setup.py -O - | python

however, once the installation starts, the fun begins:

Extracting in c:/var/temp/tmpcqr6rs
Now working in c:/var/temp/tmpcqr6rs/setuptools-5.6
Installing Setuptools
running install
running bdist_egg
running egg_info
writing requirements to setuptools.egg-info/requires.txt
writing setuptools.egg-info/PKG-INFO
writing top-level names to setuptools.egg-info/top_level.txt
writing dependency_links to setuptools.egg-info/dependency_links.txt
writing entry points to setuptools.egg-info/entry_points.txt
Traceback (most recent call last):
  File "setup.py", line 219, in <module>
    dist = setuptools.setup(**setup_params)
  File "J:/usr/lib/python2.7/distutils/core.py", line 152, in setup
    dist.run_commands()
  File "J:/usr/lib/python2.7/distutils/dist.py", line 953, in run_commands
    self.run_command(cmd)
  File "J:/usr/lib/python2.7/distutils/dist.py", line 972, in run_command
    cmd_obj.run()
  File "C:/var/temp/tmpcqr6rs/setuptools-5.6/setuptools/command/install.py", line 67, in run
    self.do_egg_install()
  File "C:/var/temp/tmpcqr6rs/setuptools-5.6/setuptools/command/install.py", line 109, in do_egg_install
    self.run_command('bdist_egg')
  File "J:/usr/lib/python2.7/distutils/cmd.py", line 326, in run_command
    self.distribution.run_command(command)
  File "J:/usr/lib/python2.7/distutils/dist.py", line 972, in run_command
    cmd_obj.run()
  File "C:/var/temp/tmpcqr6rs/setuptools-5.6/setuptools/command/bdist_egg.py", line 152, in run
    self.run_command("egg_info")
  File "J:/usr/lib/python2.7/distutils/cmd.py", line 326, in run_command
    self.distribution.run_command(command)
  File "J:/usr/lib/python2.7/distutils/dist.py", line 972, in run_command
    cmd_obj.run()
  File "C:/var/temp/tmpcqr6rs/setuptools-5.6/setuptools/command/egg_info.py", line 168, in run
    self.find_sources()
  File "C:/var/temp/tmpcqr6rs/setuptools-5.6/setuptools/command/egg_info.py", line 193, in find_sources
    mm.run()
  File "C:/var/temp/tmpcqr6rs/setuptools-5.6/setuptools/command/egg_info.py", line 277, in run
    self.add_defaults()
  File "C:/var/temp/tmpcqr6rs/setuptools-5.6/setuptools/command/egg_info.py", line 306, in add_defaults
    sdist.add_defaults(self)
  File "C:/var/temp/tmpcqr6rs/setuptools-5.6/setuptools/command/sdist.py", line 151, in add_defaults
    self.filelist.append(fn)
  File "C:/var/temp/tmpcqr6rs/setuptools-5.6/setuptools/command/egg_info.py", line 220, in append
    if self._safe_path(path):
  File "C:/var/temp/tmpcqr6rs/setuptools-5.6/setuptools/command/egg_info.py", line 240, in _safe_path
    u_path = unicode_utils.filesys_decode(path)
  File "C:/var/temp/tmpcqr6rs/setuptools-5.6/setuptools/unicode_utils.py", line 31, in filesys_decode
    return path.decode(enc)
TypeError: decode() argument 1 must be string, not None
Something went wrong during the installation.
See the error message above.

Looking at the section in question in unicode_utils.py, he see:

def filesys_decode(path):
    """
    Ensure that the given path is decoded,
    NONE when no expected encoding works
    """

    fs_enc = sys.getfilesystemencoding()
    if isinstance(path, decoded_string):
        return path

    for enc in (fs_enc, "utf-8"):
        try:
            return path.decode(enc)
        except UnicodeDecodeError:
            continue

Result: cannot install setuptools.

I've tried this under a variety of shells, too, but to no avail. Filesystem is JFS.

Change History (8)

comment:1 Changed 5 years ago by ydario

  • Owner set to ydario
  • Status changed from new to accepted

Since encoding is None, .decode fails... This workaround works for me:

for enc in (fs_enc, "utf-8"):

try:

return path #.decode(enc)

except UnicodeDecodeError?:

continue

and instead of easy setup, I followed the standard way of installing python packages

python setup.py install

I'm leaving this ticket open since we need a fix for path.decode() function.

comment:2 Changed 5 years ago by lewisr

Confirming that your edit works for me, Yuri; thanks. Unfortunately, I don't know enough python to be of much use, so I truly appreciate the pointer. However, as you say, we should keep this open as this is merely a workaround and not a fix to the underlying problem.

comment:3 Changed 4 months ago by diver

  • Priority changed from major to Feedback Pending
  • Severity set to highest

I'm not sure if this is still the case, as setuptools changed a lot since this ticket was opened

comment:4 Changed 4 months ago by lewisr

Well, we still fail during the install, but in a different place:

Traceback (most recent call last):
  File "setup.py", line 181, in <module>
    dist = setuptools.setup(**setup_params)
  File "N:/USR/lib/python2.7/distutils/core.py", line 152, in setup
    dist.run_commands()
  File "N:/USR/lib/python2.7/distutils/dist.py", line 953, in run_commands
    self.run_command(cmd)
  File "N:/USR/lib/python2.7/distutils/dist.py", line 972, in run_command
    cmd_obj.run()
  File "N:/var/temp/tmpfmn52e/setuptools-33.1.1/setuptools/command/install.py", line 67, in run
    self.do_egg_install()
  File "N:/var/temp/tmpfmn52e/setuptools-33.1.1/setuptools/command/install.py", line 109, in do_egg_install
    self.run_command('bdist_egg')
  File "N:/USR/lib/python2.7/distutils/cmd.py", line 326, in run_command
    self.distribution.run_command(command)
  File "N:/USR/lib/python2.7/distutils/dist.py", line 972, in run_command
    cmd_obj.run()
  File "N:/var/temp/tmpfmn52e/setuptools-33.1.1/setuptools/command/bdist_egg.py", line 161, in run
    cmd = self.call_command('install_lib', warn_dir=0)
  File "N:/var/temp/tmpfmn52e/setuptools-33.1.1/setuptools/command/bdist_egg.py", line 147, in call_command
    self.run_command(cmdname)
  File "N:/USR/lib/python2.7/distutils/cmd.py", line 326, in run_command
    self.distribution.run_command(command)
  File "N:/USR/lib/python2.7/distutils/dist.py", line 972, in run_command
    cmd_obj.run()
  File "N:/var/temp/tmpfmn52e/setuptools-33.1.1/setuptools/command/install_lib.py", line 12, in run
    outfiles = self.install()
  File "N:/USR/lib/python2.7/distutils/command/install_lib.py", line 115, in install
    outfiles = self.copy_tree(self.build_dir, self.install_dir, preserve_symlinks=1)
  File "N:/var/temp/tmpfmn52e/setuptools-33.1.1/setuptools/command/install_lib.py", line 90, in copy_tree
    assert preserve_mode and preserve_times and not preserve_symlinks
AssertionError
Something went wrong during the installation.
See the error message above.

install_lib.py beginning at line 86 looks like this:

def copy_tree(
        self, infile, outfile,
        preserve_mode=1, preserve_times=1, preserve_symlinks=0, level=1
):
    assert preserve_mode and preserve_times and not preserve_symlinks
    exclude = self.get_exclusions()

On the surface, I don't see anything wrong with the assertion which would cause it to be false.

comment:5 Changed 4 months ago by diver

I wonder what setuptool you use. As have version 34.4.1 installed. And I have exactly this fixed in this version.

comment:6 Changed 4 months ago by diver

And 34 is available as rpm since ages iirc

comment:7 Changed 4 months ago by diver

any news when using the rpm setuptools version 34?

comment:8 Changed 4 months ago by lewisr

  • Resolution set to fixed
  • Status changed from accepted to closed

The original issue (problem determining filesystem encoding) was resolved for the testcase by using Yuri's workaround. I can confirm that installing setuptools from RPM works; that was never the issue. I was installing setuptools from ez_setup.py. Perhaps the build of setuptools downloaded using that method is incompatible with our environment.

This issue started as an inability to install pip (and wheel) necessary to install M2Crypto. Now that setuptools works, I should open specific tickets for what is failing during the installation of pip.

Note: See TracTickets for help on using tickets.