python: (2.7.6) trouble with SSL (to access repo from https)

[Lewis Rosenthal]

I wanted to test accessing an encrypted repo over SSL. As www.2rosenthals.com is also available over SSL (with a recognized thrid-party cert), I simply edited my repo files to use https instead of http in the URI. However, yum check-update comes back with:

Traceback (most recent call last):
  File "J:\USR\BIN\YUM", line 29, in <module>
    yummain.user_main(sys.argv[1:], exit_code=True)
  File "/@unixroot/usr/share/yum-cli/yummain.py", line 254, in user_main
    errcode = main(args)
  File "/@unixroot/usr/share/yum-cli/yummain.py", line 109, in main
    result, resultmsgs = base.doCommands()
  File "/@unixroot/usr/share/yum-cli/cli.py", line 352, in doCommands
    self._getTs(needTsRemove)
  File "/@unixroot/usr/lib/python2.7/site-packages/yum/depsolve.py", line 99, in _getTs
    self._getTsInfo(remove_only)
  File "/@unixroot/usr/lib/python2.7/site-packages/yum/depsolve.py", line 110, in _getTsInfo
    pkgSack = self.pkgSack
  File "/@unixroot/usr/lib/python2.7/site-packages/yum/__init__.py", line 777, in <lambda>
    pkgSack = property(fget=lambda self: self._getSacks(),
  File "/@unixroot/usr/lib/python2.7/site-packages/yum/__init__.py", line 567, in _getSacks
    self.repos.populateSack(which=repos)
  File "/@unixroot/usr/lib/python2.7/site-packages/yum/repos.py", line 279, in populateSack
    sack.populate(repo, mdtype, callback, cacheonly)
  File "/@unixroot/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 164, in populate
    if self._check_db_version(repo, mydbtype):
  File "/@unixroot/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 223, in _check_db_version
    return repo._check_db_version(mdtype)
  File "/@unixroot/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1227, in _check_db_version
    repoXML = self.repoXML
  File "/@unixroot/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1419, in <lambda>
    repoXML = property(fget=lambda self: self._getRepoXML(),
  File "/@unixroot/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1411, in _getRepoXML
    self._loadRepoXML(text=self)
  File "/@unixroot/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1401, in _loadRepoXML
    return self._groupLoadRepoXML(text, self._mdpolicy2mdtypes())
  File "/@unixroot/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1376, in _groupLoadRepoXML
    if self._commonLoadRepoXML(text):
  File "/@unixroot/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1201, in _commonLoadRepoXML
    result = self._getFileRepoXML(local, text)
  File "/@unixroot/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 986, in _getFileRepoXML
    size=102400) # setting max size as 100K
  File "/@unixroot/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 810, in _getFile
    size=size
  File "/@unixroot/usr/lib/python2.7/site-packages/urlgrabber/mirror.py", line 411, in urlgrab
    return self._mirror_try(func, url, kw)
  File "/@unixroot/usr/lib/python2.7/site-packages/urlgrabber/mirror.py", line 397, in _mirror_try
    return func_ref( *(fullurl,), **kwargs )
  File "/@unixroot/usr/lib/python2.7/site-packages/urlgrabber/grabber.py", line 927, in urlgrab
    return self._retry(opts, retryfunc, url, filename)
  File "/@unixroot/usr/lib/python2.7/site-packages/urlgrabber/grabber.py", line 845, in _retry
    r = apply(func, (opts,) + args, {})
  File "/@unixroot/usr/lib/python2.7/site-packages/urlgrabber/grabber.py", line 913, in retryfunc
    fo = URLGrabberFileObject(url, filename, opts)
  File "/@unixroot/usr/lib/python2.7/site-packages/urlgrabber/grabber.py", line 1001, in __init__
    self._do_open()
  File "/@unixroot/usr/lib/python2.7/site-packages/urlgrabber/grabber.py", line 1072, in _do_open
    fo, hdr = self._make_request(req, opener)
  File "/@unixroot/usr/lib/python2.7/site-packages/urlgrabber/grabber.py", line 1168, in _make_request
    fo = opener.open(req)
  File "/@unixroot/usr/lib/python2.7/urllib2.py", line 404, in open
    response = self._open(req, data)
  File "/@unixroot/usr/lib/python2.7/urllib2.py", line 422, in _open
    '_open', req)
  File "/@unixroot/usr/lib/python2.7/urllib2.py", line 382, in _call_chain
    result = func(*args)
  File "/@unixroot/usr/lib/python2.7/site-packages/urlgrabber/keepalive.py", line 347, in https_open
    return self.do_open(req)
  File "/@unixroot/usr/lib/python2.7/site-packages/urlgrabber/keepalive.py", line 233, in do_open
    h = self._get_connection(host)
  File "/@unixroot/usr/lib/python2.7/site-packages/urlgrabber/keepalive.py", line 350, in _get_connection
    return self._ssl_factory.get_https_connection(host)
AttributeError: SSLFactory instance has no attribute 'get_https_connection'

Python appears to have been compiled with SSL, e.g.:

>>> import ssl

comes back with no errors, and:

>>> import httplib
>>> conn = httplib.HTTPSConnection("mail.google.com")
>>> conn.request("GET", "/")
>>> r1 = conn.getresponse()
>>> print r1.status, r1.reason

comes back with:

200 OK

which leads me to believe that SSL is indeed working.

Looking at the related SSLFactory code, it appears that without M2Crypto, we fall back on python's SSL. I attempted to install M2Crypto, but ran into other problems (related to setupttols not installing).

[Lewis Rosenthal]

This appears to be a defect in urlgrabber 3.1.0.

Building urlgrabber 3.10.1 (which does not build or use sslfactory) and upgrading the installation in site-packages results in successful ssl negotiation, both with and without sslverify enabled. I have not tested client-side certs, yet.

I'm having some issues building the rpm for 3.10.1, so if someone else would be kind enough to build it before I do, we can close this ticket. ;-)

[Lewis Rosenthal]

urlgrabber 3.10.1 has a dependency on pycurl 7.19.5, which also builds easily (though I am having the same difficulties with the python rpm packaging (need to study how to work around things like the attempts to create hard links which fail and a few other OS/2 differences).

So, what's important to note is that we need both of these packages in order to upgrade yum's capability to support SSL.

[Yuri Dario]

Done in r520, r522, r524. Reopen in case of failures.

[Lewis Rosenthal]

I can't tell whether this is as a result of my own builds of pycurl and urlgrabber not having been installed via yum (I've cleaned up as best I can, I think), but with an https transaction (from yum: urlgrabber and pycurl alone seem to negotiate fine), I am getting a 416 error:

[Errno 14] HTTPS Error 416 - Requested Range Not Satisfiable

As I say, I'm not sure whether there is something being passed from the updated yum or something not quite right with pycurl or urlgrabber, which (per above) do seem to work independent of yum.

For testing, just use the 2rosenthals.com mirrors via https.

I'll keep investigating and will do a fresh install in a separate VM.

[Lewis Rosenthal]

A second VM configuration seems to indicate that the issue I'm seeing with the 416 error must be related to some leftover from my own pycurl and/or urlgrabber build(s). Updating from an installation without those worked fine.

Apologies for the noise!