createrepo 0.10.4-2 seems terribly broken (truncated metadata and bad behavior)

[Lewis Rosenthal]

createrepo-0.10.4-2.oc00.noarch.rpm is currently in experimental. This version of the script seems to dump all output to the console (even when told to be --quiet), and out of a total of 302 packages in the current on-disc ArcaOS repository, we seem to only get metadata built on the first(?) 34 of them.

Invocation:

[j:\localrep] sh -c j:/usr/bin/createrepo --no-database --simple-md-filenames .

Result:

scrolling output to the screen, and when done, very small metadata files, which, though readable, do not contain info on all files.

--simple-md-filenames seems to be respected, as does --no-database. --quiet (or -q) has no effect on the excess screen output, though it does seem to suppress the initial report of workers starting up (tried with no additional workers, then 10, and finally 20) and the final report of the files being written.

The older 0.4.11-2 works perfectly.

I'm marking this as an "other" component instead of python itself.

[Silvan Scherrer]

the --quiet issue I doubt it's a porting issue. Please try the same on another platform and see if --quiet behaves different there. But be sure to use the same version.
That only some (first 34) rpm are taken into account I need to test first. It would be helpfull, if you could test with less and see if it's really 34.

[Lewis Rosenthal]

I set up a local repository, and populated it with 36 packages. I ended up with 23 in my metadata.

I installed Fedora 21 in a VM (last release to use yum). Unfortunately, the latest package for createrepo for that which I see is 0.10.3. I have it installed, and will test. (I don't really feel like having to install the newer createrepo manually or build a package for it.) I also have the latest Mageia 6.1 which I can set up. This seems to have the 0.10.4 package available.

I'll try to stay on this during the week, so we can figure it out.

Thanks for the quick reply, Silvan, as always.

[Lewis Rosenthal]

Mageia 6.1

installed yum 3.4.3-19
installed createrepo 0.10.4-1

[root@localhost localrep]# createrepo --no-database --simple-md-filenames .
Warning: Requested more delta workers than workers. This is insane. Limiting.
Spawning worker 0 with 33 pkgs
Workers Finished
Reading package sizes in preparation for deltarpm creation
Sorting package files by size
Worker PID(5120) - Starting 0 workers to process deltarpms - max total work size (100000000) bytes
Saving Primary metadata
Saving file lists metadata
Saving other metadata
[root@localhost localrep]# ls -l repodata/
-rw-r--r-- 1 root root 4552 Dec 19 13:33 filelists.xml.gz
-rw-r--r-- 1 root root 2976 Dec 19 13:33 other.xml.gz
-rw-r--r-- 1 root root 8380 Dec 19 13:33 primary.xml.gz
-rw-r--r-- 1 root root 1310 Dec 19 13:33 repomd.xml

This all looks as I would expect. Checking for packages processed:

[root@localhost localrep]# gzip -cd repodata/filelists.xml.gz | grep -c \<version
33

This is correct.

createrepo 0.10.4-2 on OS/2 does not work this way. What else do you suggest we examine? On this platform, we get the following deplist for createrepo:

[root@localhost localrep]# yum deplist createrepo
package: createrepo.noarch 0.10.4-1.mga6
  dependency: python(abi)
   provider: python3.i586 3.5.3-1.4.mga6
   provider: python.i586 2.7.15-1.mga6
   provider: libpython2.7-stdlib.i586 2.7.15-1.mga6
  dependency: python(abi) = 2.7
   provider: python.i586 2.7.15-1.mga6
   provider: libpython2.7-stdlib.i586 2.7.15-1.mga6
  dependency: python-deltarpm
   provider: python-deltarpm.i586 3.6.1-2.mga6
  dependency: python-libxml2
   provider: libxml2-python.i586 2.9.7-1.mga6
  dependency: python-rpm
   provider: python2-rpm.i586 1:4.13.1-3.2.mga6
  dependency: python-yum >= 3.2.23
   provider: python-yum.noarch 3.4.3-19.mga6

[Silvan Scherrer]

please give me the same output on os/2 when you also do it with the very same rpm. so it's easier to compare. And I still wonder which limit is there. Means how many rpm are good and when it starts to go wrong. As in my limited tests I always got them all.

[Lewis Rosenthal]

Okay. To be clear:

SHELL=J:/usr/bin/sh.exe
EMXSHELL=J:/usr/bin/sh.exe
CONFIG_SHELL=J:/usr/bin/sh.exe
MAKESHELL=J:/usr/bin/sh.exe
EXECSHELL=J:/usr/bin/sh.exe

[j:\] comp j:\usr\bin\sh.exe j:\usr\bin\dash.exe

Compare file j:\usr\bin\sh.exe and file j:\usr\bin\dash.exe.

The files compare OK.

So, dash is my shell. Specifically:

dash-0.5.9.1-2.oc00.pentium4
dash-sh-0.5.9.1-2.oc00.pentium4

So:

# createrepo --no-database --simple-md-filenames .
Spawning worker 0 with 33 pkgs
Worker 0:
iso-8859-1 encoding on CUPS printing system provides a portable printing layer f
or
UNIX® operating systems. It has been developed by Apple Inc.
to promote a standard printing solution for all UNIX vendors and users.
CUPS provides the System V and Berkeley command-line interfaces.
CUPS was ported to OS/2 to have the same benefit as UNIX has.

*** 3905 39883 3017

<package type="rpm">
  <name>cups</name>
  <arch>i686</arch>
  <version epoch="1" ver="2.1.3" rel="10.oc00"/>
  <checksum type="sha256" pkgid="YES">f7dd5d1af267fc292f081c12d6e09a9b9fe41677c1
a7e9231b6ae4db354aa0c2</checksum>
  <summary>CUPS</summary>
  <description>CUPS printing system provides a portable printing layer for
UNIX┬® operating systems. It has been developed by Apple Inc.
to promote a standard printing solution for all UNIX vendors and users.
CUPS provides the System V and Berkeley command-line interfaces.
CUPS was ported to OS/2 to have the same benefit as UNIX has.</description>
  <packager></packager>

(output continues)

e="1511784000">- add ETC to the env, as else tcpip32.dll doesn't find any dns na
mes</changelog>

</package>
Workers Finished
Saving Primary metadata
Saving file lists metadata
Saving other metadata

(The last package alphabetically in the local repository is cups-libs-2.1.3-10.oc00.i686.rpm.)

# ls -l repodata/
total 20
-rw-r--r-- 1 root 0 4083 Dec 23 14:54 filelists.xml.gz
-rw-r--r-- 1 root 0 3483 Dec 23 14:54 other.xml.gz
-rw-r--r-- 1 root 0 7082 Dec 23 14:54 primary.xml.gz
-rw-r--r-- 1 root 0 1310 Dec 23 14:54 repomd.xml

Now:

# gzip -cd repodata/filelists.xml.gz | grep -c \<version
23

We're missing 10 packages. Also, judging from the output to the screen, those are the missing packages.

Packages in the repo:

arcanoae-rel-0.0.0-1.oc00.noarch.rpm
ash-0.0.1-1.oc00.i686.rpm
ash-0.0.1-1.oc00.pentium4.rpm
ASPIROUT-1.1.10-1.oc00.i686.rpm
bash-3.2.0-8.oc00.i686.rpm
bc-1.06-1.oc00.i686.rpm
bc-1.06-1.oc00.pentium4.rpm
bww-resources-rpm-1.1.2-1.oc00.noarch.rpm
bzip2-1.0.6-6.oc00.i686.rpm
bzip2-1.0.6-6.oc00.pentium4.rpm
bzip2-libs-1.0.6-6.oc00.i686.rpm
bzip2-libs-1.0.6-6.oc00.pentium4.rpm
ca-certificates-2017.11-1.oc00.noarch.rpm
cairo-1.12.18-4.oc00.i686.rpm
cairo-1.12.18-4.oc00.pentium4.rpm
cdrtools-3.01.1-1.oc00.i386.rpm
coreutils-8.26-2.oc00.i686.rpm
coreutils-8.26-2.oc00.pentium4.rpm
coreutils-common-8.26-2.oc00.i686.rpm
coreutils-common-8.26-2.oc00.pentium4.rpm
cpio-2.12-1.oc00.i686.rpm
cpio-2.12-1.oc00.pentium4.rpm
cube-2.6-5.oc00.i386.rpm
cups-2.1.3-10.oc00.i686.rpm
cups-2.1.3-10.oc00.pentium4.rpm
cups-client-2.1.3-10.oc00.i686.rpm
cups-client-2.1.3-10.oc00.pentium4.rpm
cups-filesystem-2.1.3-10.oc00.noarch.rpm
cups-filters-1.17.2-3.oc00.i686.rpm
cups-filters-1.17.2-3.oc00.pentium4.rpm
cups-filters-libs-1.17.2-3.oc00.i686.rpm
cups-filters-libs-1.17.2-3.oc00.pentium4.rpm
cups-libs-2.1.3-10.oc00.i686.rpm

Those are:

2 i386 packages
15 i686 packages
12 pentium4 packages
4 noarch packages

(We're not just skipping a particular arch.)

yum list recent says:

ASPIROUT.i686
arcanoae-rel.noarch
ash.i686
ash.pentium4
bash.i686
bc.i686
bc.pentium4
bww-resources-rpm.noarch
bzip2.i686
bzip2.pentium4
bzip2-libs.i686
bzip2-libs.pentium4
ca-certificates.noarch
cairo.i686
cairo.pentium4
cdrtools.i386
coreutils.i686
coreutils.pentium4
coreutils-common.i686
coreutils-common.pentium4
cpio.i686
cpio.pentium4
cube.i386

The screen dump from createrepo would indicate that when we start dumping to the screen, we stop adding to the metadata files (the last package listed in the metadata is cube, and the first dumped to the screen is cups; there are 10 cups packages in the directory, and these are all missing from the metadata).

[Lewis Rosenthal]

Actually, this looks like something with the CUPS packages and some others which createrepo does not like.

Filling the repo with:

cups-2.1.3-10.oc00.i686.rpm
cups-client-2.1.3-10.oc00.i686.rpm
cups-filters-1.17.2-3.oc00.i686.rpm
cups-filters-libs-1.17.2-3.oc00.i686.rpm
cups-libs-2.1.3-10.oc00.i686.rpm

gets us the same screen output processing the CUPS packages, and resulting repository metadata of no packages.

It's not the file size, either. Trimming the local repository down further to just:

12-17-18 19:16 51,108 0 ___A_ cups-client-2.1.3-10.oc00.i686.rpm

yields the same result (screen output during processing and no packages in the metadata).

Empty repodata files, attached (in case there are any clues in the files which were not dumped to the screen).

[Lewis Rosenthal]

Empty repodata after running craeterepo with only cups-client package.

[Silvan Scherrer]

yes cups triggers it. I can reproduce that.

[Silvan Scherrer]

should be fixed in latest rpm

[Lewis Rosenthal]

Not quite.

It seems that 0.10.4-3 is not getting the filelists metadata created in a readable fashion. I'm also experiencing difficulty reading the metadata from netlabs-exp (has that server been updated to this build, yet?).

YUM returns:

failure: repodata/ba60dacc27853eeac33d956f326b1ea30cacd48e1cedf27b9e13e1c501e9b1a6-filelists.sqlite.bz2 from netlabs-exp: [Errno 256] No more mirrors to try.

This is (I think) consistent with an error I get from a local repo trying to read filelists.xml.gz. The xml file appears to be truncated.

[Lewis Rosenthal]

I should note that I still get more screen output during a normal createrepo pass than I would expect, though the count of processed packages reported is correct, now.

[Silvan Scherrer]

This output is right and is from SQL not createrepo. So a SQL or python or whatever issue. Createrepo just shows it.

[Lewis Rosenthal]

Okay. Unless I can produce another testcase, let's leave this closed.