fedora-modularity / libmodulemd Goto Github PK
View Code? Open in Web Editor NEWC Library for manipulating module metadata files
License: MIT License
C Library for manipulating module metadata files
License: MIT License
#!/usr/bin/env python
import gi
gi.require_version('Modulemd', '1.0') # noqa
from gi.repository import Modulemd
s = '''
---
document: modulemd
version: 2
data:
name: testmodule
stream: master
version: 1
context: 2017
summary: foo
description: >-
foo
license:
module:
- GPL
components:
rpms:
pkg:
rationale: Needed for test
multilib: [x86_64]
artifacts:
rpms:
- pkg-1.0.0-1.i686
- pkg-1.0.0-1.x86_64
...
'''
mmd = Modulemd.Module.new_from_string(s)
$ python repl.py
Traceback (most recent call last):
File "repl.py", line 34, in <module>
mmd = Modulemd.Module.new_from_string(s)
TypeError: constructor returned NULL
Right now, the parse_yaml()
function only returns the subset of results that validated the parser and reports errors using g_message()
. We should instead have some mechanism for returning an error code and the failed YAML document string to the calling function to handle its own logging.
This is a backwards compatible change.
Use cases for optional default stream: modules providing alternatives to non-modularized software and documents defining default streams under intents
User cases for optional default profiles: perhaps highly specialized modules that do not provide any profiles whatsoever (profiles are optional) and those defining their profiles under intents
By using declare_dependency()
for ease of bundling libmodulemd..
In theory, should be something like
modulemd_dep = declare_dependency(
include_directories : include_directories('.'),
link_with : modulemd_lib,
)
The following causes Python to segfault:
>>> import gi
>>> gi.require_version('Modulemd', '1.0')
>>> from gi.repository import Modulemd
>>> mmd = Modulemd.Module().new_from_file('tests/staged_data/formatted_testmodule.yaml')
>>> component = Modulemd.ComponentRpm()
>>> component.set_name('test')
>>> mmd.add_rpm_component(component)
This is the modulemd I'm using:
https://pagure.io/fm-orchestrator/blob/master/f/tests/staged_data/formatted_testmodule.yaml
Or if I generate it manually, "dup_nsvc()" of such module returns "None". Reproducer:
from gi import Repository
r = Repository.get_default() # noqa
r.require('Modulemd') # noqa
from gi.repository import Modulemd
print "Modulemd.get_version() ==", Modulemd.get_version()
data = """
document: modulemd
version: 1
data:
name: foo
stream: 0
version: 0
context: 0
summary: A test module in all its beautiful beauty.
description: This module demonstrates how to write simple modulemd files And can be used for testing the build and release pipeline.
license:
module: [ MIT ]
dependencies:
buildrequires:
platform: f28
requires:
platform: f28
references:
community: https://fedoraproject.org/wiki/Modularity
documentation: https://fedoraproject.org/wiki/Fedora_Packaging_Guidelines_for_Modules
tracker: https://taiga.fedorainfracloud.org/project/modularity
"""
mmd = Modulemd.Module.new_from_string(data)
mmd.upgrade()
print mmd.dup_nsvc()
Using path to file is fine, but not when you are dealing with something complex.
For instance, in rust ecosystem we just want to pull components for the buildroot. Yes, you can use filter
, but since that doesn't support globs/regex, it would be to complex to list 2-30 rpms per component.
g_autoptr(GPtrArray) objects = modulemd_objects_from_file (pool_tmpjoin (pool, TEST_DIR, fname, NULL), NULL);
for (guint i = 0; i < objects->len; i++)
{
if (MODULEMD_IS_MODULE (g_ptr_array_index (objects, i)))
{
ModulemdModule *module = g_ptr_array_index (objects, i);
add_module_solvables (repo, module);
}
}
code like this fails with
** Message: 11:54:53.974: GError is initialized.
fish: “and ./build/main” terminated by signal SIGSEGV (Address boundary error)
Moreover, it segfaults:
#0 0x00007ffff6e33cdf in free () at /lib64/libc.so.6
#1 0x00007ffff58a2c12 in yaml_parser_delete (parser=parser@entry=0x7fffffffdea0) at api.c:218
__PRETTY_FUNCTION__ = "yaml_parser_delete"
#2 0x00007ffff73e2495 in parse_yaml_file (path=0x62e670 "/home/brain/tmp/m/data/repodata/8e7a1438e8668c6a8a74eb6d9ff3362f8039dd5f4ab25a4bb2db30e685e639d8-modules.yaml.gz", data=data@entry=0x7fffffffe0c0, error=<optimized out>) at ../modulemd/modulemd-yaml-parser.c:113
result = 0
yaml_file = 0x0
parser =
{error = 544497952, problem = 0x74737961682f2f3a <error: Cannot access memory at address 0x74737961682f2f3a>, problem_offset = 0, problem_value = 0, problem_mark = {index = 0, line = 0, column = 0}, context = 0x0, context_mark = {index = 0, line = 0, column = 0}, read_handler = 0x0, read_handler_data = 0x0, input = {string = {start = 0x0, end = 0x0, current = 0x0}, file = 0x0}, eof = 4202117, buffer = {start = 0x629250 "\360\222b", end = 0x8000000100000007 <error: Cannot access memory at address 0x8000000100000007>, pointer = 0x610730 "", last = 0x629250 "\360\222b"}, unread = 6478576, raw_buffer = {start = 0x62af0d "\201C\216z\024\070\350f\214j\212t\353m\237\363\066/\200\071\335_J\262ZK\262\333\060\346\205\346\071\330]\r=\263\006\\3\203mvi<\373('\214\036\032\235P\030W\347\b\a\244\177\222\341\260U\252repodata/8e7a1438e8668c6a8a74eb6d9ff3362f8039dd5f4ab25a4bb2db30e685e639d8-modules.yaml.gz", end = 0x62af0f "\216z\024\070\350f\214j\212t\353m\237\363\066/\200\071\335_J\262ZK\262\333\060\346\205\346\071\330]\r=\263\006\\3\203mvi<\373('\214\036\032\235P\030W\347\b\a\244\177\222\341\260U\252repodata/8e7a1438e8668c6a8a74eb6d9ff3362f8039dd5f4ab25a4bb2db30e685e639d8-modules.yaml.gz", pointer = 0x0, last = 0x62d534 "\002"}, encoding = 6521616, offset = 195, mark = {index = 2, line = 0, column = 0}, stream_start_produced = 0, stream_end_produced = 0, flow_level = 0, tokens = {start = 0x0, end = 0x0, head = 0x0, tail = 0x0}, tokens_parsed = 0, token_available = 0, indents = {start = 0x0, end = 0x0, top = 0x0}, indent = 0, simple_key_allowed = 0, simple_keys = {start = 0x0, end = 0x0, top = 0x0}, states = {start = 0x0, end = 0x0, top = 0x0}, state = YAML_PARSE_STREAM_START_STATE, marks = {start = 0x0, end = 0x0, top = 0x0}, tag_directives = {start = 0x0, end = 0x0, top = 0x0}, aliases = {start = 0x0, end = 0x0, top = 0x0}, document = 0x0}
__func__ = "parse_yaml_file"
#3 0x00007ffff73ca168 in modulemd_objects_from_file (yaml_file=<optimized out>, error=<optimized out>) at ../modulemd/modulemd-common.c:53
data = 0x0
__func__ = "modulemd_objects_from_file"
#4 0x0000000000401cd6 in main () at ../main.c:205
fp = 0x62e510
pool = 0x610730
repo = 0x629250
chksumtype = 52
chksum = 0x62af0f "\216z\024\070\350f\214j\212t\353m\237\363\066/\200\071\335_J\262ZK\262\333\060\346\205\346\071\330]\r=\263\006\\3\203mvi<\373('\214\036\032\235P\030W\347\b\a\244\177\222\341\260U\252repodata/8e7a1438e8668c6a8a74eb6d9ff3362f8039dd5f4ab25a4bb2db30e685e639d8-modules.yaml.gz"
fname = 0x62af4f "repodata/8e7a1438e8668c6a8a74eb6d9ff3362f8039dd5f4ab25a4bb2db30e685e639d8-modules.yaml.gz"
objects = 0x7ffff7192369 <_g_signal_init+153>
libmodulemd crashes when no defaults to Prioritizer were added.
reproducer:
import gi
gi.require_version('Modulemd', '1.0')
import gi.repository.Modulemd
prioritizer = gi.repository.Modulemd.Prioritizer()
prioritizer.resolve()
This should not crash, but should throw an error.
Intents support was added for 1.5.0, but copying existing intents when copying a defaults object was missed. This needs to be fixed before 1.5.0 is released.
It'd be nice to have modulemd-style convenience functions for the defaults, too. I would like to load and dump multiple defaults (and only defaults) from/to strings and files.
When upgrading a v1 modulemd to v2 that doesn't have the eol
property set, you get the warning __main__:1: Warning: g_date_valid: assertion 'd != NULL' failed
. This warning isn't useful since a v1 modulemd is valid without having the eol
property set. It'd be friendlier to not have this warning, especially since it can pollute application logs if the upgrade
function is often run.
Here is how to produce the warning message:
>>> import gi
>>> gi.require_version('Modulemd', '1.0')
>>> from gi.repository import Modulemd
>>> mmd = Modulemd.Module().new_from_string('document: modulemd\nversion: 1\ndata:\n summary: A test module in all its beautiful beauty\n description: >-\n This module demonstrates how to write simple modulemd files And\n can be used for testing the build and release pipeline.\n license:\n module: [ MIT ]\n dependencies:\n buildrequires:\n platform: f28\n requires:\n platform: f28\n references:\n community: https://docs.pagure.org/modularity/\n documentation: https://fedoraproject.org/wiki/Fedora_Packaging_Guidelines_for_Modules\n profiles:\n default:\n rpms:\n - tangerine\n api:\n rpms:\n - perl-Tangerine\n - tangerine\n components:\n rpms:\n perl-List-Compare:\n rationale: A dependency of tangerine.\n ref: master\n perl-Tangerine:\n rationale: Provides API for this module and is a dependency of tangerine.\n ref: master\n tangerine:\n rationale: Provides API for this module.\n buildorder: 10\n ref: master\n')
>>> mmd.upgrade()
__main__:1: Warning: g_date_valid: assertion 'd != NULL' failed
True
Right now, the YAML parser and emitter logic is all in monolithic files with all the YAML parsing.
It would be better and more maintainable in the future if the parser and emitter sections were handled by the C files containing the objects they represent.
I tried this today:
import gi
gi.require_version('Modulemd', '1.0')
from gi.repository import Modulemd # noqa
# Here, load the same file twice.
# All entries in these two lists should be duplicates of each other.
objects_from_repo_a = Modulemd.objects_from_file('modules.yaml')
objects_from_repo_b = Modulemd.objects_from_file('modules.yaml')
prioritizer = Modulemd.Prioritizer()
prioritizer.add(objects_from_repo_a, 0)
prioritizer.add(objects_from_repo_b, 1)
supposedly_merged_objects = prioritizer.resolve()
# I would expect all three of these to be the same length... but they are not.
print(len(objects_from_repo_a))
print(len(objects_from_repo_b))
print(len(supposedly_merged_objects))
Am I doing something wrong?
I'm working on porting MBS to libmodulemd and noticed that when running mmd.get_dependencies()
on a v2 modulemd (became v2 via upgrade), I seem to always get a list with one element, no matter how many requires or buildrequires the modulemd specifies. To actually get those requires and buildrequires, I have to run mmd.get_dependencies()[0].get_requires()
and mmd.get_dependencies()[0].get_buildrequires()
. These functions return a dictionary with all the dependencies in the modulemd.
So with that said, what circumstance causes mmd.get_dependencies()
to return a list that is larger than one element? I would have expected that mmd.get_dependencies()
would return a ModulemdDependencies
directly instead of this one element list.
It also prints "Incompatible modulemd version" while running these methods. Reproducer:
from gi import Repository
r = Repository.get_default() # noqa
r.require('Modulemd') # noqa
from gi.repository import Modulemd
data = """
document: modulemd
version: 1
data:
name: testmodule
stream: master
version: 20180205135154
context: c2c572ec
summary: A test module in all its beautiful beauty
description: This module demonstrates how to write simple modulemd files And can
be used for testing the build and release pipeline.
license:
module:
- MIT
dependencies:
buildrequires:
platform: f28
requires:
platform: f28
references:
community: https://docs.pagure.org/modularity/
documentation: https://fedoraproject.org/wiki/Fedora_Packaging_Guidelines_for_Modules
xmd:
mbs:
buildrequires:
platform:
filtered_rpms: []
ref: virtual
stream: f28
version: '3'
context: '00000000'
commit: 65a7721ee4eff44d2a63fb8f3a8da6e944ab7f4d
requires:
platform:
filtered_rpms: []
ref: virtual
stream: f28
version: '3'
rpms:
perl-List-Compare:
ref: ac0f3bccca9dcb8465c434ac4c38974bf6205c28
perl-Tangerine:
ref: 7e96446223f1ad84a26c7cf23d6591cd9f6326c6
tangerine:
ref: c0f9a7dbd8cf823a2bdc19eeeed20d22b0aa52bf
mse: true
scmurl: https://src.fedoraproject.org/modules/testmodule.git?#65a7721ee4eff44d2a63fb8f3a8da6e944ab7f4d
profiles:
default:
rpms:
- tangerine
api:
rpms:
- perl-Tangerine
- tangerine
components:
rpms:
perl-List-Compare:
rationale: A dependency of tangerine.
repository: git://pkgs.fedoraproject.org/rpms/perl-List-Compare
ref: master
cache: http://pkgs.fedoraproject.org/repo/pkgs/perl-List-Compare
tangerine:
rationale: Provides API for this module.
buildorder: 10
repository: git://pkgs.fedoraproject.org/rpms/tangerine
ref: master
cache: http://pkgs.fedoraproject.org/repo/pkgs/tangerine
perl-Tangerine:
rationale: Provides API for this module and is a dependency of tangerine.
repository: git://pkgs.fedoraproject.org/rpms/perl-Tangerine
ref: master
cache: http://pkgs.fedoraproject.org/repo/pkgs/perl-Tangerine
"""
mmd = Modulemd.Module.new_from_string(data)
mmd.upgrade()
mmd.set_rpm_buildopts({'macros': '%my_macro 1'})
mmd = Modulemd.Module.new_from_string(mmd.dumps())
assert mmd.get_rpm_buildopts() != {}
In [1]: import gi
In [2]: gi.require_version('Modulemd', '1.0')
In [3]: from gi.repository import Modulemd
In [4]: x = '/home/ipanova/Downloads/fcb39b9a0b045c35442f1ab05dfd230148b4662fe1ab6ad7cd3e9722ffce8b84-modules.yaml'
In [5]: modules = Modulemd.objects_from_file(x)
In [6]: modules
Out[6]:
[<Modulemd.Module object at 0x7fb68a5dd2d0 (ModulemdModule at 0x5645864fb000)>,
<Modulemd.Module object at 0x7fb689cc8d70 (ModulemdModule at 0x5645864fb0f0)>,
<Modulemd.Module object at 0x7fb689cc8d20 (ModulemdModule at 0x5645864fb1e0)>,
<Modulemd.Module object at 0x7fb689cc8cd0 (ModulemdModule at 0x5645864fb2d0)>,
<Modulemd.Module object at 0x7fb689cc8c80 (ModulemdModule at 0x5645864fb3c0)>,
<Modulemd.Defaults object at 0x7fb689cc8be0 (ModulemdDefaults at 0x564586493280)>,
<Modulemd.Defaults object at 0x7fb689cc8aa0 (ModulemdDefaults at 0x5645864932c0)>,
<Modulemd.Defaults object at 0x7fb689cc8a50 (ModulemdDefaults at 0x564586493300)>]
In [7]: defaults = modules[-1]
In [8]: defaults
Out[8]: <Modulemd.Defaults object at 0x7fb689cc8a50 (ModulemdDefaults at 0x564586493300)>
In [9]: defaults.dumps()
Out[9]: '---\ndocument: modulemd-defaults\nversion: 1\ndata:\n module: django\n profiles:\n 1.6: [default]\n intents: {}\n...\n'
Meanwhile in the downloaded modules.yaml file the defaults for django look like:
---
document: modulemd-defaults
version: 1
data:
module: django
profiles:
1.6: [default]
...
Version : libmodulemd-1.5.2-1.fc28.x86_64
NOTE: Intents where NOT present in the original yaml document
Same behaviours is noticed with modulemd_defaults_dump ()
When priority is a too big number it either overflows or add() fails:
module_defaults_prioritizer.add(defaults, 99999999999)
The result is that the added defaults are not present in the prioritizer output.
You can test this with the following Python code:
import gi
from gi.repository import Modulemd
import requests
mmd_str = requests.get('https://src.fedoraproject.org/modules/testmodule/raw/master/f/testmodule.yaml').text
mmd = Modulemd.Module.new_from_string(mmd_str)
mmd.props.buildopts is None
I'm currently using libmodulemd v1.5.1.
For this particular mmd file, the new_from_string method shows following warning, but things are working, so not a critical issue :).
from gi import Repository
r = Repository.get_default() # noqa
r.require('Modulemd') # noqa
from gi.repository import Modulemd
data = """
data:
api:
rpms: [tangerine, perl-Tangerine]
components:
rpms:
perl-List-Compare: {cache: 'http://pkgs.fedoraproject.org/repo/pkgs/perl-List-Compare',
rationale: A dependency of tangerine., ref: 76f9d8c8e87eed0aab91034b01d3d5ff6bd5b4cb,
repository: 'git://pkgs.fedoraproject.org/rpms/perl-List-Compare'}
perl-Tangerine: {cache: 'http://pkgs.fedoraproject.org/repo/pkgs/perl-Tangerine',
rationale: Provides API for this module and is a dependency of tangerine.,
ref: 4ceea43add2366d8b8c5a622a2fb563b625b9abf, repository: 'git://pkgs.fedoraproject.org/rpms/perl-Tangerine'}
tangerine: {buildorder: 10, cache: 'http://pkgs.fedoraproject.org/repo/pkgs/tangerine',
rationale: Provides API for this module., ref: fbed359411a1baa08d4a88e0d12d426fbf8f602c,
repository: 'git://pkgs.fedoraproject.org/rpms/tangerine'}
dependencies:
buildrequires:
zebra: stripes
base-runtime: master
fluffy: slippers
requires:
fidget: spinner
base-runtime: master
silly: puddy
glass: water
description: This module demonstrates how to write simple modulemd files And can
be used for testing the build and release pipeline.
filter: {}
license:
module: [MIT]
name: testmodule
profiles:
default:
rpms: [tangerine]
references: {community: 'https://fedoraproject.org/wiki/Modularity', documentation: 'https://fedoraproject.org/wiki/Fedora_Packaging_Guidelines_for_Modules',
tracker: 'https://taiga.fedorainfracloud.org/project/modularity'}
stream: master
summary: A test module in all its beautiful beauty
version: 20170109091357
xmd:
mbs:
buildrequires:
zebra:
ref: 08f8671f5925ecbd7c55d83e8368dd0be81ef8ed
base-runtime:
ref: ae993ba84f4bce554471382ccba917ef16265f11
fluffy:
ref: ea83325b231f64c575ea104bb698e9d43b80469a
requires:
fidget:
ref: 5aa32bd61aadaec8295ef90f79daaf190e387509
base-runtime:
ref: ae993ba84f4bce554471382ccba917ef16265f11
silly:
ref: 0e2a1f9246118180d61aad731923ebd85154bc6e
glass:
ref: ea8753e832974dd5b0d95cd6bb7d26727d2ba742
commit: 7fea453bc362cc8e5aa41e129e689baea853653d
scmurl: git://pkgs.stg.fedoraproject.org/modules/testmodule.git?#7fea453
document: modulemd
version: 1
"""
mmd = Modulemd.Module.new_from_string(data)
This will make it possible to avoid always setting SimpleSet
s when saving, even if they have the same effective contents.
See https://github.com/sgallagher/c-modulemd/blob/a4e1617/modulemd/modulemd-profile.c#L100 for an example use-case.
When setting boolean variant in xmd, libmodulemd fails with:
** Message: Unhandled variant type
** Message: Error writing arbitrary mapping
** Message: Error emitting variant hashtable
** Message: Failed to emit xmd
** Message: Failed to emit data
** Message: Failed to process root
** Message: Could not emit YAML
** Message: Error emitting YAML string: Unhandled variant type
We would like to use this for "mse" flag in XMD section in MBS.
When loading a modulemd from a file, the following fails with "Invalid document [Unknown modulemd version]. Skipping it":
document: modulemd
data:
dependencies:
buildrequires:
platform: f28
requires:
platform: f28
version: 1
When specifying the version before dependencies
is processed, it works as expected.
document: modulemd
version: 1
data:
dependencies:
buildrequires:
platform: f28
requires:
platform: f28
I'm not sure how to workaround this in MBS as both should be valid modulemds, or at least they were in the Python implementation of modulemd.
Example data:
document: modulemd-defaults
version: 1
data:
module: testmodule
stream: 2.0
profiles:
master: [default]
This triggers following output:
** Message: 09:15:48.608: Default stream missing from profile defaults
** Message: 09:15:48.608: Invalid [ModulemdDefaults] document [Default stream missing from profile defaults].
** Message: 09:15:48.608: Invalid document [Default stream missing from profile defaults]. Skipping it.
I think this behavior is incorrect. Consider following use cases:
When reading in a YAML stream, libmodulemd must keep track of the set of modules for which a defaults object exists and throw a validation error if defaults for the same module occur twice in one stream.
Right now, libmodulemd allows the following
We'll need libmodulemd built for YUM4 CentOS SIG:
https://wiki.centos.org/SpecialInterestGroup/ConfigManagementSIG/YUM4
The build is also needed for several infrastructure tools as they run on RHEL 7.
We can't build libmodulemd as it is, because RHEL doesn't have python3 and meson.
The following script will cause a segmentation fault:
from gi.repository import Modulemd
mmdd = Modulemd.Module.new_all_from_file_ext("defaults.yaml")
print (mmdd)
The reason is incorrect memory management of the returned array.
Hello,
I am not able to compile libdnf, when including any header from libmodulemd.
Here is minimal reproducer:
main.cpp.txt
Compiling using:
$ g++ main.cpp `pkg-config --cflags --libs modulemd`
When adding support for Intents, they weren't figured into default-merging. (Presently, they are ignored during the merge, so only the Intents set by the first defaults subdocument for a particular module will be used.
The proposed logic is to treat it like the Profiles: if the key does not exist, it is added with the new document's value. If the key does exist, override with the higher-valued subdocument when in override mode, otherwise simply check that they are identical and throw an error if they are not.
If I have a module component that has a ref specified, the ref is lost when I dump the module to a string.
Here is the reproducer in Python:
>>> import gi
>>> gi.require_version('Modulemd', '1.0')
>>> from gi.repository import Modulemd
>>> mmd = Modulemd.Module().new_from_string("""
... document: modulemd
... version: 1
... data:
... summary: A test module in all its beautiful beauty.
... description: This module demonstrates how to write simple modulemd files And can be used for testing the build and release pipeline.
... license:
... module: [ MIT ]
... dependencies:
... buildrequires:
... platform: el8
... requires:
... platform: el8
... references:
... community: https://fedoraproject.org/wiki/Modularity
... documentation: https://fedoraproject.org/wiki/Fedora_Packaging_Guidelines_for_Modules
... tracker: https://taiga.fedorainfracloud.org/project/modularity
... profiles:
... default:
... rpms:
... - acl
... api:
... rpms:
... - acl
... components:
... rpms:
... acl:
... rationale: needed
... ref: rhel-8.0
... modules:
... testmodule:
... ref: private-x
... rationale: Testing module inclusion.
... buildorder: 10
... """)
>>> assert mmd.get_module_components()['testmodule'].get_ref() == 'private-x'
>>> mmd2 = Modulemd.Module().new_from_string(mmd.dumps())
>>> assert mmd2.get_module_components()['testmodule'].get_ref() == 'private-x'
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AssertionError
I can replicate this with 1.6.1 and 1.6.2 through the gobject introspection bindings:
>>> mmd.dup_nsvc()
'mod:master:125ab64228a8:9edba152'
>>> mmd.peek_version()
20180814145704L
The DNF team proposes a change in how libmodulemd handles RPM NEVRAs.
We'd like to always get records in N-E:V-R.A format.
libmodulemd SHOULD fail on reading data in invalid format.
libmodulemd MUST fail on writing data in invalid format.
As part of this, we'd like to enforce '0:' epoch when the epoch is zero or empty in RPM.
In some cases missing epoch can represented as "any epoch", which may result in bad UX.
When trying to create a Moduemd.Module
object with bad YAML, it will hang as long as the bad YAML starts with document: modulemd\n
.
For example:
from gi.repository import Modulemd
Modulemd.Module().new_from_string('document: modulemd\nBad YAML')
We need libmodulemd to be able fail on bad modulemd files that users submit to MBS.
When dealing with multiple repositories, we need to handle how to merge content. For example, if we have a release repository (fedora-modular
) and an updates repository (fedora-modular-updates
) and both provides differing profiles in a modulemd-defaults file, we need to prefer the fedora-modular-updates
version.
After conversations with GNOME folks, it was made clear that a "proper" GObject API should have both object_peek()
and object_dup()
methods. The object_peek()
methods would be the same functionality as currently provided by object_get()
in libmodulemd and should be implemented as deprecating the latter.
object_dup()
methods must return a deep copy of the property.
from gi import Repository
r = Repository.get_default() # noqa
r.require('Modulemd') # noqa
from gi.repository import Modulemd
data = """
document: modulemd
version: 1
data:
summary: A test module in all its beautiful beauty.
description: This module demonstrates how to write simple modulemd files And can be used for testing the build and release pipeline.
license:
module: [ MIT ]
dependencies:
buildrequires:
platform: el8
requires:
platform: el8
references:
community: https://fedoraproject.org/wiki/Modularity
documentation: https://fedoraproject.org/wiki/Fedora_Packaging_Guidelines_for_Modules
tracker: https://taiga.fedorainfracloud.org/project/modularity
profiles:
default:
rpms:
- tangerine
api:
rpms:
- perl-Tangerine
- tangerine
components:
rpms:
attr:
rationale: A build dependency of acl
ref: rhel-8.0
buildorder: -1
"""
mmd = Modulemd.Module.new_from_string(data)
mmd.upgrade()
print mmd.get_rpm_components().values()[0].get_buildorder()
$ python mmd.py
18446744073709551615
Lookup by NEVRA is extremely slow... See filelists.xml for example how checksum is used.
I was pretty sure I have filled this RFE before, but can't find it.
Recently the parsing code was made to enforce that the RPM artifacts are in name-epoch:version-release form, but the setters weren't changed to match, so it's possible to use libmodulemd to generate a file it can't read back in.
I would suggest that set_rpm_artifacts() should warn and not set the artifacts at all if the contained strings are invalid. Stripping only non-validating elements would be possible too, but I don't think it's better to have a "corrupted" set of artifacts then no set at all.
The alternative of a replacement function that returns a GError doens't strike me as much better - it's not like we need to propagate the result back to a dialog, and there's typically nothing reasonable that a programmer could do to "recover" from this. Most likely the programmer just needs to fix the code that generates the artifacts list. The only pro is that in Python/Javascript it would turn the error into a backtrace rather than warning - which is nice - unless the caller uses props.artifacts_rpm =
...
Also, where to draw the line is unclear - GError for this is somewhat reasonable but GError for modulemd_module_set_name(module, NULL) is not.
In [1]: import gi
In [2]: gi.require_version('Modulemd', '1.0')
In [3]: from gi.repository import Modulemd
In [4]: x = '/home/ipanova/Downloads/fcb39b9a0b045c35442f1ab05dfd230148b4662fe1ab6ad7cd3e9722ffce8b84-modules.yaml'
In [5]: modules = Modulemd.objects_from_file(x)
In [6]: modules
Out[6]:
[<Modulemd.Module object at 0x7fc91c111370 (ModulemdModule at 0x55f1aeadf040)>,
<Modulemd.Module object at 0x7fc91c08d640 (ModulemdModule at 0x55f1aeadf130)>,
<Modulemd.Module object at 0x7fc91c08d5f0 (ModulemdModule at 0x55f1aeadf220)>,
<Modulemd.Module object at 0x7fc91c08d410 (ModulemdModule at 0x55f1aeadf310)>,
<Modulemd.Module object at 0x7fc91c08d2d0 (ModulemdModule at 0x55f1aeadf400)>,
<Modulemd.Defaults object at 0x7fc91c08d3c0 (ModulemdDefaults at 0x55f1aea3c280)>,
<Modulemd.Defaults object at 0x7fc91c08d5a0 (ModulemdDefaults at 0x55f1aea3c2c0)>,
<Modulemd.Defaults object at 0x7fc91c08d550 (ModulemdDefaults at 0x55f1aea3c300)>]
In [7]: module = modules[0]
In [8]: module.dump('/tmp/lal')
$ diff temp-django.yaml /tmp/lal
54a55,56
> buildopts:
> rpms: {}
Same behaviour is observed with dump() method.
Version: libmodulemd-1.5.2-1.fc28.x86_64
As discussed on IRC: Currently, components have a name property and profiles don't. They might have more reason than profiles to carry that information (it identifies them "more") but it's surprising for a user of the API to remember that one thing has it and the other doesn't.
When Ubuntu 17.04 disappeared from the Ubuntu mirrors, the automated builds (including the one that submits the code to Coverity for static analysis) broke. Instead of relying on Ubuntu, we should update the Travis environment to use a Fedora Rawhide container.
Only one of buildorder
or buildafter
should be used. Then MBS could take advantage of this submitting all builds at once and let koji handle build queue and priorities.
I picked first name which came to my mind, you can choose different one.
I'm not sure if it's expected that provided example ends with 'master' default stream.
I was assuming that this behavior is allowed only when 2nd record has higher priority (lower number).
Shouldn't the Prioritizer raise an error instead?
document: modulemd-defaults
version: 1
data:
module: testmodule
stream: 2.0
profiles:
master: [default]
---
document: modulemd-defaults
version: 1
data:
module: testmodule
stream: master
profiles:
master: [default]
When running dumps
on a Modulemd.Module
object, the string output doesn't include the RPM components.
This is blocking the Module Build Service from migrating to libmodulemd, which inherently blocks work on Module Stream Expansion.
Here is an example:
>>> import gi
>>> gi.require_version('Modulemd', '1.0')
>>> from gi.repository import Modulemd
>>> mmd = Modulemd.Module().new_from_string('document: modulemd\nversion: 1\ndata:\n summary: A test module in all its beautiful beauty\n description: >-\n This module demonstrates how to write simple modulemd files And\n can be used for testing the build and release pipeline.\n license:\n module: [ MIT ]\n dependencies:\n buildrequires:\n platform: f28\n requires:\n platform: f28\n references:\n community: https://docs.pagure.org/modularity/\n documentation: https://fedoraproject.org/wiki/Fedora_Packaging_Guidelines_for_Modules\n profiles:\n default:\n rpms:\n - tangerine\n api:\n rpms:\n - perl-Tangerine\n - tangerine\n components:\n rpms:\n perl-List-Compare:\n rationale: A dependency of tangerine.\n ref: master\n perl-Tangerine:\n rationale: Provides API for this module and is a dependency of tangerine.\n ref: master\n tangerine:\n rationale: Provides API for this module.\n buildorder: 10\n ref: master\n')
>>> mmd.get_rpm_components()
{'perl-Tangerine': <Modulemd.ComponentRpm object at 0x7fc8ec39eeb0 (ModulemdComponentRpm at 0x558af5756820)>, 'tangerine': <Modulemd.ComponentRpm object at 0x7fc8ee5746e0 (ModulemdComponentRpm at 0x558af5756e00)>, 'perl-List-Compare': <Modulemd.ComponentRpm object at 0x7fc8ee574690 (ModulemdComponentRpm at 0x558af56222c0)>}
>>> mmd2 = Modulemd.Module().new_from_string(mmd.dumps())
>>> mmd2.get_rpm_components()
{}
I'm going to implement proper multilib in Pungi, spec.v2.yaml says:
# A list of architectures with multilib
# installs, i.e. both i686 and x86_64
# versions will be installed on x86_64.
# TODO: This needs to be reworked or dropped as it's not at all useful in the current state.
# Optional, defaults to no multilib.
multilib: [ x86_64 ]
I talked with @contyk about this and he says the TODO basically means it is not clear what the arches should be in multilib field and if the current way how multilib field is defined is enough. We should have the final way how to define multilib so I can implement that in Pungi.
From the Pungi perspective, the current way with multilib: [x86_64]
would be enough. It would work like this:
multilib
strategy (see "multilib" here) would really be the option defining whether the particular RPMs end up in compose or not. So it might be possible that component with "multilib" set in modulemd won't appear in resulting variant, because the multilib
Pungi option will disallow that for particular variant.The get_as_strv()
routine currently returns in the order provided by the GHashTable
iterator. We should allow passing a sorting comparison function to get the results in an expected order.
When parsing a modules.yaml file and one of the modulemd entries doesn't contain a context
element, I get:
ValidationError (Modulemd:3cc94ec5-27ea-4d19-b942-a80ec0317d84) (Field is required: ['context'])
However, this seems to disagree with what is present in the spec.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.