oca / openupgradelib Goto Github PK

Hello,

This appears to be a regression from commit 22a6409, as I did a few upgrades last year without the error.
The line

produces this error:

2018-02-11 10:27:31,701 15701 INFO bs_05_migrated OpenUpgrade: base: post-migration script called with version 6.1.1.3
2018-02-11 10:27:31,701 15701 ERROR bs_05_migrated OpenUpgrade: base: error in migration script base/migrations/7.0.1.3/post-migration.py: TypeError('get() takes exactly 2 arguments (3 given)',)
2018-02-11 10:27:31,702 15701 ERROR bs_05_migrated OpenUpgrade: get() takes exactly 2 arguments (3 given)
Traceback (most recent call last):
  File "/var/tmp/openupgrade/openupgradelib/openupgradelib/openupgrade.py", line 1391, in wrapped_function
    if use_env2 else cr, version)
  File "/var/tmp/openupgrade/7.0/server/openerp/addons/base/migrations/7.0.1.3/post-migration.py", line 514, in migrate
    openupgrade.set_defaults(cr, pool, force_defaults, force=True)
  File "/var/tmp/openupgrade/openupgradelib/openupgradelib/openupgrade.py", line 811, in set_defaults
    obj = pool.get(model, False)
TypeError: get() takes exactly 2 arguments (3 given)
2018-02-11 10:27:31,702 15701 ERROR bs_05_migrated openerp.modules.loading: Error executing post migration script for module base
: get() takes exactly 2 arguments (3 given)


2018-02-11 10:27:31,718 15701 ERROR bs_05_migrated openerp: Failed to initialize database `bs_05_migrated`.
Traceback (most recent call last):
  File "/var/tmp/openupgrade/7.0/server/openerp/cli/server.py", line 97, in preload_registry
    db, registry = openerp.pooler.get_db_and_pool(dbname,update_module=update_module)
  File "/var/tmp/openupgrade/7.0/server/openerp/pooler.py", line 33, in get_db_and_pool
    registry = RegistryManager.get(db_name, force_demo, status, update_module)
  File "/var/tmp/openupgrade/7.0/server/openerp/modules/registry.py", line 203, in get
    update_module)
  File "/var/tmp/openupgrade/7.0/server/openerp/modules/registry.py", line 233, in new
    openerp.modules.load_modules(registry.db, force_demo, status, update_module)
  File "/var/tmp/openupgrade/7.0/server/openerp/modules/loading.py", line 329, in load_modules
    loaded_modules, processed_modules = load_module_graph(cr, graph, status, perform_checks=update_module, report=report, registry=registry)
  File "/var/tmp/openupgrade/7.0/server/openerp/modules/loading.py", line 231, in load_module_graph
    migrations.migrate_module(package, 'post')
  File "/var/tmp/openupgrade/7.0/server/openerp/modules/migration.py", line 191, in migrate_module
    mod.migrate(self.cr, pkg.installed_version)
  File "/var/tmp/openupgrade/openupgradelib/openupgradelib/openupgrade.py", line 1391, in wrapped_function
    if use_env2 else cr, version)
  File "/var/tmp/openupgrade/7.0/server/openerp/addons/base/migrations/7.0.1.3/post-migration.py", line 514, in migrate
    openupgrade.set_defaults(cr, pool, force_defaults, force=True)
  File "/var/tmp/openupgrade/openupgradelib/openupgradelib/openupgrade.py", line 811, in set_defaults
    obj = pool.get(model, False)
TypeError: get() takes exactly 2 arguments (3 given)

Is there something I could do to circumvent this situation, as I see no branch per version, and I have updated the old script. Of course, I can go and undo the changes, but this is not an elegant way...

If I simply take out the , False I get into a new error few lines down, as the new api is called...

Thank you in advance

commit 868dcc6 adapted logged_query to work with psycopg2.sql.SQL and prints decoded query in order to make newline appear as a newline rather than a raw \n string (for python 3). This however fails when query has unicode characters with UnicodeDecodeError.

https://pypi.org/project/openupgradelib/#history

Renaming a module is something that is useful for community modules.
A particular case that I'm concerned about are previously built modules that are then proposed the OCA, and must change their technical name to meet the OCA guidelines.

The lib already provides all the helpers needed to achieve this, but I miss a higher level function to do that.
Would that be a good addition to the library?

Now that the compare noupdate xml script detects the value of forcecreate (1), and we can have a mode that allows to update data on existing items only (2), I'm thinking that these best be combined and that the load_data method should always pick apart each item and, if the item has the forcecreate='0' attribute, it is only actually loaded if the item already exists in the database.

What do you think @pedrobaeza, @hbrunn?

And do we need to keep the init_no_create mode in that case?

(1) OCA/OpenUpgrade#1060
(2) #80

I know it's a little bit late, but I wrote a few functions to activate a workflow without the workflow engine. As I dislike my original idea of making this a library addon, how do you feel about adding them here? I use this for in import, but I can imagine cases where this can come handy for migrations too.

Using latest OCA/OpenUpgrade 12.0 branch, and pip install openupgradelib>=2.0.0 ...

2019-06-25 04:00:46,369 23631 CRITICAL ? odoo.modules.module: Couldn't load module base 
2019-06-25 04:00:46,369 23631 CRITICAL ? odoo.modules.module: module 'odoo.tools' has no attribute 'yaml_import' 
2019-06-25 04:00:46,370 23631 ERROR ? odoo.service.server: Failed to load server-wide module `base`. 
Traceback (most recent call last):
  File "/odoo/OpenUpgrade/odoo/service/server.py", line 1063, in load_server_wide_modules
    odoo.modules.module.load_openerp_module(m)
  File "/odoo/OpenUpgrade/odoo/modules/module.py", line 368, in load_openerp_module
    __import__('odoo.addons.' + module_name)
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 656, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 626, in _load_backward_compatible
  File "/odoo/OpenUpgrade/odoo/modules/module.py", line 82, in load_module
    exec(open(modfile, 'rb').read(), new_mod.__dict__)
  File "<string>", line 5, in <module>
  File "/odoo/OpenUpgrade/odoo/addons/base/models/__init__.py", line 7, in <module>
    from . import ir_model
  File "/odoo/OpenUpgrade/odoo/addons/base/models/ir_model.py", line 17, in <module>
    from openupgradelib import openupgrade
  File "/usr/local/lib/python3.6/dist-packages/openupgradelib/openupgrade.py", line 84, in <module>
    yaml_import = tools.yaml_import

I can't see the yaml_import function defined in either OCA/OpenUpgrade or upstream odoo 12.0 source trees. Where is this function defined?

Please release on PyPI

There is strange license name displayed on PyPI

This is really just a gimmick, but:

• the json-like spec approach seems to be best practice in the library.
• It could be more generalized and feature enriched by a small and simple data validation schema
• also this has small positive side effects on documentation.

Would this be something useful? (PR's accepted?)

In version 10, there's no more pool object, so we need to change a lot of openupgradelib functions to adapt to the new architecture. The problem here is the backwards compatibility. I think right now in several possibilities:

• Create a separate branch for openupgradelib v10 version. This one is the cleanest option.
• Add keyword arguments for passing env and adapt code to detect not null values.
• Some kind of decorator to change the signature.

What do you think, @hbrunn, @StefanRijnhart, @jbeficent ?

Migration to 12.0 fails on user constraint on category user type, if there are users with portal + [user or public]

First of all, thx for maintaining and publishing the OpenUpgrade effort.

My suggestion/wish is to create a fresh release of openupgradelib.

Rationale:

This commit
954121e introduced a new api "rename_fields", that seems not to be part of the latest pip release (1.3.2). However this API is used by the OpenUpgrade Repo (on 9.0 with OCA/OpenUpgrade@ba519edb915bf3eaf8a). So currently we need a source install of openupgradelib to use OpenUpgrade. ;(

So about shipping out a fresh release?

Regards,
Peter

As per the docs, I want to sum up the rationale behind my recent seris of PR, who mainly are still WIP.

The series comprises:
#40 with it's sister #43 (concerning get_legacy_columns())
#38 (concerning wrapping of Env)
#37 (concerning backward-compatibly shelling out map_value_orm to make it able to map m2m)
#36 (small clean up helper for minor migrations, where no full blown migration is warranted)
#35 (concerning xml mapping in map_value as a general improvement)
#45 (concerning the ability of reliably copying m2m fields through the ORM)

Let's start with the ultimate use case:

As a user of openupgradelib (not OpenUpgrade!), I want to use openupgradelib to help me standardize and optimize the following task:

Goal:
I want to update account.tags on account.account (m2m) based on detecting weather tags of account.template have been altered through an updated xml file during migration. (You might question, why I want to do that, but let's stay on topic, I have my reasons.)

Explanation:
In order to be able to detect changes in the m2m field I need to know during post migration, what where the previous values. Therefore copying the m2m field preserving all it's relations is achieved by #45 . Clearly, I could manually manipulate the rel_table, but then again, I only can construct a little generic SQL script only for this specific use case and would need to throw it away afterwards. Additionally, manipulating rel_tables is not always intuitive, so this wrapper let's me focus on actual business logic while making future migrations scripts instead of getting stuck by the low level details of SQL queries of mid range complexity to the unexperienced SQLer. Also this task's would get complicated as it is not legit to assume that the target model of a m2m migration remains the same. So forcibly, one would need to keep old column "as is". But, Without salvaging the needed column through copying, it would get lost during module loading. On the simple table level it's easy, but on m2m it's slightly more complicated. That is why I built #45.

Details:
Now as the ORM does already quite a good job in housekeeping tracked fields and columns, despite in the case of m2m (alleged reason for #36), in order for ORM copies to be persisted across module updates, we need to mark them as "manual", which in some ORM routines is indicated by the manual flag in ir_model_fields yet in others simply by the x_ prefix. Hence essence of #40 . As often times, constructing mapping dicts by database id's is not something where regex can really be helpful, contrary to xml_ids, #35 tries to solve this. Imagine the need to construct a 2000 lines mapping table based on db id's, I think it is self evident, why this can be quite a headache.
As both, mapping m2m and copying them can only be (afaik) conveniently done through the ORM, dividing functions into a _orm style and a _sql style became a natural conclusion. This is not directly related to the Rational of this issue, but includes several benefits. For example a clear separation throughout openupgradelib, when orm mechanisms are to be used and when the lower level pendant is necessary. Think about the ORM is being purposely fooled by the migration script or we need to handle stale columns/tables/values through direct SQL. Having both options side by side permits opting for the best of both worlds and generally increases the usefulness and generality of opernupgradelib. This being said, #38 tries to set the path and establish a common interface for future ORM style methods or refactoring of old ones.

Backward Compatibility:
All this has been done with utmost care for backwards compatibility, so that exposed behaviour and known interface would not change, properly emitting deprecation-warnings at the appropriate points, while at the same time permitting the user taking advantage of the new subtleties, in case he needs to.

Documentation:
I can agree, that all this needs proper documentation, which has been included into the docstrings, where applicable. But I'm also motivated to undertake a first step towards improving this.

Further Implications / Thoughts:
If you are inclined, you can read in the signature of this work the idea of deprecating methods in favor of others. I'm not aware what is/should be openupgradelib's stance on deprecation. I understand that compatibility is very important, yet I do not know if it always wins against maintainability. In our case for example, the majority are or will be enterprise clients, so the main reason why we would commit to contributing to openupgradelib is in order to manage migration of own modules which we do routinely every week since beginning of this year. There is, no doubt, a stance in the code about a wider understanding of openupgradelib's raison d'etre. Some comments throughout the discussion warrant for the interpretation that openupgradelib is kind of a "third wheel" of OpenUpgrade, this is historically true indeed. Yet, I guess, there has been some deeper reason why lib was shelled out. And it's for this reason, I'm actually contributing, because it serves me very well in another, slightly different use case. This slightly different interpretation of openupgradelib together with a deprecation approach, could solve the compatibility problem also in another more flexible and dynamic way: by pinning down openupgradelib versions to specific OpenUpgrade Migrations.
This could help clean the codebase and ease maintenance while at the same time attracting more people to a dynamic and well conceived project (which as I stated in the mailing list, to my firm belief is an undervalued treasure).

If consens can be drawn on the general rational, I would be very happy to proceed with rigorous compliance with PR-Guidlines and get everything nice and clean.

I hope, by this resume I can comply with the relvant recommendations for proposing new features.

Thanks everyone for your time and effort, your collaboration and help, and your benevolent contribution to get this initiative proper.

Best Regards, David

Impacted versions: 9.0

Steps to reproduce:
Make a migration script using @openupgrade.migrate(use_env=True) decorator
Call env.ref('module.xml_id') function or env['model'].search() and it doesn't work.

Current behavior:
For instance, calling env['res.partner'].search([('ref', '=', '123')])
{ValueError}Invalid field 'ref' in leaf "<osv.ExtendedLeaf: ('ref', '=', 'lsv') on res_partner (ctx: )>"

Expected behavior:
Find a res.partner record.

I tried this in a custom module pre-migration and I'm only doing this search in the migrate function which gives me this error.

Playing around with openupgrade and found a small issue here:

The environment's invalidate_all method has been removed as of odoo/odoo@32a58c0db

You probably want to do something like

env.cache.invalidate()

instead

Otherwise this method crashes and chunked operations are not possible ;)

When I launch an upgrade from 7.0 to 8.0 with OpenUpgrade migration.py script, openupgradelib rise an error:

Traceback (most recent call last):
  File "/var/tmp/openupgrade/8.0/server/openerp/service/server.py", line 941, in prelo
ad_registries
    registry = RegistryManager.new(dbname, update_module=update_module)
  File "/var/tmp/openupgrade/8.0/server/openerp/modules/registry.py", line 370, in new
    openerp.modules.load_modules(registry._db, force_demo, status, update_module)
  File "/var/tmp/openupgrade/8.0/server/openerp/modules/loading.py", line 420, in load
_modules
    force, status, report, loaded_modules, update_module, upg_registry)
  File "/var/tmp/openupgrade/8.0/server/openerp/modules/loading.py", line 312, in load
_marked_modules
    loaded, processed = load_module_graph(cr, graph, progressdict, report=report, skip
_modules=loaded_modules, perform_checks=perform_checks, upg_registry=upg_registry)
  File "/var/tmp/openupgrade/8.0/server/openerp/modules/loading.py", line 208, in load
_module_graph
    migrations.migrate_module(package, 'post')
  File "/var/tmp/openupgrade/8.0/server/openerp/modules/migration.py", line 179, in mi
grate_module
    mod.migrate(self.cr, pkg.installed_version)
  File "/var/tmp/openupgrade/openupgradelib/openupgradelib/openupgrade.py", line 1495,
 in wrapped_function
    if use_env2 else cr, version)
  File "/var/tmp/openupgrade/8.0/addons/stock/migrations/8.0.1.1/post-migration.py", l
ine 1150, in migrate
    migrate_stock_qty(cr, registry)
  File "/var/tmp/openupgrade/8.0/addons/stock/migrations/8.0.1.1/post-migration.py", l
ine 950, in migrate_stock_qty
    create_index.format(table=table, col=col),
  File "/var/tmp/openupgrade/openupgradelib/openupgradelib/openupgrade.py", line 979, 
in logged_query
    cr.execute(query, args)
  File "/var/tmp/openupgrade/8.0/server/openerp/sql_db.py", line 171, in wrapper
    return f(self, *args, **kwargs)
  File "/var/tmp/openupgrade/8.0/server/openerp/sql_db.py", line 247, in execute
    res = self._obj.execute(query, params)
ProgrammingError: ERREUR:  erreur de syntaxe sur ou près de « NOT »
LIGNE 2 :         CREATE INDEX IF NOT EXISTS stock_pack_operation_loca...

I'm using add_fields on a v8 database, but this breaks because select_level is required there and there's no default. This field doesn't exist in newer versions, so we can't just change the statement.

We could do some version detection and adapt the statement accordingly, but why don't we use the ORM here? The function wants an environment anyways, and if you've an environment, then ir.model.fields et al exist already in the registry.

TLDR: Why don't we use the ORM in add_fields?

str(e) will crash if there is special characters in the exception message (because I am in fr_CH.UTF8 with my user on my system I think). Then the error that occurs is not displayed in the log. I will try to run the process with another temporary session locale but this is not very clean.

Traceback (most recent call last):
File "/[…]/9.0/server/openerp/service/server.py", line 897, in preload_registries
registry = RegistryManager.new(dbname, update_module=update_module)
File "/[…]/9.0/server/openerp/modules/registry.py", line 390, in new
openerp.modules.load_modules(registry._db, force_demo, status, update_module)
File "/[…]/9.0/server/openerp/modules/loading.py", line 342, in load_modules
loaded_modules, processed_modules = load_module_graph(cr, graph, status, perform_checks=update_module, report=report, upg_registry=upg_registry)
File "/[…]/9.0/server/openerp/modules/loading.py", line 189, in load_module_graph
migrations.migrate_module(package, 'post')
File "/[…]/9.0/server/openerp/modules/migration.py", line 169, in migrate_module
mod.migrate(self.cr, pkg.installed_version)
File "/[…]/openupgradelib/openupgradelib/openupgrade.py", line 1391, in wrapped_function
message = str(e)
UnicodeEncodeError: 'ascii' codec can't encode character u'\xab' in position 69: ordinal not in range(128)

This repo is missing the improvements from OCA/maintainer-quality-tools#613. It should have them, although some configs might have to be slightly modified for this specific case.

@sbidoul @Tecnativa

we get:

select module,model,name from ir_model_data where module='base';
module    |       model      |        name
----------+------------------+--------------------
base      | ir.module.module | module_foobar

Should https://github.com/OCA/openupgradelib/blob/develop/openupgradelib/openupgrade.py#L543-L566 be this way under this circumstances? At least a hint in the docs???

It seems as if the specific query in https://github.com/OCA/openupgradelib/blob/develop/openupgradelib/openupgrade.py#L559-L561 needs to be adapted, as else the other queris might break, can anyone confirm?

Hi there are recent commits here, but no recent version tag here.

Some of these commits are required by odoo account module.

What can I do to help ?

Another interesting missing merged value is to get first not null on many2one/binary fields. Example: you merge countries, and you have the flag image. The selected target record doesn't have image, but one of the rest of merged records has. If we put target, the image will be missing.

I've decorated the migration method like so:

from openupgradelib import openupgrade as ou
@ou.migrate()
def migrate(cr, version):

Yet, when the migration fails (on the post script of a module), nothing is rolled back. Is that expected behaviour? Would there be a way to roll back a failing migration automatically.

Hi @hbrunn @pedrobaeza !

I want to use the 'set_defaults' method in v10.

In v10 this method breaks in this line https://github.com/OCA/openupgradelib/blob/master/openupgradelib/openupgrade.py#L652 because .search() in new api only has domain but not cr nor user_id.

Could you update this method to be usable in old versions but also in v10?

Let's update this method ASAP, if possible 😃

Thanks!

As in OCA/pylint-odoo#2, we can enable auto-deployment to Pypi with Travis, like explained here:

http://docs.travis-ci.com/user/deployment/pypi/

And only activate it when making tags, so we can:

• Remove master and development branch.
• Control the moment when we want to make a release.
• Avoid to have someone with the corresponding credentials.

It seems Odoo is publishing similar tools that those we have in this library in the core:

https://github.com/odoo/odoo/pull/26548/files#diff-88bb0c3f4288be63142d7287759e3101

There are the beginnings of a unittest structure in tests/test_openupgradelib.py

The goal is to increase coverage and guarantee code stability of functions in openupgradelib by:

1. running each function
2. mocking a openerp installation, database, modules
3. checking the calls run on the mocks
4. having a mock database built in travis or in setup
5. checking that openupgradelib correctly manipulates the data

Things to avoid:

1. Implementation tests. This means testing a whole migration at once. A unittest is supposed to test by units. Implementation testing for whole migrations should be done in https://github.com/OCA/openupgrade
2. Using mock to give out the expected answer. Mock should take care of the background details. We don't want to be testing the mock object, we want to test the calls.
3. Testing the best cases. Test edge cases. Example: single id, duplicate ids, no ids, unicode instead of str, None as a parameter.
4. Testing version specific migrations. As said in 1. these scripts should be seen as version agnostic and must be usable for all migrations including non-major migrations.

In Postgres, a Boolean can have 3 states:
true, false and unknown: https://www.postgresql.org/docs/10/datatype-boolean.html

However in in the code of the map_values function we have:

    for old, new in mapping:
        new = "'%s'" % new

        if old is True:
            old = 'NOT NULL'
            op = 'IS'
        elif old is False:
            old = 'NULL'
            op = 'IS'
        else:
            old = "'%s'" % old
            op = '='

        values = {
            'table': table,
            'source': source_column,
            'target': target_column,
            'old': old,
            'new': new,
            'op': op,
        }

        if write == 'sql':
            query = """UPDATE %(table)s
                       SET %(target)s = %(new)s
                       WHERE %(source)s %(op)s %(old)s""" % values
        else:
            query = """SELECT id FROM %(table)s
                       WHERE %(source)s %(op)s %(old)s""" % values
        logged_query(cr, query, values)

So if we want to do a mapping such as defined here:

def map_product_attribute_create_variant(cr):
    openupgrade.map_values(
        cr,
        openupgrade.get_legacy_name('create_variant'),
        'create_variant',
        [(False, 'no_variant'),
         (None, 'no_variant'),
         (True, 'always'),
         ],
        table='product_attribute', write='sql')

( see https://github.com/Eficent/OpenUpgrade/blob/12.0-mig-product/addons/product/migrations/12.0.1.2/post-migration.py the branch has errors which I am fixing but it can be used as reference to understand)

We cannot because:

1. False will be transformed to IS NULL (which different of False for a Boolean Postgres type)
2. We have no way to use the NULL value as the source to be mapped

cc @mreficent (I used you branch to discover this issue, I don't know how you worked around it)

If I make: sudo pip install openupgradelib, I get:

Downloading/unpacking openupgradelib
  Downloading openupgradelib-0.1.0.tar.gz (40kB): 40kB downloaded
  Running setup.py (path:/tmp/pip_build_root/openupgradelib/setup.py) egg_info for package openupgradelib
    Traceback (most recent call last):
      File "<string>", line 17, in <module>
      File "/tmp/pip_build_root/openupgradelib/setup.py", line 17, in <module>
        with open('requirements.txt') as requirements_file:
    IOError: [Errno 2] No such file or directory: 'requirements.txt'
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):

  File "<string>", line 17, in <module>

  File "/tmp/pip_build_root/openupgradelib/setup.py", line 17, in <module>

    with open('requirements.txt') as requirements_file:

IOError: [Errno 2] No such file or directory: 'requirements.txt'

And that's because it's downloading version 0.1.0 instead of version 0.1.2.

Hello

i've tried to install the master version of openupgradelib in a docker container with odoo 10; i got this error.

sudo pip install git+https://github.com/OCA/openupgradelib.git
Downloading/unpacking git+https://github.com/OCA/openupgradelib.git
  Cloning https://github.com/OCA/openupgradelib.git to /tmp/pip-K4kIat-build
  Running setup.py (path:/tmp/pip-K4kIat-build/setup.py) egg_info for package from git+https://github.com/OCA/openupgradelib.git
    /tmp/pip-K4kIat-build/setup.py:12: PkgResourcesDeprecationWarning: Parameters to load are deprecated.  Call .resolve and .require separately.
    
    warning: no previously-included files matching '__pycache__' found under directory '*'
    warning: no previously-included files matching '*.py[co]' found under directory '*'
Requirement already satisfied (use --upgrade to upgrade): lxml in /usr/lib/python2.7/dist-packages (from openupgradelib==2.0.0)
Requirement already satisfied (use --upgrade to upgrade): cssselect in /usr/local/lib/python2.7/dist-packages (from openupgradelib==2.0.0)
Installing collected packages: openupgradelib
  Running setup.py install for openupgradelib
    warning: no previously-included files matching '__pycache__' found under directory '*'
    warning: no previously-included files matching '*.py[co]' found under directory '*'
      File "/usr/local/lib/python2.7/dist-packages/openupgradelib/openupgrade_120.py", line 55
        *(_r("col%s-%s" % (t3, col), "col%s-%s" % (t4, col))
        ^
    SyntaxError: invalid syntax
    
  Could not find .egg-info directory in install record for openupgradelib==2.0.0 from git+https://github.com/OCA/openupgradelib.git
Successfully installed openupgradelib
Cleaning up...

Customized views - ir.ui.view.custom - (used on dashboards), can contain a field in the domain that, being renamed through the rename_fields method, are not touched.

It would be good to have them renamed as well if they are first level domain field at least.

I want to split some models definition in different new modules.

I don't find any openupgrade function that allows to move a model to another module. I think there is an entry in ir_model_data that associates model and module. Is the entry deleted on new module install?

Maybe there is nothing to do... Is there anybody faced the same situation

I run a migration from Odoo 8 to Odoo 9 with openupgrade inside the
Official (but no more supported) Odoo docker image 8.0.

When I run the migrate.py script, I got this error:

Traceback (most recent call last):
  File "/var/tmp/openupgrade/9.0/server/openerp-server", line 2, in <module>
    import openerp
  File "/var/tmp/openupgrade/9.0/server/openerp/__init__.py", line 64, in <module>
    import modules
  File "/var/tmp/openupgrade/9.0/server/openerp/modules/__init__.py", line 8, in <module>
    from . import db, graph, loading, migration, module, registry
  File "/var/tmp/openupgrade/9.0/server/openerp/modules/loading.py", line 28, in <module>
    from openerp.openupgrade import openupgrade_loading, deferred_90
  File "/var/tmp/openupgrade/9.0/server/openerp/openupgrade/deferred_90.py", line 37, in <module>
    from openupgradelib import openupgrade, openupgrade_90, openupgrade_tools
  File "/var/tmp/openupgrade/openupgradelib/openupgradelib/openupgrade.py", line 57, in <module>
    from psycopg2 import sql
ImportError: cannot import name sql

Apparently, openupgradelib assumes that you run a recent version of
psycopg2, which it's not the case in environment with old version of
Odoo. This can cause problem when you do migration from old Odoo.

Hi all, sorry for asking here (where is the right place?), where can I find an example to change a column from a type to another saving values? (boolean to selection in this case)
I'm proposing a PR on a module and would like to put the migration script too.
Thanks

Please, can somebody fix this? Example in https://travis-ci.org/OCA/openupgradelib/builds/565402166.

Traceback (most recent call last):
  File "setup.py", line 56, in <module>
    tests_require=test_requirements
  File "/home/travis/virtualenv/python2.7.15/lib/python2.7/site-packages/setuptools/__init__.py", line 145, in setup
    return distutils.core.setup(**attrs)
  File "/opt/python/2.7.15/lib/python2.7/distutils/core.py", line 151, in setup
    dist.run_commands()
  File "/opt/python/2.7.15/lib/python2.7/distutils/dist.py", line 953, in run_commands
    self.run_command(cmd)
  File "/opt/python/2.7.15/lib/python2.7/distutils/dist.py", line 972, in run_command
    cmd_obj.run()
  File "/home/travis/virtualenv/python2.7.15/lib/python2.7/site-packages/setuptools/command/test.py", line 228, in run
    self.run_tests()
  File "/home/travis/virtualenv/python2.7.15/lib/python2.7/site-packages/setuptools/command/test.py", line 250, in run_tests
    exit=False,
  File "/opt/python/2.7.15/lib/python2.7/unittest/main.py", line 94, in __init__
    self.parseArgs(argv)
  File "/opt/python/2.7.15/lib/python2.7/unittest/main.py", line 149, in parseArgs
    self.createTests()
  File "/opt/python/2.7.15/lib/python2.7/unittest/main.py", line 158, in createTests
    self.module)
  File "/opt/python/2.7.15/lib/python2.7/unittest/loader.py", line 130, in loadTestsFromNames
    suites = [self.loadTestsFromName(name, module) for name in names]
  File "/opt/python/2.7.15/lib/python2.7/unittest/loader.py", line 91, in loadTestsFromName
    module = __import__('.'.join(parts_copy))
  File "/home/travis/build/OCA/openupgradelib/tests/__init__.py", line 2, in <module>
    from . import test_openupgradelib  # noqa: F401
  File "/home/travis/build/OCA/openupgradelib/tests/test_openupgradelib.py", line 44, in <module>
    from openupgradelib import openupgrade
  File "/home/travis/virtualenv/python2.7.15/lib/python2.7/site-packages/mock/mock.py", line 1062, in __call__
    return _mock_self._mock_call(*args, **kwargs)
  File "/home/travis/virtualenv/python2.7.15/lib/python2.7/site-packages/mock/mock.py", line 1128, in _mock_call
    ret_val = effect(*args, **kwargs)
  File "/home/travis/build/OCA/openupgradelib/tests/test_openupgradelib.py", line 26, in import_mock
    return orig_import(name, *args)
  File "/home/travis/build/OCA/openupgradelib/openupgradelib/openupgrade.py", line 57, in <module>
    from lxml import etree
  File "/home/travis/virtualenv/python2.7.15/lib/python2.7/site-packages/mock/mock.py", line 1062, in __call__
    return _mock_self._mock_call(*args, **kwargs)
  File "/home/travis/virtualenv/python2.7.15/lib/python2.7/site-packages/mock/mock.py", line 1128, in _mock_call
    ret_val = effect(*args, **kwargs)
  File "/home/travis/build/OCA/openupgradelib/tests/test_openupgradelib.py", line 26, in import_mock
    return orig_import(name, *args)
  File "src/lxml/serializer.pxi", line 622, in init lxml.etree
  File "/home/travis/virtualenv/python2.7.15/lib/python2.7/codecs.py", line 1011, in getwriter
    return lookup(encoding).streamwriter
  File "/home/travis/virtualenv/python2.7.15/lib/python2.7/encodings/__init__.py", line 100, in search_function
    level=0)
  File "/home/travis/virtualenv/python2.7.15/lib/python2.7/site-packages/mock/mock.py", line 1062, in __call__
    return _mock_self._mock_call(*args, **kwargs)
  File "/home/travis/virtualenv/python2.7.15/lib/python2.7/site-packages/mock/mock.py", line 1128, in _mock_call
    ret_val = effect(*args, **kwargs)
TypeError: import_mock() got an unexpected keyword argument 'fromlist'
The command "coverage run setup.py test" exited with 1.

pypi Travis stage (for publishing to pypi) is failing and I'm not sure why:

https://travis-ci.org/OCA/openupgradelib/jobs/270853280#L547

@moylop260 any idea?

@pedrobaeza Maybe it's time to make another release to pypi ?

When i execute openupgradelib from an openupgrade 7.0 version, i found errors due to this commit:
22a6409

Lines like:
obj = pool.get(model, False) or..
if isinstance(pool, api.Environment): are failing because no api defined in 7.0.
I need to take a older commit to avoid this errors

As suggested by @pedrobaeza on this comment OCA/account-invoice-reporting#107 (comment):

This code seems a very good candidate for a function in https://github.com/OCA/openupgradelib called convert_field_to_property or similar.

https://github.com/OCA/account-invoice-reporting/pull/107/files/884b26cd0217cf5ff49688d4015b933083f02810#diff-35de6c1d400309c265512f253022f2d0R19

    cr.execute("SELECT id FROM res_company")
    company_ids = [d[0] for d in cr.fetchall()]

    cr.execute(
        "SELECT id FROM ir_model_fields WHERE model=%s AND name=%s",
        ('res.partner', 'property_comment_template_id'))
    [field_id] = cr.fetchone()

    for company_id in company_ids:
        cr.execute("""
            INSERT INTO ir_property(
                name,
                type,
                fields_id,
                company_id,
                res_id,
                value_reference
            )
            SELECT
                {field},
                'many2one',
                {field_id},
                {company_id},
                CONCAT('{model},',id),
                CONCAT('{target_model},',{oldfield})
            FROM {table} t
            WHERE {oldfield} IS NOT NULL
            AND (company_id IS NULL OR company_id = {company_id})
            AND NOT EXISTS(
                SELECT 1
                FROM ir_property
                WHERE fields_id={field_id}
                AND company_id={company_id}
                AND res_id=CONCAT('{model},',t.id)
            )
        """.format(
            oldfield='comment_template_id',
            field='property_comment_template_id',
            field_id=field_id,
            company_id=company_id,
            model='res.partner',
            target_model='base.comment.template',
            table='res_partner',
        ))

I have faced this problem using the feature for merging other records that contains a translatable field.

example:

2019-02-08 23:25:15,944 144 ERROR openupgrade-to-11 odoo.modules.registry: Failed to load registry                                                                                                           
Traceback (most recent call last):                                                                                                                                                                           
  File "/odoo/src/odoo/modules/registry.py", line 85, in new                                                                                                                                                 
    odoo.modules.load_modules(registry._db, force_demo, status, update_module)                                                                                                                               
  File "/odoo/src/odoo/modules/loading.py", line 463, in load_modules                                                                                                                                        
    migrations.migrate_module(package, 'end')                                                                                                                                                                
  File "/odoo/src/odoo/modules/migration.py", line 187, in migrate_module                                                                                                                                    
    migrate(self.cr, installed_version)                                                                                                                                                                      
  File "/src/openupgradelib/openupgradelib/openupgrade.py", line 1445, in wrapped_function                                                                                                                   
    if use_env2 else cr, version)                                                                                                                                                                            
  File "/odoo/src/addons/stock/migrations/11.0.1.1/end-migration.py", line 31, in migrate                                                                                                                    
    merge_quants(env)                                                                                                                                                                                        
  File "/odoo/src/addons/stock/migrations/11.0.1.1/end-migration.py", line 25, in merge_quants                                                                                                               
    env, 'stock.quant', quants[1:].ids, quants[0].id, QUANT_MERGE_OPS,                                                                                                                                       
  File "/src/openupgradelib/openupgradelib/openupgrade_merge_records.py", line 505, in merge_records
    _adjust_merged_values_orm(*args2)
  File "/src/openupgradelib/openupgradelib/openupgrade_merge_records.py", line 354, in _adjust_merged_values_orm
    vals[field.name] = min(l)
TypeError: unorderable types: bool() < str()
2019-02-08 23:25:15,945 144 CRITICAL openupgrade-to-11 odoo.service.server: Failed to initialize database `openupgrade-to-11`.
Traceback (most recent call last):
  File "/odoo/src/odoo/service/server.py", line 932, in preload_registries
    registry = Registry.new(dbname, update_module=update_module)
  File "/odoo/src/odoo/modules/registry.py", line 85, in new
    odoo.modules.load_modules(registry._db, force_demo, status, update_module)
  File "/odoo/src/odoo/modules/loading.py", line 463, in load_modules
    migrations.migrate_module(package, 'end')
  File "/odoo/src/odoo/modules/migration.py", line 187, in migrate_module
    migrate(self.cr, installed_version)
  File "/src/openupgradelib/openupgradelib/openupgrade.py", line 1445, in wrapped_function
    if use_env2 else cr, version)
  File "/odoo/src/addons/stock/migrations/11.0.1.1/end-migration.py", line 31, in migrate
    merge_quants(env)
  File "/odoo/src/addons/stock/migrations/11.0.1.1/end-migration.py", line 25, in merge_quants
    env, 'stock.quant', quants[1:].ids, quants[0].id, QUANT_MERGE_OPS,
  File "/src/openupgradelib/openupgradelib/openupgrade_merge_records.py", line 505, in merge_records
    _adjust_merged_values_orm(*args2)
  File "/src/openupgradelib/openupgradelib/openupgrade_merge_records.py", line 354, in _adjust_merged_values_orm
    vals[field.name] = min(l)
TypeError: unorderable types: bool() < str()

probably some heuristics, like filter null date/datetime from the list could help. The same may happen with the comparisons of other types...

As this package is now on Pypi, we need to increment the version number on each improvement, but if I have multiple PRs, I don't know the sequence in which they are going to be accepted. How others work with this question? Do they add the version change at merge time? Do they make dependent PRs?

Hello when I ran the server, I have this issue.
File "/home/vagrant/.local/lib/python3.5/site-packages/openupgradelib/openupgrade.py", line 1204, in wrapped_function
(module, filename, str(e).decode('utf8')))
AttributeError: 'str' object has no attribute 'decode'

odoo12 CE module odoo.tools loading yaml_import, but there is no file /odoo/tools/yaml_import.py
The test when the file yaml_import.py don't exist for the corresponding versions of Odoo, must be included in /openupgradelib/openupgrade.py

As stated, I'm working on the implementation of a ORM rename_column to enable 2many and possibly property fields. While seems right now easy to rename a column to a manual one, which then gets initialized properly by the ORM at module loading time (so the plan), copying a column is slightly more complex (because construction and resolution of 2many table and columns, as well as data copying as the new field would initialize empty at first).
On the other hand the only use case of copy column, I can really think about is:

1. Copy a column
2. Do something with one of the columns, one functioning as source and one as destination)
3. source means basically "backup" in most cases, and in all sane cases means "inmutable"
4. destination means "transformation and storage for future use"

This can also be achieved in an ORM compatible way like this:

1. Rename column (ORM)
2. Let your model create the new column
3. map_values_orm 1 to 2

In this order of ideas, it could be argued, that copy_column could be deprecated in future version for the sake of uniformity and simplicity.

However, it is not imminent to create a compatibility wrapper, as the proposed workflow plays with the ORM loading behaviour and is not easily alterable in the scope of a wrapper.

What do you think in general and in detail?

Indexes named according to the 'table_field_index' pattern are being renamed here.

However when this leads to a very long (>63 chars) index name eg. this one leading to the command:

ALTER INDEX IF EXISTS "product_attribute_line_product_attribute_value_rel_val_id_index" RENAME TO "product_attribute_line_product_attribute_value_rel_product_attribute_value_id_index"

Then Postgres's maximum identifier length kicks in and the target name is now abbreviated to product_attribute_line_product_attribute_value_rel_product_attr, which can already exist - in this case because of the earlier rename for product_attribute_line_id.

Look at this code we have:

As you can see, the sql_type is always ignored. However, the docstring of the method says:

So, it should be:

-        sql_type = sql_type_mapping.get(field_type)
+        sql_type = sql_type or sql_type_mapping.get(field_type)

However I'm wondering if fixing this bug would mean breaking existing migration scripts out there in the wild who didn't realize they were working because of this bug... 🤔

Also, since the pypi package is almost never released, we don't have this ability of adding a new major version and introducing backwards-incompatible fixes...

So what do we do here?

Suppose you are merging records of a model A with merge_records method. Then, if those records are used in another model B using a many2one with ondelete=restrict, then it breaks.

To solve this, the method should be adapted to include this case: before doing the deletion of the record of model A, it should change the records of model B that reference that record with the new id.

I would do it myself, but I am a bit confused on where the exact code change should be done.

The method must check if it's a translatable field (and what kind of translatable field - although in the end, it will mean more or only one ir.translation records -), and perform the conversion also on that translations. Example, field website_description in product.template.