Git Product home page Git Product logo

Comments (8)

daiyip avatar daiyip commented on May 5, 2024

Hmm, could you give me example code for repro? Actually sym_sealed is defined in pyglove.core.symbolic.Symbolic.

from pyglove.

D-X-Y avatar D-X-Y commented on May 5, 2024

Unfortunately, I can not easily reproduce it, that I can say is that the pyglove Object that involved is a pretty simple one such as

class AA(pg.Object):
    tokenizer: Annotated[AbstractTokenizer, "Tokenizer"] = get_tokenizer(
        "..."
    )

    max_num: Annotated[
        int,
        "The maximum number of tokens for prefix, where -1 means infinity.",
    ] = -1

The error occurs in some weird spark runs and can not easily be debugged. However, replacing pg.Object with Object works fine on the codes.

from pyglove.

D-X-Y avatar D-X-Y commented on May 5, 2024

I'm not sure whether it conflicts with some underling serialization / deserialization functions used in other libraries.

from pyglove.

daiyip avatar daiyip commented on May 5, 2024

Not sure. Can you share longer callstacks?

from pyglove.

D-X-Y avatar D-X-Y commented on May 5, 2024

Seems not more callstacks, it was like:

23/07/21 16:41:04 WARN TaskSetManager: Lost task 0.0 in stage 9.0 (TID 165) (216.153.61.120 executor 14): org.apache.spark.api.python.PythonException: Traceback (most recent call last):
  File "~/.local/lib/python3.9/site-packages/pyglove/core/symbolic/dict.py", line 628, in __setitem__
    if base.treats_as_sealed(self):
...
    ```

from pyglove.

D-X-Y avatar D-X-Y commented on May 5, 2024

I found a minimum reproducible example:

import pickle
from typing import Annotated, Dict
import pyglove as pg

class A(pg.Object):

  x: Annotated[Dict, "."] = dict()
  y: Annotated[str, "."] = ".."

# Define a simple dictionary
# data = A(x={1:1})
data = A()

# Serialization
serialized_data = pickle.dumps(data)
print(f"Serialized data: {serialized_data}")

# Deserialization
deserialized_data = pickle.loads(serialized_data)
print(f"Deserialized data: {deserialized_data}")

from pyglove.

daiyip avatar daiyip commented on May 5, 2024

Great, it turns out pickle does not work well with pg.Symbolic types previously.
Fixed by #194.

from pyglove.

D-X-Y avatar D-X-Y commented on May 5, 2024

Nice, thanks for your supports!

from pyglove.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.