Git Product home page Git Product logo

ceph-dev-docker's Issues

AttributeError: module 'rados' has no attribute 'Rados'

Hi,

I am trying to setup a ceph dev env, and found this repo via https://insujang.github.io/2020-11-03/deploying-a-ceph-development-environment-cluster/

It is very helpful, thank you.

However, at step "4. Ceph Dashboard", I got:

╭─root@ceph-dev /ceph ‹master› 
╰─# npm-start.sh 
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
Traceback (most recent call last):
  File "./bin/ceph", line 1318, in <module>
    retval = main()
  File "./bin/ceph", line 978, in main
    conffile = rados.Rados.DEFAULT_CONF_FILES
AttributeError: module 'rados' has no attribute 'Rados'
jq: error: syntax error, unexpected $end (Unix shell quoting issues?) at <top-level>, line 1:
.["/api/"].target=                 
jq: 1 compile error
jq: error: syntax error, unexpected $end (Unix shell quoting issues?) at <top-level>, line 1:
.["/ui-api/"].target=                    
jq: 1 compile error

ceph -s also report same error:

╭─root@ceph-dev /ceph/build ‹master› 
╰─# bin/ceph -s
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
Traceback (most recent call last):
  File "bin/ceph", line 1318, in <module>
    retval = main()
  File "bin/ceph", line 978, in main
    conffile = rados.Rados.DEFAULT_CONF_FILES
AttributeError: module 'rados' has no attribute 'Rados'

It seems the rados module has updated and break the code ?

`install-deps.sh` fails due to missing `pip2` command

install-deps.sh fails due to missing pip2 command.

╭─root@ceph-dev /ceph ‹master*› 
╰─setup-setup-ceph.sh                                                                                                                                                                                       
[...]
Nothing to do.
Collecting python3-saml
  Downloading https://files.pythonhosted.org/packages/1c/5a/fdb873d1f89b031b24a078fd3a2de8da887407e98cb2e651f13d97a10aea/python3_saml-1.6.0-py3-none-any.whl (72kB)
    100% |████████████████████████████████| 81kB 4.0MB/s 
Collecting xmlsec>=0.6.0 (from python3-saml)
  Downloading https://files.pythonhosted.org/packages/35/42/d7cd323c91d4706f3cc32ffe7d5f851ab8ef9898ccb350f6ba593dd8b89a/xmlsec-1.3.3.tar.gz
Collecting isodate>=0.5.0 (from python3-saml)
  Downloading https://files.pythonhosted.org/packages/9b/9f/b36f7774ff5ea8e428fdcfc4bb332c39ee5b9362ddd3d40d9516a55221b2/isodate-0.6.0-py2.py3-none-any.whl (45kB)
    100% |████████████████████████████████| 51kB 5.9MB/s 
Collecting defusedxml==0.5.0 (from python3-saml)
  Downloading https://files.pythonhosted.org/packages/87/1c/17f3e3935a913dfe2a5ca85fa5ccbef366bfd82eb318b1f75dadbf0affca/defusedxml-0.5.0-py2.py3-none-any.whl
Collecting pkgconfig (from xmlsec>=0.6.0->python3-saml)
  Downloading https://files.pythonhosted.org/packages/b4/2c/bf434cb5a6590417e1d4468050ec317ea17fd6231c2a256df4646c11e588/pkgconfig-1.5.1-py2.py3-none-any.whl
Collecting lxml>=3.0 (from xmlsec>=0.6.0->python3-saml)
  Downloading https://files.pythonhosted.org/packages/66/20/49201c7bcb1c92942ac98658b09fb4a0c0dcd064d439489349bc5891207c/lxml-4.3.3-cp37-cp37m-manylinux1_x86_64.whl (5.7MB)
    100% |████████████████████████████████| 5.7MB 4.6MB/s 
Requirement already satisfied: six in /usr/lib/python3.7/site-packages (from isodate>=0.5.0->python3-saml) (1.12.0)
Installing collected packages: pkgconfig, lxml, xmlsec, isodate, defusedxml, python3-saml
  Running setup.py install for xmlsec ... done
Successfully installed defusedxml-0.5.0 isodate-0.6.0 lxml-4.3.3 pkgconfig-1.5.1 python3-saml-1.6.0 xmlsec-1.3.3
/shared/bin/setup-ceph.sh: line 26: pip2: command not found
╭─root@ceph-dev /ceph ‹master*› 
╰─#                                                                                                                                                                                                     127 ↵

Installing the python2-pip package using zypper in python2-pip solves the problem.

Errors occurred during compling ceph

Hi, after following the steps in the documentation, I have now a running container.

But when I try to build the ceph using setup-ceph.sh script, it fails.

The error message is below:

``txt
git version 2.20.1
WITH_PYTHON 3
-- Building with ccache: /usr/bin/ccache, CCACHE_DIR=
-- NSS_LIBRARIES: /usr/lib64/libssl3.so;/usr/lib64/libsmime3.so;/usr/lib64/libnss3.so;/usr/lib64/libnssutil3.so
-- NSS_INCLUDE_DIRS: /usr/include/nss3
'--host=x86_64-suse-linux-gnu' '--build=x86_64-suse-linux-gnu' '--program-prefix=' '--prefix=/usr' '--exec-prefix=/usr' '--bindir=/usr/bin' '--sbindir=/usr/sbin' '--sysconfdir=/etc' '--datadir=/usr/share' '--includedir=/usr/include' '--libdir=/usr/lib64' '--libexecdir=/usr/lib' '--localstatedir=/var' '--sharedstatedir=/var/lib' '--mandir=/usr/share/man' '--infodir=/usr/share/info' '--disable-dependency-tracking' '--enable-ipv6' '--with-ssl' '--with-ca-fallback' '--without-ca-path' '--without-ca-bundle' '--with-gssapi=/usr/lib/mit' '--with-libidn2' '--with-libssh' '--with-libmetalink' '--enable-hidden-symbols' '--disable-static' '--enable-threaded-resolver' 'build_alias=x86_64-suse-linux-gnu' 'host_alias=x86_64-suse-linux-gnu' 'CFLAGS=-O2 -Wall -fstack-protector-strong -funwind-tables -fasynchronous-unwind-tables -fstack-clash-protection -g -fPIE' 'LDFLAGS= -pie' 'CPPFLAGS=-D_FORTIFY_SOURCE=2'
-- libcurl is linked with openssl: explicitly setting locks
-- ssl soname: libssl.so.1.1
-- crypto soname: libcrypto.so.1.1
-- Found PythonInterp: /usr/bin/python3 (found suitable version "3.7.2", minimum required is "3")
-- BUILDING Boost Libraries at j 2
-- boost will be downloaded...
-- Found Yasm: good -- capable of assembling x86_64
-- Found PythonInterp: /usr/bin/python3 (found version "3.7.2")
-- Setting civetweb to use OPENSSL >= 1.1
CMake Error at /usr/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:137 (message):
Could NOT find RabbitMQ (missing: rabbitmq_INCLUDE_DIR rabbitmq_LIBRARY)
Call Stack (most recent call first):
/usr/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:378 (_FPHSA_FAILURE_MESSAGE)
cmake/modules/FindRabbitMQ.cmake:9 (find_package_handle_standard_args)
src/rgw/CMakeLists.txt:182 (find_package)

-- Configuring incomplete, errors occurred!
``

The Cmake error log file is also attached.
CMakeError.log

Since I am completely new to cmake and compiling of ceph, it's the error of script or something wrong in the environment relating to Dockerfile? Thanks for any advice and help.

Enable Dashboard debug mode

Dashboard debug mode should be automatically enabled using the following command:

  • bin/ceph dashboard debug enable

(or bin/ceph config set mgr mgr/dashboard/debug true)

"Can't find librados2 package" error during the building

I find this error when just running the building command.

It seems in the source repos, only librados3 is available but not librados2. So I modified it in the Dockerfile, and finally builded the image successfully. The reason is just that there is not official librados2 package for openSUSE Tumbleweed.

Don't know if it matters...Is librados2 necessary?

change the base image to a more stable one

openSUSE Tumbleweed is a rolling release so something could easily break after an update. Yesterday I tried to setup a ceph dev environment and encountered the issue openssl/openssl#10015 with the latest opensuse/tumbleweed image. It was caused by openssl 1.1.1d. Maybe we should use a non-rolling update release?

`npm install` fails due to permission problems

Please note that this issue has originally been reported in oftc#ceph-dashboard by using ceph-dev-docker. I just happened to be able reproduce it on my customized fork using sudo. As ceph-dev-docker runs as root by default, it is not necessary to use sudo to reproduce the problem here!

My proposed solution (as I do not have this problem with my fork) is to switch from root to some non-root user inside the container. Ceph doesn't require root anyway and it will prevent mixed owners in the ceph repo.

The workaround, which fixed the problem for ceph-dev-docker users is to install the frontend dependencies by calling npm install --unsafe-perm in /ceph/src/pybind/mgr/dashboard/frontend.

user@ceph-4 /ceph/src/pybind/mgr/dashboard/frontend (master*) $ sudo npm install   
Unhandled rejection Error: Command failed: /usr/bin/git submodule update -q --init --recursivec/pybind/mgr/dashboard/frontend/node_modules/.staging/typescript-7d3fd7c7 (11733ms)
fatal: failed to stat '/root/.npm/_cacache/tmp/git-clone-44f5e7c8': Permission denied

    at ChildProcess.exithandler (child_process.js:281:12)
    at emitTwo (events.js:126:13)
    at ChildProcess.emit (events.js:214:7)
    at maybeClose (internal/child_process.js:915:16)
    at Socket.stream.socket.on (internal/child_process.js:336:11)
    at emitOne (events.js:116:13)
    at Socket.emit (events.js:211:7)
    at Pipe._handle.close [as _onclose] (net.js:561:12)


> [email protected] install /ceph/src/pybind/mgr/dashboard/frontend/node_modules/node-sass
> node scripts/install.js

Unable to save binary /ceph/src/pybind/mgr/dashboard/frontend/node_modules/node-sass/vendor/linux-x64-57 : { Error: EACCES: permission denied, mkdir '/ceph/src/pybind/mgr/dashboard/frontend/node_modules/node-sass/vendor'
    at Object.fs.mkdirSync (fs.js:885:18)
    at sync (/ceph/src/pybind/mgr/dashboard/frontend/node_modules/mkdirp/index.js:71:13)
    at Function.sync (/ceph/src/pybind/mgr/dashboard/frontend/node_modules/mkdirp/index.js:77:24)
    at checkAndDownloadBinary (/ceph/src/pybind/mgr/dashboard/frontend/node_modules/node-sass/scripts/install.js:114:11)
    at Object.<anonymous> (/ceph/src/pybind/mgr/dashboard/frontend/node_modules/node-sass/scripts/install.js:157:1)
    at Module._compile (module.js:653:30)
    at Object.Module._extensions..js (module.js:664:10)
    at Module.load (module.js:566:32)
    at tryModuleLoad (module.js:506:12)
    at Function.Module._load (module.js:498:3)
  errno: -13,
  code: 'EACCES',
  syscall: 'mkdir',
  path: '/ceph/src/pybind/mgr/dashboard/frontend/node_modules/node-sass/vendor' }

> [email protected] postinstall /ceph/src/pybind/mgr/dashboard/frontend/node_modules/node-sass
> node scripts/build.js

Building: /usr/bin/node8 /ceph/src/pybind/mgr/dashboard/frontend/node_modules/node-gyp/bin/node-gyp.js rebuild --verbose --libsass_ext= --libsass_cflags= --libsass_ldflags= --libsass_library=
gyp info it worked if it ends with ok
gyp verb cli [ '/usr/bin/node8',
gyp verb cli   '/ceph/src/pybind/mgr/dashboard/frontend/node_modules/node-gyp/bin/node-gyp.js',
gyp verb cli   'rebuild',
gyp verb cli   '--verbose',
gyp verb cli   '--libsass_ext=',
gyp verb cli   '--libsass_cflags=',
gyp verb cli   '--libsass_ldflags=',
gyp verb cli   '--libsass_library=' ]
gyp info using [email protected]
gyp info using [email protected] | linux | x64
gyp verb command rebuild []
gyp verb command clean []
gyp verb clean removing "build" directory
gyp verb command configure []
gyp verb check python checking for Python executable "python2" in the PATH
gyp verb `which` succeeded python2 /usr/bin/python2
gyp verb check python version `/usr/bin/python2 -c "import sys; print "2.7.16
gyp verb check python version .%s.%s" % sys.version_info[:3];"` returned: %j
gyp verb get node dir no --target version specified, falling back to host node version: 8.15.1
gyp verb command install [ '8.15.1' ]
gyp verb install input version string "8.15.1"
gyp verb install installing version: 8.15.1
gyp verb install --ensure was passed, so won't reinstall if already installed
gyp WARN EACCES user "root" does not have permission to access the dev dir "/root/.node-gyp/8.15.1"
gyp WARN EACCES attempting to reinstall using temporary dev dir "/ceph/src/pybind/mgr/dashboard/frontend/node_modules/node-sass/.node-gyp"
gyp verb tmpdir == cwd automatically will remove dev files after to save disk space
gyp verb command install [ '--node_gyp_internal_noretry', '8.15.1' ]
gyp verb install input version string "8.15.1"
gyp verb install installing version: 8.15.1
gyp verb install --ensure was passed, so won't reinstall if already installed
gyp verb install version not already installed, continuing with install 8.15.1
gyp verb ensuring nodedir is created /ceph/src/pybind/mgr/dashboard/frontend/node_modules/node-sass/.node-gyp/8.15.1
gyp WARN install got an error, rolling back install
gyp verb command remove [ '8.15.1' ]
gyp verb remove using node-gyp dir: /ceph/src/pybind/mgr/dashboard/frontend/node_modules/node-sass/.node-gyp
gyp verb remove removing target version: 8.15.1
gyp verb remove removing development files for version: 8.15.1
gyp WARN install got an error, rolling back install
gyp verb command remove [ '8.15.1' ]
gyp verb remove using node-gyp dir: /ceph/src/pybind/mgr/dashboard/frontend/node_modules/node-sass/.node-gyp
gyp verb remove removing target version: 8.15.1
gyp verb remove removing development files for version: 8.15.1
gyp ERR! configure error 
gyp ERR! stack Error: EACCES: permission denied, mkdir '/ceph/src/pybind/mgr/dashboard/frontend/node_modules/node-sass/.node-gyp'
gyp ERR! System Linux 4.15.0-47-generic
gyp ERR! command "/usr/bin/node8" "/ceph/src/pybind/mgr/dashboard/frontend/node_modules/node-gyp/bin/node-gyp.js" "rebuild" "--verbose" "--libsass_ext=" "--libsass_cflags=" "--libsass_ldflags=" "--libsass_library="
gyp ERR! cwd /ceph/src/pybind/mgr/dashboard/frontend/node_modules/node-sass
gyp ERR! node -v v8.15.1
gyp ERR! node-gyp -v v3.8.0
gyp ERR! not ok 
Build failed with error code: 1

alerting rules file contains wrong expressions

Fail to start Ceph in Nautilus container

Recent change enabled logging in start-ceph.sh script.

./bin/ceph config set mgr mgr/dashboard/log_level info
./bin/ceph config set mgr mgr/dashboard/log_to_file true

These configs are not available in Nautilus, which leads to start-ceph.sh error.

dashboard urls: https://172.18.0.1:41700
  w/ user/pass: admin / admin
restful urls: https://172.18.0.1:42700
  w/ user/pass: admin / be67b364-a629-4763-bc8c-a150b4ea7f2b


export PYTHONPATH=./pybind:/ceph/src/pybind:/ceph/build/lib/cython_modules/lib.3:
export LD_LIBRARY_PATH=/ceph/build/lib
CEPH_DEV=1
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
2020-02-05 07:40:05.968 7fafc7e68700 -1 WARNING: all dangerous and experimental features are enabled.
2020-02-05 07:40:06.000 7fafc7e68700 -1 WARNING: all dangerous and experimental features are enabled.
Error EINVAL: unrecognized config option 'mgr/dashboard/log_level'

Ceph MGR fails to load MGR modules after rebuild

2018-10-31 13:52:33.781 7f978235e740 10 mgr[py] loaded 10 options
2018-10-31 13:52:33.781 7f978235e740  4 mgr[py] Standby mode not provided by module 'balancer'
2018-10-31 13:52:33.781 7f978235e740  1 mgr[py] Loading python module 'crash'
2018-10-31 13:52:33.801 7f978235e740 10 mgr[py] Computed sys.path '/ceph/src/pybind:/ceph/build/lib/cython_modules/lib.3::/usr/lib/python36.zip:/usr/lib64/python3.6:/usr/lib64/python3.6:/usr/lib64/python3.6/lib-dynload:/usr/lib64/python3.6/site-packages:/usr/lib/python3.6/site-packages:/usr/local/lib64/python3.6/site-packages:/usr/local/lib/python3.6/site-packages:/ceph/src/pybind/mgr'
2018-10-31 13:52:33.813 7f978235e740 -1 mgr[py] Module not found: 'mgr_module'
2018-10-31 13:52:33.813 7f978235e740 -1 mgr[py] Traceback (most recent call last):
  File "/ceph/src/pybind/mgr/mgr_module.py", line 8, in <module>
    import rados
ImportError: Interpreter change detected - this module can only be loaded into one interpreter per process.

2018-10-31 13:52:33.813 7f978235e740 -1 mgr[py] Class not found in module 'crash'
2018-10-31 13:52:33.813 7f978235e740 -1 mgr[py] Error loading module 'crash': (22) Invalid argument
2018-10-31 13:52:33.813 7f978235e740  1 mgr[py] Loading python module 'dashboard'
2018-10-31 13:52:33.837 7f978235e740 10 mgr[py] Computed sys.path '/ceph/src/pybind:/ceph/build/lib/cython_modules/lib.3::/usr/lib/python36.zip:/usr/lib64/python3.6:/usr/lib64/python3.6:/usr/lib64/python3.6/lib-dynload:/usr/lib64/python3.6/site-packages:/usr/lib/python3.6/site-packages:/usr/local/lib64/python3.6/site-packages:/usr/local/lib/python3.6/site-packages:/ceph/src/pybind/mgr'
2018-10-31 13:52:33.849 7f978235e740 -1 mgr[py] Module not found: 'mgr_module'
2018-10-31 13:52:33.849 7f978235e740 -1 mgr[py] Traceback (most recent call last):
  File "/ceph/src/pybind/mgr/mgr_module.py", line 8, in <module>
    import rados
ImportError: Interpreter change detected - this module can only be loaded into one interpreter per process.

2018-10-31 13:52:33.849 7f978235e740 -1 mgr[py] Class not found in module 'dashboard'
2018-10-31 13:52:33.849 7f978235e740 -1 mgr[py] Error loading module 'dashboard': (22) Invalid argument
2018-10-31 13:52:33.849 7f978235e740  1 mgr[py] Loading python module 'devicehealth'
2018-10-31 13:52:33.873 7f978235e740 10 mgr[py] Computed sys.path '/ceph/src/pybind:/ceph/build/lib/cython_modules/lib.3::/usr/lib/python36.zip:/usr/lib64/python3.6:/usr/lib64/python3.6:/usr/lib64/python3.6/lib-dynload:/usr/lib64/python3.6/site-packages:/usr/lib/python3.6/site-packages:/usr/local/lib64/python3.6/site-packages:/usr/local/lib/python3.6/site-packages:/ceph/src/pybind/mgr'
2018-10-31 13:52:33.889 7f978235e740 -1 mgr[py] Module not found: 'mgr_module'
2018-10-31 13:52:33.889 7f978235e740 -1 mgr[py] Traceback (most recent call last):
  File "/ceph/src/pybind/mgr/mgr_module.py", line 8, in <module>
    import rados
ImportError: Interpreter change detected - this module can only be loaded into one interpreter per process.

2018-10-31 13:52:33.889 7f978235e740 -1 mgr[py] Class not found in module 'devicehealth'
2018-10-31 13:52:33.889 7f978235e740 -1 mgr[py] Error loading module 'devicehealth': (22) Invalid argument
2018-10-31 13:52:33.889 7f978235e740  1 mgr[py] Loading python module 'diskprediction'
2018-10-31 13:52:33.913 7f978235e740 10 mgr[py] Computed sys.path '/ceph/src/pybind:/ceph/build/lib/cython_modules/lib.3::/usr/lib/python36.zip:/usr/lib64/python3.6:/usr/lib64/python3.6:/usr/lib64/python3.6/lib-dynload:/usr/lib64/python3.6/site-packages:/usr/lib/python3.6/site-packages:/usr/local/lib64/python3.6/site-packages:/usr/local/lib/python3.6/site-packages:/ceph/src/pybind/mgr'
2018-10-31 13:52:33.929 7f978235e740 -1 mgr[py] Module not found: 'mgr_module'
2018-10-31 13:52:33.929 7f978235e740 -1 mgr[py] Traceback (most recent call last):
  File "/ceph/src/pybind/mgr/mgr_module.py", line 8, in <module>
    import rados
ImportError: Interpreter change detected - this module can only be loaded into one interpreter per process.

2018-10-31 13:52:33.929 7f978235e740 -1 mgr[py] Class not found in module 'diskprediction'
2018-10-31 13:52:33.929 7f978235e740 -1 mgr[py] Error loading module 'diskprediction': (22) Invalid argument
2018-10-31 13:52:33.929 7f978235e740  1 mgr[py] Loading python module 'hello'
2018-10-31 13:52:33.953 7f978235e740 10 mgr[py] Computed sys.path '/ceph/src/pybind:/ceph/build/lib/cython_modules/lib.3::/usr/lib/python36.zip:/usr/lib64/python3.6:/usr/lib64/python3.6:/usr/lib64/python3.6/lib-dynload:/usr/lib64/python3.6/site-packages:/usr/lib/python3.6/site-packages:/usr/local/lib64/python3.6/site-packages:/usr/local/lib/python3.6/site-packages:/ceph/src/pybind/mgr'
2018-10-31 13:52:33.961 7f978235e740 -1 mgr[py] Module not found: 'mgr_module'
2018-10-31 13:52:33.961 7f978235e740 -1 mgr[py] Traceback (most recent call last):
  File "/ceph/src/pybind/mgr/mgr_module.py", line 8, in <module>
    import rados
ImportError: Interpreter change detected - this module can only be loaded into one interpreter per process.

Current build flags in this project:
-DWITH_PYTHON3=ON -DWITH_PYTHON2=OFF -DMGR_PYTHON_VERSION=3 -DWITH_TESTS=ON -DWITH_CCACHE=ON

Another combination which does not trigger the issue:
-D WITH_PYTHON3=ON -D WITH_TESTS=ON -D WITH_CCACHE=ON -D ENABLE_GIT_VERSION=OFF WITH_MGR_DASHBOARD_FRONTEND=OFF

Likely relevant (bad):
-D WITH_PYTHON2=OFF and -D MGR_PYTHON_VERSION=3

This looks like something in master has changed and prevents us from using a Python3 only Ceph, but as I'm not sure about that, I opened the issue here first.

Workaround:

Use the build flags known to work and adapt the setup-ceph.sh file inside or outside of your container. Adapting it outside requires you to build a new image and create a new container out of that image.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.