The JCMS consists of 32,310 sentences annotated with word boundary and POS tag information for 27 specialized domains. The corpus adopted two segmentation criteria (and corresponding POS tag sets):

• short unit word (SUW), which was designed by the NINJAL, and
• SUW-SC, which we defined by separating conjugate words into stems and conjugation endings.

The corpus statistics is as follows.

The JCMS consists of sentences from the following corpora.

• ASPEC http://lotus.kuee.kyoto-u.ac.jp/ASPEC/
• BCCWJ https://pj.ninjal.ac.jp/corpus_center/bccwj/subscription.html
• NTCIR-9 PatentMT test collection http://research.nii.ac.jp/ntcir/permission/ntcir-9/perm-ja-PatentMT.html
• NTCIR-11 MedNLP2 test collection http://research.nii.ac.jp/ntcir/permission/ntcir-11/perm-ja-MedNLP.html

To use the JCMS, it is required to sign the data use agreements and obtain the original data of BCCWJ, ASPEC, and NTCIR-9 PatentMT and NTCIR-11 MedNLP2 test collectins.

• The original data of ASPEC, BCCWJ, NTCIR-9 PatentMT test collection, and NTCIR-11 MedNLP2 test collection
• Python 3
• sed
• awk
• zenhan https://pypi.org/project/zenhan/0.5/

+-- data_source      ... Directory for placing original data
+-- script           ... Scripts for generating annotated data
+-- suw              ... SUW anntation data
| +-- mask           ... Annotated data with masked words 
| +-- proc           ... Processed annotated data will be placed here.
+-- suw-sc           ... SUW-SC annotation data
| +-- proc           ... Processed annotated data will be placed here.
+-- text             ... Raw text data
  +-- org            ... Raw text data extracted from original data will be placed here.
  +-- proc           ... Processed text data will be placed here.
  +-- tmp            ... Temporary directory

The following steps generate the complete JCMS data from the original text data and masked JCMS data. This requires 300MB of disk space, but step 6 deletes 200MB of data.

1. Create aliases in data_source

   ln -s /path/to/ASPEC          data_source/ASPEC
   ln -s /path/to/BCCWJ          data_source/BCCWJ
   ln -s /path/to/NTCIR_PatentMT data_source/NTCIR_PatentMT
   ln -s /path/to/NTCIR_MedNLP2  data_source/NTCIR_MedNLP2
   

2. Extract text from original data and save it in text/org

   ./script/extract_text_ASPEC.sh
   ./script/extract_text_BCCWJ.sh
   ./script/extract_text_PatentMT.sh
   ./script/extract_text_MedNLP.sh
   

3. Generate SUW-SC annotated data from text and masked data, and save it in suw-sc/proc

   ./script/restore_suw-sc_data.sh
   

4. Generate SUW annotated data from SUW-SC data and save it in suw/proc

   ./script/restore_suw_data.sh
   

5. Generate raw text data from SUW-SC data and save it in text/proc.
   The resulting data differs from the data in text/org w.r.t.
   character normalization and character error correction.

   ./script/extract_text_from_suw-sc_data.sh
   

6. Remove unnecessary data (Optional)

   rm -r text/tmp/*
   

Shohei Higashiyama
National Institute of Information and Communications Technology (NICT), Seika-cho, Kyoto, Japan
shohei.higashiyama [at] nict.go.jp

Part of this work was supported by the collaborative research between NICT and Fujitsu. We used the Asian Scientific Paper Excerpt Corpus, NITCIR-9 PatentMT test collection, NTCIR-11 MedNLP-2 test collection, and Balanced Corpus of Contemporary Written Japanese to construct the JCMS.

Please cite the following paper.

@inproceedings{higashiyama2022,
    title = {A Japanese Corpus of Many Specialized Domains for Word Segmentation and Part-of-Speech Tagging}
    author = {Higashiyama, Shohei and Ideuchi, Masao and Utiyama, Masao and Oida, Yoshiaki and Sumita, Eiichiro}
    booktitle = {Proceedings of the 3rd Workshop on Evaluation and Comparison of NLP Systems (Eval4NLP)}
    month = Nov,
    year = 2022,
}