metasmile / grit-i18n Goto Github PK

What steps will reproduce the problem?
1. Include the "&" char in one of the policy descriptions in chrome 
(chrome/app/policy/policy_templates.json)
2. Generate the policy_templates target

An intermediate XML representation fails to be built due to the special "&" 
character. This leads to build failures on the try bots when new policies are 
introduced (e.g. http://goo.gl/6CAmL):

Traceback (most recent call last):
  File "../tools/grit/grit.py", line 15, in <module>
    grit.grit_runner.Main(sys.argv[1:])
  File "/mnt/data/b/build/slave/linux_chromeos_clang/build/src/tools/grit/grit/grit_runner.py", line 207, in Main
    toolobject.Run(options, args[1:])
  File "/mnt/data/b/build/slave/linux_chromeos_clang/build/src/tools/grit/grit/tool/build.py", line 118, in Run
    self.res.RunGatherers(recursive = True)
  File "/mnt/data/b/build/slave/linux_chromeos_clang/build/src/tools/grit/grit/node/base.py", line 321, in RunGatherers
    child.RunGatherers(recursive=recursive, debug=debug)
  File "/mnt/data/b/build/slave/linux_chromeos_clang/build/src/tools/grit/grit/node/base.py", line 321, in RunGatherers
    child.RunGatherers(recursive=recursive, debug=debug)
  File "/mnt/data/b/build/slave/linux_chromeos_clang/build/src/tools/grit/grit/node/base.py", line 321, in RunGatherers
    child.RunGatherers(recursive=recursive, debug=debug)
  File "/mnt/data/b/build/slave/linux_chromeos_clang/build/src/tools/grit/grit/node/structure.py", line 153, in RunGatherers
    self.gatherer.Parse()
  File "/mnt/data/b/build/slave/linux_chromeos_clang/build/src/tools/grit/grit/gather/policy_json.py", line 239, in Parse
    self._AddItems(self.data['policy_definitions'], 'policy', None, 2)
  File "/mnt/data/b/build/slave/linux_chromeos_clang/build/src/tools/grit/grit/gather/policy_json.py", line 213, in _AddItems
    self._AddPolicyKey(item1, item_type, parent_item, key, depth + 1)
  File "/mnt/data/b/build/slave/linux_chromeos_clang/build/src/tools/grit/grit/gather/policy_json.py", line 178, in _AddPolicyKey
    self._GetDescription(item, item_type, parent_item, key))
  File "/mnt/data/b/build/slave/linux_chromeos_clang/build/src/tools/grit/grit/gather/policy_json.py", line 72, in _ParseMessage
    node = minidom.parseString(xml).childNodes[0]
  File "/usr/lib/python2.6/xml/dom/minidom.py", line 1928, in parseString
    return expatbuilder.parseString(string)
  File "/usr/lib/python2.6/xml/dom/expatbuilder.py", line 940, in parseString
    return builder.parseString(string)
  File "/usr/lib/python2.6/xml/dom/expatbuilder.py", line 223, in parseString
    parser.Parse(string, True)
xml.parsers.expat.ExpatError: not well-formed (invalid token): line 1, column 16

1. In the chromium sources, edit src/chrome/app/policy/policy_templates.json
2. Rename some name within <ph> so that it becomes a valid infix string of 
another name

(e.g. rename EXTENSION_NAME_EXAMPLE to EXTENSION_POLICY_EXAMPLE_NAME, which now 
has EXTENSION_POLICY_EXAMPLE as a valid prefix)

3. build policy_templates

BaseMessage in tools/grit/grit/tclib.py will build a regex in an undefined 
order, that can split on EXTENSION_POLICY_EXAMPLE first and will break 
EXTENSION_POLICY_EXAMPLE_NAME into "EXTENSION_POLICY_EXAMPLE" and "_NAME", 
which is not what is intended.

Nico claims that:
"grit is very slow because it has all these module-scope 're.compile' calls 
which
get executed at module load time. grit imports all its modules but uses only a
few of these regexen on every run depending on various flags. So as a result
grit has slow startup because of its 'global' re.compile statements."

git grep counts 57 lines with re.compile (outside of unittest.py files).  If 
someone has time to profile, that would be great.

What steps will reproduce the problem?
1. Put in a directive like the following:
<include
    src="foobaz.html"
</include>
2. Run the grit tool on that file
3. Observe the </include> is left behind

What is the expected output? What do you see instead?
</include> should no longer be there. It should have been removed when the file 
was inlined.

What version of the product are you using? On what operating system?
r126

Please provide any additional information below.
The proposed fix is below:

For the tags that get replaced wholesale, change the .*> at the end to .*?> and 
use the modifier re.DOTALL to include newlines.

--- grit/format/html_inline.py  (revision 126)
+++ grit/format/html_inline.py  (working copy)
@@ -33,10 +33,10 @@
 # Used by DoInline to replace various links with inline content.
 _STYLESHEET_RE = lazy_re.compile(
     '<link rel="stylesheet"[^>]+?href="(?P<filename>[^"]*)".*?>',
-    re.MULTILINE)
+    re.DOTALL)
 _INCLUDE_RE = lazy_re.compile(
-    '<include[^>]+?src="(?P<filename>[^"\']*)".*>',
-    re.MULTILINE)
+    '<include[^>]+?src="(?P<filename>[^"\']*)".*?>',
+    re.DOTALL)
 _SRC_RE = lazy_re.compile(
     r'<(?!script)(?:[^>]+?\s)src=(?P<quote>")(?P<filename>[^"\']*)\1',
     re.MULTILINE)

(As talked about on IM with thakis@.)

Chrome has multiple resource paks for different purposes. There is a set of 
{default,touch}_{100,200,300,...}.pak files which contain data which is 
supposed to be keyed to certain properties of the display, and a normal 
resource.pak which is a supposed to be used for items which aren't display 
dependent.

It is entirely possible to enter a world of pain by unintentionally mixing 
these two concepts. Given that the definitions for, ie, the chrome_html and 
chrome_scaled_image structures live in grit, maybe a warning or error could be 
emitted if there's mixing of a chrome_scaled_image structure and a normal, 
non-image include?

This may not be possible (or adding this here might be the wrong architectural 
layer).

In this sentence: 
"Use SVN to fetch the GRIT sources, and run [grit.py] for help. GRIT integrates 
easily with GYP, see examples."

"examples" point to 
http://codesearch.google.com/#search/&q=grit%20file:%5C.gyp&type=cs, but when 
we navigate to it an 404 error occurs (404. That’s an error.).

May be we should update this link to cs.chromium.org?

On at least one version of Python (2.6.4 on Windows), when running [grit build] 
on files that contain Unicode characters, I get errors like the stack trace 
below.  Switching to a regular file open and util.WrapOutputStream seems to fix 
the issue.

Traceback (most recent call last):
  File "D:\src\tb/googleclient/tools/new_grit.py", line 56, in <module>
    grit.grit_runner.Main(['-h', 'transconsole.base.FP'] + args)
  File "D:\src\tb/googleclient/tools\..\..\google3\third_party\py\grit\grit_runner.py", line 274, in Main
    toolobject.Run(options, args[1:])
  File "D:\src\tb/googleclient/tools\..\..\google3\third_party\py\grit\tool\build.py", line 110, in Run
    self.Process()
  File "D:\src\tb/googleclient/tools\..\..\google3\third_party\py\grit\tool\build.py", line 255, in Process
    self.ProcessNode(self.res, output, outfile)
  File "D:\src\tb/googleclient/tools\..\..\google3\third_party\py\grit\tool\build.py", line 181, in ProcessNode
    RcBuilder.ProcessNode(child, output_node, outfile)
  File "D:\src\tb/googleclient/tools\..\..\google3\third_party\py\grit\tool\build.py", line 181, in ProcessNode
    RcBuilder.ProcessNode(child, output_node, outfile)
  File "D:\src\tb/googleclient/tools\..\..\google3\third_party\py\grit\tool\build.py", line 181, in ProcessNode
    RcBuilder.ProcessNode(child, output_node, outfile)
  File "D:\src\tb/googleclient/tools\..\..\google3\third_party\py\grit\tool\build.py", line 173, in ProcessNode
    output_dir=base_dir)
  File "D:\src\tb/googleclient/tools\..\..\google3\third_party\py\grit\format\rc.py", line 432, in Format
    filename = os.path.abspath(item.FileForLanguage(lang, output_dir))
  File "D:\src\tb/googleclient/tools\..\..\google3\third_party\py\grit\node\structure.py", line 301, in FileForLanguage
    file_object.write(file_contents)
UnicodeEncodeError: 'ascii' codec can't encode character u'\xbb' in position 
10833: ordinal not in range(128)

What steps will reproduce the problem?
1. add a message with the same id in two different conditional branches and set 
'use_name_for_id' to true
2. run grit 'compiler'


What is the expected output? What do you see instead?

Actual:  I got the following error : 

MakeClique
    assert c.GetMessage().GetPresentableContent() == presentable_text

Expected: The message in a branch meeting the condition is processed while 
other branches are ignored. 


What version of the product are you using? On what operating system?

grit-i18n @ r=84 on Linux, but it should be platform-independent. 

Please provide any additional information below.

See https://chromiumcodereview.appspot.com/11338052/#ps1

The documentation generated for the policy templates does not recognize the 
mobile platforms ("android" and "ios").

It also shows the registry path even for policies that aren't supported on 
Windows. Same for plists for Mac and JSON for Linux.

It is possible to have 'sub' grd files, the ones with .grdp extension.

They can be include in .grd files with:

<part file="bookmarks_strings.grdp" />

An example is here:
https://code.google.com/p/chromium/codesearch#chromium/src/chrome/app/bookmarks_
strings.grdp&sq=package:chromium&type=cs

But UsersGuide does not say a word about this. It would be cool to inform the 
users that grit has this feature.

This seems non-trivial and might require testing with Chrome to ensure the fix 
is OK.  Chrome is the only user of the use_base_dir attribute that I know of.


======================================================================
FAIL: testGetPathNoBasedir (grit.node.include_unittest.IncludeNodeUnittest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "S:\src\grit-i18n\grit\node\include_unittest.py", line 71, in testGetPathNoBasedir
    os.path.join(ur'../', ur'flugel/kugel.pdf')))
AssertionError: u'..\\grit-i18n\\flugel\\kugel.pdf' != u'..\\flugel\\kugel.pdf'

======================================================================
ERROR: Tests that ".src=" doesn't treat as a tag.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "S:\src\grit-i18n\grit\format\html_inline_unittest.py", line 95, in testCompressedJavaScript
    resources = html_inline.GetResourceFilenames(tmp_dir.GetPath('index.js'))
  File "S:\src\grit-i18n\grit\format\html_inline.py", line 378, in GetResourceFilenames
    rewrite_function=rewrite_function).inlined_files
  File "S:\src\grit-i18n\grit\format\html_inline.py", line 306, in DoInline
    flat_text)
  File "C:\python_26_amd64\files\lib\re.py", line 151, in sub
    return _compile(pattern, 0).sub(repl, string, count)
  File "C:\python_26_amd64\files\lib\re.py", line 241, in _compile
    raise TypeError, "first argument must be string or compiled pattern"
TypeError: first argument must be string or compiled pattern

======================================================================
ERROR: Tests that all included files are returned by GetResourceFilenames.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "S:\src\grit-i18n\grit\format\html_inline_unittest.py", line 76, in testGetResourceFilenames
    resources = html_inline.GetResourceFilenames(tmp_dir.GetPath('index.html'))
  File "S:\src\grit-i18n\grit\format\html_inline.py", line 378, in GetResourceFilenames
    rewrite_function=rewrite_function).inlined_files
  File "S:\src\grit-i18n\grit\format\html_inline.py", line 306, in DoInline
    flat_text)
  File "C:\python_26_amd64\files\lib\re.py", line 151, in sub
    return _compile(pattern, 0).sub(repl, string, count)
  File "C:\python_26_amd64\files\lib\re.py", line 241, in _compile
    raise TypeError, "first argument must be string or compiled pattern"
TypeError: first argument must be string or compiled pattern

======================================================================
ERROR: Tests that @import directives in inlined CSS files are inlined too.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "S:\src\grit-i18n\grit\format\html_inline_unittest.py", line 149, in testInlineCSSImports
    result = html_inline.DoInline(tmp_dir.GetPath('index.html'), None)
  File "S:\src\grit-i18n\grit\format\html_inline.py", line 306, in DoInline
    flat_text)
  File "C:\python_26_amd64\files\lib\re.py", line 151, in sub
    return _compile(pattern, 0).sub(repl, string, count)
  File "C:\python_26_amd64\files\lib\re.py", line 241, in _compile
    raise TypeError, "first argument must be string or compiled pattern"
TypeError: first argument must be string or compiled pattern

======================================================================
ERROR: Tests that only CSS files referenced via relative URLs are inlined.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "S:\src\grit-i18n\grit\format\html_inline_unittest.py", line 197, in testInlineCSSLinks
    result = html_inline.DoInline(tmp_dir.GetPath('index.html'), None)
  File "S:\src\grit-i18n\grit\format\html_inline.py", line 306, in DoInline
    flat_text)
  File "C:\python_26_amd64\files\lib\re.py", line 151, in sub
    return _compile(pattern, 0).sub(repl, string, count)
  File "C:\python_26_amd64\files\lib\re.py", line 241, in _compile
    raise TypeError, "first argument must be string or compiled pattern"
TypeError: first argument must be string or compiled pattern

======================================================================
ERROR: testChromeHtmlNodeOutputfile (grit.format.rc_unittest.FormatRcUnittest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "S:\src\grit-i18n\grit\format\rc_unittest.py", line 269, in testChromeHtmlNodeOutputfile
    root.RunGatherers()
  File "S:\src\grit-i18n\grit\node\misc.py", line 471, in RunGatherers
    node.RunPreSubstitutionGatherer(debug=debug)
  File "S:\src\grit-i18n\grit\node\structure.py", line 192, in RunPreSubstitutionGatherer
    self.gatherer.Parse()
  File "S:\src\grit-i18n\grit\gather\chrome_html.py", line 290, in Parse
    rewrite_function=lambda fp, t, d: ProcessImageSets(
  File "S:\src\grit-i18n\grit\format\html_inline.py", line 344, in InlineToString
    rewrite_function=rewrite_function).inlined_data
  File "S:\src\grit-i18n\grit\format\html_inline.py", line 306, in DoInline
    flat_text)
  File "C:\python_26_amd64\files\lib\re.py", line 151, in sub
    return _compile(pattern, 0).sub(repl, string, count)
  File "C:\python_26_amd64\files\lib\re.py", line 241, in _compile
    raise TypeError, "first argument must be string or compiled pattern"
TypeError: first argument must be string or compiled pattern

======================================================================
ERROR: testRcIncludeFlattenedHtmlFile (grit.format.rc_unittest.FormatRcUnittest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "S:\src\grit-i18n\grit\format\rc_unittest.py", line 206, in testRcIncludeFlattenedHtmlFile
    buf)
  File "S:\src\grit-i18n\grit\tool\build.py", line 205, in ProcessNode
    outfile.writelines(formatted)
  File "C:\python_26_amd64\files\lib\StringIO.py", line 250, in writelines
    for line in iterable:
  File "S:\src\grit-i18n\grit\format\rc.py", line 35, in Format
    yield FormatInclude(item, lang, output_dir)
  File "S:\src\grit-i18n\grit\format\rc.py", line 433, in FormatInclude
    filename = item.Process(output_dir)
  File "S:\src\grit-i18n\grit\node\include.py", line 104, in Process
    outfile.write(self._GetFlattenedData())
  File "S:\src\grit-i18n\grit\node\include.py", line 38, in _GetFlattenedData
    allow_external_script=allow_external_script))
  File "S:\src\grit-i18n\grit\format\html_inline.py", line 344, in InlineToString
    rewrite_function=rewrite_function).inlined_data
  File "S:\src\grit-i18n\grit\format\html_inline.py", line 306, in DoInline
    flat_text)
  File "C:\python_26_amd64\files\lib\re.py", line 151, in sub
    return _compile(pattern, 0).sub(repl, string, count)
  File "C:\python_26_amd64\files\lib\re.py", line 241, in _compile
    raise TypeError, "first argument must be string or compiled pattern"
TypeError: first argument must be string or compiled pattern

======================================================================
ERROR: Tests inlined image file resources with available high DPI assets.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "S:\src\grit-i18n\grit\gather\chrome_html_unittest.py", line 66, in testFileResources
    html.Parse()
  File "S:\src\grit-i18n\grit\gather\chrome_html.py", line 290, in Parse
    rewrite_function=lambda fp, t, d: ProcessImageSets(
  File "S:\src\grit-i18n\grit\format\html_inline.py", line 344, in InlineToString
    rewrite_function=rewrite_function).inlined_data
  File "S:\src\grit-i18n\grit\format\html_inline.py", line 306, in DoInline
    flat_text)
  File "C:\python_26_amd64\files\lib\re.py", line 151, in sub
    return _compile(pattern, 0).sub(repl, string, count)
  File "C:\python_26_amd64\files\lib\re.py", line 241, in _compile
    raise TypeError, "first argument must be string or compiled pattern"
TypeError: first argument must be string or compiled pattern

======================================================================
ERROR: Tests inlined image file resources if url() filename is double quoted.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "S:\src\grit-i18n\grit\gather\chrome_html_unittest.py", line 166, in testFileResourcesDoubleQuotes
    html.Parse()
  File "S:\src\grit-i18n\grit\gather\chrome_html.py", line 290, in Parse
    rewrite_function=lambda fp, t, d: ProcessImageSets(
  File "S:\src\grit-i18n\grit\format\html_inline.py", line 344, in InlineToString
    rewrite_function=rewrite_function).inlined_data
  File "S:\src\grit-i18n\grit\format\html_inline.py", line 306, in DoInline
    flat_text)
  File "C:\python_26_amd64\files\lib\re.py", line 151, in sub
    return _compile(pattern, 0).sub(repl, string, count)
  File "C:\python_26_amd64\files\lib\re.py", line 241, in _compile
    raise TypeError, "first argument must be string or compiled pattern"
TypeError: first argument must be string or compiled pattern

======================================================================
ERROR: Tests inlined image file resources with available high DPI assets on
----------------------------------------------------------------------
Traceback (most recent call last):
  File "S:\src\grit-i18n\grit\gather\chrome_html_unittest.py", line 107, in testFileResourcesImageTag
    html.Parse()
  File "S:\src\grit-i18n\grit\gather\chrome_html.py", line 290, in Parse
    rewrite_function=lambda fp, t, d: ProcessImageSets(
  File "S:\src\grit-i18n\grit\format\html_inline.py", line 344, in InlineToString
    rewrite_function=rewrite_function).inlined_data
  File "S:\src\grit-i18n\grit\format\html_inline.py", line 306, in DoInline
    flat_text)
  File "C:\python_26_amd64\files\lib\re.py", line 151, in sub
    return _compile(pattern, 0).sub(repl, string, count)
  File "C:\python_26_amd64\files\lib\re.py", line 241, in _compile
    raise TypeError, "first argument must be string or compiled pattern"
TypeError: first argument must be string or compiled pattern

======================================================================
ERROR: Tests inlined image file resources without available high DPI assets.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "S:\src\grit-i18n\grit\gather\chrome_html_unittest.py", line 230, in testFileResourcesNoFile
    html.Parse()
  File "S:\src\grit-i18n\grit\gather\chrome_html.py", line 290, in Parse
    rewrite_function=lambda fp, t, d: ProcessImageSets(
  File "S:\src\grit-i18n\grit\format\html_inline.py", line 344, in InlineToString
    rewrite_function=rewrite_function).inlined_data
  File "S:\src\grit-i18n\grit\format\html_inline.py", line 306, in DoInline
    flat_text)
  File "C:\python_26_amd64\files\lib\re.py", line 151, in sub
    return _compile(pattern, 0).sub(repl, string, count)
  File "C:\python_26_amd64\files\lib\re.py", line 241, in _compile
    raise TypeError, "first argument must be string or compiled pattern"
TypeError: first argument must be string or compiled pattern

======================================================================
ERROR: Tests inlined image file resources when url() filename is unquoted.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "S:\src\grit-i18n\grit\gather\chrome_html_unittest.py", line 193, in testFileResourcesNoQuotes
    html.Parse()
  File "S:\src\grit-i18n\grit\gather\chrome_html.py", line 290, in Parse
    rewrite_function=lambda fp, t, d: ProcessImageSets(
  File "S:\src\grit-i18n\grit\format\html_inline.py", line 344, in InlineToString
    rewrite_function=rewrite_function).inlined_data
  File "S:\src\grit-i18n\grit\format\html_inline.py", line 306, in DoInline
    flat_text)
  File "C:\python_26_amd64\files\lib\re.py", line 151, in sub
    return _compile(pattern, 0).sub(repl, string, count)
  File "C:\python_26_amd64\files\lib\re.py", line 241, in _compile
    raise TypeError, "first argument must be string or compiled pattern"
TypeError: first argument must be string or compiled pattern

======================================================================
ERROR: Tests removing an unsupported scale factor from an explicit image-set.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "S:\src\grit-i18n\grit\gather\chrome_html_unittest.py", line 328, in testRemoveUnsupportedScale
    html.Parse()
  File "S:\src\grit-i18n\grit\gather\chrome_html.py", line 290, in Parse
    rewrite_function=lambda fp, t, d: ProcessImageSets(
  File "S:\src\grit-i18n\grit\format\html_inline.py", line 344, in InlineToString
    rewrite_function=rewrite_function).inlined_data
  File "S:\src\grit-i18n\grit\format\html_inline.py", line 306, in DoInline
    flat_text)
  File "C:\python_26_amd64\files\lib\re.py", line 151, in sub
    return _compile(pattern, 0).sub(repl, string, count)
  File "C:\python_26_amd64\files\lib\re.py", line 241, in _compile
    raise TypeError, "first argument must be string or compiled pattern"
TypeError: first argument must be string or compiled pattern

======================================================================
ERROR: Tests inserting high DPI chrome://theme references.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "S:\src\grit-i18n\grit\gather\chrome_html_unittest.py", line 275, in testThemeResources
    html.Parse()
  File "S:\src\grit-i18n\grit\gather\chrome_html.py", line 290, in Parse
    rewrite_function=lambda fp, t, d: ProcessImageSets(
  File "S:\src\grit-i18n\grit\format\html_inline.py", line 344, in InlineToString
    rewrite_function=rewrite_function).inlined_data
  File "S:\src\grit-i18n\grit\format\html_inline.py", line 306, in DoInline
    flat_text)
  File "C:\python_26_amd64\files\lib\re.py", line 151, in sub
    return _compile(pattern, 0).sub(repl, string, count)
  File "C:\python_26_amd64\files\lib\re.py", line 241, in _compile
    raise TypeError, "first argument must be string or compiled pattern"
TypeError: first argument must be string or compiled pattern

======================================================================
FAIL: testLowresFallback 
(grit.gather.chrome_scaled_image_unittest.ChromeScaledImageUnittest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "S:\src\grit-i18n\grit\gather\chrome_scaled_image_unittest.py", line 162, in testLowresFallback
    _RunBuildTest(*args, skip_rc=True)
  File "S:\src\grit-i18n\grit\gather\chrome_scaled_image_unittest.py", line 109, in _RunBuildTest
    build.RcBuilder().Run(options, [])
  File "S:\src\grit-i18n\grit\tool\build.py", line 149, in Run
    self.Process()
  File "S:\src\grit-i18n\grit\tool\build.py", line 268, in Process
    self.ProcessNode(self.res, output, outfile)
  File "S:\src\grit-i18n\grit\tool\build.py", line 204, in ProcessNode
    formatted = formatter(node, output_node.GetLanguage(), output_dir=base_dir)
  File "S:\src\grit-i18n\grit\format\data_pack.py", line 46, in Format
    id, value = node.GetDataPackPair(lang, UTF8)
  File "S:\src\grit-i18n\grit\node\structure.py", line 168, in GetDataPackPair
    data = self.gatherer.GetData(lang, encoding)
  File "S:\src\grit-i18n\grit\gather\chrome_scaled_image.py", line 136, in GetData
    data = _MoveSpecialChunksToFront(data)
  File "S:\src\grit-i18n\grit\gather\chrome_scaled_image.py", line 53, in _MoveSpecialChunksToFront
    for chunk in _ChunkifyPNG(data):
  File "S:\src\grit-i18n\grit\gather\chrome_scaled_image.py", line 67, in _ChunkifyPNG
    assert data.startswith(_PNG_MAGIC)
AssertionError

======================================================================
FAIL: testNormalFallback 
(grit.gather.chrome_scaled_image_unittest.ChromeScaledImageUnittest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "S:\src\grit-i18n\grit\gather\chrome_scaled_image_unittest.py", line 134, in testNormalFallback
    'tactile_123_percent': set([t123a, d123b]),
  File "S:\src\grit-i18n\grit\gather\chrome_scaled_image_unittest.py", line 109, in _RunBuildTest
    build.RcBuilder().Run(options, [])
  File "S:\src\grit-i18n\grit\tool\build.py", line 149, in Run
    self.Process()
  File "S:\src\grit-i18n\grit\tool\build.py", line 268, in Process
    self.ProcessNode(self.res, output, outfile)
  File "S:\src\grit-i18n\grit\tool\build.py", line 204, in ProcessNode
    formatted = formatter(node, output_node.GetLanguage(), output_dir=base_dir)
  File "S:\src\grit-i18n\grit\format\data_pack.py", line 46, in Format
    id, value = node.GetDataPackPair(lang, UTF8)
  File "S:\src\grit-i18n\grit\node\structure.py", line 168, in GetDataPackPair
    data = self.gatherer.GetData(lang, encoding)
  File "S:\src\grit-i18n\grit\gather\chrome_scaled_image.py", line 136, in GetData
    data = _MoveSpecialChunksToFront(data)
  File "S:\src\grit-i18n\grit\gather\chrome_scaled_image.py", line 53, in _MoveSpecialChunksToFront
    for chunk in _ChunkifyPNG(data):
  File "S:\src\grit-i18n\grit\gather\chrome_scaled_image.py", line 67, in _ChunkifyPNG
    assert data.startswith(_PNG_MAGIC)
AssertionError

======================================================================
FAIL: testGetPathNoBasedir (grit.node.include_unittest.IncludeNodeUnittest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "S:\src\grit-i18n\grit\node\include_unittest.py", line 70, in testGetPathNoBasedir
    os.path.join(ur'../', ur'flugel/kugel.pdf')))
AssertionError: u'..\\grit-i18n\\flugel\\kugel.pdf' != u'..\\flugel\\kugel.pdf'

======================================================================
FAIL: testOutputs (grit.tool.android2grd_unittest.Android2GrdUnittest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "S:\src\grit-i18n\grit\tool\android2grd_unittest.py", line 166, in testOutputs
    'header/dir/chrome_android_strings.h')
AssertionError: 'header/dir\\chrome_android_strings.h' != 
'header/dir/chrome_android_strings.h'

----------------------------------------------------------------------
Ran 265 tests in 1.852s

FAILED (failures=4, errors=13)

On OS X, run

  time python ../tools/grit/grit.py -i app/generated_resources.grd build -f ../tools/gritsettings/resource_ids -o ../out_android/Release/gen/chrome -D _chromium -E "CHROMIUM_BUILD=chromium" -t android -E "ANDROID_JAVA_TAGGED_ONLY=true" -D enable_printing -D use_concatenated_impulse_responses -D enable_webrtc

from the 'chrome' directory. It'll die with

  grit.exception.DuplicateKey: A duplicate key attribute was found.: IDS_CONTENT_CONTEXT_PASTE, IDS_PASTE

which is caused by EvaluateExpression() thinking that is_macosx is true, even 
though `-t android` is passed in. I'm guessing this is broken on linux too but 
doesn't cause problems since there's not much linux-specific stuff in grd files 
on linux?

This issue was raised by crbug.com/372175 and was fixed in crrev.com/282495 
(https://codereview.chromium.org/381353002/).

<emit> tag can contain an emit_type attribute, but the documentation doe not 
mention it.

In chromium this is used in more than one place and the value is prepend.

<link> tags to load .css files aren't inlined when wrapping, i.e.

  <link rel="stylesheet" href="../../blah.css">

works, but

    <link rel="stylesheet"
        href="../../../../../../../../../../../../blah.css">

doesn't.

This is useful because we like to keep HTML files in Chromium to the 80 col 
wrapping rule, so there may be the need to wrap (if possible).

I think the regexp in grit could be updated to handle this case relatively 
easily:
https://code.google.com/p/chromium/codesearch#chromium/src/tools/grit/grit/forma
t/html_inline.py&l=278

Here's a CL to do this:
https://codereview.chromium.org/12289018

In Python 2.7 a bug in dom.xml.minixml.toprettyxml() has been fixed that 
changes the way strings are formatted removing indentation for them. This 
breaks our adml tests.

Internal to Google, there is a separate branch of grit with several features 
that the open-source grit is missing.  The open-source grit, which was branched 
away from the internal trunk a few years ago, also has several features that 
are not in the internal branch.

This bug tracks the work being done to add any important features from the 
internal branch to the open-source grit (this project) and make the open-source 
grit able to replace the Google-internal version of grit, so that grit may be 
maintained in just this one place.

======================================================================
FAIL: Tests removing an unsupported scale factor from an explicit image-set.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "S:\src\grit-i18n\grit\gather\chrome_html_unittest.py", line 345, in testRemoveUnsupportedScale
    '''))
AssertionError: "\n      <!DOCTYPE HTML>\n      <html>\n        <head>\n        
  <style>\n      .image {\n        background: -webk
it-image-set(url('data:image/png;base64,UE5HIERBVEE=') 1x,\n                    
                  url('data:image/png;base64,MS40eCB
QTkcgREFUQQ==') 1.4x,\n                                      
url('data:image/png;base64,MS44eCBQTkcgREFUQQ==') 1.8x);\n      }\n
  </style>\n        </head>\n        <body>\n          <!-- Don't need a body. -->\n        </body>\n      </html>\n      " != "\n
    <!DOCTYPE HTML>\n      <html>\n        <head>\n          <style>\n      .image {\n        background: -webkit-image-set(url('dat
a:image/png;base64,UE5HIERBVEE=') 1x,\n                                      
url('data:image/png;base64,MS44eCBQTkcgREFUQQ==') 1.8x)
;\n      }\n      </style>\n        </head>\n        <body>\n          <!-- 
Don't need a body. -->\n        </body>\n      </html>\n
      "

For most other python stuff I need I can install by just issuing:

 $ pip install <package>

It would be great if grit-i18n were available via pip as well. I'm not deeply 
familiar with the Python world so I don't know any of the technical details to 
make this happen though.

What steps will reproduce the problem?
1. Have a html resource with a tag like <script type="foo" src="path">
2. Embed it in a resource with flatten_html=true
3. Look at at the resulting tag

What is the expected output? What do you see instead?
Expect <script type="foo">..contents..</script>
See instead <script>...contents...</script>

It would be nice if grit warned about duplicate files in the PAK files 
(included normally or through html_inline).

For example Chrome for Android has the following duplicates now:

Count   | File
--------------------------------------------------------------------------------
-------------------------------------
      2 [ /usr/local/google/ssd1/clankium/src/chrome/app/theme/default_100_percent/common/error_network_generic.png
      2 [ /usr/local/google/ssd1/clankium/src/chrome/app/theme/default_100_percent/common/error_network_offline.png
      2 [ /usr/local/google/ssd1/clankium/src/chrome/app/theme/default_100_percent/extensions_section.png
      2 [ /usr/local/google/ssd1/clankium/src/chrome/app/theme/default_200_percent/extensions_section.png
      2 [ /usr/local/google/ssd1/clankium/src/chrome/browser/resources/about_welcome_android/images/holo_logo_150.png
      2 [ /usr/local/google/ssd1/clankium/src/chrome/browser/resources/ntp_android/images/default_thumbnail.png
      2 [ /usr/local/google/ssd1/clankium/src/chrome/browser/resources/ssl/roadblock_background.png
      2 [ /usr/local/google/ssd1/clankium/src/chrome/browser/resources/ssl/ssl_errors_common.js
      2 [ /usr/local/google/ssd1/clankium/src/chrome/browser/resources/uber/uber_shared.css
      2 [ /usr/local/google/ssd1/clankium/src/chrome/renderer/resources/neterror.css
      2 [ /usr/local/google/ssd1/clankium/src/chrome/renderer/resources/neterror.js
      2 [ /usr/local/google/ssd1/clankium/src/chrome/renderer/resources/plugin_blocked.png
      2 [ /usr/local/google/ssd1/clankium/src/third_party/jstemplate/jstemplate_compiled.js
      2 [ /usr/local/google/ssd1/clankium/src/ui/resources/default_100_percent/common/folder_closed.png
      2 [ /usr/local/google/ssd1/clankium/src/ui/resources/default_100_percent/common/folder_closed_rtl.png
      2 [ /usr/local/google/ssd1/clankium/src/ui/webui/resources/images/star_small.png
      2 [ /usr/local/google/ssd1/clankium/src/ui/webui/resources/js/cr.js
      2 [ /usr/local/google/ssd1/clankium/src/ui/webui/resources/js/i18n_template_no_process.js
      3 [ /usr/local/google/ssd1/clankium/src/chrome/browser/resources/disclosure_triangle_small.png
      3 [ /usr/local/google/ssd1/clankium/src/chrome/browser/resources/ssl/images/roadblock.png
      3 [ /usr/local/google/ssd1/clankium/src/chrome/browser/resources/uber/uber_utils.js
      3 [ /usr/local/google/ssd1/clankium/src/chrome/renderer/resources/plugin_placeholders.css
      3 [ /usr/local/google/ssd1/clankium/src/ui/resources/default_200_percent/close_2_hover.png
      3 [ /usr/local/google/ssd1/clankium/src/ui/resources/default_200_percent/close_2.png
      4 [ /usr/local/google/ssd1/clankium/src/ui/resources/default_100_percent/close_2_hover.png
      4 [ /usr/local/google/ssd1/clankium/src/ui/resources/default_100_percent/close_2.png
      4 [ /usr/local/google/ssd1/clankium/src/ui/resources/default_100_percent/close_2_pressed.png
      4 [ /usr/local/google/ssd1/clankium/src/ui/resources/default_200_percent/close_2_pressed.png
      4 [ /usr/local/google/ssd1/clankium/src/ui/webui/resources/images/select.png
      6 [ /usr/local/google/ssd1/clankium/src/chrome/browser/resources/about_welcome_android/images/holo_logo_200.png
      6 [ /usr/local/google/ssd1/clankium/src/ui/webui/resources/js/util.js
      9 [ /usr/local/google/ssd1/clankium/src/ui/webui/resources/js/assert.js

Could you please document how GRIT's language codes are constructed in 
GritUserGuide?  I guess it's ISO 639-1, but I've seen things like "zh-CN", too, 
so I think it would be great to have a formal definition of these codes.

Jocsibal

grit should follow the same convention as gyp,
which is that iOS and Mac are different.

https://code.google.com/searchframe#OAMlx_jo-ck/src/tools/grit/grit/node/base.py
&exact_package=chromium&q=is_macosx&l=467

The specific case is for messages that are iOS only.
Current workaround is to test for pp_ifdef('ios').

Original discussion:
https://codereview.chromium.org/11966004/

To generate a grd file that is equivalent to the android strings.xml file, 
android2grd should include non-translatable strings in the output grd.

In resource_map.py the _FormatSource function builds a map from either path or 
name to resource id. If it encounters duplicate values it silently skips the 
duplicates. For name, duplicate values likely represent an error and should be 
reported. For path, duplicate values mean we are packing the same resource more 
than once (See https://code.google.com/p/grit-i18n/issues/detail?id=27). This 
is not necessarily an error but I feel that if we want to explicitly support 
this we should include both resource ids in the "map" which would be more of a 
multi-map at that point.

When hash value randomization is turned on in Python 2.7.3. and up (it is on by 
default for Python 3.3 and up), the following tests fail once every 10 runs or 
so.  See http://bugs.python.org/issue13703 for context on why hash value 
randomization is done.

Easiest way to test is on a Google-internal machine. See Google-internal info 
on this at http://go/lixvg


======================================================================
FAIL: testDictionaryPolicy 
(grit.format.policy_templates.writers.json_writer_unittest.JsonWriterUnittest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/build/work/f99236681aa92b17eec3f07013049c15/google3/runfiles/google3/third_party/py/grit/../grit/format/policy_templates/wr
iters/json_writer_unittest.py", line 272, in testDictionaryPolicy
    self.CompareOutputs(output, expected_output)
  File "/build/work/f99236681aa92b17eec3f07013049c15/google3/runfiles/google3/third_party/py/grit/../grit/format/policy_templates/wr
iters/json_writer_unittest.py", line 48, in CompareOutputs
    expected_output.strip())
AssertionError: u'// Policy template for Linux.\n// Uncomment the policies you 
wish to activate and change their values to\n// somet
hing useful for your case. The provided values are for reference only\n// and 
do not provide meaningful defaults!\n{\n  // Example D
ictionary Policy\n  
//-------------------------------------------------------------------------\n  
// Example Dictionary Policy\n\n
 //"DictionaryPolicy": {\'int\': 10, \'dict\': {\'a\': 1, \'b\': 2}, \'string\': \'abc\', \'list\': [1, 2, 3], \'bool\': True}\n\n}'
 != '// Policy template for Linux.\n// Uncomment the policies you wish to activate and change their values to\n// something useful f
or your case. The provided values are for reference only\n// and do not provide 
meaningful defaults!\n{\n  // Example Dictionary Pol
icy\n  
//-------------------------------------------------------------------------\n  
// Example Dictionary Policy\n\n  //"Dictionar
yPolicy": {\'int\': 10, \'list\': [1, 2, 3], \'string\': \'abc\', \'dict\': 
{\'a\': 1, \'b\': 2}, \'bool\': True}\n\n}'

======================================================================
FAIL: testDictionaryPolicy 
(grit.format.policy_templates.writers.reg_writer_unittest.RegWriterUnittest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/build/work/f99236681aa92b17eec3f07013049c15/google3/runfiles/google3/third_party/py/grit/../grit/format/policy_templates/writers/reg_writer_unittest.py", line 251, in testDictionaryPolicy
    self.CompareOutputs(output, expected_output)
  File "/build/work/f99236681aa92b17eec3f07013049c15/google3/runfiles/google3/third_party/py/grit/../grit/format/policy_templates/writers/reg_writer_unittest.py", line 37, in CompareOutputs
    expected_output.strip())
AssertionError: 'Windows Registry Editor Version 
5.00\r\n\r\n[HKEY_LOCAL_MACHINE\\Software\\Policies\\Chromium]\r\n"DictionaryPol
icy"="{\'int\': 10, \'dict\': {\'a\': 1, \'b\': 2}, \'string\': \'abc\', 
\'list\': [1, 2, 3], \'bool\': True}"' != 'Windows Registry Editor Version 
5.00\r\n\r\n[HKEY_LOCAL_MACHINE\\Software\\Policies\\Chromium]\r\n"DictionaryPol
icy"="{\'int\': 10, \'list\': [1, 2, 3], \'string\': \'abc\', \'dict\': {\'a\': 
1, \'b\': 2}, \'bool\': True}"'

Two consecutive runs of [ grit xmb ] produce .xmb files that are semantically 
equivalent, but may change the order in which messages appear.

This makes regression testing harder, where we compare the output of [ grit xmb 
] for certain projects before and after an update to GRIT.

There are two options to solve this:
a) Write a tool that normalizes the order;
b) Change GRIT to output XMBs in a stable order.

Solution (b) is probably less work.

What steps will reproduce the problem?
1. Have a message spanning multiple lines.
2. Use GRIT to generate 'chrome_messages_json' output.
3. Open the generated json and observe that '\n' wasn't properly escaped.

What version of the product are you using? On what operating system?

http://grit-i18n.googlecode.com/svn/trunk@127

Windows 7

Please provide any additional information below.

I'll be uploading the fix for review shortly.

The GRIT User's Guide (at 
https://code.google.com/p/grit-i18n/wiki/GritUsersGuide) has a section called 
"GRIT Cookbook" which contains example snippets that would belong in entire 
.grd files.

It would be nice if for each of these, we had a .grd file under trunk/examples 
where that example snippet is used in a full .grd file that you are able to 
successfully run [ grit build ] on.

Creating one of these example files is a good first bug for anybody who wants 
to start contributing to GRIT.  To create an example file, you need to read the 
user's guide, understand how GRIT works, and get slightly familiar with the 
project's revision control and code review guidelines (see 
https://code.google.com/p/grit-i18n/wiki/HowToContribute).

If you're interested in any one or more of these examples, please add a comment 
saying which example you intend to implement.  Reviews may be sent to 
[email protected].

======================================================================
FAIL: testLowresFallback 
(grit.gather.chrome_scaled_image_unittest.ChromeScaledImageUnittest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "S:\src\grit-i18n\grit\gather\chrome_scaled_image_unittest.py", line 162, in testLowresFallback
    _RunBuildTest(*args, skip_rc=True)
  File "S:\src\grit-i18n\grit\gather\chrome_scaled_image_unittest.py", line 109, in _RunBuildTest
    build.RcBuilder().Run(options, [])
  File "S:\src\grit-i18n\grit\tool\build.py", line 149, in Run
    self.Process()
  File "S:\src\grit-i18n\grit\tool\build.py", line 268, in Process
    self.ProcessNode(self.res, output, outfile)
  File "S:\src\grit-i18n\grit\tool\build.py", line 204, in ProcessNode
    formatted = formatter(node, output_node.GetLanguage(), output_dir=base_dir)
  File "S:\src\grit-i18n\grit\format\data_pack.py", line 46, in Format
    id, value = node.GetDataPackPair(lang, UTF8)
  File "S:\src\grit-i18n\grit\node\structure.py", line 168, in GetDataPackPair
    data = self.gatherer.GetData(lang, encoding)
  File "S:\src\grit-i18n\grit\gather\chrome_scaled_image.py", line 136, in GetData
    data = _MoveSpecialChunksToFront(data)
  File "S:\src\grit-i18n\grit\gather\chrome_scaled_image.py", line 53, in _MoveSpecialChunksToFront
    for chunk in _ChunkifyPNG(data):
  File "S:\src\grit-i18n\grit\gather\chrome_scaled_image.py", line 67, in _ChunkifyPNG
    assert data.startswith(_PNG_MAGIC)
AssertionError

======================================================================
FAIL: testNormalFallback 
(grit.gather.chrome_scaled_image_unittest.ChromeScaledImageUnittest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "S:\src\grit-i18n\grit\gather\chrome_scaled_image_unittest.py", line 134, in testNormalFallback
    'tactile_123_percent': set([t123a, d123b]),
  File "S:\src\grit-i18n\grit\gather\chrome_scaled_image_unittest.py", line 109, in _RunBuildTest
    build.RcBuilder().Run(options, [])
  File "S:\src\grit-i18n\grit\tool\build.py", line 149, in Run
    self.Process()
  File "S:\src\grit-i18n\grit\tool\build.py", line 268, in Process
    self.ProcessNode(self.res, output, outfile)
  File "S:\src\grit-i18n\grit\tool\build.py", line 204, in ProcessNode
    formatted = formatter(node, output_node.GetLanguage(), output_dir=base_dir)
  File "S:\src\grit-i18n\grit\format\data_pack.py", line 46, in Format
    id, value = node.GetDataPackPair(lang, UTF8)
  File "S:\src\grit-i18n\grit\node\structure.py", line 168, in GetDataPackPair
    data = self.gatherer.GetData(lang, encoding)
  File "S:\src\grit-i18n\grit\gather\chrome_scaled_image.py", line 136, in GetData
    data = _MoveSpecialChunksToFront(data)
  File "S:\src\grit-i18n\grit\gather\chrome_scaled_image.py", line 53, in _MoveSpecialChunksToFront
    for chunk in _ChunkifyPNG(data):
  File "S:\src\grit-i18n\grit\gather\chrome_scaled_image.py", line 67, in _ChunkifyPNG
    assert data.startswith(_PNG_MAGIC)
AssertionError

I have a JS file containing such comment:
         // stamp template                                                        
         // which includes parsing and applying MDV bindings before being         
         // inserted (to avoid {{}} in attribute values)                          
         // e.g. to prevent <img src="images/{{icon}}"> from generating a 404.

When this file is imported into a HTML file, GRIT treats <img> tag as a real 
HTML tag and generates an dependency for 'src'. That shouldn't happen. Here is 
an unit test demonstrating the error:

  def testTagsInJSComments(self):
    '''Tests that "<img src=...>" tags inside JS comment are ignored.'''

    files = {
      'index.js': '''
      // <img src="example.png">
      ''',
    }

    source_resources = set()
    tmp_dir = util.TempDir(files)
    for filename in files:
      source_resources.add(tmp_dir.GetPath(filename))

    resources = html_inline.GetResourceFilenames(tmp_dir.GetPath('index.js'))
    resources.add(tmp_dir.GetPath('index.js'))
    self.failUnlessEqual(resources, source_resources)
    tmp_dir.CleanUp()