matroska-org / libebml Goto Github PK
View Code? Open in Web Editor NEWa C++ library to parse EBML files
License: GNU Lesser General Public License v2.1
libebml's People
Contributors
![dependabot[bot] avatar](https://avatars.githubusercontent.com/in/29110?v=4 "dependabot[bot]")
Stargazers
Watchers
Forkers
sportzworld martinmenardca vikingmedia mediaarea playdirty 0-wiz-0 mwgoldsmith disigma-network generic-library roche-emmanuel maddthesane stevenhoving evpobr exploresystems chouquette acidburn0zzz darthmoulbyte weinbery minghuadev autilus begasus jiapei100 culhane itmanlee hainam2101 alexandre-janniaux hesnotaveryniceman neheb toutatis2 sizeofvoid randy408 kant wprodell telegie ljani m-dguzman jeruka9 wydengyre sitedata miachel-park sysfce2 ebell495 sathishkumaras ds-innovation robux4 scantist-ossops-m2libebml's Issues
Subtitles no longer visible in VLC after updating libebml
I'm running Arch Linux btw.
Subtitles in VLC can't be seen any longer after updating libebml to the newest (1.4.3) version.
Also mentioned on archlinux.org
Some symbols are not exported when build as Windows DLL
Got problem when building libMatroska with CMake & Visual Studio 2017.
When libEBML is built as DLL, class SafeReadIOCallback::EndOfStreamX is not exported and i have linking errors. It is public, but not marked as EBML_DLL_API. After that fixing that Matroska was built without problems.
Probably EbmlCodeVersion & EbmlCodeDate should be exported too.
Make a special class for deprecated elements
That class would be split from EbmlElement so it doesn't have a write function. So it would be technically impossible to try to write such an element. Maybe an EbmlReadElement.
An EbmlElement could be an EbmlReadElement with the Render methods.
Add profile availability in the semantic
One of the things necessary in mkvalidator is the possibility to know in which Matroska profiles an Element is available or not. This information is available in libmatroska2 but not in libmatroska. To do this we need to stored the allowed (or disabled as in libmatroska2) profiles for each element. This is done at the EBML Semantic level of each element.
1.4.2: build fauls with gcc 11
[ 4%] Building CXX object CMakeFiles/ebml.dir/src/EbmlString.cpp.o
/usr/bin/g++ -Debml_EXPORTS -I/home/tkloczko/rpmbuild/BUILD/libebml-1.4.2/x86_64-redhat-linux-gnu -I/home/tkloczko/rpmbuild/BUILD/libebml-1.4.2 -O2 -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -fdata-sections -ffunction-sections -flto=auto -flto-partition=none -O2 -g -DNDEBUG -fPIC -fvisibility=hidden -fvisibility-inlines-hidden -std=gnu++11 -o CMakeFiles/ebml.dir/src/EbmlString.cpp.o -c /home/tkloczko/rpmbuild/BUILD/libebml-1.4.2/src/EbmlString.cpp
/home/tkloczko/rpmbuild/BUILD/libebml-1.4.2/src/EbmlString.cpp: In member function ‘virtual filepos_t libebml::EbmlString::ReadData(libebml::IOCallback&, libebml::ScopeMode)’:
/home/tkloczko/rpmbuild/BUILD/libebml-1.4.2/src/EbmlString.cpp:147:41: error: ‘numeric_limits’ is not a member of ‘std’
147 | auto Buffer = (GetSize() + 1 < std::numeric_limits<std::size_t>::max()) ? new (std::nothrow) char[GetSize() + 1] : nullptr;
| ^~~~~~~~~~~~~~
/home/tkloczko/rpmbuild/BUILD/libebml-1.4.2/src/EbmlString.cpp:147:67: error: expected primary-expression before ‘>’ token
147 | auto Buffer = (GetSize() + 1 < std::numeric_limits<std::size_t>::max()) ? new (std::nothrow) char[GetSize() + 1] : nullptr;
| ^
/home/tkloczko/rpmbuild/BUILD/libebml-1.4.2/src/EbmlString.cpp:147:70: error: ‘::max’ has not been declared; did you mean ‘std::max’?
147 | auto Buffer = (GetSize() + 1 < std::numeric_limits<std::size_t>::max()) ? new (std::nothrow) char[GetSize() + 1] : nullptr;
| ^~~
| std::max
In file included from /usr/include/c++/11/algorithm:62,
from /home/tkloczko/rpmbuild/BUILD/libebml-1.4.2/ebml/EbmlEndian.h:41,
from /home/tkloczko/rpmbuild/BUILD/libebml-1.4.2/ebml/EbmlTypes.h:38,
from /home/tkloczko/rpmbuild/BUILD/libebml-1.4.2/ebml/EbmlString.h:41,
from /home/tkloczko/rpmbuild/BUILD/libebml-1.4.2/src/EbmlString.cpp:38:
/usr/include/c++/11/bits/stl_algo.h:3467:5: note: ‘std::max’ declared here
3467 | max(initializer_list<_Tp> __l, _Compare __comp)
| ^~~
/home/tkloczko/rpmbuild/BUILD/libebml-1.4.2/src/EbmlString.cpp:157:17: error: type ‘<type error>’ argument given to ‘delete’, expected pointer
157 | delete [] Buffer;
| ^~~~~~
Tell when an element can have an infinite size or not
Since in the latest EBML it's a feature per element.
RenderData does not write non-mandatory elements having a default value
Background
Let's take one of our new flags, KaxFlagHearingImpaired. It isn't mandatory, but it has a default value of 0.
Semantically there are three different states:
- The element is not present, meaning no information is known about the question whether the track is suitable for hearing impaired users.
- The element is present and set to 0: the track is NOT suitable for hearing impaired users.
- The element is present and set to 1: the track IS suitable for hearing impaired users.
Bug
The various Render and RenderData functions in libebml contain a parameter governing whether or not libebml should write elements whose current value equals their default value. Unfortunately this function does NOT take into account whether or not the element is mandatory. They only compare the current value with the default value, both for mandatory and for non-mandatory elements.
So the situation is:
- Element not present in data structure, no matter what
bWithDefaultis set to → element is not present in file. OK. - Element present in data structure,
bWithDefault = true→ element is present in file. OK. - Element present in data structure,
bWithDefault = false→ element is NOT present in file. Not OK, as the file representation is now semantically different than the representation in memory.
Proposed change
EbmlElement::Render() should take into account whether or not an element is mandatory. If it is not mandatory, its value should always be written, no matter what the bWithDefault parameter is set to.
How to reproduce
Here's an example program:
#include "ebml/EbmlHead.h"
#include "ebml/EbmlSubHead.h"
#include "ebml/StdIOCallback.h"
#include "matroska/KaxSegment.h"
#include "matroska/KaxSemantic.h"
#include "matroska/KaxTracks.h"
#include "matroska/KaxTrackEntryData.h"
using namespace libebml;
using namespace libmatroska;
int
main(int,
char **) {
auto write_defaults = false;
StdIOCallback out{"test.mkv", MODE_CREATE};
EbmlHead head;
GetChild<EDocType>(head).SetValue("matroska");
GetChild<EDocTypeVersion>(head).SetValue(4);
GetChild<EDocTypeReadVersion>(head).SetValue(4);
head.Render(out, write_defaults);
KaxSegment segment;
auto &info = GetChild<KaxInfo>(segment);
auto &tracks = GetChild<KaxTracks>(segment);
GetChild<KaxTimecodeScale>(info).SetValue(1000000);
GetChild<KaxMuxingApp>(info).SetValue(L"dummy");
GetChild<KaxWritingApp>(info).SetValue(L"dummy");
tracks.PushElement(*new KaxTrackEntry);
auto &track1 = dynamic_cast<EbmlMaster &>(*tracks[0]);
GetChild<KaxTrackNumber>(track1).SetValue(1);
GetChild<KaxTrackUID>(track1).SetValue(1);
GetChild<KaxTrackType>(track1).SetValue(track_video);
GetChild<KaxCodecID>(track1).SetValue("dummy1");
GetChild<KaxFlagHearingImpaired>(track1).SetValue(1);
tracks.PushElement(*new KaxTrackEntry);
auto &track2 = dynamic_cast<EbmlMaster &>(*tracks[1]);
GetChild<KaxTrackNumber>(track2).SetValue(2);
GetChild<KaxTrackUID>(track2).SetValue(2);
GetChild<KaxTrackType>(track2).SetValue(track_audio);
GetChild<KaxCodecID>(track2).SetValue("dummy2");
GetChild<KaxFlagHearingImpaired>(track2).SetValue(0);
segment.WriteHead(out, 5, write_defaults);
info.Render(out, write_defaults);
tracks.Render(out, write_defaults);
segment.ForceSize(out.getFilePointer() - segment.GetElementPosition() - segment.HeadSize());
segment.OverwriteHead(out);
return 0;
}Compile with current libebml & libmatroska. Run it. Run mkvinfo test.mkv and observe:
[0 mosu@sweet-chili ~/test] ./matroska1 && mkvinfo test.mkv
+ EBML head
|+ Document type version: 4
|+ Document type read version: 4
+ Segment: size 70
|+ Segment information
| + Multiplexing application: dummy
| + Writing application: dummy
|+ Tracks
| + Track
| + Track number: 1 (track ID for mkvmerge & mkvextract: 0)
| + Track UID: 1
| + Track type: video
| + Codec ID: dummy1
| + 'Hearing impaired' flag: 1
| + Track
| + Track number: 2 (track ID for mkvmerge & mkvextract: 1)
| + Track UID: 2
| + Track type: audio
| + Codec ID: dummy2
For both tracks the flag is explicitly set. However, libebml only writes the element if it is set to a value that is NOT its default value as is the case for track 1. For track 2 the element, even though it is present in the data structure to be written to disk, is not written as the value that was set equals the element's default value.
Validate EbmlId on creation/Don't set the size on creation
The EbmlId class is very loose. It's possible to create an EbmlId with a bogus size and use it in a class without anyone noticing. We just assume the values are correct in the spec/custom code.
There are easy tools like gcc/clang clz (https://en.cppreference.com/w/cpp/numeric/countl_zero in C++20) and even in MSVC (https://stackoverflow.com/questions/355967/how-to-use-msvc-intrinsics-to-get-the-equivalent-of-this-gcc-code) that can quickly find the proper size. We should not have to set the size manually at all.
The constructors only allows 4 byte values so we already have constraints on the max ID size. We just need to generate the length automatically. EbmlId should be only created once per class. After that they should be used as const references. So the cost on creation shouldn't be too much.
The spectool generates the ID size based on the size of the id attribute string.
Add simple usage example
I think that it would be a lot easier for beginners like myself if we had a usage example :)
Warnings when building with cl on windows
ebmlcrc32.h(104): warning C4138: '*/' found outside of comment
ebmlsinteger.cpp(119): warning C4146: unary minus operator applied to unsigned type, result still unsigned
Library compatibility version number decreased in 1.3.6
Hi, I'm the maintainer of libebml in MacPorts. I see that for version 1.3.6, you switched to cmake. This has the unfortunate side effect that the compatibility version number of libebml unexpectedly decreased from 5.0.0 to 4.0.0:
$ port installed libebml
The following ports are currently installed:
libebml @1.3.5_0+universal (active)
libebml @1.3.6_0+universal
$ otool -L /opt/local/lib/libebml.dylib
/opt/local/lib/libebml.dylib:
/opt/local/lib/libebml.4.dylib (compatibility version 5.0.0, current version 5.0.0)
/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 307.5.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1238.60.2)
$ sudo port activate libebml @1.3.6_0+universal
---> Deactivating libebml @1.3.5_0+universal
---> Cleaning libebml
---> Activating libebml @1.3.6_0+universal
---> Cleaning libebml
$ otool -L /opt/local/lib/libebml.dylib
/opt/local/lib/libebml.dylib:
/opt/local/lib/libebml.4.dylib (compatibility version 4.0.0, current version 4.0.0)
/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 307.5.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1238.60.2)
This means that any program linked with the old libebml 1.3.5 no longer works, because it thinks the new libebml 1.3.6 is older:
$ /opt/local/bin/mkvtoolnix-gui
dyld: Library not loaded: /opt/local/lib/libebml.4.dylib
Referenced from: /opt/local/bin/mkvtoolnix-gui
Reason: Incompatible library version: mkvtoolnix-gui requires version 5.0.0 or later, but libebml.4.dylib provides version 4.0.0
Abort trap: 6
Library symbols are not visible, because 'EBML_DLL_API' macro get redefined
Building libebml from 1.3.8 release produces unusable library, where symbols are invisible. This is because EBML_DLL_API is defined twice: by CMakeFile.txt and ebml/EbmlConfig.h. The first one (correctly) generates ebml_export.h containing # define EBML_DLL_API __attribute__((visibility("default"))), while the latter discards that definition.
Clang reports:
libebml-1.3.8/ebml/EbmlConfig.h:85:10: warning: 'EBML_DLL_API' macro redefined [-Wmacro-redefined]
# define EBML_DLL_API
^
libebml-1.3.8/ebml_export.h:12:15: note: previous definition is here
# define EBML_DLL_API __attribute__((visibility("default")))
^
1 warning generated.
The following patch fixes the problem:
--- ebml/EbmlConfig.h.orig
+++ ebml/EbmlConfig.h
@@ -81,8 +81,6 @@
# ifdef _MSC_VER
# pragma warning(disable:4786) // length of internal identifiers
# endif // _MSC_VER
-#else
-# define EBML_DLL_API
#endif // WIN32 || _WIN32
Use the CMake endian detection rather than our wild guess
https://cmake.org/cmake/help/v3.0/module/TestBigEndian.html
This works, even with an emscripten toolchain.
handle non-mandatory Empty Elements with default values correctly
In section "6.1. Data Size Format" the EBML RFC states:
If an Empty Element has a default value declared, then the EBML Reader MUST interpret the value of the Empty Element as the default value.
At the moment this simply isn't done. For example, the EbmlUInteger::ReadData function always uses 0 if it's an Empty Element. This works for elements whose default value is 0, of course, but not for others.
BTW: the EbmlUInteger::RenderData function doesn't take default values into account either. To be honest, I prefer it that way as it will be less confusing to not fully spec-compliant EBML readers (such as libebml in its current state).
pkg-config file is broken when CMAKE_INSTALL_{INCLUDE,LIB}DIR is absolute
As per title: CMakeLists.txt has
set(prefix ${CMAKE_INSTALL_PREFIX})
set(exec_prefix "\$\{prefix\}")
set(libdir "\$\{prefix\}/${CMAKE_INSTALL_LIBDIR}")
set(includedir "\$\{prefix\}/${CMAKE_INSTALL_INCLUDEDIR}")
and so can’t handle absolute paths in CMAKE_INSTALL_{INCLUDE,LIB}DIR. This leads to a broken .pc file on NixOS in particular.
Identical to Matroska-Org/libmatroska#62. Similar to open-source-parsers/jsoncpp#1199. See “Concatenating paths when building pkg-config files” for a discussion of the problem in a somewhat different context and a suggested fix (I don’t know CMake myself, sorry).
libebml 1.4.3 breaks MKVToolNix
I'm running Arch Linux btw.
mkvtoolnix-gui/mkvtoolnix-cli (ver. 70.0) won't work anymore after updating libebml to the newest (1.4.3) version.
$ ... symbol lookup error: mkvtoolnix-gui: undefined symbol ...
Also mentioned on archlinux.org
use standard types instead of custom ones
Let's have an issue focussed on the upcoming change to using standard types (e.g. std::uint32_t) instead of our custom ones (e.g. uint32).
There's an older PR, #86, which does a lot of what I'd like to see, but it's somewhat controversial. The points I've seen there & in other places are:
- switch all
(u)int(8|16|32|64)tostd::(u)int(8|16|32|64)_t(probably uncontroversial) - switch
size_ttostd::size_t - keep the old
(u)int(8|16|32|64)types around for backwards compatibility but derive them from the official types instead of doing our own compiler- & OS-based#ifdefthing - the header
libebml_t.his suposed to be usable from C applications as well, leading to a couple of follow-up questions:- are there any known C applications that actually use it?
- do we want to keep it around? If so, we must derive our backwards-compatible types from the C types, not the C++ ones (meaning…
stdint.h) - do we want to remove it? → somewhat serious API break if there are actual users outside of our libraries.
Anything else I've missed?
Question about CVE-2015-8789, Backports for Debian
Hello,
I'm currently in the process to backport the patches for CVE-2015-8789 to Debian's stable and oldstable releases. Debian ships version 1.2.2 in oldstable and version 1.3.0 in stable. I think both versions are affected by CVE-2015-8789 but I'm not totally sure that the fix from
can be applied as is. Could you take a look at the patches and tell me whether they are correct or not?
Patch for 1.2.2: ftp://46.182.19.245/CVE-2015-8789_wheezy.patch
Patch for 1.3.0: ftp://46.182.19.245/CVE-2015-8789.jessie.patch
Thanks
MemIOCallback::read has a integer overflow bug
function MemIOCallback::read at line 70 has an integer overflow:
POC code as below:
#include "ebml/MemIOCallback.h"
using namespace libebml;
int main() {
char buff[128] = {};
MemIOCallback memoryBuffer;
memoryBuffer.write(buff, 124);
char outBuff[128];
memoryBuffer.read(outBuff, 0xfffffffffffffff0);
return 0;
}
terminal output is:
Process finished with exit code 139 (interrupted by signal 11: SIGSEGV)
Setting EbmlString from an EbmlMaster to default value wont work
I created my own EbmlHead class subclassing from EbmlMaster and did this:
DEFINE_EBML_UINTEGER_DEF(EVersion, 0x4286, 2, OrwellEbmlHead, "EBMLVersion", 1)
DEFINE_EBML_UINTEGER_DEF(EReadVersion, 0x42F7, 2, OrwellEbmlHead, "EBMLReadVersion", 1)
DEFINE_EBML_UINTEGER_DEF(EMaxIdLength, 0x42F2, 2, OrwellEbmlHead, "EBMLMaxIdLength", 4)
DEFINE_EBML_UINTEGER_DEF(EMaxSizeLength, 0x42F3, 2, OrwellEbmlHead, "EBMLMaxSizeLength", 8)
DEFINE_EBML_STRING_DEF(EDocType, 0x4282, 2, OrwellEbmlHead, "EBMLDocType", "orwell")
DEFINE_EBML_UINTEGER_DEF(EDocTypeVersion, 0x4287, 2, OrwellEbmlHead, "EBMLDocTypeVersion", 1)
DEFINE_EBML_UINTEGER_DEF(EDocTypeReadVersion, 0x4285, 2, OrwellEbmlHead, "EBMLDocTypeReadVersion", 1)
Then if I try to do
EbmlHead FileHead;
*static_cast<EbmlUInteger *>(&GetChild<EVersion>(FileHead)) = 1;
*static_cast<EbmlUInteger *>(&GetChild<EReadVersion>(FileHead)) = 1;
*static_cast<EbmlUInteger *>(&GetChild<EMaxIdLength>(FileHead)) = 4;
*static_cast<EbmlUInteger *>(&GetChild<EMaxSizeLength>(FileHead)) = 8;
*static_cast<EbmlString *>(&GetChild<EDocType>(FileHead)) = "something";
*static_cast<EbmlUInteger *>(&GetChild<EDocTypeVersion>(FileHead)) = 1;
*static_cast<EbmlUInteger *>(&GetChild<EDocTypeReadVersion>(FileHead)) = 1;
FileHead.RenderData(stdIOCallback, true);
I get the expected binary file. But if
something
in
*static_cast<EbmlString *>(&GetChild<EDocType>(FileHead)) = "something";
is renamed to
orwell
which is the default value defined for EDocType as you can see, then I get a file with 0 bytes. Shouldn't it be a bug?
I also tried to simply do
EbmlHead FileHead;
FileHead.RenderData(stdIOCallback, true);
assuming it would write to the file since I already gave default values for every class, but it won't work, I get a file with 0 bytes
Fix name of EbmlElement::DefaultISset()
This is just a reminder/TODO for when we break the API/ABI the next time: the function name is wrong. The member variable, on the other hand, is named DefaultIsSet and should be renamed in order not to clash with the then-fixed function name.
provide toggle to cause element to be written even if set to its default value
We need a toggle for an element to signal that it should be written regardless of whether its value is set to its default value. Why? For compatibility reasons with existing players out there.
At the moment there are two ways to write elements: call a master's Render() method the true or false for the write even if set to default value argument. Using true here causes all elements set to their defaults to be written, and a LOT of players fall flat on their face when they encounter a lot of those elements as they're very seldomly used if ever. Using false causes none of them to be written. However, sometimes the application wants more fine-grained control for compatibility purposes with bad software/hardware players out there and only write that one element.
Therefore a toggle at the element level would be nice so that I can still call Render() on a top-level master (for me it's mostly the Tracks element) with the write even if default value parameter set to false, but that one instance of that element will still be written regardless of its value.
At the moment what I do is deriving a class from the corresponding Matroska class & forcing it not to have a default value. This is ugly and will likely not work well, depending on how #154 will be implemented.
Won't break the API: only new functions & new data fields will be introduced; will likely break the ABI due to the same reason, though (likely change in member variable layout).
Add support for building against system utf8cpp
libebml bundles utf8cpp in src/lib/utf8cpp. Please add support for building against system copy (headers in /usr/include/utf8).
ebml spec missing mandate for element ids to be variable length encoded
Perhaps I'm missing it, but I don't think the EBML spec is strong enough to say that Element IDs must use variable length encoding. The closest is where it says "Element ID coded with an UTF-8 like system", but it's missing a declaration of mandate or "MUST".
Drop old GCC versions support
Anything before 4.3 is probably not working anymore, nor used much.
So we don't forget after #82 is merged (or before).
EbmlElement::FindNextID doesn't support global elements
The programs in the MKVToolNix package use the FindNextId function for finding the first elements: the EBML Head & the Matroska Segment.
However, there may be other elements present: EBML Void & EBML CRC32 1. EBML Void is something that mkvpropedit used to create as a top-level element in certain situations. Unfortunately, there are quite a lot of programs that simply don't support EBML Void elements in the top-level position in the EBML Body, including libEBML & ffmpeg.
In the case of libEBML the following happens:
- A call to
ebml_stream->FindNextID(EBML_INFO(EbmlHead), 0xFFFFFFFFL)is run first.EbmlStream::FindNextIDdispatches toEbmlElement::FindNextIDunder the hood. - If it succeeds, great: a call to
ebml_stream->FindNextID(EBML_INFO(KaxSegment), 0xFFFFFFFFFFFFFFFFLL)is run next. - The second call finds an EBML Void. As it doesn't support global elements, it'll return an instance of
EbmlDummyinstead ofEbmlVoid.
Note that the related function FindNextElement does support global elements. Unfortunately I cannot use FindNextElement for the KaxSegment as FindNextElement attempts to read the whole element into memory, which is obviously not going to work with big Matroska files.
Here is a test file created with a slightly modified mkvmerge: there's a small EBML Void element between the EBML Head & the Matroska Segment. The current mkvinfo output shows how this fails:
[0 mosu@sweet-chili ~/prog/video/data] mkvinfo v.mkv
+ EBML head
|+ EBML version: 1
|+ EBML read version: 1
|+ Maximum EBML ID length: 4
|+ Maximum EBML size length: 8
|+ Document type: matroska
|+ Document type version: 4
|+ Document type read version: 1
+ (Known element, but invalid at this position: EBML void; ID: 0xec size: 6)
+ Segment: size 5760
|+ Seek head (subentries will be skipped)
…
Currently mkvmerge doesn't support such files, but I'll fix that soonish. ffmpeg also fails, but that isn't our problem, of course.
Footnotes
Make the default value part of the semantic
The default value is tight to the semantic of an element. It doesn't need to be stored in each instance of the class.
The possible default values can only be integer/string/float values, maybe binary might be possible although we don't support that in Matroska (for now). We can use a union in the semantic to hold the default value or make each semantic classes with a typed default value. The an EbmlElement can only have a semantic of that type. We should be able to do that with templates and keeps constexpr where possible.
Add a mode to allow reading dummy elements but only when properly positioned
The current dummy mode can read a lot of things in damaged file. Most people should allow reading dummies for forward compatibility but on the other hand they probably want to read only valid data, not damaged data recovered in unlikely places (not elements boundaries).
Ebml[Unicode]String::ReadData heap overflow bug on 32bit builds
--[ 1. Summary
An extremely exploitable heap overflow bug exists in the implementation
of EbmlString::ReadData and EbmlUnicodeString::ReadData in libebml. This
bug is reachable from the current stable release of vlc.
--[ 2. Discussion
The issue exists in the calculation of the required buffer size to store
the string. The following calculation is performed:
310 auto Buffer = new (std::nothrow) char[GetSize()+1];
...
315 input.readFully(Buffer, GetSize());
The value returned from GetSize is guaranteed to be an unisigned 64 bug
number, and due to the way in which intetgers are stored and parsed in
Ebml will only use the lowest 48 bits. This guarantees that the integer
cannot overflow on 64but builds.
However, on 32bit builds, the value is implicitly cast to a size_t in
the call to new, meaning that the truncated length can be significantly
shorter than the amount of data to be copied. For example, if the length
of the string element is claimed to be 0xffffffff, the resultant
allocation will be 0x100000000. The cast to a 32bit size_t drops the top
1 bit, meaning an array of size zero is allocated.
In the event that the string element is placed maliciously at the end of
the file to be parsed, an extremely exploitable controlled heap overflow
can occur.
--[ 3. Resolution
The fix for this bug is relatively straightforward, a check must be
added to ensure that the value of GetData() + 1 is less than SIZE_MAX to
ensure that it will not be truncated in the call to new.
--[ 4. Proof of Concept
The following proof of concept file shows the behaviour in the latest
(at time of writing) version of vlc on Ubuntu 10.10.
$ sudo apt install vlc:i386 gdb
$ wget https://kryc.uk/libebml-poc.mkv
$ gdb -q vlc
$$ r libebml-poc.mkv
Make `Data` private in EbmlBinary
Add protected accessors if needed.
This will make the use of Data/Size safer, allowing asserts to detect mismatches early.
Add CMake project
Casting 0xFFFFFFFF to handle generates a warning in MSVC
Background
In src/platform/win32/WinIOCallback.cpp, a file handle is created using CreateFileA. We check that the result is valid before we try to use the handle as such:
mFile = CreateFileA(Path, AccessMode, ShareMode, NULL, Disposition, dwFlags, NULL);
if ((mFile == INVALID_HANDLE_VALUE) || (mFile == (HANDLE)0xffffffff)) {
Bug
MSVC does not like the casting of 0xFFFFFFFF to type handle. This generates the warning
warning C4312: 'type cast': conversion from 'unsigned int' to 'HANDLE' of greater size
Open question: Is this also a bug?
We check that the handle is valid by comparing it to (I assume) -1, written as 0xFFFFFFFF. According to the Microsoft documentation for CreateFileA, the two return values are either a valid handle, or INVALID_HANDLE_VALUE. Why do we also check for 0xFFFFFFFF?
Proposed fix
If the 0xFFFFFFFF check is unnecessary, we can simply remove that check. If it is necessary, we'll need to update the value so the cast can be done safely/correctly.
Build process
Can you please provide some information on the README on how to build this lib?
New soname needed with 1.4.3 ?
Debian unstable amd64
After ebml 1.4.3 installation all applications linked with libebml return
symbol lookup error: mkvextract: undefined symbol: _ZN7libebml8EbmlDate14UnixEpochDelayE
migrate ebml specs from sourceforge to github pages
I propose moving the EBML specs from http://ebml.sourceforge.net to a 'github pages' in the libebml repo (potentially to be at http://matroska-org.github.io/libebml). The EBML homepage has various issues (broken links, typos) and it would be easier to fix these issues if we could send pull requests through github and have the change process documented.
I suggest moving the pages from http://ebml.sourceforge.net to github pages as-is and adding a notice to the sourceforge pages to say that they have moved to the new repo.
EbmlId doesn't have a copy assignment operator anymore
I'm trying to adjust MKVToolNix to the current libEBML/libMatroska changes. The one with the constexpre changes causes the most damage:
src/common/kax_analyzer.cpp:1087:30: fatal error: object of type 'libebml::EbmlId' cannot be assigned because its copy assignment operator is implicitly deleted
m_data[data_idx]->m_id = EbmlId(*e);
^
/home/mosu/prog/video/mkvtoolnix/lib/libebml/ebml/EbmlId.h:53:25: note: copy assignment operator of 'EbmlId' is implicitly deleted because field 'Value' is of const-qualified type 'const std::uint32_t' (aka 'const unsigned int')
const std::uint32_t Value;
Meaning you'll have to define your own copy assignment operator & copy constructor & deal with this.
Incorrect FSF address
RPMLint reports this issue:
W: incorrect-fsf-address /usr/include/ebml/MemReadIOCallback.h
The Free Software Foundation address in this file seems to be outdated or
misspelled. Ask upstream to update the address, or if this is a license file,
possibly the entire file with a new copy available from the FSF.
Basically the FSF Address changed from 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA to 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
Deprecate CodedSizeLengthSigned/CodedValueLengthSigned/ReadCodedSizeSignedValue
- CodedSizeLengthSigned
- CodedValueLengthSigned
- ReadCodedSizeSignedValue
This code doesn't correspond to anything used in EBML. It's for libmatroska custom signed-integers. It should be moved there.
libebml 1.3.6 fails to build on RHEL7 with gcc 4.8.5
I'm getting the following error when building libebml-1.3.6 on RHEL 7.5 using gcc-4.8.5 from the standard repositories:
[ 4%] Building CXX object CMakeFiles/ebml.dir/src/EbmlSInteger.cpp.o
/bin/c++ -DEBML_DLL -DEBML_DLL_EXPORT -I/builddir/build/BUILD/libebml-1.3.6 -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -fPIC -fvisibility-inlines-hidden -o CMakeFiles/ebml.dir/src/EbmlSInteger.cpp.o -c /builddir/build/BUILD/libebml-1.3.6/src/EbmlSInteger.cpp
/builddir/build/BUILD/libebml-1.3.6/src/EbmlSInteger.cpp: In function 'int64_t {anonymous}::ToSigned(uint64_t)':
/builddir/build/BUILD/libebml-1.3.6/src/EbmlSInteger.cpp:48:12: error: 'numeric_limits' is not a member of 'std'
if (u <= std::numeric_limits<int64_t>::max())
^
/builddir/build/BUILD/libebml-1.3.6/src/EbmlSInteger.cpp:48:39: error: expected primary-expression before '>' token
if (u <= std::numeric_limits<int64_t>::max())
^
/builddir/build/BUILD/libebml-1.3.6/src/EbmlSInteger.cpp:48:40: error: '::max' has not been declared
if (u <= std::numeric_limits<int64_t>::max())
^
/builddir/build/BUILD/libebml-1.3.6/src/EbmlSInteger.cpp:48:40: note: suggested alternative:
In file included from /usr/include/c++/4.8.2/algorithm:61:0,
from /builddir/build/BUILD/libebml-1.3.6/ebml/EbmlEndian.h:41,
from /builddir/build/BUILD/libebml-1.3.6/ebml/EbmlTypes.h:38,
from /builddir/build/BUILD/libebml-1.3.6/ebml/EbmlSInteger.h:43,
from /builddir/build/BUILD/libebml-1.3.6/src/EbmlSInteger.cpp:38:
/usr/include/c++/4.8.2/bits/stl_algobase.h:260:5: note: 'std::max'
max(const _Tp& __a, const _Tp& __b, _Compare __comp)
^
/builddir/build/BUILD/libebml-1.3.6/src/EbmlSInteger.cpp:51:35: error: 'numeric_limits' is not a member of 'std'
return static_cast<int64_t>(u - std::numeric_limits<int64_t>::min()) + std::numeric_limits<int64_t>::min();
^
/builddir/build/BUILD/libebml-1.3.6/src/EbmlSInteger.cpp:51:62: error: expected primary-expression before '>' token
return static_cast<int64_t>(u - std::numeric_limits<int64_t>::min()) + std::numeric_limits<int64_t>::min();
^
/builddir/build/BUILD/libebml-1.3.6/src/EbmlSInteger.cpp:51:63: error: '::min' has not been declared
return static_cast<int64_t>(u - std::numeric_limits<int64_t>::min()) + std::numeric_limits<int64_t>::min();
^
/builddir/build/BUILD/libebml-1.3.6/src/EbmlSInteger.cpp:51:63: note: suggested alternative:
In file included from /usr/include/c++/4.8.2/algorithm:61:0,
from /builddir/build/BUILD/libebml-1.3.6/ebml/EbmlEndian.h:41,
from /builddir/build/BUILD/libebml-1.3.6/ebml/EbmlTypes.h:38,
from /builddir/build/BUILD/libebml-1.3.6/ebml/EbmlSInteger.h:43,
from /builddir/build/BUILD/libebml-1.3.6/src/EbmlSInteger.cpp:38:
/usr/include/c++/4.8.2/bits/stl_algobase.h:239:5: note: 'std::min'
min(const _Tp& __a, const _Tp& __b, _Compare __comp)
^
/builddir/build/BUILD/libebml-1.3.6/src/EbmlSInteger.cpp:51:74: error: 'numeric_limits' is not a member of 'std'
return static_cast<int64_t>(u - std::numeric_limits<int64_t>::min()) + std::numeric_limits<int64_t>::min();
^
/builddir/build/BUILD/libebml-1.3.6/src/EbmlSInteger.cpp:51:101: error: expected primary-expression before '>' token
return static_cast<int64_t>(u - std::numeric_limits<int64_t>::min()) + std::numeric_limits<int64_t>::min();
^
/builddir/build/BUILD/libebml-1.3.6/src/EbmlSInteger.cpp:51:102: error: '::min' has not been declared
return static_cast<int64_t>(u - std::numeric_limits<int64_t>::min()) + std::numeric_limits<int64_t>::min();
^
/builddir/build/BUILD/libebml-1.3.6/src/EbmlSInteger.cpp:51:102: note: suggested alternative:
In file included from /usr/include/c++/4.8.2/algorithm:61:0,
from /builddir/build/BUILD/libebml-1.3.6/ebml/EbmlEndian.h:41,
from /builddir/build/BUILD/libebml-1.3.6/ebml/EbmlTypes.h:38,
from /builddir/build/BUILD/libebml-1.3.6/ebml/EbmlSInteger.h:43,
from /builddir/build/BUILD/libebml-1.3.6/src/EbmlSInteger.cpp:38:
/usr/include/c++/4.8.2/bits/stl_algobase.h:239:5: note: 'std::min'
min(const _Tp& __a, const _Tp& __b, _Compare __comp)
^
/builddir/build/BUILD/libebml-1.3.6/src/EbmlSInteger.cpp: In member function 'virtual filepos_t libebml::EbmlSInteger::ReadData(libebml::IOCallback&, libebml::ScopeMode)':
/builddir/build/BUILD/libebml-1.3.6/src/EbmlSInteger.cpp:147:45: error: 'numeric_limits' is not a member of 'std'
uint64_t TempValue = Buffer[0] & 0x80 ? std::numeric_limits<uint64_t>::max() : 0;
^
/builddir/build/BUILD/libebml-1.3.6/src/EbmlSInteger.cpp:147:73: error: expected primary-expression before '>' token
uint64_t TempValue = Buffer[0] & 0x80 ? std::numeric_limits<uint64_t>::max() : 0;
^
/builddir/build/BUILD/libebml-1.3.6/src/EbmlSInteger.cpp:147:74: error: '::max' has not been declared
uint64_t TempValue = Buffer[0] & 0x80 ? std::numeric_limits<uint64_t>::max() : 0;
^
/builddir/build/BUILD/libebml-1.3.6/src/EbmlSInteger.cpp:147:74: note: suggested alternative:
In file included from /usr/include/c++/4.8.2/algorithm:61:0,
from /builddir/build/BUILD/libebml-1.3.6/ebml/EbmlEndian.h:41,
from /builddir/build/BUILD/libebml-1.3.6/ebml/EbmlTypes.h:38,
from /builddir/build/BUILD/libebml-1.3.6/ebml/EbmlSInteger.h:43,
from /builddir/build/BUILD/libebml-1.3.6/src/EbmlSInteger.cpp:38:
/usr/include/c++/4.8.2/bits/stl_algobase.h:260:5: note: 'std::max'
max(const _Tp& __a, const _Tp& __b, _Compare __comp)
^
/builddir/build/BUILD/libebml-1.3.6/src/EbmlSInteger.cpp: In function 'int64_t {anonymous}::ToSigned(uint64_t)':
/builddir/build/BUILD/libebml-1.3.6/src/EbmlSInteger.cpp:52:1: warning: control reaches end of non-void function [-Wreturn-type]
}
^
make[2]: *** [CMakeFiles/ebml.dir/src/EbmlSInteger.cpp.o] Error 1
The fix is to add -std=c++11 to compile flags, but this should be done automatically by cmake.
Check the malloc() return values
(doh!)
Document C++11 requirement
Per #65 C++11 is now required but README.md still says
libebml [...] requires only a C++ compiler
Change the I/O uint32/size_t combination
They should be the same type.
We should also document than returning less than requested size means EOF.
Does not compile under MSVC
When linking against libmatroska, linking errors occur:
src_KaxTracks.cpp.obj : error LNK2001: unresolved external symbol "protected: static unsigned __int64 const libebml::EbmlDate::UnixEpochDelay" (?UnixEpochDelay@EbmlDate@libebml@@1_KB)
src_KaxInfoData.cpp.obj : error LNK2001: unresolved external symbol "protected: static unsigned __int64 const libebml::EbmlDate::UnixEpochDelay" (?UnixEpochDelay@EbmlDate@libebml@@1_KB)
src_KaxSeekHead.cpp.obj : error LNK2001: unresolved external symbol "protected: static unsigned __int64 const libebml::EbmlDate::UnixEpochDelay" (?UnixEpochDelay@EbmlDate@libebml@@1_KB)
src_KaxSegment.cpp.obj : error LNK2001: unresolved external symbol "protected: static unsigned __int64 const libebml::EbmlDate::UnixEpochDelay" (?UnixEpochDelay@EbmlDate@libebml@@1_KB)
src_KaxSemantic.cpp.obj : error LNK2001: unresolved external symbol "protected: static unsigned __int64 const libebml::EbmlDate::UnixEpochDelay" (?UnixEpochDelay@EbmlDate@libebml@@1_KB)
src_KaxCluster.cpp.obj : error LNK2001: unresolved external symbol "protected: static unsigned __int64 const libebml::EbmlDate::UnixEpochDelay" (?UnixEpochDelay@EbmlDate@libebml@@1_KB)
src_KaxContexts.cpp.obj : error LNK2001: unresolved external symbol "protected: static unsigned __int64 const libebml::EbmlDate::UnixEpochDelay" (?UnixEpochDelay@EbmlDate@libebml@@1_KB)
src_KaxCues.cpp.obj : error LNK2001: unresolved external symbol "protected: static unsigned __int64 const libebml::EbmlDate::UnixEpochDelay" (?UnixEpochDelay@EbmlDate@libebml@@1_KB)
src_KaxCuesData.cpp.obj : error LNK2001: unresolved external symbol "protected: static unsigned __int64 const libebml::EbmlDate::UnixEpochDelay" (?UnixEpochDelay@EbmlDate@libebml@@1_KB)
src_KaxAttached.cpp.obj : error LNK2001: unresolved external symbol "protected: static unsigned __int64 const libebml::EbmlDate::UnixEpochDelay" (?UnixEpochDelay@EbmlDate@libebml@@1_KB)
src_KaxAttachments.cpp.obj : error LNK2001: unresolved external symbol "protected: static unsigned __int64 const libebml::EbmlDate::UnixEpochDelay" (?UnixEpochDelay@EbmlDate@libebml@@1_KB)
src_KaxBlock.cpp.obj : error LNK2001: unresolved external symbol "protected: static unsigned __int64 const libebml::EbmlDate::UnixEpochDelay" (?UnixEpochDelay@EbmlDate@libebml@@1_KB)
src_KaxBlockData.cpp.obj : error LNK2001: unresolved external symbol "protected: static unsigned __int64 const libebml::EbmlDate::UnixEpochDelay" (?UnixEpochDelay@EbmlDate@libebml@@1_KB)
src_KaxTracks.cpp.obj : error LNK2001: unresolved external symbol "public: static class libebml::EbmlCallbacks const libebml::EbmlCrc32::ClassInfos" (?ClassInfos@EbmlCrc32@libebml@@2VEbmlCallbacks@2@B)
src_KaxInfoData.cpp.obj : error LNK2001: unresolved external symbol "public: static class libebml::EbmlCallbacks const libebml::EbmlCrc32::ClassInfos" (?ClassInfos@EbmlCrc32@libebml@@2VEbmlCallbacks@2@B)
src_KaxSeekHead.cpp.obj : error LNK2001: unresolved external symbol "public: static class libebml::EbmlCallbacks const libebml::EbmlCrc32::ClassInfos" (?ClassInfos@EbmlCrc32@libebml@@2VEbmlCallbacks@2@B)
src_KaxSegment.cpp.obj : error LNK2001: unresolved external symbol "public: static class libebml::EbmlCallbacks const libebml::EbmlCrc32::ClassInfos" (?ClassInfos@EbmlCrc32@libebml@@2VEbmlCallbacks@2@B)
src_KaxSemantic.cpp.obj : error LNK2001: unresolved external symbol "public: static class libebml::EbmlCallbacks const libebml::EbmlCrc32::ClassInfos" (?ClassInfos@EbmlCrc32@libebml@@2VEbmlCallbacks@2@B)
src_KaxCluster.cpp.obj : error LNK2001: unresolved external symbol "public: static class libebml::EbmlCallbacks const libebml::EbmlCrc32::ClassInfos" (?ClassInfos@EbmlCrc32@libebml@@2VEbmlCallbacks@2@B)
src_KaxContexts.cpp.obj : error LNK2001: unresolved external symbol "public: static class libebml::EbmlCallbacks const libebml::EbmlCrc32::ClassInfos" (?ClassInfos@EbmlCrc32@libebml@@2VEbmlCallbacks@2@B)
src_KaxCues.cpp.obj : error LNK2001: unresolved external symbol "public: static class libebml::EbmlCallbacks const libebml::EbmlCrc32::ClassInfos" (?ClassInfos@EbmlCrc32@libebml@@2VEbmlCallbacks@2@B)
src_KaxCuesData.cpp.obj : error LNK2001: unresolved external symbol "public: static class libebml::EbmlCallbacks const libebml::EbmlCrc32::ClassInfos" (?ClassInfos@EbmlCrc32@libebml@@2VEbmlCallbacks@2@B)
src_KaxAttached.cpp.obj : error LNK2001: unresolved external symbol "public: static class libebml::EbmlCallbacks const libebml::EbmlCrc32::ClassInfos" (?ClassInfos@EbmlCrc32@libebml@@2VEbmlCallbacks@2@B)
src_KaxAttachments.cpp.obj : error LNK2001: unresolved external symbol "public: static class libebml::EbmlCallbacks const libebml::EbmlCrc32::ClassInfos" (?ClassInfos@EbmlCrc32@libebml@@2VEbmlCallbacks@2@B)
src_KaxBlock.cpp.obj : error LNK2001: unresolved external symbol "public: static class libebml::EbmlCallbacks const libebml::EbmlCrc32::ClassInfos" (?ClassInfos@EbmlCrc32@libebml@@2VEbmlCallbacks@2@B)
src_KaxBlockData.cpp.obj : error LNK2001: unresolved external symbol "public: static class libebml::EbmlCallbacks const libebml::EbmlCrc32::ClassInfos" (?ClassInfos@EbmlCrc32@libebml@@2VEbmlCallbacks@2@B)
src_KaxContexts.cpp.obj : error LNK2001: unresolved external symbol "public: static class libebml::EbmlCallbacks const libebml::EbmlHead::ClassInfos" (?ClassInfos@EbmlHead@libebml@@2VEbmlCallbacks@2@B)
src_KaxSegment.cpp.obj : error LNK2001: unresolved external symbol "public: static class libebml::EbmlCallbacks const libebml::EbmlHead::ClassInfos" (?ClassInfos@EbmlHead@libebml@@2VEbmlCallbacks@2@B)
subprojects\libmatroska-release-1.6.3\libmatroska.dll : fatal error LNK1120: 3 unresolved externals
for UnixEpochDelay, defining it directly fixes the error. I'm not sure about the others.
Keep the information about elements that can be infinite
In Matroska only certain elements are allowed to have infinite size. SetSizeInfinite() should fail for elements that are not allowed to be infinite.
CVE-2016-1514
I've just got a report that libebml is vulnerable to CVE-2016-1514, but looking at the code in question (https://github.com/Matroska-Org/libebml/blob/master/src/EbmlUnicodeString.cpp#L156), it doesn't seem to be vulnerable. Please confirm which commit fixed it.
Add a way to tell if reading an element returned NULL because it reached the end parent or EOF
Adding an isEof() callback to IOCallback may be the simplest way.
add copyright and license info to the ebml spec
It's unclear if the spec is under the same license and copyright as libEBML. I suggest clarifying this on the spec page: http://matroska-org.github.io/libebml/specs.html
Any plans to make new release?
I think that it would be good to flush currently committed changes and make new release :)
OSS-Fuzz integration
OSS-Fuzz can be used to automate fuzz testing for this project. Once integrated issues will be automatically filed on bugs.chromium.org when bugs are found and closed when they're fixed.
Integration consists of one or more fuzz targets and a build script that runs in an oss-fuzz environment (those could all go under test/).
I intend to integrate this project, which would involve 2-3 PRs.
Are the maintainers interested in OSS-Fuzz integration?
minimum supported C++ version
We've never talked about the minimum C++ language version we'd like to support in libEBML and libMatroska. Recent pull requests regarding visibility and the default keyword make such a determination necessary.
Personally I'm rather aggressive in MKVToolNix where I require a lot of C++17 features already. I don't want to be that aggressive with our two libraries, though.
For me C++11 offers so many compelling features such as lambdas, range-based for loops, the default keyword for constructions & destructors and many more, that I'd like to require at least C++11, maybe even C++14. C++11 support in compilers is ten years old by now. It really is available everywhere.
For the following refer to https://en.cppreference.com/w/cpp/compiler_support
C++14 becomes somewhat more difficult. If we look at currently supported Linux versions, the oldest, widely supported version is RedHat/CentOS 7 where gcc 4.8.5 is the version bundled with the OS. That one only supports C++11, not 14. However, newer compiler versions up to and including gcc 9 (!) are readily available in the devtoolset packages from the widely-used EPEL repository. (That's what I use for compiling MKVToolNix on CentOS 7.)
The oldest supported Ubuntu is currently 16.04 which comes with gcc 5.3.1 which does support C++14 fully.
C++14 being smaller, it doesn't offer that many compelling new features (though I definitely like e.g. the number separator: 1'000'000'000).
So — opinions? C++11 at least, maybe C++14?
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.