Git Product home page Git Product logo

aqtoolkit's Introduction

AQ Toolkit

Overview

This toolkit is comprised of various bits of utility code I’ve written for odd projects here & there, including iPhone projects such as Outpost. Most of these classes, with the possible exception of those which came from iPhone projects, are designed to be usable with both garbage collection and manual memory management.

If you have feature requests, or wish to file bugs, you can do so via Lighthouse

Projects Using AQToolkit

Outpost iTunes link - HTTPMessage, StreamingXMLParser, Compression, Extensions, LowMemoryDownload, TempFiles

Atomium: Periodic Table iTunes link - CommonCrypto

Contents

ASLogger

This implements an Objective-C interface to the Apple System Logger C API.

  • ASLogger — An interface to a specific logger client connection. Also defines a default logger. Use this class and its associated macros to send messages to the logger, and to set its attributes.
  • ASLMessage — A wrapper for a particular message. For plain messages, you’ll likely never need this class, but if you want to use template messages or set advanced/custom properties on a message, this class provides the interface.
  • ASLQuery — Used to construct a query. Actually a subclass of ASLMessage with a single new function to allow you to explicitly specify a comparison operation.
  • ASLResponse — Encapsulates the result of querying a log using ASLQuery. Use while ((msg = [response next]) != nil) { ... } to loop through the returned messages.

ChunkedXMLParser

This is a slight optimization to Apple’s NSXMLParser class. This version includes an NSData subclass which returns blobs of data by reading from an NSInputStream, and an NSXMLParser subclass which overrides -parse to call xmlDataChunk() multiple times, passing a discrete chunk each time.

To initialize an AQChunkedXMLData, you pass it an NSInputStream instance. You then use the resulting object to initialize your AQChunkedXMLParser, which you use as though it were a regular NSXMLParser instance.

CommonCrypto

This implements a category on NSData for performing digest, HMAC, and cryptographic operations on the contents of the receiver, all of which are based on the CommonCrypto C API.

For an example of its usage, look at example.m. This file implements a command-line utility which can encrypt/decrypt data to/from files or standard input/output.

Extensions

A collection of small categories on standard FoundationKit classes.

  • NSData+Base64 — Implements Base64 string/data conversions. This code is based on the implementation from the Omni Group’s OmniFoundation framework, located here. This is because the alternative is to use OpenSSL, which requires linking against that library. That seemed a bit much just to get Base64 encoding support.
  • NSError+CFStreamError — A category on NSError which attempts to convert the CFStreamError constructs still used by some CFNetwork code to proper NSError instances, using the newer CFErrorRef constants defined by that where possible.
  • NSObject+Properties — A set of C functions and wrapping class/instance methods used to get information on the Objective-C 2.0 properties implemented by a class. It includes getting information on all attributes of the properties, as well as obtaining their types.
  • NSString+PropertyKVC — A utility for the Property support above.

FSEventsWrapper

A project implementing a simple Objective-C wrapper around the FSEvents C API. It’s based on delegation right now, although may have something more generic and event-based later on.

You use it by creating an instance of AQFSEventStream, passing a set of paths to watch. You can then schedule it with runloops, start, and stop the stream. Events which happen on the stream are passed to the delegate, which should conform to the AQFSEventStreamDelegate protocol. Only one method is required, to get basic folder-updated events. The optional methods in the protocol are called to notify the delegate of special types of events; these may not be interesting, hence they are optional.

A sample command-line application is included in Monitor.m.

HTTPMessage

The classes in this folder provide wrappers around the CFHTTPMessageRef C API. Included are wrappers for CFHTTPMessageRef itself as well as CFHTTPAuthenticationRef, used for handling authentication responses. They fully support garbage-collection or managed memory.

A HTTPMessage can represent a request or a response; which one must be specified at creation. Most properties are specified indirectly through the creation function arguments, but a couple can be set directly, such as the body data. HTTP header fields for requests can be set using -setValue:forHeaderField:.

Access to the underlying CF object is not provided, except as a convenience between the two classes themselves.

If you’re thinking that CFNetwork is in different places on the Mac and the iPhone, don’t worry, that’s already covered.

LowLevelFSEvents

This section implements an event-based reader for the low-level FSEvents device, as used by Spotlight and fseventsd. The code is based on the code & information from Mac OS X Internals by Amit Singh. The events themselves are read using a background thread, and are immediately passed on to the main thread, to avoid the fsevents queue from filling up in the kernel, causing dropped events.

The fsevents.h header comes from xnu (the OS X kernel) and is reproduced verbatim.

  • FSEventManager — this class implements the top-level interface to the system, and has a very sparse public API. It maintains a singleton object, accessible via +sharedManager, and has a class method +shutdown used to ensure it shuts down cleanly when an application terminates. You set a single event handler object using the handler property. The event handler should conform to the FSEventHandler protocol.
  • FSEvent — another sparse class, an instance of FSEvent is passed to the -handleEvent: method of the event handler. From this you can get the event code, the process ID of the process which caused the event, a timestamp indicating when it happened, and a dictionary of arguments (which are event-specific). A loggable string value for an event code can be obtained using +stringForEventCode:.

LowMemoryDownload

This set of classes is based on code from Outpost, and represents one way of reducing the memory footprint when downloading XML data on the iPhone. The main idea is to have all NSURLConnection instances run on a single background thread to reduce the number of thread-local variables being allocated by that (famously memory-hungry) API, whilst writing all received data directly out to a temporary file, instead of accumulating it in memory as is traditional. You can think of this as a form of manual pagefile. When the download is complete, the data is returned using memory-mapping, such that the kernel can manage loading and purging the data directly in discrete page-sized blocks.

While this approach works marvellously at its intended aim (memory consumption in Outpost dropped from a peak of 26MB to 4MB when testing against some large accounts kindly provided by our users), it doesn’t appear to solve every issue. Firstly it should be noted that using any form of tree-based XML parsing almost completely negates the benefits— you should really only use tree-based XML parsers when you know you’re dealing with small amounts of data. Secondly, since NSXMLParser hands the entire blob of data into libxml2 at once, in a single xmlParseChunk() call, this enables libxml to hop around the data in memory enough to keep too much of it loaded, at least when multiplexing multiple XML downloads/parses (as Outpost must in order to remain remotely usable).

  • AQLowMemoryDownloadHelper — this is designed with a minimal API such that it ultimately provides two one-shot functions for downloading data: one synchronous, another asynchronous. It defines a protocol for an authentication provider object, which will be used to handle 401 and 407 responses from the web server if supplied. It then gives read-only access to its request, response, and any returned data.
  • AQConnectionMultiplexer — this class is a singleton used to manage a single download thread. Its only useful public function is +cancelPendingTransfers, which can be used when terminating an application to cancel any running downloads.

StreamingXMLParser

This is the latest attempt to manage the download/parse sequence in Outpost. It’s so new that at the time of this writing it’s not been fully integrated into the codebase! It revolves around a new event-based XML parser, designed to function in nearly the same manner as NSXMLParser, with the same delegation routines. The main difference is that this version uses an NSInputStream as its data source, and it provides data in discrete blocks to the libxml2 parser library.

Beyond the requirement of a stream, it is initialized, setup, and used in the same way as a standard NSXMLParser. Note that, for compile-time type checking, the delegate routines for AQXMLParser are declared again here, with AQXMLParser replacing NSXMLParser. These routines are now also declared inside a protocol using the @optional keyword, to match the new delegation API constructs used in the iPhone SDK.

AQXMLParser is fully gc-compliant, or can be used in a managed-memory environment.

TempFiles

This folder contains three categories designed to be useful when creating temporary files:

  • NSString — adds a uuidString function to return a UUID string. Uses CFUUIDRef internally to generate the UUID.
  • NSFileManager — adds three functions to generate a temporary file name. Uses mktemp() internally to generate a name which is guaranteed to be unique at the time of the call.
  • NSFileHandle — adds three methods similar to those in NSFileManager to create a new temporary file. Uses mkstemp() internally to test and create the file in one step, avoiding the race condition between test & creation.

The NSFileManager and NSFileHandle temp-file routines have three forms:

  1. Create a randomly-named file within NSTemporaryDirectory() (usually somewhere inside /tmp/[user_id]), under a new folder named the same as the current process (i.e. NSTemporaryDirectory()/app_name/temp_file)
  2. Create a randomly-named file within a folder of your choosing (i.e. folder_name/temp_file)
  3. Create a randomly-named file within NSTemporaryDirectory(), under a new folder whose name you specify (i.e. NSTemporaryDirectory()/folder_name/temp_file).

All three categories are compatible with garbage collection or manual memory management.

Contacts

In here are some fairly simple ObjC wrapper classes for the iPhone’s CF-only AddressBook framework. Mostly they provide ‘syntactic sugar’ over the C APIs and return autoreleased rather than retained variables. So far the only non-wrapper function I’ve implemented is a routine to return an NSIndexSet containing persistent ABRecordIDs for all members of an ABGroup. This will probably be followed by a few more similar routines as I begin to use this in real-world situations where contacts will be saved and manipulated more frequently and in particular ways.

aqtoolkit's People

Contributors

alanquatermain avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

aqtoolkit's Issues

trouble using AQXMLParser

hi there,

i've added -lxml2 to linker flags and /usr/include/libxml2 to header search paths but i'm still having trouble getting it to work:

Undefined symbols:
"_kCFStreamPropertyHTTPResponseHeader", referenced from:
_kCFStreamPropertyHTTPResponseHeader$non_lazy_ptr in AQXMLParser.o
"_CFHTTPMessageCopyHeaderFieldValue", referenced from:
-[AQXMLParser(Internal) _setupExpectedLength] in AQXMLParser.o
"_kCFStreamPropertyFTPResourceSize", referenced from:
_kCFStreamPropertyFTPResourceSize$non_lazy_ptr in AQXMLParser.o
ld: symbol(s) not found
collect2: ld returned 1 exit status

any ideas of what's wrong?

AQGzipInputStream pauses unexpectedly while parsing XML

While running AQXMLParserWithTimeout in a dispatch_async() block, AQGzipInputStream pause on line number 261:

kern_return_t kr = mach_msg( (mach_msg_header_t *) &msg, MACH_SEND_MSG, msg.header.msgh_size, 0, MACH_PORT_NULL, 0, MACH_PORT_NULL );

The stream is never closed, and the parser is waiting for input for an indefinite period

Unchecked Error Condition - security issue

We are using “NSData+CommonCrypto”. We are facing security issue raised By Veracode Scanning Report. Please check the following details for issue provided by Veracode.

Description:
Error handling problems occur when an application does not properly handle errors that occur during processing. If a function does not generate the correct return/status codes, or if the product does not handle all possible return/status codes that could be generated by a function, then security issues may result. Similarly, failing to catch an exception thrown by a function can potentially cause the program to crash or to behave in an unexpected manner.
This type of problem is most often found in edge conditions that are rarely encountered during normal application use. Presumably, most bugs related to common conditions are found and eliminated during development and testing. In some cases, the attacker can directly control or influence the environment to trigger these edge conditions.

Recommendations:
Never ignore return codes, assuming that a function will always succeed. Check for and handle all possible return codes to ensure that all scenarios are covered, including boundary or edge conditions. Subject the application to extensive testing to discover some of the possible instances of where and how errors or return values are not handled.
Use a standard exception handling mechanism to be sure that the application properly handles all types of processing errors. Do not allow the application to throw errors up to the application container, generally the web application server.

Associated Flaws by CWE ID:

Unchecked Error Condition (CWE ID 391)(1 flaw) 
Description :  The result of this call are not captured. Failing to check the return code can result in unexpected behavior. 
Recommendations : Check the function return code for success 
Class - NSData+CommonCrypto.m : Location line number 346

`void * buf = malloc( bufsize );`

kSecAttrAccount not used in -[AQKeychainItem copyQueryDictionary:] ?

Not sure if this is by design or not, but at least I found myself wanting to include the kSecAttrAccount attribute to be included when querying (-[AQKeychainItem queryKeychain]).

So in -[AQKeychainItem copyQueryDictionary: (AQKeychainOptions)] , around AQKeychainItem.m:142,
why not also include kSecAttrAccount in addition to kSecAttrSecurityDomain, kSecAttrPort, kSecAttrPath etc when building the set of attributes used in the query?

AQXMLParser/NSXMLParser delegate call behavior discrepancy

I'm parsing RSS1/2/Atom feeds with AQXMLParser, and I find a discrepancy between the behavior for the delegate call parser:didEndElement:namespaceURI:qualifiedName:. AQXMLParser sends nil for qualifiedName when the element is unqualified, but NSXMLParser sends qualifiedName equal to element name when it is unqualified.

I fixed this for myself pretty naively by returning completeStr for qualifiedName in the AQXMLParser.m line 377 didEndElement delegate call for the prefixStr==nil case.

StreamingXMLParser: Getting Byte Position on End of Document?

On parserDidEndDocument: is it possible to get the position/byte-offset that the document ended on?

I'm trying to parse several XML documents from the same socket. They are not delimited so in order to parse the next document I need to know where the current document finished parsing.

Warnings when using LLVM compiler

When using the LLVM 2.0 compiler included with Xcode 4, I get warnings when compiling NSError+CFStreamError.m:

 Weak identifier 'x' never declared

I fixed this by moving the weak symbol definitions below the import statements.

AQXMLParser random crashes when parsing some xml documents

We had the issue that some of the xml documents which we were parsing caused a crash on iOS 6. The fix is to add the following line to the AQXMLParser line 1141:

p->serror = __errorCallback2;

And adding the following method to the same file:

static void __errorCallback2(void *userData, xmlErrorPtr error) {

}

The main problem was that the serror field is not initialized to any thing valid so the call stack you get when it is crashing is more or less random.
Our crashes were also only happening when we were parsing on a background thread.

downloaderCompletedTask: never called if completed successfully (includes simple fix)

Just change the _markComplete method in AQLowMemoryDownloadHelper.m as follows:

- (void) _markComplete
{
    self.complete = YES;
    if (self.asyncDelegate != nil) {
        [self.asyncDelegate performSelector:@selector(downloaderCompletedTask:) withObject:self];
    }
}

And voila, it appears to work fine now. (I haven't tried testing multiple downloads at one time, etc., so you may need to experiment. In addition, it seems like downloaderCompletedTask is never called by the existing code, so I'll also want to add this line or another, to indicate failure. ymmv.)

Warnings and Analysis warnings

First of all, great work on aqxmlparser, a real win for my project.
I try to create as clean as code I can and can't stand warnings and analysis warnings.
Sadly aqxmlparser does throw some of them, when I compile I get the following warning and analysis warnings:

Analysis

/Users/gerteunis/Documents/NZBVortex/Classes/XML/AQXMLParser.m:1138:9 Incorrect decrement of the reference count of an object is not owned at this point by the caller

/Users/gerteunis/Documents/NZBVortex/Classes/XML/AQXMLParser.m:1187:3 Value stored to 'saxHandler' is never read

Warnings

/Users/gerteunis/Documents/NZBVortex/Classes/XML/AQXMLParser.m:115:0 /Users/gerteunis/Documents/NZBVortex/Classes/XML/AQXMLParser.m:115: warning: 'NSObject' may not respond to '-stream:handleEvent:'

/Users/gerteunis/Documents/NZBVortex/Classes/XML/AQXMLParser.m:115:0 /Users/gerteunis/Documents/NZBVortex/Classes/XML/AQXMLParser.m:115: warning: (Messages without a matching method signature

/Users/gerteunis/Documents/NZBVortex/Classes/XML/AQXMLParser.m:153:0 /Users/gerteunis/Documents/NZBVortex/Classes/XML/AQXMLParser.m:153: warning: 'NSObject' may not respond to '-stream:handleEvent:'

/Users/gerteunis/Documents/NZBVortex/Classes/XML/AQXMLParser.m:159:0 /Users/gerteunis/Documents/NZBVortex/Classes/XML/AQXMLParser.m:159: warning: 'NSObject' may not respond to '-stream:handleEvent:'

/Users/gerteunis/Documents/NZBVortex/Classes/XML/AQXMLParser.m:1098:0 /Users/gerteunis/Documents/NZBVortex/Classes/XML/AQXMLParser.m:1098: warning: class 'AQXMLParser' does not implement the 'NSStreamDelegate' protocol

/Users/gerteunis/Documents/NZBVortex/Classes/XML/AQXMLParser.m:1228:0 /Users/gerteunis/Documents/NZBVortex/Classes/XML/AQXMLParser.m:1228: warning: incompatible Objective-C types 'struct AQXMLParser *', expected 'struct NSXMLParser *' when passing argument 1 of 'parser:parseErrorOccurred:' from distinct Objective-C type

/Users/gerteunis/Documents/NZBVortex/Classes/XML/AQXMLParser.m:1332:0 /Users/gerteunis/Documents/NZBVortex/Classes/XML/AQXMLParser.m:1332: warning: incompatible Objective-C types 'struct AQXMLParser *', expected 'struct NSXMLParser *' when passing argument 1 of 'parser:didStartMappingPrefix:toURI:' from distinct Objective-C type

/Users/gerteunis/Documents/NZBVortex/Classes/XML/AQXMLParser.m:1354:0 /Users/gerteunis/Documents/NZBVortex/Classes/XML/AQXMLParser.m:1354: warning: incompatible Objective-C types 'struct AQXMLParser *', expected 'struct NSXMLParser *' when passing argument 1 of 'parser:didEndMappingPrefix:' from distinct Objective-C type

AES encrypt options

In the AES256EncryptedDataUsingKey method, I can't decryt the data using java when the options is set to kCCOptionPKCS7Padding. But when I modify the options to "kCCOptionPKCS7Padding | kCCOptionECBMode", I can decrypt correctly.
I haven't used the decryptedAES256DataUsingKey method in the iOS client, but I think the options in this method should alos be modified to "kCCOptionPKCS7Padding | kCCOptionECBMode".

Problem parsing XML with encoding "US-ASCII" in AQXMLparser

The parsing seems to keep starting over and over.
Perhaps the US-ASCII isn't detected/used correctly.
"US-ASCII is upwards-compatible with UTF-8 (an US-ASCII string is also a UTF-8 string, see [RFC 3629])"

When I manually change the encoding inside the xml to "utf-8" it seems to work okay.

Fix key length function has a bug

if you try decrypting with a 16 byte key it converts the keylength to 24 bytes

the following function should be changed to add <= for 16 and 24 byte comparisons

static void FixKeyLengths( CCAlgorithm algorithm, NSMutableData * keyData, NSMutableData * ivData )
{
NSUInteger keyLength = [keyData length];
switch ( algorithm )
{
case kCCAlgorithmAES128:
{
if ( keyLength <= 16 )
{
[keyData setLength: 16];
}
else if ( keyLength <= 24 )
{
[keyData setLength: 24];
}
else
{
[keyData setLength: 32];
}

        break;

Handling of input stream error

I am new to Objective C and your streaming XML parser, but I ran into problem when network error occurs. I wanted to know if this is correct:
In AQXMLParser.m at line 963:
_internal->error = [input streamError];

Should this be:
_internal->error = [[input streamError] retain];

When this error occurs, then the parser is released, I get a crash in -dealloc at the line:
[_internal->error release];

with this error message:
-[CFError release]: message sent to deallocated instance

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.