Comments (9)
@dehesa yeah, I'd like to give it a try. How does this interface look?
extension CSVDecoder {
/// Decode CSV data row by row.
///
/// Does not throw since parsing does not start until requested by iterator.
///
/// - Returns: Iterator for decoding each row.
func decodeSequence<T: Decodable>(of type: T.Type, from url: URL) -> CSVDecodableIterator<T>
}
struct CSVDecodableIterator<Element: Decodable> {
/// Decode next row.
/// - Returns: Decoded row as `Element` or `nil` if no next row exists.
/// - Throws: `DecodingError` if row can not be decoded.
func next() throws -> Element?
}
from codablecsv.
Hi @josh,
Yeah, last week was pretty productive 😄 I didn't manage to do everything I wanted (the .unrequested
decoding buffering strategy is still missing), but I succeeded on having full Codable adoption, which makes me very happy.
Referring to your idea:
-
I quite like it.
Having a counterpart to
CSVReader.readRow()
on the decoder seems to be worth while. It would help parsing huge files, since the result is decoded on demand (currently you can only receive the whole decoded file). That said... -
Foundation's
JSONDecoder
andPropertyListDecoder
have established the expectation that calling.decode(...)
on a decoder will return the whole instance being decoded.We would need to come up with a different API that makes it clear you are decoding on demand. The solution you propose is, under my opinion, confusing because the user would expect to fully decode the type they are writing as argument. However, you are just returning a sequence to decode later on.
-
I don't understand why your
DecodableSequence
needs to conform toDecodable
. There doesn't seem to be a need for that.At a high-level:
let decoder = CSVDecoder() let sequence = decoder.__nameForOnDemandRowDecoding__(CustomRowType.self, from: url) for row in sequence { /* Do what you want with it */ }
The sequence could be just a wrapper over the
shadowDecoder
returning a unkeyed container, so you can reuse everything else in the library.extension CSVDecoder { internal ShadowSequence<Element>: Sequence, IteratorProtocol { private let decoder: ShadowDecoder func next() -> Result<Element,Error>? { guard !__ask_self.decoder.source_if_is_at_end() else { return nil } do { try self.decoder.unkeyedContainer().decode(Element.self) } catch let error { return .failure(error) } } } }
-
I am not yet 100% convinced that wrapping the decoded row twice is a great idea. One wrapping for
Result<T,Error>
and another wrapping forOptional<Result<T,Error>>
.If feels ugly 😅 that you have to send two failure signal on error: one for the
Result.failure
and another one sending anil
to tell any sequence syntax consumer that the sequence has finished.The only other solution I can think of, is to get away completely from the Sequence syntax and have
ShadowSequence
have a throwing method to be called in awhile
loop. For example.let decoder = CSVDecoder() let sequence = decoder.__nameForOnDemandRowDecoding__(CustomRowType.self, from: url) while let row = try sequence.next() { /* Do what you want with it */ }
It would be a pity to throw away all the nice
Sequence
extension, though. Maybe it is something we can bring up in the Swift forums. The Compiler devs are pretty active there (specifically @itaiferber, who is the creator of the Codable API).
from codablecsv.
I haven't gotten a chance to look at any of this decoder work at all, so take my comments with a grain of salt, but two small notes:
-
In general, it is not safe to hold on to a decoding container outside of the scope of an
init(from:)
call:private var container: UnkeyedDecodingContainer required init(from decoder: Decoder) throws { container = try decoder.unkeyedContainer() }
Decoders may rely on the fact that once an
init(from:)
has returned, all container references are no longer valid. Attempting to decode from the container is undefined — it might work, but more likely than not, the container will decode something you don't expect, or point to something that no longer exists. IfCSVDecoder
guarantees safety for this, then you can consider creating a type scoped specifically toCSVDecoder
, perhaps? -
To make this sort of operation safe, you'd really need to expose an interface on
CSVDecoder
that guarantees this operation being valid. e.g. instead of writing a generalDecodableSequence
, you should go with something likelet sequence = CSVDecoder().decodeSequence(of: CustomRowType.self, from: url) for row in sequence { ... }
The
Sequence
you return would need to hold on to the internal state of theDecoder
so it can safely be used without forcing any internal components (like containers) to outlive their intended scope.
In general, I agree with @dehesa that when an object is decoded, it is expected to be fully initialized. If lazily decoding is entirely transparent (e.g. a CSVLazyDecodableSequence
holds on to the underlying CSV data and decodes it internally as I iterate it without exposing that to me directly), then that could be safe.
from codablecsv.
Hi @itaiferber, thank you for pitching in here. I completely agree with all the points you've raised. My doubt hangs more over my point 4. Let me rephrase it:
For a lazily sequence decoder, as the one you have described:
let sequence = CSVDecoder().decodeSequence(of: CustomRowType.self, from: url)
for row in sequence { ... }
How could we iterate through its elements when the operation may throw at any point? We would like to keep the Sequence
syntax and benefit from all the Sequence
extensions, but IteratorProtocol
's next()
function doesn't support throwing an error.
-
One option is to wrap the sequence's element in a
Result
type, but that actually doesn't stop sequence. The next logic step is to wrap the result in an optional.In case of error, the sequence would receive a failure and next time is queried again, it will forward
nil
. This is problematic for cases where the user is counting the elements (for example). -
Another option is to wrap the sequence's element in an optional, and when a
nil
is received, check an error status on the sequence. Which would force the user to always be wary about the decoding state.
Do you see any other option for lazy sequence decoding?
from codablecsv.
Exposing something like decodeSequence
would be great. The only reason I'm using this work around is the exposed public decoding APIs provide the ability to decode a single row. The DecodableSequence
is just a hack to get a handle on that internal shadow decoder's container.
from codablecsv.
@dehesa Good point, I wasn't thinking about that.
I was going to write the following:
Personally, I would go the
Result
route, wrapping the sequence element (such thatnext()
returnsOptional<Result<...>>
—nil
when the sequence is over, andResult<...>
per element).In case of an error, though, I would consider not returning
nil
but instead repeatedly returning the same error again. This is consistent withUnkeyedDecodingContainer
, which will not advance in case of an error, allowing you to retry decoding as another type. In order to skip an element, you need to decode a dummy type which always works. In theSequence
world, you can probablydropFirst()
to achieve the same goal.
But the more I think about it, the more Sequence
conformance feels somewhat limiting in this case. It feels very easy to accidentally end up in an infinite loop if you're not paying attention:
for row in sequence {
guard case let .success(row) = row else {
// Skip invalid rows.
// <oops, this doesn't actually advance the sequence>
continue
}
// Do something with `row`.
}
Alternatively, if you do want to dropFirst()
correctly, you can't easily use for-in
syntax because dropFirst()
returns a new Sequence
to iterate.
So, it sounds like your original suggestion of
let decoder = CSVDecoder()
let sequence = decoder.__nameForOnDemandRowDecoding__(CustomRowType.self, from: url)
while let row = try sequence.next() { /* Do what you want with it */ }
is more apt. Getting rid of Sequence
/IteratorProtocol
conformance is unfortunate, but easier to get right from the client side.
Two alternative ideas that might be interesting:
- You keep
IteratorProtocol
conformance but add a manualskip()
method in case decoding fails (this allows for the equivalent ofdropFirst()
while still letting you keep the loop simple) - You keep
Sequence
conformance but have theElement
type be an opaque value which has an explicitdecode()
method which itself throws. This lets you keep thefor-in
syntax and gives you an easier out by not having to wrap withResult
. In this case, I would consider having a faileddecode()
call allow the loop to continue (unlike my previous suggestion above in which theSequence
would not advance) — in this case, theSequence
you are iterating is of opaque values which are divorced from the underlying encoded representation.
This last idea, to me, seems the more interesting one to investigate.
from codablecsv.
Ohh, I like the last idea you mentioned. Thank you @itaiferber (and congratulations for coming up with the amazing Codable interface) 😉
What do you say, @josh? Are you still up for giving it a go with all the information gathered?
By the way @itaiferber, if you find yourself with a bit of time and you end up checking the library. I will be very happy to receive any comment/criticism that you may have 😄
from codablecsv.
If I understand correctly, @josh's proposed API will look as follows on regular usage:
let decoder = CSVDecoder(configuration: ...)
for row in decoder.decodeSequence(of: Student.self, from: url) {
let student = try row.decode()
// Do something
}
I would suggest the following changes:
-
Let's use the
lazy
moniker as in the Swift Standard Library.We are actually providing that kind of functionality. I believe it is pretty descriptive.
-
Don't force the outcome type to be passed at the sequence level, but at the row decoding level.
This will provide the user with the freedom to decode different rows with different types if she wanted.
With those changes in mind. What do you think about this?
let decoder = CSVDecoder(configuration: ...)
for row in decoder.lazy(from: url) {
let student = try row.decode(Student.self)
// Do something
}
The benefit of point 2 to is that the user can choose different row representation as she wants. For example, you may have a students CSV where the students from 0 to 99 are represented as a different type than the students from 100 to 199. Although the underlying CSV representation is the same.
from codablecsv.
Now, on an implementation perspective. There are some challenges that you will face:
-
Most users will likely write the code as we intended. However it is technically possible to do the following:
let decoder = CSVDecoder(configuration: ...) let sequence = decoder.lazy(from: url) guard let rowA = sequence.next(), let rowB = sequence.next() else { ... } let studentB = try rowB.decode(Student.self) let studentA = try rowA.decode(Student.self)
Meaning, the second row is actually decoded before the first. What happens if
studentA
contains an error (whether a problem in the underlying string encoding, or the user mistype an escaping character)? We cannot recover from that.The solution for this is buffering and decoding the previous rows. But this will require a none trivial amount of state management. You could reuse the buffers I implemented (which will be optimized for performance later on), but even so, it will be some work.
-
The
Sequence
protocol encompasses the destructive case as it is ours. Meaning, that only one pass is allowed.We should be clear in the documentation that that is the case and decide what to do with some
Sequence
properties such asunderestimatedCount
. Should we always return0
?, or should we return the number of decoded rows? or should we do a full pass and give the actual number of rows? -
Take to heart what @itaiferber said about keeping decoding containers alive. They are intended for scoped in-order use and they all reuse the same
ShadowDecoder.Source
.They are used like that in my current decoder implementation. Check it out.
I am happy for you to give it a go, but if it becomes too much, we can always fall back in the easier-to-implement API, which doesn't conform to the Sequence
protocol and doesn't use buffering.
let decoder = CSVDecoder()
let sequence = decoder.lazy(from: url)
while let row = try sequence.next() {
// Do what you want with it
}
from codablecsv.
Related Issues (20)
- Make it conform to TopLevelDecoder/Encoder to use with Combine HOT 2
- Encoding a single object produces a confusing error message. HOT 2
- Lazy decoding a CSV with CRLF line endings fails without an error HOT 4
- Extra comma in header or data line causes failure to parse subsequent lines HOT 8
- Decoding CSV file with CRLF line endings fail with error if the last column is quoted HOT 9
- trimStrategy does not trim characters inside of a quoted field HOT 2
- Value of type 'CSVReader' has no member 'columns'
- doesn't handle strings with "," HOT 1
- Option to escape Excel-unfriendly strings HOT 5
- How can i install on linux? HOT 1
- Add support for field delimiter detection HOT 1
- Give CSVWriter.Configuration its own delimiter type which has no inference API
- Skip Column in Encoder (and Decoder)
- Last column decodes as blank
- Last column decodes as blank HOT 1
- question about escaping
- Define headers, but suppress header output?
- Customise cell parsing for declarative decoder
- in iOS17 can't read file from CloudDocs
- Build error in Xcode v16.0 beta 5 HOT 5
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from codablecsv.