Use factories and not fixtures about betterspecs HOT 42 OPEN

Factories fall apart when your application logic requires complex and numerous relationships set up in order to test - it makes the tests SLOW. I think there is much to be gained by setting up fixtures for a base system that has most of the relationships in place, and then modifying that as needed for specific tests, whether with factories or basic object creation.

I don't know why everyone says fixtures are hard to maintain. When your data gets complex, factories are equally hard to maintain. When dealing with over 50 models (government reporting here, it cannot be less complex), you have to put them in different files, and using factories means yet another DSL syntax to learn. YAML isn't that hard.

What is really missing is better tools to help manage those fixture files. There should be an easy way to load up the test db fixtures, edit them, save, and re-run tests to make sure you didn't break anything. (You say that's brittle? The same thing is possible by editing factory definitions. You deal with it by running tests.)

I like FactoryGirl, but only as an addition to the standard fixtures, to create border cases, test invalid data, etc.

from betterspecs.

I appreciate the reply, and all points are taken with consideration. You're right that my apps are mostly small data management applications for ETL and client management -- fixtures work great. But I want to correct this part as it's directly related to this issue, and my original comment:

If you don't have a lot of experience with factories, then why are you making sweeping statements about them?

That's actually the exact opposite of what's going on here. I'm not making any assumptions at all about rspec/FactoryGirl, or criticizing them. In fact, I'm fully admitting I don't know them well. This is how I ended up on this project's page in the first place, and found this exact issue to discuss this point in the project.

When you are learning rails, minitest and fixtures are what's included -- so that's what you learn.

The project is making the sweeping statement, asserting that fixtures are bad and you should opt for factories without helping a user understand why. The given example has nothing to do with fixtures.

My goal with these comments isn't to criticize or argue about the merits of Fixtures v FactoryGirl, but to bring to light that for someone who uses fixtures regularly, the content of the subject at hand simply doesn't make sense. It makes a dramatic statement ("don't use the thing that comes bundled with Rails and is well documented in the official documentation") while intentionally misrepresenting the example.

So, like I mentioned -- I don't know enough about this subject to be able to write a PR for this to improve it. I just know that as it stands, this doesn't make much sense, and this is the forum for discussing that.

from betterspecs.

I will concede that factories are nice for cucumber, but that demands to ask if cucumber is worth anything. In all my experience, it is just extra steps, because only programmers write it, and it just adds extra steps and extra knowledge to the process. Cucumber only makes sense if you have QA, managers, or clients writing the stories... and even then I suspect programmers have to tweak it. I can take the same conversation with a client, write the story in pseudocode, and then program rspec to do the same thing with capybara and save the extra steps of regex parsing.

As I said before, fixtures are great for the base system, when there are a lot of dependencies due to the complexity of the domain model. Let the fixtures define the basic setup - almost a seed like structure, but a little more. Then use factories/object creation to define the corner cases.

Properly named fixtures help define the components that you want to test for, and proper tests know how to exclude the extra data (which is something that you should be testing for!). If the presence of additional data is throwing off your test, or if changing a fixture or factory breaks tests, you need to evaluate those tests and set them up to not be so brittle.

For example, if a test currently checks that there are 4 comments to a post after completing an addition, and you add a comment in the fixtures or factories, of course that breaks the test. The test should instead query the number before the action, and then verify that the action added 1 comment. This insulates your tests from changes in the fixtures OR factories.

@RobSaunders says: "Structure should always be built up within the test itself." Well, that is nice until you have 500 tests (only 500?) that all need to build up 30 relationships before each test can run. Personally, I hate waiting four an hour for tests to finish. Mocking and stubbing don't help at all, because it is the very nature of these relationships that need the most testing. Fixtures and transactional tests take a lot of pain out the equation. I can depend on the base system and add 1-3 objects per test and achieve a huge speed boost.

And again, what is truly missing is proper tools to manage those fixtures - not just a dump of the test database, but to facilitate proper naming of the various models and organize it. But in the end, it is just editing text files, and yaml is no more difficult to understand than factory girl's DSL; in fact, I think yaml is far easier to understand!

Coming back around to cucumber: If you have decently named factories, I think my project managers would better understand the fixtures, because they could say: (I'm going to alter the syntax a little, this isn't necessarily cucumber)

With child_x_with_abc_registration
Sign up for y_class at x_site
x_site should show registration for child_x in y_class

My project managers have a great memory for the existence of predefined test data like child_x and abc_registration. This would make a lot of sense to them and doesn't require factories. It also doesn't require cucumber. That little pseudocode can be converted to rspec that is readable by managers, even if they cannot create it directly.

For me, tests that run in a decent amount of time, and not wasting time on unecessary programming steps, is pure gold.

from betterspecs.

@nruth That article you linked to drives me crazy, because it seems to go out of its way to show some of the ways that fixtures suck, but then unaccountably says "oh, they're not so bad"!

They are so bad; they encourage inflexible test data that's overly separated from your test functions. What you probably want instead is more factory templates to easily cover different cases.

from betterspecs.

It's been five years of discussion, and the rule in the guide still makes a pretty dramatic claim about fixtures without stating why. The example given has nothing to do with fixtures. I've recently been trying to research the reasons people flatly say "don't use fixtures" but in every case I've come across, they are used in an arduous way that isn't necessary. If the example were actually using fixtures, it would look like this:

Use factories and not fixtures

This is an old topic, but it's still good to remember it. Do not use fixtures because they are difficult to control, use factories instead. Use them to reduce the verbosity on creating new data.

BAD
user = users(:guy)

GOOD
user = FactoryGirl.create :user

Well, that's not really compelling at all. In the published example, you aren't even using fixtures, so of course it's more verbose. A common argument is often that you have to maintain fixtures and give them default values and whatnot. But... don't you also do that with FactoryGirl?

Being that the fixtures can also include generated data if you really need it to (such as name: <%= Faker::Name.first_name %>, or create 1000 random objects in quick loop), I struggle to understand the argument here. What's difficult to control about sourcing an object with default values, and when necessary, changing them to fit your test using things like user = users(:tim); user.name = 'special name for this case'? Even complex associations can either be saved in the Fixture, or composed on the fly in your test, just like in FactoryGirl.

I fully admit that I don't have a lot of experience with rspec and FactoryGirl, as I've found MiniTest and fixtures very sufficient in my few years of study. It's very possible that my lack of experience in huge apps has me blind to something. That said, the motivation of my comment here is the search for that experience -- but everything I've found so far has been extremely unconvincing.

from betterspecs.

While talking about fixtures, bad example shows plain object creation, not fixture. As comparison of fixtures and factories may be controversial, plain creation vs. factories has no contenders. Perhaps it makes sense to rename?

from betterspecs.

I'd like to see a definition of "difficult to control". I have no idea what that means.

from betterspecs.

@RobSaunders:

One caveat is to watch out you don't build up complex associations in your factory files - as this gives them the same downsides as fixtures. Structure should always be built up within the test itself.

I completely disagree. A major purpose of factories is to be able to build complex associations easily—otherwise you'd just use User.new instead of FactoryGirl.create :user. This does not have the same downside as fixtures, because the object that you care about is specified and created right in your test (sure, associated objects are created too, but you can ignore them). Fixtures get less useful as your associations get more complex, while factories actually get more useful. Not using associations in your factories removes most of the benefit of using factories at all.

How do you test with complex associations, then? It seems to me you'd either have to build up the associations in your tests (making your tests ugly) or mock a lot (making your tests implementation-coupled). Is there a third way that I'm missing?

@mpalmer:

In-memory sqlite3 is awesome for testing. I use it in all my projects.

Careful! SQLite is single-user and therefore not suitable for production. If you're using it for testing, then that means you're testing on a different DB from production, and your test environment may behave significantly differently from your dev environment: although ActiveRecord does a good job of abstracting differences between DBs, it cannot do so completely, and SQLite is just plain missing a lot of useful features (such as foreign key constraints and true booleans), so that testing any part of your app involving those features will be automatically incorrect.

IMHO, for any but the most trivial uses of the database, this difference is too dangerous to bear, so I'm willing to live with the lower speed of my tests actually talking to Postgres—because then I know they're correct for my production environment.

from betterspecs.

I think this illustrates the pain point with fixtures pretty well.

TL;DR: Changing a fixture, or even adding a new fixture will often break a vast number of previous tests*, and fixtures introduces mystery guests.

* - Sometimes you need to test methods that f.ex. outputs statistic based on the DB state. Even adding 1 new fixture later on, would break all those tests of methods relying on counting records in the DB. Maybe there's a workaround (?), but that is what I am experiencing.

For data sets that are a bit more involved, YAML fixtures are difficult to maintain and hard to change without affecting other tests. I mean, you can make them work, of course—after all, developers used them plenty in the past—but tons of developers will agree that the price to pay for managing fixtures is just a bit stingy.

One scenario we definitely want to avoid is changing little details on an existing fixture and causing tons of tests to fail. If these failing tests are unrelated, the situation is even worse—a good example of tests being too brittle. In order to “protect” existing tests from this scenario, this can also lead to growing your fixture set beyond any reasonable size—being DRY with fixtures is most likely not on the table anymore at that point.

To avoid breaking your test data when the inevitable changes occur, developers were happy to adopt newer strategies that offered more flexibility and dynamic behaviour. That’s where Factory Girl came in and kissed the YAML days goodbye.

Another issue is the heavy dependency between the test and the .yml fixture file. Since the fixtures are defined in a separate .yml file, mystery guests are also a major pain waiting to bite you due to being obscure.

Source: https://code.tutsplus.com/articles/antipatterns-basics-rails-tests--cms-26011

It also describes what Mystery Guests are, and the problem with them (their obscurity - especially to new developers coming into the project - are akin to a mode, and we all know why we generally want "no modes"). One counter-argument is that mystery guests is a result of stringently applying Convention over Configuration.

That said, I see the benefits of fixtures as well, and I'm torn between that and the admittedly more sexy FactoryGirl.

I tried finding an elaboration on DHH's stance on it, but the most detailed I could find was this:

Use the power of testing fixtures and the case-transaction loop to avoid having to do expensive and excessive setup work for every case, and merely tailor your prototypical fixtures to the case at hand by changing the single state needed. (More on this technique in that promised future post).

Source: http://david.heinemeierhansson.com/2014/test-induced-design-damage.html

I couldn't find that promised future post. Let me know if any of you find it.

The case-transaction loop he is referring to (a novel expression, I believe), is probably ActionModel::Base.transaction.
By "tailor your prototypical fixtures to the case at hand by changing the single state needed", I presume he means using Model.update_attributes (which will update the record in the DB which the YAML fixtures had ensured was already there.) in each test (or in the setup method).

I think this is doing exactly what @marnen advised against above:
"You're abusing fixtures by treating them like factories as suggested above (in which case, why not use factories?)" But to DHH that seems to be the golden standard, perhaps oddly enough. Maybe you can actually get the best of both worlds with this approach? Using a complete set of fully fleshed out fixtures (using real world data), in all their complexity and interconnectedness, while overriding in the few cases where it is necessary..

I think the whole issue with the betterspec page though, is that it uses "fixtures" in an ambiguous way: "fixtures" in the broad sense which means pre-created fixed data (like Model.create), and not "fixtures" in the narrow sense that is common in Rails, which refers to the YAML files approach. The page should probably be updated to remove this ambiguity.

I second the suggestion by @dgm "What I want is a VCR like gem for databases." to try to get the best from both factories and fixtures and the worst of neither.

(If I'm allowed to dream, outside of this discussion, I would also want at VCR-like gem that would allow a non-developer to click through a use case in the app web interface, and store that as a test case which then capybara or similar could run. Automatically generated/macroed integration tests. Sorry for digressing, I just wanted that idea to be out there.)

from betterspecs.

@redbar0n FWIW, I no longer pay much attention to how DHH wants me to use Rails. Of course he did fantastic work in creating (the early versions of) the framework, but perhaps because of that, he's become so attached to his earlier vision that he can't seem to understand that there are other (IMHO better) ways of doing things than he envisioned. It's got to the point where by default, I take his recommendations as a slight negative these days.

That said, I see the benefits of fixtures as well

Then would you kindly enlighten me on what you think they are? IMHO there are none. :)

from betterspecs.

I'm not a fan of the binary good/bad statement here. There are good reasons not to use factories.

Really, the best practice is to use neither as much as possible: instead put as much of your domain logic in PORO that can be tested without needing complex, time consuming setup with either factories or fixtures. You'll need some tests that use DB records of course, and either factories or fixtures are valid tools to use there.

from betterspecs.

One caveat is to watch out you don't build up complex associations in your factory files - as this gives them the same downsides as fixtures. Structure should always be built up within the test itself.

Having complex structure in factory objects to DRY your tests obscures the object structure, adds unnecessary associations for tests that don't use them, and creates problems when you need a test which changes one small association or value, and all your other tests fail. It's an example of bad DRY.

from betterspecs.

@marnen:

If that's how you feel about factories, you may be abusing them. What you are doing is pushing complex and brittle structure that actually BELONGS to the test, into another, communal area. I may have misunderstood you though, so if I did this is not relevant, but I am assuming you mean to put 'helpers' to build up associated data (complex structures of models with specific sets of children) inside your factory files, which are used by multiple tests at once, rather than in the model test file itself.

It's fine in a small project with a few people who all respect the code. But more than a handful of devs and tests and you run into chaos quickly. This isn't about 'pretty' code or convenient short term code. This is about robust code that works in the long run, and that is about modularising areas so you have fewer moving parts (areas of extreme churn) and less functions that start to look like 'The Blob', which are changed every time you need a little bit more functionality from new consumers of said function and potentially breaking other consumers in the process.

I think it does take being in projects where this is a problem to truly respect how bad it can get, I am always on the look out for this anti-pattern and the single biggest one I find is mis-using factories.

In my opinion (I do understand that other people differ) factories provide the following benefits:

• They provide a nice syntactic sugar layer for quickly creating models in your tests
• They allow for 'default' values for your objects so you don't have to worry so much about validations which are a cause of brittleness themselves.
• They allow you to focus purely on what the test itself is testing (for example, you can create a 'default' object and each test unit can focus on values for its area knowing the other values are correct). This abstraction is probably the main value of factories.
• They reduce bugs by separating off what is considered a valid model (eg. when you change something about the model so it is no longer valid for your tests, the factory is often the first thing that breaks, so you know your model is failing correctly, it's a good 'first place' to look when you're unsure if your model is valid).

This is why I don't just use Model.new instead of factories, and I think these are sensible benefits rather than shortcuts that satiate short-term developer laziness but create long term problems.

from betterspecs.

The kinds of functions I am talking about are things like functions called create_user with crazy levels of arguments to define how the user factory instance is made

Yes this does happen, e.g. wrapping the must-be-in-one-transaction-so-cant-be-belongs-to shortcoming of old tools. Anyway, it seems the document should be updated to say "Don't use Rails helpers spec/support helpers", instead of don't use fixtures.

In fact, for complex setup cases fixtures sound pretty good and I'm going to give them a spin for climbing out of a 4-7 year old hole of overlapping helper mess & slow factories.

Beyond that it'd mean showing sustainable ways to use both approaches, which is probably no good for a pithy guide like this.

from betterspecs.

@ttilberg:

A common argument is often that you have to maintain fixtures and give them default values and whatnot. But... don't you also do that with FactoryGirl?

Not in the same way. A factory is a template, with default values that you're expected to override as necessary when you call it. A fixture, OTOH, is a fixed record that you're expected to fetch as is. (At least, this was the case when I last used fixtures, admittedly a long time ago.)

So with fixtures, you define your test data far away from your test functions, and you need to predeclare every test record. With factories, you define your test data right in the test, and you do so on the fly without much predeclaration. The difference may look small at the beginning, but gets more apparent as the tests get more complex.

Predeclaring is time-consuming and takes you out of the creative flow when you're writing your test. If you need a different case of test record with fixtures, you'll be tempted to reuse an existing one because of the overhead of predeclaring, so your test data won't be sufficiently specific to actually test what you think you're testing. If you need a different case with factories, there is no overhead; you just say exactly what you want, so you can test exactly what you think you're testing.

What's difficult to control about sourcing an object with default values, and when necessary, changing them to fit your test

Nothing as stated. But if you really wanted a user named John, why did you have to first create Tim and then change his name? Something like Factory Girl would let you just create John right at the beginning without declaring a new template or fixture. In your example, there's not a huge difference, but in a system that had an audit log for each change to a record, there would be a big difference, and it could mess up your tests.

Moreover, if you only have a few fixtures and you're changing their data for every test, then you have no reason not to use factories: that is more or less the usage pattern that Factory Girl was designed for, and it provides a nicer interface for it.

Even complex associations can either be saved in the Fixture, or composed on the fly in your test, just like in FactoryGirl.

Saving a complex association in the fixture is usually bad, because it makes your test data too rigid. Composing on the fly without syntactic help is awkward as soon as there's more than one level of association. Factory Girl strikes a happy medium where the association pattern is assisted, but the association data is flexible on the fly.

I fully admit that I don't have a lot of experience with rspec and FactoryGirl, as I've found MiniTest and fixtures very sufficient in my few years of study.

If you don't have a lot of experience with factories, then why are you making sweeping statements about them? (Also, check out RSpec and/or MiniTest::Spec: the syntax is more readable and helps you focus your tests on interface, not implementation.)

Anyway, this sounds like classic Blub Paradox, but I know that's not an especially satisfactory answer. So: for me, a big reason that I like factories is that I can create exactly the test data I need, right where I need it, without predeclaring or confusing the test data.

I'll grant you that the differences may initially seem subtle and may not be immediately apparent to a novice. But they grow bigger as projects get more complex and as tests get more complete.

I'd go so far as to say that if you find fixtures sufficient, you probably fall into one of three cases:

• Your project is trivial (so it doesn't matter which method you use right now, but your project will grow)
• You're not testing well enough (because fixtures make it very hard to do so)
• You're abusing fixtures by treating them like factories as suggested above (in which case, why not use factories?)

In closing, a thought: some people who switch from fixtures to factories try to use factories like fixtures by predeclaring every fixture as a factory template. If you're doing that, stop now: that's not how factories are meant to be used.

If you have sample test code available, I would be happy to have a look at how you use fixtures and see what I can tell you about the differences for your use case.

from betterspecs.

Ah, good point; the example on the betterspecs website is terrible. I was responding more to the general principle than to the website content.

When you are learning rails, minitest and fixtures are what's included -- so that's what you learn.

Yes, and then you quickly discard them when you realize how much they suck. :) FWIW, I've worked as a Rails developer since about 2008, at a variety of companies. I think every one of them has used RSpec and factories. They are kind of community defaults at this point, even if the Rails core team doesn't want to distribute them with the Rails gem. :)

Now, I'm not recommending them because they're widely used, but rather because I think they're better than what comes standard with Rails. But I do think their near-universal acceptance is an interesting point.

from betterspecs.

@marnen What benefits does fixtures have?

"We avoid file system, database, simply because these 'shared fixture' elements prevent us running in isolation from other tests, or cause our tests to be slow." - Ian Cooper (36:42) https://youtu.be/EZ05e7EMOLM?t=2202

This was rebutted by DHH in https://dhh.dk/2014/slow-database-test-fallacy.html since DB runs can be fast, and fixtures can be run in isolation.

But isolation needs not only to happen at runtime, but the data itself need to be isolated, so that changing the fixture for one test doesn't inadvertently break another test. Shared fixtures introduce coupling, as mentioned in the previous post.

I think this tweet by dhh illustrates how he circumvents that problem:

"Only write a handful of fixtures per class to spin up the world. Then mutate base fixtures as required per test case." - dhh on twitter: https://twitter.com/dhh/status/449673835353477120

Then there are some benefits to fixtures, since you can describe a general "world" once, which can be shared between test cases without actually introducing any meaningful coupling. Because whenever you change anything in that world which is not shared amongst all test cases, then you use a small mutation in the diverging test case in question. Then you achieve Kent Beck's principle that all tests are isolated from other tests.

("Kent uses 'unit test' in a very specific way. When he uses the phrase, all he means is that: You should be able to run all tests together, in one suite, without one test being run impacting the other test. The unit of isolation is the test. A lot of mistakes has been made, because people think that the unit of isolation is the class under test. It is not." - Ian Cooper, 35:00, https://www.youtube.com/watch?v=EZ05e7EMOLM )

The problem with this approach is, of course, that if not all developers are on the same page, someone will change the base fixtures themselves, instead of mutating them in their test, and then crash a bunch of other tests. Hopefully this should be caught at inception when tests are immediately run afterwards, and by inferring the principle dhh tweeted from how other developers wrote their tests.

from betterspecs.

@redbar0n I think you might possibly have missed my point. It seems like you’ve mostly listed advantages of testing with fixtures as opposed to no DB management. My claim, though, was about fixtures as opposed to factories—and there I claim that anything that fixtures do, factories do better. Your comment mostly doesn’t seem to address this, unless I’m misreading.

Shared fixtures introduce coupling, as mentioned in the previous post.

Then there are some benefits to fixtures, since you can describe a general "world" once, which can be shared between test cases without actually introducing any meaningful coupling.

These two points are mutually contradictory. You can’t really do this with shared fixtures without introducing coupling between tests. The right way to do this—if you really must—is to describe the world once, but to create it for each test. This is what factories in an RSpec before block would do, for example.

But I’m not sure it’s a good idea anyway. I find that a general “world” is far too much setup and far too many assumptions for unit testing. I prefer to customize the world for each test case. Sure, there will be some
common aspects between cases, but I generally want as little of that as I can get away with. That way I’m making as few assumptions as possible for each test. And that’s much easier to do with factories than with fixtures, because it’s easier to use factories dynamically.

The problem with this approach is, of course, that if not all developers are on the same page, someone will change the base fixtures themselves, instead of mutating them in their test, and then crash a bunch of other tests.

Right. Why do that to yourself in the first place? It gives no compensating advantage and just introduces the temptation to bring a lot of unnecessary garbage into your test setup.

from betterspecs.

@marnen Actually, I was thinking fixtures vs. factories myself when I wrote it. It is my prime motivation here to find out the important differences, and try to deduce which is better when. DHH's tweet was particularly elucidating on how he uses fixtures successfully. In that way you can actually use fixtures without meaningful coupling, because the fixtures are actually "fixed", and only local copies of them in each tests are mutated. The "world" you describe with fixtures would be equivalent to the factories way of using shared instantiations in the "before test hooks" (particularly before(:suite) in rspec) at the top of the test file. In both cases these are then mutated only in each test (and in case of fixtures then mutations are automatically rolled back after each test). And in both cases (fixtures, or with factories using the before test hooks) there is coupling, but not 'meaningful coupling' in the sense that changing something inside one test will break another test (the unit of isolation is the test). I think we are very much on the same page in preferring to customise the world for each test case. My point was that this is even what DHH is doing with fixtures. Which avoids how people typically misuse fixtures when coming into them: making single-test-modifications inside the global fixtures. Instead of just making changes to the preloaded models inside each test like DHH seems to do.

I have found that creating a general "world" can be easier with fixtures than with factories. Because with fixtures that world could be a simple copy of your production DB, if you'd like. Or you could create it manually, but with seemingly real-world data. The benefit to the programmer testing is that he/she would only have to know "one world" really well, and be able to reason about it quite easily, to be able to write tests. Instead of when using factories typically creating "many small worlds", with typically bogus data, "just to get a that model up and running". Of course, you could create "one world" with factories as well, but then it would typically be spread out over several factory instantiation files.

Why do that to yourself in the first place? It gives no compensating advantage and just introduces the temptation to bring a lot of unnecessary garbage into your test setup.

All the above said, I agree with you that it's easier to use factories dynamically, and to keep your test surface lean and small for each individual test, so I see a lot of good reasons to use them. My only point was that the issue was not as clear-cut as I initially thought it to be. DHH is heavily on the side of pragmatism and for least-effort-approaches (the ethos of RoR), so it carried some weight with me that he was such a stark proponent for sticking with fixtures (and using them in the right way, which the rest of us might have simply missed from the get-go).

More on DHH's bottom line suggestions for testing Rails/MVC apps: https://dhh.dk/2014/test-induced-design-damage.html

from betterspecs.

I think you should check out fixture_builder and reconsider this one. The basic idea is to auto-generate fixtures (using factories or whatever) at the beginning of the test suite and only when necessary. I've gotten huge test performance improvements using this method with a transaction-based db clean strategy.

from betterspecs.

I don't use either in unit tests. I tend to just build objects using the actual classes. I've used fixtures, and factories. My preferential factory gem is Fabrication. I prefer the speed, clarity, and simple communication of just using the objects I need directly in unit tests.

If I'm doing functional testing, I'll definitely mix in some fabrication. The ease of using a factory in that case is apparent.

from betterspecs.

Where are the upvote options ?? dgm - amen. YAML files are just data - a business person can write them, a QA - anyone. Factories are executable code. Executable code is a poor serialization of an object graph. For my money, fixtures solve that problem far more elegantly than whatever Factory-Gem-Of-The-Day.

from betterspecs.

I, for one, love the concept of VCR. Perhaps a nice factory2fixture gem could be made? Something like that?

from betterspecs.

+1 @RobSaunders @myronmarston

The trick is not to abuse anything, and for me abusing the fixtures is just little bit easier.

And when cucumbering, good factories are pure gold.

from betterspecs.

I've added a section where I describe the fact that if possible we should avoid both fixtures and factories.
Any correction and comment to the updated guideline is appreciated.

from betterspecs.

I've read that "why I don't like factory girl" article before, and it has never felt "whole" to me, because it doesn't give any concrete way to do it better. So in some ways, this point has gotten worse for me now. I still don't understand why fixtures are "difficult to control" -- they've never seemed particularly painful to me (although I will concede that may be Stockholm Syndrome talking). In addition to that, I'm now being pointed to an article that says that factories are a bad idea too, but doesn't give me any way for me to tell if I'm suffering from the same problems the author of that article is worried about, and even if I am, they don't give any suggestions for exactly how to solve those problems.

A solid, real-world-applicable explanation of what we should be doing instead of both factories and fixtures is sorely needed, I think. I don't know of any relevant articles on the subject, though. Perhaps it still needs to be written...

from betterspecs.

It is possible to mix traits together to create a similar behaviour to fixture-factories, but having the dependency lie on the test-side rather than the fixture side. It is more limited and less flexible, but the overall force here is to do the right thing rather than the easy thing. When it comes down to it, putting structure into your factory creation is the equivalent of writing fixtures and you WILL end up with a messy pile of unmaintainable tests.

As a slight panacea you could, for example, use traits and do something like this:

create(:user, :with_posts)

It is difficult to then go ahead and create sub-associations (:with_comments_on_posts for example), but this limitation is a good thing - it prevents you from having crazy levels of sub-associations within your tests. And if you need sub associations, you write them in your test setup. This is in no way slower, in fact it's slower creating a pile of unused associative objects EVERY time you call your 'monster factory create' function.

@dgm if you have 500+ tests that use crazy levels of associations, you're doing it wrong. The bulk of your tests should be testing models (unit) or controllers (functional) where you really don't want to be testing a complete set of associations at once. The tests that you do have this kind of setup (integration tests), you should be confident that the associations work and really just be testing behaviour. It takes time to discover this, but the more complex your test setup, the more you end up testing nothing, because you can't account for the trillions of different edge cases and permutations that occur. Setting up tests this way also reduces the time it takes your suite to run (as you have fewer slow and bulky integration tests).

The complexity of the above paragraph probably requires giving a whole talk on the subject and there are a few good ones out there on this topic.

@mpalmer hopefully this answers your question - it's not a matter of finding a 'better way to do the wrong thing', it's a matter of understanding why building associations in a global way is bad, and how you can write more efficient, more specific tests.

from betterspecs.

@RobSaunders I totally agree. I see it as a sign of test and code smell when I start seeing crazy levels of associations needing to be done in order to test one thing. Which a lot of times the solution is to stub the crap out of that model and make sure the boundaries of that method are right.

from betterspecs.

I prefer to use build instead of create as much as possible, as not hitting the db speeds up tests greatly. If you test behaviour instead of state, this should not be a problem as you don't want to test ActiveRecord or DataMapper or whatever.

from betterspecs.

@mpalmer, @RobSaunders would you like to send a pull request better describing this point? It feels like this is the weakest point in all betterspecs and I want to fix it.

from betterspecs.

I disagree that one should only use fixtures.

There are many reasons why you'd use both, and it's a bit presumptuous to
lay a blanket statement like that down.
I can't even decide if 'prefer factories' is a good way forward either.
Things like VCR which record data you're using to test against are almost
like a hybrid. Dynamically generating data and saving it as a fixture.

I don't believe this is an either/or situation, it seems to me that it's a
question of when and how much.

~Johnneylee

On Thu, May 16, 2013 at 6:47 AM, Andrea Reginato
[email protected]:

@mpalmer https://github.com/mpalmer, @robsaundershttps://github.com/robsaunderswould you like to send a pull request better describing this point? It
feels like this is the weakest point in all betterspecs and I want to fix
it.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/11#issuecomment-17993971
.

from betterspecs.

@andreareginato I don't really have enough knowledge to be able to write a pull request. In my opinion, the point should probably just be removed entirely.

@RobSaunders I'm afraid that doesn't answer my question at all -- I still lack "understanding why building associations in a global way is bad". I see some furious hand-waving, but not a lot of concrete example. A simple "here is what happens when you use fixtures, and here is how factories make it all better" would be nice.

from betterspecs.

I don't think fixtures are necessarily bad, but their current implementation could be better.

Right now I'm working on a report that requires a lot of data in the database in order to fully test - it takes 5 minutes for the factories to whip up the test data. As a result, all the assertions are done in one massive test, as no one wants to wait for it to regenerate.

It would be much better if there was a way to use the factories to generate the data, and then snapshot the whole database into a cached sql file; then the test case could use that to load the database directly in a before section, and use transactions to roll back between tests that are augmented by further factory calls.

This would be a hybrid between fixtures and factories - but I would have to agree that using yaml fixtures would be painful... What I want is a VCR like gem for databases.

The reason there is no pull request for this, as far as I am concerned, is because there is still no truly good solution yet.

from betterspecs.

@dgm if you are using SQLite3 for your database, there is an in memory option. I have never gotten it to work but I've seen it done and it can speed tests up. I know it really doesn't fix the problem but it is definitely something to look in to in order to speed up your tests.

from betterspecs.

In-memory sqlite3 is awesome for testing. I use it in all my projects. I'm not sure it's an argument either for or against fixtures or factories, though -- it'll just make everything database-related significantly faster.

from betterspecs.

well he was arguing that the build up time for his tests with factories was
causing him to consider using fixtures and possible database switching.

just curious, do you have an example sqlite3 memory example I can look at?

from betterspecs.

You guys think is a good pattern use the FactoryGirl::Syntax::Methods to wirite only create(:user) over FactoryGirl.create(:user)? I use as default on all projects the short syntax.

from betterspecs.

@RobSaunders Thanks very much for a thoughtful reply! I'm always trying to make my testing practices better.

What you are doing is pushing complex and brittle structure that actually BELONGS to the test, into another, communal area.

I'm not sure that the complex structure belongs explicitly in a unit test. Perhaps it does, but...

# user.rb
class User
  validates_presence_of :account_id
  validates_presence_of :preferred_language_id
  validates_presence_of 10 other things
  ...
end

#user_spec.rb
describe User do
  describe "#full_name" do
    it "should return the first and last names" do
      User.new(first_name: 'John', last_name: 'Smith').full_name.should == 'John Smith'
    end
  end
end

The unit spec will fail, because all the required attributes and associations have to be set up even though they're not relevant to this spec. So you either have to build them all explicitly (ugly), mock them out (ugly and risky), or put them in a factory. I'll take the factory every time.

I may have misunderstood you though, so if I did this is not relevant, but I am assuming you mean to put 'helpers' to build up associated data (complex structures of models with specific sets of children) inside your factory files, which are used by multiple tests at once, rather than in the model test file itself.

Certainly not. I've never heard of this helper technique. My factory file would be

Factory.define :user do
  association :account
  association :preferred_language
  # 10 more associations and default fields
end

So when I do FactoryGirl.create :user, I get all the associations needed to make a valid User, but I don't have to clutter up specs with them where they're not relevant.

from betterspecs.

@marnen I may have misunderstood you. The kinds of functions I am talking about are things like functions called create_user with crazy levels of arguments to define how the user factory instance is made. Defining single-level associations within a factory is ok but not great (you're making much more database-heavy work than you need, and kind of starts to break the unit tests reason for being a unit test, but there are some situations where, without it you don't have a valid factory). I think as long as you're only doing the minimal work needed to create a valid factory, you won't run into what equates to a 'god function' (the equivalent of the god model antipattern).

from betterspecs.

@RobSaunders:

The kinds of functions I am talking about are things like functions called create_user with crazy levels of arguments to define how the user factory instance is made.

People do that?!? That really defeats the point of factories. I could see doing that if you weren't using factories, but.

Defining single-level associations within a factory is ok but not great (you're making much more database-heavy work than you need, and kind of starts to break the unit tests reason for being a unit test, but there are some situations where, without it you don't have a valid factory).

Rails unit tests hit the DB. I understand the theoretical arguments for not having them do so, but so far, I haven't found a good way of making that convenient in practice. Even if you use something like NullDB, the presence or absence of the DB starts to leak into the tests, even though it should be a private implementation detail.

That being the case, then, I'm perfectly happy to have as many associations as necessary in my factories as necessary for a valid record of the class I care about. If that means that FactoryGirl.create :user makes 15 associated records, I really don't care very much. I'll use as many levels of association in my factories as I have to in order to make my test code easy to read.

Do you do otherwise? If so, how do you make it convenient?

I think as long as you're only doing the minimal work needed to create a valid factory,

That's my practice. In a complex application, that can still be a lot of work, but it's necessary.

you won't run into what equates to a 'god function' (the equivalent of the god model antipattern).

What do you consider to be a "god function" here?

from betterspecs.

@redbar0n:

I would also want at VCR-like gem that would allow a non-developer to click through a use case in the app web interface, and store that as a test case which then capybara or similar could run.

This is not the topic for that, but: maybe https://www.selenium.dev/selenium-ide/ ? Or Cucumber with a rich set of step definitions?

from betterspecs.

@redbar0n:

I would also want at VCR-like gem that would allow a non-developer to click through a use case in the app web interface, and store that as a test case which then capybara or similar could run.

This is not the topic for that, but: maybe https://www.selenium.dev/selenium-ide/ ? Or Cucumber with a rich set of step definitions?

Thanks for the tip. :) I knew about Selenium and Cucumber, but Selenium IDE seems to be close to my idea! Sadly, Selenium IDE is deprecated, but other tools such as applitools.com and katalon.com has taken its place, apparently. Actually Katalon Automation Recorder seems to be exactly what I was thinking about! <3

from betterspecs.