solidusio-contrib / solidus_importer Goto Github PK

Due to a recent change users don't have a billing or shipping address anymore. The address is only saved in the user's .addresses relationship which is not showed anywhere in /admin.

I think the importer should set the only address for the customer to both shipping and billing address.

Example of CSV files provided in the Shopify documentation are fine for those coming from Shopify or Magento (using shopify_transporter), but for people coming from other platforms, we should not require to be compliant with that format if we don't use all the columns.

It takes a lot of time and effort to create a CSV file and we need to be sure we are not over-complicating the format to provide an onboarding experience as smooth as possible.

The OptionValue processor can create an OptionValue without an OptionType. This can lead to problems when querying Variants which are associated with it, resulting in 500 on both Solidus API and Admin (and I'm afraid, more concerning, the Frontend).

This problem has happened quite a few times on a project I'm working on; unfortunately I haven't been able to reproduce on the sandbox yet.
There's an open pull request on Solidus addressing this problem

--
Maybe we could tweak

You can see that the financial_status inquiry doesn't actually do anything. I would expect it to be used to create Pending payments and make the status of the Order balance_due.

We should either process it somehow or remove it from the CSV examples to avoid confusion when using the extension 💡

A few ideas worth pursuing as part of a future release

• Be independent from CSV (ex. JSON source)
• Tests with Magento
• Tests with other solutions

Originally part of the roadmap in #5

Two specs are failing: import_spec and customer_address_spec. The issue appears to be with the data setup in the specs rather than the underlying code.

I opened PR #90 which I resolves these.

Originally part of the roadmap in #5

• Main structure (#1)
  • Base extension using dev_tools
  • Models to store the state of an import / every import rows
  • PORO classes with import logic based on plugins
  • Plugins to check, process and log imported data for: customers, products, orders
  • Plugins configuration via initializer
  • Related specs
• Background processing (#2)
  • Active Job integration
  • Single job to process every row of an import
  • Related specs
• Improve structure to simplify some logics (#9)
• Remove unnecessary code for database_cleaner (#10)
• Image processing (#8)
  • Attach the required images to the row target entity
  • Related specs
• Improve processors context removing unused keys and using exceptions (#13)
• Documentation and packaging (#11)
  • README improvements
  • Usage examples (ex. a plugin to notify by email)
  • Gemspec update

Recording the "message" is useful, but perhaps we could make it possible to report the exception to the error reporter in use on the app.

Whether from within the application or console, none of the import works. I used the example file provided by the gem (product.csv). I also used the standard Shopify template (csv). No success.

From with the application this is the error message:

1 error prohibited this record from being saved:
There were problems with the following fields:

File has contents that are not what they are reported to be

The console error message is the following:

[paperclip] Trying to link /Users/georg/Desktop/solidus-importer-example-2.csv to /var/folders/41/yljlyw7s4q54r21bnr_rfby40000gn/T/696d120d4b6ceb12d85c44f8596ed2c420220815-76077-1u85pz2.csv
Command :: file -b --mime '/Users/georg/Desktop/solidus-importer-example-2.csv'
[paperclip] Trying to link /var/folders/41/yljlyw7s4q54r21bnr_rfby40000gn/T/696d120d4b6ceb12d85c44f8596ed2c420220815-76077-1u85pz2.csv to /var/folders/41/yljlyw7s4q54r21bnr_rfby40000gn/T/696d120d4b6ceb12d85c44f8596ed2c420220815-76077-1w61p5r.csv
Command :: file -b --mime '/var/folders/41/yljlyw7s4q54r21bnr_rfby40000gn/T/696d120d4b6ceb12d85c44f8596ed2c420220815-76077-1w61p5r.csv'
[paperclip] Content Type Spoof: Filename solidus-importer-example-2.csv (application/csv from Headers, ["text/csv", "text/comma-separated-values"] from Extension), content type discovered from file command: application/csv. See documentation to allow this combination.
[paperclip] Trying to link /var/folders/41/yljlyw7s4q54r21bnr_rfby40000gn/T/696d120d4b6ceb12d85c44f8596ed2c420220815-76077-1u85pz2.csv to /var/folders/41/yljlyw7s4q54r21bnr_rfby40000gn/T/696d120d4b6ceb12d85c44f8596ed2c420220815-76077-idyp0p.csv
Command :: file -b --mime '/var/folders/41/yljlyw7s4q54r21bnr_rfby40000gn/T/696d120d4b6ceb12d85c44f8596ed2c420220815-76077-idyp0p.csv'
[paperclip] Content Type Spoof: Filename solidus-importer-example-2.csv (application/csv from Headers, ["text/csv", "text/comma-separated-values"] from Extension), content type discovered from file command: application/csv. See documentation to allow this combination.
Traceback (most recent call last):
        2: from (irb):1
        1: from (irb):2:in `rescue in irb_binding'
ActiveRecord::RecordInvalid (Validation failed: File has contents that are not what they are reported to be)

This piece of code that finds the Spree::State to be saved within the Spree::Address is picking the wrong state.

This is mainly due to the specs not including enough setup data (or seeding the database altogether).

An example would be if you're trying to import a US address from Florida (FL). The find_by would return the state named "Florești" with abbr: "FL" instead of Florida.

This is probably an easy fix as the code should look within the single country list of states instead of accessing the Spree::State model directly.

Hi,

When importing a product, I have used the csv: https://help.shopify.com/en/manual/products/import-export/using-csv#product-csv-file-format

I have a header 'Body (HTML)' as well as 'description' and neither seem to be parsed when importing. They are visible in the details of the import, but not in the product (after import) itself.

ActiveStorage is now the default in Solidus, so it should either be the default for this gem or at least configurable like it is in Solidus.

Paperclip is deprecated but it's also the current default of Solidus, consider doing the same that's done for Spree::Image

Originally posted by @blocknotes in 9b4c90b

We're currently using Shopify's HQ as the default stock location address, but we'd probably better off not setting an address at all, making the behavior more simple and understandable.

@MinasMazar wdyt?

    I am having problems getting this function to work. In my CSV, I have a column Tags (or Type); however, when I import, the taxon is not added to the product. Any insights?

Originally posted by @Mufc2202 in #14 (comment)

Hi @kennyadsl @aldesantis, Bryan asked me to mention this to you. We noticed a potential bug in the order importer where it does not generate payment records to account for shipment costs. The result was imported orders were left in a balance_due payment state.

You'll see in the attached screenshot, after the import of the template, there are only payment records for the line items.

I wrote a patch for our needs but am happy to contribute it upstream.

I imagine this is a extreme edge case but I’ve managed to hit it a few times when I was testing the lib on a client project.

• import a products csv with one row for the Master product and 1+ rows for its Variants
• delete the Master Product in the console
• rerun the import. The Master product row will be created, but the Variant rows will fail with Validation failed: Product must exist, Product can't be blank

Maybe we could tweak this line a bit so that it checks if the Product exists and, if not, assign context[:product] to it?

It looks like error messages and status are leaked from one row to the other. In the screenshot below 713 is invalid but 714 is valid.

In the messages section they both show the same error (the one generated by 713).

I think this could be related to this piece of code because the merge! will keep the error data for future rows.

I try to import 2 products, with Size and Color attributes. I check the first product with no problems, the second product can't see variants because it can't load option_type. I think the reason is probably due to the duplicate attribute, so I can't create a new option_type

Processors::Product finds or initializes a new product based on the @data["Handle"]. Then, later on, it assigns the slug again to the @data["Handle"]. This is normally not a problem. In a recent import, however, we came across a handle/slug data issue that was difficult to debug. We think this is due to how Solidus generates valid slugs from invalid input.

Reproduction steps

Here's how you can reproduce:

1. In your product import CSV, add a Handle with an invalid character. i.e. arts-&-crafts-box
2. Run the product import.
3. Check the slug of the created product. (It should be arts-crafts-box.)
4. Run the import again.

Current behaviour

A new product is initialized for arts-&-crafts-box, then the slug is re-generated to be arts-crafts-box, and we end up with a ActiveRecord::InvalidRecord Slug has already been taken error.

Expected behaviour

The arts-crafts-box product is found and loaded. No new product is initialized.

Solution

I am not sure what the best solution would be. Perhaps there is a more nuanced problem in Solidus where Product.find_or_initialize_by(slug: slug) is not the best way to find or initialize a product.

I think at the very least we could validate that the slug input in the CSV would be a valid slug as written. In this example, the first import would complete successfully, but the user would expect their product to be at /product/arts-&-crafts-box and they might be confused and think their import didn't work. The user's second import would fail, and it would be hard for them to figure out why.

CSVs exported from Excel normally contain a byte-order-marker at the beginning of the file. It manifests as the first column header having invisible characters at the start of it, which will mean that it'll look like "handle" in your console but won't actually be equal to "handle" due to the invisible characters.

There's some info on the issue here: https://estl.tech/of-ruby-and-hidden-csv-characters-ef482c679b35

It's kind of annoying to tell clients to open their CSVs in Numbers.app or Google Sheets and export them from there.

We've been working a store that has their data all over the place and have build a bunch of custom processors on top of the existing ones to make everything work. One issue we've seen is that if there is a significant number of failures on any particular type of import, usually that means that someone needs to go follow up in some capacity.

If it were possible to reconstruct the failed rows of the CSV and allowed that to be downloaded, that could really be useful in those failure cases.

I've personally done that with some console-fu, but it would be pretty easy to add a "Download failed rows" button to the import page. Would this enhancement be useful to others?

Update this extension so that it works with Solidus 3.0 by making sure the specs pass.

Right now SolidusImporter::ProcessImport is reading the CSV file by doing a File.read on the import file path.

def scan
      data = CSV.parse(
        File.read(@import.file.path),
        headers: true,
        encoding: 'UTF-8',
        header_converters: ->(h) { h.strip }
      )
      prepare_rows(data)
end

However, on some projects that path may not be available depending on which customizations have been applied to Paperclip or ActiveStorage.
On a project I'm working on the files are sent directly to AWS, even on development, and their bucket path is interpolated in such a way that the @import.file local path can't be determined.

Therefore it would be useful to allow user-customizable strategies for reading the content from the file. E.g.:

SolidusImporter.config do |c|
  c.import_file_reader = -> (import) {
     open(@import.file.public_url)
  }
end

def scan
      raw_content = SolidusImporter::Config.import_file_reader.call(@import)
      data = CSV.parse(
        raw_content,
        headers: true,
        encoding: 'UTF-8',
        header_converters: ->(h) { h.strip }
      )
      prepare_rows(data)
end

Or any other approach may also be viable as long as we allow this to me easily customizable.

• Release version 1.0 (#4)
• Improve structure
  • Replace Paperclip (#6)
• Improve plugins
  • Taxon processor (for products) (#23)
  • Options processor (for products) (#15)
• Admin UI (#3)
  • Import listing
  • Import details with import rows
  • Import row details
  • Pagination and filters to improve search functionalities
  • Basic specs (at least load every new route)
• Improve background processing
  • Attachment processor: transfer the files in separate jobs
  • Process rows in the background to improve concurrency
  • Group the rows by common changes (ex. for products by product slug)
• Miscellaneous
  • Be independent from CSV (ex. JSON source)
  • Tests with Magento
  • Tests with other solutions

• Attachment processor: transfer the files in separate jobs
• Process rows in the background to improve concurrency
• Group the rows by common changes (ex. for products by product slug)

Originally part of the roadmap in #5

This will put the great work that has been done into test and help in smoothing out any rough edge that we might otherwise miss.

Originally part of the roadmap in #5

Hi did you plan to support solidus v3 ?

When i run bundle install , i get this error :

Fetching gem metadata from https://rubygems.org/.........
Resolving dependencies...
Could not find compatible versions

Because every version of solidus_importer depends on solidus_core >= 2.0.0, < 3
  and Gemfile depends on solidus_core = 3.4.3,
  every version of solidus_importer is forbidden.
So, because Gemfile depends on solidus_importer >= 0,
  version solving has failed.

ruby -v
ruby 2.7.3p183 (2021-04-05 revision 6847ee089d) [arm64-darwin21]

and gemfile:

source "https://rubygems.org"
git_source(:github) { |repo| "https://github.com/#{repo}.git" }

ruby "2.7.3"

# Bundle edge Rails instead: gem "rails", github: "rails/rails", branch: "main"
gem "rails", "~> 7.0.6"

# The original asset pipeline for Rails [https://github.com/rails/sprockets-rails]
gem "sprockets-rails"

# Use sqlite3 as the database for Active Record
gem "sqlite3", "~> 1.4"

# Use the Puma web server [https://github.com/puma/puma]
gem "puma", "~> 5.0"

# Use JavaScript with ESM import maps [https://github.com/rails/importmap-rails]
gem "importmap-rails"

# Hotwire's SPA-like page accelerator [https://turbo.hotwired.dev]
gem "turbo-rails"

# Hotwire's modest JavaScript framework [https://stimulus.hotwired.dev]
gem "stimulus-rails"

# Build JSON APIs with ease [https://github.com/rails/jbuilder]
gem "jbuilder"

# Use Redis adapter to run Action Cable in production
gem "redis", "~> 4.0"
gem "connection_pool"

# Use Kredis to get higher-level data types in Redis [https://github.com/rails/kredis]
# gem "kredis"

# Use Active Model has_secure_password [https://guides.rubyonrails.org/active_model_basics.html#securepassword]
# gem "bcrypt", "~> 3.1.7"

# Windows does not include zoneinfo files, so bundle the tzinfo-data gem
gem "tzinfo-data", platforms: %i[ mingw mswin x64_mingw jruby ]

# Reduces boot times through caching; required in config/boot.rb
gem "bootsnap", require: false

# Use Sass to process CSS
# gem "sassc-rails"

# Use Active Storage variants [https://guides.rubyonrails.org/active_storage_overview.html#transforming-images]
# gem "image_processing", "~> 1.2"

group :development, :test do
  # See https://guides.rubyonrails.org/debugging_rails_applications.html#debugging-with-the-debug-gem
  gem "debug", platforms: %i[ mri mingw x64_mingw ]
end

group :development do
  # Use console on exceptions pages [https://github.com/rails/web-console]
  gem "web-console"

  # Add speed badges [https://github.com/MiniProfiler/rack-mini-profiler]
  # gem "rack-mini-profiler"

  # Speed up commands on slow machines / big apps [https://github.com/rails/spring]
  # gem "spring"
  gem 'letter_opener', '~> 1.8.1'
end

# FIXME: Please remove this line if `solidus_auth_devise` appears anywhere else in the gemfile
#        or replace it with a simple `gem 'solidus_auth_devise'` otherwise.
gem 'solidus_auth_devise' unless File.read(__FILE__).lines[__LINE__..-1].grep(/solidus_auth_devise/).any?

gem "solidus_auth_devise", "~> 2.5"

gem "solidus_core", "3.4.3"
gem "solidus_backend", "3.4.3"
gem "solidus_api", "3.4.3"
gem "solidus_sample", "3.4.3"
gem "canonical-rails"
gem "solidus_support"
gem "truncate_html"
gem "view_component", "~> 2.46"

group :test do
  gem "capybara"
  gem "selenium-webdriver"
  gem "webdrivers"
  gem "capybara-screenshot", "~> 1.0"
  gem "database_cleaner", "~> 1.7"
end

group :development, :test do
  gem "rspec-rails"
  gem "rails-controller-testing", "~> 1.0.5"
  gem "rspec-activemodel-mocks", "~> 1.1.0"
  gem "factory_bot", ">= 4.8"
  gem "factory_bot_rails"
  gem "ffaker", "~> 2.13"
  gem "rubocop", "~> 1.0"
  gem "rubocop-performance", "~> 1.5"
  gem "rubocop-rails", "~> 2.3"
  gem "rubocop-rspec", "~> 2.0"
end

group :production, :staging do
  gem 'pg'
end

# additional project gems
gem 'solidus_i18n'
gem 'rails-i18n'
gem 'kaminari-i18n'
gem "aws-sdk-s3"
gem 'jquery-ui-rails'
gem "font-awesome-sass", "~> 6.2.1"
gem 'sitemap_generator', require: false
gem 'whenever', require: false
gem 'solidus_importer'

Thanks

This bug was first discovered by @aamyot

SolidusImporter::Processors::Shipment creates only one InventoryUnit per LineItem, when it should, in fact, create InventoryUnits equivalent to the LineItem amount.

Here's a suggestion:

def inventory_units
    sku = line_items_attributes[:sku]

    return [] if sku.blank?

    quantity = line_items_attributes[:quantity].to_i
    [{ sku: sku }] * quantity
  end

Currently, when importing a product that has a specific option type, all existing products associated with that option type are touched.

This is happening because we are doing these operations at each product import:

I think we should try to change the option type only if needed. It is not even clear why we need to update its position, since that position is a global value against other option types and not something specific for the product imported.

Originally part of the roadmap in #5

I was unable to import 2 line items for an Order using the samples included here #59.
I imported the customer, products and then the Order, but only one of the line items will be included in the Order.

I put a byebug on this line and it is clear that on the second row the line_items_attributes is still empty, instead of containing the previous row's attributes.

To further check you can add the following expectation to spec/features/solidus_importer/import_spec.rb, ln 172:

it 'imports order with line items' do
      import
      expect(imported_order.line_items).not_to be_blank
      expect(imported_order.line_items.count).to eq 3
    end

Which will fail - 3 variants are provided but only 1 LiteItem is created.

Doesn't create an option type when 'Option1 Name' is specified.
This causes the user to be unable to navigate to the product -> images, variants, prices, properties.

NoMethodError in Spree::Admin::ImagesController#index 
NoMethodError in Spree::Admin::Variants#index 
NoMethodError in Spree::Admin::Prices#index 

Showing /usr/share/rvm/gems/ruby-2.7.2/gems/solidus_backend-2.11.5/app/views/spree/admin/variants/_table.html.erb where line #29 raised: 
Showing /usr/share/rvm/gems/ruby-2.7.2/gems/solidus_backend-2.11.5/app/views/spree/admin/prices/index.html.erb where line #23 raised: 
undefined method `position' for nil:NilClass
    def options_text
      values = option_values.includes(:option_type).sort_by do |option_value|
        option_value.option_type.position
      end

solidus_core (2.11.5) app/models/spree/variant.rb:197:in `block in options_text' 

NoMethodError in Spree::Admin::ProductProperties#index 
Showing /usr/share/rvm/gems/ruby-2.7.2/gems/solidus_backend-2.11.5/app/views/spree/admin/product_properties/index.html.erb where line #53 raised: 
undefined method `presentation' for nil:NilClass
      <% @option_types.each do |option_type, option_values| %>
        <div class="field">
          <%= label :option_type_presentation, option_type.presentation %>
          <%= select_tag "ovi[]", options_from_collection_for_select(option_values, :id, :presentation, params[:ovi]), class: 'custom-select fullwidth', include_blank: true, id: "#{option_type.name}_option_type_select" %>
        </div>
      <% end %>