solidusio-contrib / solidus_importer Goto Github PK
View Code? Open in Web Editor NEWSolidus importer extension to migrate data from other eCommerce systems
License: BSD 3-Clause "New" or "Revised" License
Solidus importer extension to migrate data from other eCommerce systems
License: BSD 3-Clause "New" or "Revised" License
Due to a recent change users don't have a billing or shipping address anymore. The address is only saved in the user's .addresses
relationship which is not showed anywhere in /admin
.
I think the importer should set the only address for the customer to both shipping and billing address.
Example of CSV files provided in the Shopify documentation are fine for those coming from Shopify or Magento (using shopify_transporter
), but for people coming from other platforms, we should not require to be compliant with that format if we don't use all the columns.
It takes a lot of time and effort to create a CSV file and we need to be sure we are not over-complicating the format to provide an onboarding experience as smooth as possible.
The OptionValue
processor can create an OptionValue without an OptionType. This can lead to problems when querying Variants which are associated with it, resulting in 500 on both Solidus API and Admin (and I'm afraid, more concerning, the Frontend).
This problem has happened quite a few times on a project I'm working on; unfortunately I haven't been able to reproduce on the sandbox yet.
There's an open pull request on Solidus addressing this problem
--
Maybe we could tweak
nil
?A few ideas worth pursuing as part of a future release
Originally part of the roadmap in #5
Two specs are failing: import_spec
and customer_address_spec
. The issue appears to be with the data setup in the specs rather than the underlying code.
I opened PR #90 which I resolves these.
Originally part of the roadmap in #5
Recording the "message" is useful, but perhaps we could make it possible to report the exception to the error reporter in use on the app.
Whether from within the application or console, none of the import works. I used the example file provided by the gem (product.csv). I also used the standard Shopify template (csv). No success.
From with the application this is the error message:
1 error prohibited this record from being saved:
There were problems with the following fields:
File has contents that are not what they are reported to be
The console error message is the following:
[paperclip] Trying to link /Users/georg/Desktop/solidus-importer-example-2.csv to /var/folders/41/yljlyw7s4q54r21bnr_rfby40000gn/T/696d120d4b6ceb12d85c44f8596ed2c420220815-76077-1u85pz2.csv
Command :: file -b --mime '/Users/georg/Desktop/solidus-importer-example-2.csv'
[paperclip] Trying to link /var/folders/41/yljlyw7s4q54r21bnr_rfby40000gn/T/696d120d4b6ceb12d85c44f8596ed2c420220815-76077-1u85pz2.csv to /var/folders/41/yljlyw7s4q54r21bnr_rfby40000gn/T/696d120d4b6ceb12d85c44f8596ed2c420220815-76077-1w61p5r.csv
Command :: file -b --mime '/var/folders/41/yljlyw7s4q54r21bnr_rfby40000gn/T/696d120d4b6ceb12d85c44f8596ed2c420220815-76077-1w61p5r.csv'
[paperclip] Content Type Spoof: Filename solidus-importer-example-2.csv (application/csv from Headers, ["text/csv", "text/comma-separated-values"] from Extension), content type discovered from file command: application/csv. See documentation to allow this combination.
[paperclip] Trying to link /var/folders/41/yljlyw7s4q54r21bnr_rfby40000gn/T/696d120d4b6ceb12d85c44f8596ed2c420220815-76077-1u85pz2.csv to /var/folders/41/yljlyw7s4q54r21bnr_rfby40000gn/T/696d120d4b6ceb12d85c44f8596ed2c420220815-76077-idyp0p.csv
Command :: file -b --mime '/var/folders/41/yljlyw7s4q54r21bnr_rfby40000gn/T/696d120d4b6ceb12d85c44f8596ed2c420220815-76077-idyp0p.csv'
[paperclip] Content Type Spoof: Filename solidus-importer-example-2.csv (application/csv from Headers, ["text/csv", "text/comma-separated-values"] from Extension), content type discovered from file command: application/csv. See documentation to allow this combination.
Traceback (most recent call last):
2: from (irb):1
1: from (irb):2:in `rescue in irb_binding'
ActiveRecord::RecordInvalid (Validation failed: File has contents that are not what they are reported to be)
This piece of code that finds the Spree::State
to be saved within the Spree::Address
is picking the wrong state.
This is mainly due to the specs not including enough setup data (or seeding the database altogether).
An example would be if you're trying to import a US address from Florida (FL). The find_by
would return the state named "Florești" with abbr: "FL" instead of Florida.
This is probably an easy fix as the code should look within the single country list of states instead of accessing the Spree::State
model directly.
Hi,
When importing a product, I have used the csv: https://help.shopify.com/en/manual/products/import-export/using-csv#product-csv-file-format
I have a header 'Body (HTML)' as well as 'description' and neither seem to be parsed when importing. They are visible in the details of the import, but not in the product (after import) itself.
ActiveStorage is now the default in Solidus, so it should either be the default for this gem or at least configurable like it is in Solidus.
Paperclip is deprecated but it's also the current default of Solidus, consider doing the same that's done for Spree::Image
Originally posted by @blocknotes in 9b4c90b
DEPRECATION WARNING: Class level methods will no longer inherit scoping from `create` in Rails 6.1. To continue using the scoped relation, pass it into the block directly. To instead access the full set of models, as Rails 6.1 will, use `Spree::Taxonomy.default_scoped`. (called from options at /Users/elia/Code/Nebulab/solidus_importer/lib/solidus_importer/processors/taxon.rb:21)
DEPRECATION WARNING: Class level methods will no longer inherit scoping from `create` in Rails 6.1. To continue using the scoped relation, pass it into the block directly. To instead access the full set of models, as Rails 6.1 will, use `Spree::Taxonomy.default_scoped`. (called from options at /Users/elia/Code/Nebulab/solidus_importer/lib/solidus_importer/processors/taxon.rb:22)
We're currently using Shopify's HQ as the default stock location address, but we'd probably better off not setting an address at all, making the behavior more simple and understandable.
@MinasMazar wdyt?
I am having problems getting this function to work. In my CSV, I have a column Tags (or Type); however, when I import, the taxon is not added to the product. Any insights?
Originally posted by @Mufc2202 in #14 (comment)
Hi @kennyadsl @aldesantis, Bryan asked me to mention this to you. We noticed a potential bug in the order importer where it does not generate payment records to account for shipment costs. The result was imported orders were left in a balance_due payment state.
You'll see in the attached screenshot, after the import of the template, there are only payment records for the line items.
I wrote a patch for our needs but am happy to contribute it upstream.
I imagine this is a extreme edge case but I’ve managed to hit it a few times when I was testing the lib on a client project.
Maybe we could tweak this line a bit so that it checks if the Product exists and, if not, assign context[:product]
to it?
It looks like error messages and status are leaked from one row to the other. In the screenshot below 713 is invalid but 714 is valid.
In the messages section they both show the same error (the one generated by 713).
I think this could be related to this piece of code because the merge!
will keep the error data for future rows.
Processors::Product
finds or initializes a new product based on the @data["Handle"]
. Then, later on, it assigns the slug again to the @data["Handle"]
. This is normally not a problem. In a recent import, however, we came across a handle/slug data issue that was difficult to debug. We think this is due to how Solidus generates valid slugs from invalid input.
Here's how you can reproduce:
Handle
with an invalid character. i.e. arts-&-crafts-box
arts-crafts-box
.)A new product is initialized for arts-&-crafts-box
, then the slug is re-generated to be arts-crafts-box
, and we end up with a ActiveRecord::InvalidRecord
Slug has already been taken
error.
The arts-crafts-box
product is found and loaded. No new product is initialized.
I am not sure what the best solution would be. Perhaps there is a more nuanced problem in Solidus where Product.find_or_initialize_by(slug: slug)
is not the best way to find or initialize a product.
I think at the very least we could validate that the slug input in the CSV would be a valid slug as written. In this example, the first import would complete successfully, but the user would expect their product to be at /product/arts-&-crafts-box
and they might be confused and think their import didn't work. The user's second import would fail, and it would be hard for them to figure out why.
CSVs exported from Excel normally contain a byte-order-marker at the beginning of the file. It manifests as the first column header having invisible characters at the start of it, which will mean that it'll look like "handle"
in your console but won't actually be equal to "handle"
due to the invisible characters.
There's some info on the issue here: https://estl.tech/of-ruby-and-hidden-csv-characters-ef482c679b35
It's kind of annoying to tell clients to open their CSVs in Numbers.app or Google Sheets and export them from there.
We've been working a store that has their data all over the place and have build a bunch of custom processors on top of the existing ones to make everything work. One issue we've seen is that if there is a significant number of failures on any particular type of import, usually that means that someone needs to go follow up in some capacity.
If it were possible to reconstruct the failed rows of the CSV and allowed that to be downloaded, that could really be useful in those failure cases.
I've personally done that with some console-fu, but it would be pretty easy to add a "Download failed rows" button to the import page. Would this enhancement be useful to others?
Update this extension so that it works with Solidus 3.0 by making sure the specs pass.
Right now SolidusImporter::ProcessImport
is reading the CSV file by doing a File.read
on the import file path.
def scan
data = CSV.parse(
File.read(@import.file.path),
headers: true,
encoding: 'UTF-8',
header_converters: ->(h) { h.strip }
)
prepare_rows(data)
end
However, on some projects that path may not be available depending on which customizations have been applied to Paperclip or ActiveStorage.
On a project I'm working on the files are sent directly to AWS, even on development, and their bucket path is interpolated in such a way that the @import.file
local path can't be determined.
Therefore it would be useful to allow user-customizable strategies for reading the content from the file. E.g.:
SolidusImporter.config do |c|
c.import_file_reader = -> (import) {
open(@import.file.public_url)
}
end
def scan
raw_content = SolidusImporter::Config.import_file_reader.call(@import)
data = CSV.parse(
raw_content,
headers: true,
encoding: 'UTF-8',
header_converters: ->(h) { h.strip }
)
prepare_rows(data)
end
Or any other approach may also be viable as long as we allow this to me easily customizable.
Originally part of the roadmap in #5
This will put the great work that has been done into test and help in smoothing out any rough edge that we might otherwise miss.
Originally part of the roadmap in #5
Hi did you plan to support solidus v3 ?
When i run bundle install
, i get this error :
Fetching gem metadata from https://rubygems.org/.........
Resolving dependencies...
Could not find compatible versions
Because every version of solidus_importer depends on solidus_core >= 2.0.0, < 3
and Gemfile depends on solidus_core = 3.4.3,
every version of solidus_importer is forbidden.
So, because Gemfile depends on solidus_importer >= 0,
version solving has failed.
ruby -v
ruby 2.7.3p183 (2021-04-05 revision 6847ee089d) [arm64-darwin21]
and gemfile:
source "https://rubygems.org"
git_source(:github) { |repo| "https://github.com/#{repo}.git" }
ruby "2.7.3"
# Bundle edge Rails instead: gem "rails", github: "rails/rails", branch: "main"
gem "rails", "~> 7.0.6"
# The original asset pipeline for Rails [https://github.com/rails/sprockets-rails]
gem "sprockets-rails"
# Use sqlite3 as the database for Active Record
gem "sqlite3", "~> 1.4"
# Use the Puma web server [https://github.com/puma/puma]
gem "puma", "~> 5.0"
# Use JavaScript with ESM import maps [https://github.com/rails/importmap-rails]
gem "importmap-rails"
# Hotwire's SPA-like page accelerator [https://turbo.hotwired.dev]
gem "turbo-rails"
# Hotwire's modest JavaScript framework [https://stimulus.hotwired.dev]
gem "stimulus-rails"
# Build JSON APIs with ease [https://github.com/rails/jbuilder]
gem "jbuilder"
# Use Redis adapter to run Action Cable in production
gem "redis", "~> 4.0"
gem "connection_pool"
# Use Kredis to get higher-level data types in Redis [https://github.com/rails/kredis]
# gem "kredis"
# Use Active Model has_secure_password [https://guides.rubyonrails.org/active_model_basics.html#securepassword]
# gem "bcrypt", "~> 3.1.7"
# Windows does not include zoneinfo files, so bundle the tzinfo-data gem
gem "tzinfo-data", platforms: %i[ mingw mswin x64_mingw jruby ]
# Reduces boot times through caching; required in config/boot.rb
gem "bootsnap", require: false
# Use Sass to process CSS
# gem "sassc-rails"
# Use Active Storage variants [https://guides.rubyonrails.org/active_storage_overview.html#transforming-images]
# gem "image_processing", "~> 1.2"
group :development, :test do
# See https://guides.rubyonrails.org/debugging_rails_applications.html#debugging-with-the-debug-gem
gem "debug", platforms: %i[ mri mingw x64_mingw ]
end
group :development do
# Use console on exceptions pages [https://github.com/rails/web-console]
gem "web-console"
# Add speed badges [https://github.com/MiniProfiler/rack-mini-profiler]
# gem "rack-mini-profiler"
# Speed up commands on slow machines / big apps [https://github.com/rails/spring]
# gem "spring"
gem 'letter_opener', '~> 1.8.1'
end
# FIXME: Please remove this line if `solidus_auth_devise` appears anywhere else in the gemfile
# or replace it with a simple `gem 'solidus_auth_devise'` otherwise.
gem 'solidus_auth_devise' unless File.read(__FILE__).lines[__LINE__..-1].grep(/solidus_auth_devise/).any?
gem "solidus_auth_devise", "~> 2.5"
gem "solidus_core", "3.4.3"
gem "solidus_backend", "3.4.3"
gem "solidus_api", "3.4.3"
gem "solidus_sample", "3.4.3"
gem "canonical-rails"
gem "solidus_support"
gem "truncate_html"
gem "view_component", "~> 2.46"
group :test do
gem "capybara"
gem "selenium-webdriver"
gem "webdrivers"
gem "capybara-screenshot", "~> 1.0"
gem "database_cleaner", "~> 1.7"
end
group :development, :test do
gem "rspec-rails"
gem "rails-controller-testing", "~> 1.0.5"
gem "rspec-activemodel-mocks", "~> 1.1.0"
gem "factory_bot", ">= 4.8"
gem "factory_bot_rails"
gem "ffaker", "~> 2.13"
gem "rubocop", "~> 1.0"
gem "rubocop-performance", "~> 1.5"
gem "rubocop-rails", "~> 2.3"
gem "rubocop-rspec", "~> 2.0"
end
group :production, :staging do
gem 'pg'
end
# additional project gems
gem 'solidus_i18n'
gem 'rails-i18n'
gem 'kaminari-i18n'
gem "aws-sdk-s3"
gem 'jquery-ui-rails'
gem "font-awesome-sass", "~> 6.2.1"
gem 'sitemap_generator', require: false
gem 'whenever', require: false
gem 'solidus_importer'
Thanks
This bug was first discovered by @aamyot
SolidusImporter::Processors::Shipment creates only one InventoryUnit
per LineItem
, when it should, in fact, create InventoryUnits equivalent to the LineItem amount.
Here's a suggestion:
def inventory_units
sku = line_items_attributes[:sku]
return [] if sku.blank?
quantity = line_items_attributes[:quantity].to_i
[{ sku: sku }] * quantity
end
Currently, when importing a product that has a specific option type, all existing products associated with that option type are touched.
This is happening because we are doing these operations at each product import:
solidus_importer/lib/solidus_importer/processors/option_types.rb
Lines 18 to 23 in c89d279
I think we should try to change the option type only if needed. It is not even clear why we need to update its position, since that position is a global value against other option types and not something specific for the product imported.
Originally part of the roadmap in #5
I was unable to import 2 line items for an Order using the samples included here #59.
I imported the customer, products and then the Order, but only one of the line items will be included in the Order.
I put a byebug
on this line and it is clear that on the second row the line_items_attributes is still empty, instead of containing the previous row's attributes.
To further check you can add the following expectation to spec/features/solidus_importer/import_spec.rb
, ln 172:
it 'imports order with line items' do
import
expect(imported_order.line_items).not_to be_blank
expect(imported_order.line_items.count).to eq 3
end
Which will fail - 3 variants are provided but only 1 LiteItem is created.
Doesn't create an option type when 'Option1 Name' is specified.
This causes the user to be unable to navigate to the product -> images, variants, prices, properties.
NoMethodError in Spree::Admin::ImagesController#index
NoMethodError in Spree::Admin::Variants#index
NoMethodError in Spree::Admin::Prices#index
Showing /usr/share/rvm/gems/ruby-2.7.2/gems/solidus_backend-2.11.5/app/views/spree/admin/variants/_table.html.erb where line #29 raised:
Showing /usr/share/rvm/gems/ruby-2.7.2/gems/solidus_backend-2.11.5/app/views/spree/admin/prices/index.html.erb where line #23 raised:
undefined method `position' for nil:NilClass
def options_text
values = option_values.includes(:option_type).sort_by do |option_value|
option_value.option_type.position
end
solidus_core (2.11.5) app/models/spree/variant.rb:197:in `block in options_text'
NoMethodError in Spree::Admin::ProductProperties#index
Showing /usr/share/rvm/gems/ruby-2.7.2/gems/solidus_backend-2.11.5/app/views/spree/admin/product_properties/index.html.erb where line #53 raised:
undefined method `presentation' for nil:NilClass
<% @option_types.each do |option_type, option_values| %>
<div class="field">
<%= label :option_type_presentation, option_type.presentation %>
<%= select_tag "ovi[]", options_from_collection_for_select(option_values, :id, :presentation, params[:ovi]), class: 'custom-select fullwidth', include_blank: true, id: "#{option_type.name}_option_type_select" %>
</div>
<% end %>
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.