Magento integrations using Xtento

One of the most common tasks I have to deal with in Magento are integrations. Almost any system I worked with needs to push order or product data to an ERP or marketing software, or get stock/product info/tracking numbers from an external source. Every integration is unique in a few ways…

the communication channel. REST, SOAP, sending files via SFTP are common variations
the data format. JSON, XML, CSV to name a few
the actual data mapping. We either pass fields as-is or combine/transform them.

However, any good integration has a lot in common:

a log must exist so we can refer back to when and how each piece of information was synchronized
errors must be logged, with an alert system so we are aware of any failures
the system should be able to queue data in case the integration is down. We cannot lose info especially when dealing with financial records.
the system must be able to retry failed records
the actual field mapping must be easy to change, ideally without changing code

I have been looking for good solution to build integrations for a while. On the lower end, there is the do it yourself custom shell script. Easy to build, but usually missing key elements, like retries or flexible data mapping. On the higher end, we have full ETL solutions. They tend to be expensive and add new software components to the mix.

Almost accidentally I stumbled upon Xtento’s Import/Export suite – https://www.xtento.com/magento-2-extensions.html . They flew under my radar as I had bad experiences in the past with such extensions and for a while, concluded that the best import/export is the one you built on top of Magento’s default.

Let’s go over the steps involved to export orders from Magento via Xtento. First, one starts with defining destinations. A destination is the place where your orders will go. You have a few options:

on the local server in a specified folder
on a remote server via a SFTP or FTP connection
to an email
to a HTTP server (i.e. REST)
to a web service. Use this for XML-RPC or SOAP integrations
via a custom class, where you can define the communication with your exotic system

So the above options should cover any possible target. One nice thing is that you can have multiple destinations, so you could place the orders on a SFTP but also mail a copy of the files to you for later review.

After defining the destinations, the next step is to define the export profile. Most options are obvious so I will go only over the important ones:

you can choose the entity to export, i.e. orders. Usually each exportable entity corresponds to a specific extension that you need to buy from Xtento.
you can define filters. For example, you can decide that you only want to export “processing” orders, keeping the “pending” ones in queue until they are reviewed
You can define the export mapping as an XSLT template. This was the feature that I was sold on. XSLT templates allow you to be as expressive as you need to. You can use all the standard fields, apply transformations, use all the fields in the order/related tables (your custom ones too). All this with a nice autocomplete interface and a dead-simple way to test the template with an existing order. Once you get the hang of it, you almost never need to refer to the docs/examples, it’s that easy. If you do need help though, https://support.xtento.com/wiki/Magento_2_Extensions:Magento_Order_Export_Module#XSL_Template has you covered.
You can define when your export runs. Do you want to export orders on schedule? How often? Do you export all orders in the last N hours or only what’s new? If the export process is time consuming, you have a CLI command to run it outside the main Magento cron.
You have a flexible manual export option in case you need to replay any of the missed exports, or simply test the process.

Everything comes with logging. You have filterable grids where you can see the exported entries and errors. You also have the option to locally store the export files for later review.

If you need to import entities, Xtento has you covered too, The process is very similar in that you still have sources where you can pull from, profiles you can define, same logging capabilities, a way to map the data. In addition to imports, you have an actions section where you can define what happens when an entry is imported. For example, when you import a tracking number, you can have Xtento ship and bill the order automatically.

I should mention that currently Xtento does not offer a product import solution. You can import stocks, but not product data. I’d love to see that on their offer sometimes.

What I really like about the extensions is that they are developer-friendly. Almost everything in their system has a fallback to a custom class. You have a very exotic export destination? You can define a class to implement the communication logic. Need to map your data in a way the XSLT template does not support? You can define a class and a method just for that. Finally, having logs for all operations make it easy to identify random issues. It scales ok too, I have been exporting importing 100k records per day with no performance issues.

Here are a few usecases where I have used Xtento successfully, usually without writing any line of code:

product exports to marketplaces/product aggregators. I still prefer specific extensions for big marketplaces, like Amazon or Google Shopping, but will use Xtento for the ones that have poor extensions or none at all.
pushing order data to ERPs and getting back tracking info, billing and shipping automatically. That’s a huge one, before Xtento I used to spend a lot of time on these type of implementations.
pushing order and product data to marketing software, like mailing list managers.
importing stocks from an external source (usually ERP).

Xtento might be lacking all the bells and whistles of an ETL solution, but in an ecosystem where not everyone has a fortune to spend on an integration, their extension suite is more than decent to get things done.

Algolia and Magento 2 – a perfect match

Magento 2 has never been ultra-fast. Even with careful customisation and not a very large catalog, it still takes somewhere near 2s to render the average category view page. This slowness is usually made better by the full page cache, but navigation and filtering patterns are too different to hope that in a browsing session, a user will hit only cached pages.

Now, 1s-2s server-side generation time is not too bad. But more and more merchants expect an almost-instant browsing experience. Magento is hard at work with the PWA implementation, which should deliver a very smooth experience, but that’s still in progress. And when it’s done, time will be needed for the extension vendors to catch up. In my opinion, all the approaches for Magento PWA, either official or community based, call for a big implementation budget.

And here comes Algolia. Stop now and go here, check the load speed as you filter through the left-hand navigation. Almost instant.

Algolia is a search service. They integrate with Magento via an open source module. Basically, it will replace all your search and category pages with an empty catalog page that at load time, will do a client-side call to Algolia, get a json-formatted list of products, then use html templates that are under your control to render the results. So the initial page load time includes Magento bootstrapping (which should be under 500ms), plus a few hundred milliseconds for the Algolia call. Even better, the navigation is ajax-based, so any further filtering is almost instant. Of course, they also have very relevant (and customisable) search results.

That’s a huge selling point for me. While I still have to deal with performance issues on product detail pages, cart and checkout, using Algolia makes catalog navigation ultra-fast. And it takes a lot of load from the server, since we don’t process the catalog there.

All this sounds great, but there are a few reasons for which you may not be able to use it.

It is a paid service. Their pricing scheme is good though, I am almost positive the increase in conversion rate will pay off. Not to mention the countless hours you’d spend in trying to optimize the catalog page, if you get to that point.
It is a SaaS. You are fully dependent on them. When I was working on a project, my free trial ended and the site instantly became unusable. That got me thinking that an Algolia outage will break my site. But they look well funded and have a great uptime.
If you have an existing project, it’s way harder to move to Algolia. Since they replace all templates on the catalog pages, any skinning/custom features will be gone. Of course you can re-do them using their templates, but it will mean a lot more than just installing the Algolia module. Don’t expect any community extension that touches the category page to work out of the box either.

I also had to customise Algolia a bit and luckily, there are options. Roughly, the module works by creating a product indexer that pushes data to Algolia. Via the admin, you get to decide what attributes are searchable, filterable etc. All changes to products will be pushed at indexing time. When rendering the results, you have access to the json and can render them as you wish using html and a proprietary markup language.

Overall, it’s very easy to change CSS styling and add more attributes or custom logic, but you should be aware that at render time, you don’t have access to the product collection, so you cannot call Magento for any logic. Or well, I guess you could, but it defeats the purpose of using Algolia. The only approach should be to send any data you need at index time (even precalculate some attributes if needed), then simply display them in the templates. Algolia has you covered for default more complex features, like different pricing per customer group, but I can see how things could get complicated if you have custom logic in displaying data based on the current customer’s session. Still doable though.

All in all, Algolia is in my top 3 recommendations for a new Magento 2 project. It’s also in the top 3 recommendations for websites that have performance issues. I do hope that Magento will provide a similar experience based on elasticsearch once PWA is finalised, but till then Algolia is one of the best integrations you can have for your store.

A few gotchas for the Magento 2 product import

Six-seven years ago when I started with Magento, the import feature was lacking. While it improved, it still has quite a few issues even today. Mostly every import task I worked on was delayed due to unexpected core bugs. Extensions that promise to help with this come with their own set of issues. There is hope with the https://github.com/magento-engcom/import-export-improvements project, but till that’s done we have to get by.

First, the imports do not have a programmatic interface you can hook into, i.e. pass an array with data and get it imported. Luckly, Firegento fixed this with https://github.com/firegento/FireGento_FastSimpleImport2 . The module is just a simple wrapper over the standard import, so not many chances to break. Of course there is the standard Magento 2 product API, but it’s pretty useless for anything else than a few products due to slowness.

The second thing I ran into is that tier price imports are a special entity, i.e. you first import the product data in a format, then import the tier pricing in a totally different one. This is not what a merchant would expect and sadly, not even the Firegento extension has a programmatic interface for tiers.

Then another gotcha is that importing empty values does nothing by default. So if you set a product color, then decide to unset it by importing an empty value, it will still keep the same value as before. Similarly, if you import category associations and want to remove an older association, you are out of luck, the old link remains. This is almost a “feature” in Magento – https://github.com/magento/magento2/issues/7930 . The proposed fix is to use the “Replace” behaviour of imports, but that creates another issue – with “Replace”, product data is wiped out and recreated. Not something you want todo if you care about stats/old orders since all the product IDs will change.

Another nuisance is that all custom attributes need to be crammed in a single csv field like:

adhesive=,application=,application_description=,available_lengths=,available_widths=,color_group=,colors=,durability=,item_size=,part_name=,print_compatibility=,product_tagline=,release_liner=,thicknes=

For most of my clients, additional attributes is where the bulk of the data lives, so having them formatted as above is clunky.

For the good parts, the import is pretty fast, it can import images correctly and once you get the format right, most of the stuff works.

The conclusion? If you’re working on a import task, pad your estimates substantially, you will need the time to go through a various set of quirks you never thought of. By the way, all of the above refers strictly to the simple products, other product types might come with their own set of problems.

Using a composer library in Magento 1, the quick way

Today I had to generate native XLS files from Magento 1. The https://github.com/PHPOffice/PhpSpreadsheet is the best library for the job, but it’s modern in that it uses composer and PSR. Since I really wanted to use this specific package, here is a quick way to include it in the now dated Magento 1 codebase:

First, create an empty directory where the library will be kept:

mkdir lib/PhpSpreadsheet
cd PhpSpreadsheet

Then, create a composer.json file:

#lib/PhpSpreadsheet/composer.json
{
  "require": {
    "phpoffice/phpspreadsheet": "~1.3"
  }
}

Then, still in lib/PhpSpreadsheet, let composer download the needed files:

composer install

To actually use the library, we will have to tell Magento how to include the directory in the path and also require the generated composer autoloader. In your helper/model/controller, add a init method to do just that:

protected function initPhpSpreadSheetLib() {
  //Add library directory to include path
  set_include_path(get_include_path() . PATH_SEPARATOR . Mage::getBaseDir('lib') . DS . 'PhpSpreadsheet' . DS . 'vendor');
  //Include composer autoloader
  require_once(Mage::getBaseDir('lib') . DS . 'PhpSpreadsheet' . DS . 'vendor' . DS . 'autoload.php');
}

Then, use the lib. Remember to use fully qualified class names as Magento is not namespace-aware:

$this->initPhpSpreadSheetLib();
$spreadsheet = new \PhpOffice\PhpSpreadsheet\Spreadsheet();
$sheet = $spreadsheet->getActiveSheet();
$sheet->fromArray($data,null,'A1');
$writer = new \PhpOffice\PhpSpreadsheet\Writer\Xlsx($spreadsheet);
$writer->save($tmpFileName);

Probably there are a million other better ways to achieve the same thing, but this looks like a quick solution to use a library without tinkering with the project’s structure too much.

US continental shipping and Magento

A lot of websites in the US offer free ground shipping nowadays. However, most restrict to US continental. State-wise, this means all states except for Alaska an Hawaii. However, the US has a lot of protectorates that do not qualify as continental: American Samoa, all the “Armed Forces”, Micronesia, Guam, Marshall Islands, Mariana Islands, Palau, Puerto Rico, Virgin Islands.

That’s one way to look at it. The other is looking at the zip codes that are not qualified for free shipping. It looks like the continental USA is:

between 00900 and 96200 OR
between 97000 and 99500

Putting all this together as a set of conditions for a Magento shopping cart rule:

If you’re using Magento 1, here is the string you can paste in the salesrule table, conditions_serialized field:

a:7:{s:4:"type";s:32:"salesrule/rule_condition_combine";s:9:"attribute";N;s:8:"operator";N;s:5:"value";s:1:"1";s:18:"is_value_processed";N;s:10:"aggregator";s:3:"all";s:10:"conditions";a:18:{i:0;a:5:{s:4:"type";s:32:"salesrule/rule_condition_address";s:9:"attribute";s:10:"country_id";s:8:"operator";s:2:"==";s:5:"value";s:2:"US";s:18:"is_value_processed";b:0;}i:1;a:5:{s:4:"type";s:32:"salesrule/rule_condition_address";s:9:"attribute";s:9:"region_id";s:8:"operator";s:2:"!=";s:5:"value";s:1:"2";s:18:"is_value_processed";b:0;}i:2;a:5:{s:4:"type";s:32:"salesrule/rule_condition_address";s:9:"attribute";s:9:"region_id";s:8:"operator";s:2:"!=";s:5:"value";s:2:"21";s:18:"is_value_processed";b:0;}i:3;a:5:{s:4:"type";s:32:"salesrule/rule_condition_address";s:9:"attribute";s:9:"region_id";s:8:"operator";s:2:"!=";s:5:"value";s:1:"3";s:18:"is_value_processed";b:0;}i:4;a:5:{s:4:"type";s:32:"salesrule/rule_condition_address";s:9:"attribute";s:9:"region_id";s:8:"operator";s:2:"!=";s:5:"value";s:1:"6";s:18:"is_value_processed";b:0;}i:5;a:5:{s:4:"type";s:32:"salesrule/rule_condition_address";s:9:"attribute";s:9:"region_id";s:8:"operator";s:2:"!=";s:5:"value";s:1:"7";s:18:"is_value_processed";b:0;}i:6;a:5:{s:4:"type";s:32:"salesrule/rule_condition_address";s:9:"attribute";s:9:"region_id";s:8:"operator";s:2:"!=";s:5:"value";s:1:"8";s:18:"is_value_processed";b:0;}i:7;a:5:{s:4:"type";s:32:"salesrule/rule_condition_address";s:9:"attribute";s:9:"region_id";s:8:"operator";s:2:"!=";s:5:"value";s:1:"9";s:18:"is_value_processed";b:0;}i:8;a:5:{s:4:"type";s:32:"salesrule/rule_condition_address";s:9:"attribute";s:9:"region_id";s:8:"operator";s:2:"!=";s:5:"value";s:2:"10";s:18:"is_value_processed";b:0;}i:9;a:5:{s:4:"type";s:32:"salesrule/rule_condition_address";s:9:"attribute";s:9:"region_id";s:8:"operator";s:2:"!=";s:5:"value";s:2:"11";s:18:"is_value_processed";b:0;}i:10;a:5:{s:4:"type";s:32:"salesrule/rule_condition_address";s:9:"attribute";s:9:"region_id";s:8:"operator";s:2:"!=";s:5:"value";s:2:"17";s:18:"is_value_processed";b:0;}i:11;a:5:{s:4:"type";s:32:"salesrule/rule_condition_address";s:9:"attribute";s:9:"region_id";s:8:"operator";s:2:"!=";s:5:"value";s:2:"20";s:18:"is_value_processed";b:0;}i:12;a:5:{s:4:"type";s:32:"salesrule/rule_condition_address";s:9:"attribute";s:9:"region_id";s:8:"operator";s:2:"!=";s:5:"value";s:2:"30";s:18:"is_value_processed";b:0;}i:13;a:5:{s:4:"type";s:32:"salesrule/rule_condition_address";s:9:"attribute";s:9:"region_id";s:8:"operator";s:2:"!=";s:5:"value";s:2:"46";s:18:"is_value_processed";b:0;}i:14;a:5:{s:4:"type";s:32:"salesrule/rule_condition_address";s:9:"attribute";s:9:"region_id";s:8:"operator";s:2:"!=";s:5:"value";s:2:"50";s:18:"is_value_processed";b:0;}i:15;a:5:{s:4:"type";s:32:"salesrule/rule_condition_address";s:9:"attribute";s:9:"region_id";s:8:"operator";s:2:"!=";s:5:"value";s:2:"52";s:18:"is_value_processed";b:0;}i:16;a:5:{s:4:"type";s:32:"salesrule/rule_condition_address";s:9:"attribute";s:9:"region_id";s:8:"operator";s:2:"!=";s:5:"value";s:2:"60";s:18:"is_value_processed";b:0;}i:17;a:7:{s:4:"type";s:32:"salesrule/rule_condition_combine";s:9:"attribute";N;s:8:"operator";N;s:5:"value";s:1:"1";s:18:"is_value_processed";N;s:10:"aggregator";s:3:"any";s:10:"conditions";a:2:{i:0;a:7:{s:4:"type";s:32:"salesrule/rule_condition_combine";s:9:"attribute";N;s:8:"operator";N;s:5:"value";s:1:"1";s:18:"is_value_processed";N;s:10:"aggregator";s:3:"all";s:10:"conditions";a:2:{i:0;a:5:{s:4:"type";s:32:"salesrule/rule_condition_address";s:9:"attribute";s:8:"postcode";s:8:"operator";s:1:">";s:5:"value";s:5:"00900";s:18:"is_value_processed";b:0;}i:1;a:5:{s:4:"type";s:32:"salesrule/rule_condition_address";s:9:"attribute";s:8:"postcode";s:8:"operator";s:1:"<";s:5:"value";s:5:"96200";s:18:"is_value_processed";b:0;}}}i:1;a:7:{s:4:"type";s:32:"salesrule/rule_condition_combine";s:9:"attribute";N;s:8:"operator";N;s:5:"value";s:1:"1";s:18:"is_value_processed";N;s:10:"aggregator";s:3:"all";s:10:"conditions";a:2:{i:0;a:5:{s:4:"type";s:32:"salesrule/rule_condition_address";s:9:"attribute";s:8:"postcode";s:8:"operator";s:2:">=";s:5:"value";s:5:"97000";s:18:"is_value_processed";b:0;}i:1;a:5:{s:4:"type";s:32:"salesrule/rule_condition_address";s:9:"attribute";s:8:"postcode";s:8:"operator";s:1:"<";s:5:"value";s:5:"99500";s:18:"is_value_processed";b:0;}}}}}}}

I’m guessing all this changes and may not even be 100% accurate, but it’s the best I could find as far as “free continental shipping” goes.

Thoughts on the Magento 2 Certified Professional exam

On March 1st 2018, the new Magento 2 Certified Professional Developer exam became available. Wanting to keep the tradition of getting my certifications early, I have attempted and passed the exam.

The exam experience is nice and easy as usual. You buy a voucher from https://u.magento.com/magento-2-certified-professional-developer#.WrtdjdNuZsY , then you can schedule an exam at a certified center. The exam has 60 multiple choice questions, a 90 minutes time limit and you get your results on the spot. It’s also possible to take the exam online, did not try it myself but Andreas did a nice write-up on this.

The exam is aimed at developers with about 1.5 years of experience with Magento 2. My opinion is that in order to pass you need to:

Have at least one full year of experience writing Magento 2 code.
Had the occasion to customize most of the Magento 2 areas, with a focus on backend
Are aware of the Magento best practices and apply them through your projects

Compared to the Magento 1 exam, this time a lot more value is put on proposing the correct architecture for a change. A fair number of questions is about doing things “the right way”. Sometimes solutions that work but are not correct are presented and you are expected to choose the ones that respect the best practices.

You can get away with very little CSS and JS knowledge (I guess a fronted developer certification will be coming out soon), but you are expected to know how to setup a theme, how to override templates or how to configure the ui components.

Besides the official guides, I encourage you reading the study guide and attempting the test exam from Swift Otter. The test exam is realistic, although I think it is a bit easier than the real thing. Personally I would have not passed without their guide. Other than that, having one year of hands-on Magento 2 experience is the only way to pass, no courses/magic formula to make up for it.

Overall, I consider this the best Magento 2 exam to date. More than ever, answering the questions correctly proves that you have experience with the platform and can deliver elegant solutions.