Development
Phase2
April 17, 2019
We’ve written a lot about content migration on our blog here—it’s something we have more than a passing interest in, because we do it a lot! The posts below cover the project management, estimation, and basics of content migration from Drupal to Drupal, and other sources too.
If you’ve been following along with this series, you will have a lot of good information at your fingertips. (If you haven’t, I highly recommend you do so now. We’re building on their foundation here.)
If you’ve tried to implement the code samples, you might even have a functional migration!
But what if you don’t? What if you’re getting errors, or unexpected results, or just… nothing at all? Well, that’s what this post is all about. What do you do, when what you did isn’t doing what you thought it was going to do?
Here are some tips, tricks, and starters for figuring out what went wrong.
Drupal 8’s migrate system is very responsible. When you create and first run a migration, it adds two new tables to your database: migrate_map_[migration_id] and migrate_message_[migration_id].
The migrate_map_ tables are comparison tables. They store three items of interest:
The purpose of storing this information is two-fold. First, it allows the MigrationLookup plugin to do its thing, associating old references to migrated content at the new ID. Second, and more relevant for this post, the Status field is a handy reference for you to quickly determine if a given item of content successfully made the transition to the new system. Imported is 0, Ignored is 2, Failed is 3.
What’s the difference between ignored and failed? Ignored means that something in your migration told Drupal to skip that row. Usually, that’s the SkipOnEmpty or SkipRowIfNotSet plugins; some other process plugins will call SkipOnEmpty, as part of their own working. This is usually intentional and not a reason to worry.
Failed means that Drupal tried, and it didn’t work. This is often because a necessary field isn’t set, or because there was a PHP error or other problems along the way.
Failures are where the migrate_message_ tables come in. In most cases, when Drupal records a failed migration row, it will also provide a message giving you a clue as to why that happened. The two tables are cross-referenced by source_ids_hash. In the event of a failed migration, migrate_message is the first stop in your diagnosis.
For example, a common one when dealing with files is
File public://example_file.pdf' does not exist
Obviously, this is a simple fix—either replace the file, or fix the file path in the source data. Look at what Drupal’s telling you here, and see if it’s something you can easily fix. If nothing jumps out at you, keep reading.
Why isn’t the change you made in your migration.yml file working? Well, Drupal probably doesn’t know about it. By default, Drupal imports a module’s configuration files only when the module is initially enabled. The ‘active’ configuration lives in the database, so changes you make to your migration module’s YML files are not registered by Drupal.
In order to overcome this, you have to run a configuration import in drush. It looks like this:
drush cim --partial --source=modules/custom/example_migration/config/install/
Of course, typing that out every time you make a small change in your config is a hassle, so script it! A good migration script will stop the migration, reset it, reimport the configuration, roll the data import back, and run it again. Having a script like that will save you the hassle of typing the same five drush commands over and over, and possibly forgetting to import config or reset your migration. You can find ours in our D8 Examples repository. Run it from the command line and relax, secure in the knowledge that your scripts are using the most current configuration.1
This tip is a little obscure, but useful. When you are migrating from an XML source, there are a whole bunch of things you have to specify in your source config. Notably, you have to specify the fields you will be using from the XML. The details of how this works are spelled out in a previous blog post, Migrating to Drupal From Alternate Sources, linked at the start of this article.
example_xml_migrate/config/install/migrate_plus.migration.example_xml_articles.yml
id: example_xml_articles
label: 'Import articles'
status: true
source:
plugin: url
data_fetcher_plugin: http
urls: 'https://www.phase2technology.com/ideas/rss.xml'
data_parser_plugin: simple_xml
item_selector: /rss/channel/item
fields:
-
name: guid
label: GUID
selector: guid
-
name: title
label: Title
selector: title
-
name: pub_date
label: 'Publication date'
selector: pubDate
The migrate_plus module gives us the ability to create migration groups, which allow us to consolidate configuration. That’s covered in detail in our blog post Drupal 8 Migrations: Taxonomy and Nodes, also linked above.
These two ideas combine together pretty nicely: You can in fact call out common fields in your XML by putting them in a migration group, and then use those fields in multiple migrations.
example_xml_migrate/config/install/migrate_plus.migration_group.example_xml_group.yml
id: example_xml_group
label: General Content Imports
description: Common configuration for node migrations from XML.
source_type: XML File
shared_configuration:
source:
plugin: url
data_fetcher_plugin: http
urls: 'https://www.phase2technology.com/ideas/rss.xml'
data_parser_plugin: simple_xml
item_selector: /rss/channel/item
fields:
1:
name: guid
label: GUID
selector: guid
2:
name: title
label: Title
selector: title
3:
name: pub_date
label: 'Publication date'
selector: pubDate
However—there’s a catch. Note that, unlike any other YML file shown in this series so far, the fields here have numerical array keys. This is because, if you specify a fields section in your individual migrations as well, and both arrays are keyed with the normal -, the migration fields section will completely override the group fields section. Keying them by number in both YML files allows them to be additive. Just make sure that your group and migration keys don’t collide.
Weirdly, the process section of migration config seems to be additive already; you can specify process plugins in both the group and the migration. The migration will only override duplicates on the field level, not the whole shebang.
This is the big one, the lynchpin of debugging a migration. I’m not going to tell you how to set up XDebug with your IDE and dev environment. Let’s face it, there are a berjillion different dev environments, and every one of them is set up differently. Fortunately, there seem to be two berjillion tutorials on making XDebug work with whatever your dev setup is. So, go figure out how to make that part of things work, then you can start looking for issues in the code.2
OK, got that part? Good.
A good IDE will allow you to set “breakpoints”. Wikipedia defines them thus:
...a breakpoint is an intentional stopping or pausing place in a program, put in place for debugging purposes.
When XDebug is enabled, and the IDE is ‘listening’ to your server, the execution of the code will halt at the breakpoint(s), and you should get some tools to examine the state of variables in that moment. In this case, we’ll be setting breakpoints in a few key files in the migration process.
First up, the main executable file in migrate module: core/modules/migrate/src/MigrateExecutable.php. This class file has the massively important import() method.
The import() method is the spider in the center of the web of a migration. It calls tons of other methods as it checks requirements, gets the source data, gets the destination configuration, and loops through the data to create & save new content in the target environment.3
MigrateExecutable.php also defines a bunch of exception error message. This is the source of many of the messages seen in migrate_message_ tables, as well as command line errors when running migrations with Drush. The messages can also be a good way to figure out where to set breakpoints - track down the error message, then backtrack to the try statement that’s associated with that exception’s catch.
When you’re searching through the code, bear in mind that Drupal does a lot of string substitution in error messages, like so: 'Migration @id did not meet the requirements. @message @requirements'. Make sure you edit your search terms to exclude things that are specific to your situation, like the migration @id.
Drupal core defines a lot of source plugins—one for pretty much every entity type present in Drupal 6 and 7, in fact, plus revisions and translations. And, confusingly, these files are not stored in the migrate or migrate_drupal modules. Source plugins are stored in the folder of the module that defines the entity type. For example, the source plugins for node entities are at drupal/core/modules/node/src/Plugin/migrate/source.
Fortunately, Drupal 8 is object oriented, which means that each of these plugins will have the same base set of methods in them. The most important is prepareRow().
The migrate_plus module also provides a URL source plugin, which is used for XML, JSON, and RSS imports. It’s substantially more abstracted than the core DB-based entity plugins. In addition to the URL source, it makes use of data fetchers, which grab the data from either a file or an http request, and data parsers, which are responsible for reading and understanding the format of the data. They do not directly invoke the prepareRow() method; instead, your debugging will likely need to poke into the data parsers.
The code that does the work of formatting and saving entities is substantially more abstracted than the source plugins, because all entity types in Drupal 8 are structurally the same. This particular functionality is pretty battle-hardened, so it’s unlikely that your issues will be here. That said, if you do have need of debugging it, start with drupal/core/modules/migrate/src/Plugin/migrate/destination/Entity.php.
Migration’s a pretty involved process, with a lot of moving parts. If you’ve tried all of this, and nothing is making it any better, well, it might be time to seek help. The #migration channel on Drupal Slack is a great place to start. I can be found there as @srjosh.
If you’ve discovered a legitimate bug in the core migrate code, you’ll want to peruse the issue queue. The migrate_plus module has its own issue queue, as well.
If you’re interested in the latest on the efforts to stabilize and improve migrate in Drupal, the Migration Initiative is the group responsible. They can be found at @MigrateDrupal.
If you have a tip, trick, or snippet that just plain makes your migration life easier, please drop it in the comments below. Happy migrating!