Migrating from non-P4 Server data management systems

This page describes migrating DM level data from non-Perforce based systems into Perforce. This guide will outline the strategies and steps needed to migrate your designs from 3rd-party tools to Perforce IPLM + Perforce. 

Migration concepts

There are two distinct sets of tasks that need to be performed in order to migrate to Perforce IPLM/P4. The first task is to migrate the data itself - the various files and versions of these files from the source data management system to an equivalent structure in Perforce. This step is called 'Data Migration'. The second step is to set up IPs, create releases and generate the IP (or Project) hierarchy in Perforce IPLM. This step is called 'Hierarchy Migration'.

Depending on how much history users care about when moving data from source DM tool to Perforce, this part of the migration can be simple or complex. However, putting together the hierarchies and dependencies usually requires some custom scripting because there are almost no standards that can be used. 

Perforce IPLM terminology mapping

Perforce IPLM is a hierarchical IP management system.

Blocks -> IPs

The base object in Perforce IPLM is an IP. An IP can refer to other IPs (also known as resources) to create hierarchical IPs. Projects are also hierarchical IPs in Perforce IPLM, so migrating a project is just the same as creating any hierarchical IP.

The key fields tracked by Perforce IPLM IPs are:

• The IP name
• An IP Library: An IP Library in Perforce IPLM is like a 'folder' that contains IPs. Each IP belongs to one and only one IP Library. These Libraries need to be pre-created in Perforce IPLM.
• A Perforce directory: Unless an IP is of type 'container', it needs a repository path location in Perforce.
• Description: A text based description of the IP.
• Version Message: Each IP that is created starts at version '0', and each version requires a text based version message.
• Resources (Optional): Other IPs that are needed. This creates a hierarchical IP.

In the simplest terms, each block in a project or design maps to a Perforce IPLM IP. Projects can be built up in a hierarchical fashion after blocks are mapped to individual IPs, and relevant hierarchies are created from these IPs.

Tags -> releases

Each special 'Tag' or 'Milestone' in a block's lifecycle can be mapped to an equivalent Perforce IPLM Release ( see: IP Versions (Releases)). There are some inherent differences between how a Release is created in Perforce IPLM Vs. a release in a 3rd party system, but conceptually they are the same

Branches -> lines

Branches can be generally mapped to Lines in Perforce IPLM (see IP Lines (Branches)). Note that Lines are generally treated as long-lived variants of a block. These variants are created for specific reasons, for example to modify the block to fit the needs of a different project. Lines are not designed to be individual 'work-in-progress' branches that will later be merged back onto trunk. Instead, Perforce IPLM has a Trunk based workflow. Please refer to Perforce' white paper on Hardware Design concepts for more details on this topic.

Project -> hierarchical IP 

Even a project is essentially an IP in Perforce IPLM, albeit one that has other IPs as 'resources' (see IP Hierarchy). These resources can themselves have other IPs are resources, resulting in a hierarchical tree of arbitrary depth and width. Projects are treated as IPs by design, because today's project is tomorrow's IP.

Data migration

Perforce has several resources to help users with Data Migration. The Perforce Migration Center is a great place to start. Another specific document that helps understand the migration strategy is Baseline And Branch Import.

Directory structure considerations

One of the key decision points here is the directory structure used in the Perforce repository. Perforce is a 'Mono Repository' data management system, which means that all the data from all projects reside in one repository. It does not support 'external' repositories unlike other some other DM tools.

When planning the repository structure the following points should be considered:

• Simplicity: The flatter the directory structure, the easier it is to import data and create new projects
• Flexibility: Allows for new projects and needs to be accommodated without having to modify existing directory structures. 
• Permissions: If it is important to restrict access to certain parts of the repository, then those should be unified at a low level

 An example directory structure is shown below:

//depot
	/protected
		/common/<block_name>/TRUNK
			/files...
		/PROJA/<block_name>/TRUNK
			/files...
		/PROJB/<block_name>/TRUNK
			/files...
		/...
	/common
		/<block_name>/TRUNK
			/files...
		/<block_name>/TRUNK
			/files...
		/...
	/projects
		/PROJA
			/<block_name>/TRUNK
				/files...
			/<block_name>/TRUNK
				/files...
		/PROJB
		/...

In the directory structure shown above, new projects can be added quite easily under the projects section. However, all restricted data stays in the 'protected' area, under relevant projects there if needed. Also, common blocks that are designed to be used by multiple different projects, that have central teams working on them, for example, are put into a common area.

The last three sections of each IP block's repository path should correspond to the IP Library name, IP name, and Line name of the IP in Perforce IPLM. Because these names are unique using this convention ensures that there won't be any overlap between IP repository path locations. See the 'repo_path' field on the Creating New IPs page for additional information on repo_path naming. By default repo_path uniqueness is enforced for all IPs managed in Perforce IPLM, in order to avoid security issues that might crop up from allowing overlapping IP repo paths. See the Neo4j Configuration page for details. Note that any arbitrary path can be used for an IP's repo_path (Perforce IPLM supports it technically) but it is best practice to follow the above recommendation.

Migrating only top of tree

The simplest migration path is to only migrate the latest snapshot of each project into Perforce. Naturally, this will erase all the history of these projects - all the various commits and associated meta-data etc, so it might not be suitable in most cases. However, there might be a subset of committed projects where the history is not important.

Migrating key 'labels' or 'tags' (releases)

This is the most common scenario of data migration. Here, instead of only moving the latest snapshot of the data, or replaying every single commit from source to Perforce, users can pick certain key points in each project's life, and create Perforce equivalent commits that reflect those points. All the history between key points is not migrated, but it might not be necessary either. This method provides the best return on migration time investment.

Replaying every commit 

In almost every case it is not practical to migrate every commit in the source DM system into Perforce. Some DM systems don't capture accurate enough logs to truly recreate all the details of their histories, and even in cases where they do it would be a massive investment to migrate every commit. Migrating key tags, or only the top of tree are generally much better options.

Migrating data with Perforce IPLM

As noted above, each block in the design maps to a Perforce IPLM IP. A convenient method to migrate a block into Perforce IPLM is to first create an equivalent IP in Perforce IPLM, pointing to the location in Perforce where the migrated data from the block will reside. The following steps will describe a sample migration of a block - titled ADC - into a Perforce IPLM IP named 'analog.ADC'. Note that the name indicates that the IP 'ADC' belongs to the 'analog' Library, which needs to be pre-created. Let's also assume that the 'ADC' is a common block, not specific to any project.

In this case, the following steps would apply:

Create the new IP (see: Creating New IPs)

1. Create a new IP in Perforce IPLM ('pi ip add analog.ADC' from CLI). Note that this step actually creates the first version or 'Release' of this IP, referred to as 'analog.ADC@0.TRUNK'. This IP is currently empty since it points to an empty location in Perforce.
   
   1. Set the dm_type of this IP to be 'P4'
   2. Set the repo_path of the IP to be the target location in Perforce. Based on our directory structure above, that might translate to '//depot/common/analog/ADC/TRUNK'
   3. Add description, version message etc. as needed.
   4. There is no need to specify any 'resources', since this block is a stand-alone, leaf level IP. Translating IP Hierarchies will be addressed later.

Load the new IP Workspace (see: Loading a Workspace)

1. Create a new workspace from this block ('pi ip load analog.ADC /path/to/temp/ws/location')
   1. If you have IPLM Cache enabled (see IP Caching, you will need to run 'pi ip local' in your workspace after it is created. This makes the IP local to the workspace - i.e. syncs the contents of the repo path into the workspace directly

Load the Data from the Legacy System onto Disk

1. Check out project data from the legacy source DM into a temporary directory

Transfer Data from the Legacy system to Perforce IPLM

1. At this point the user can decide whether they wish to move only the latest top-of-tree, or some key tags/labels.
   1. If only the top of tree is to be moved
      1. Run a unix copy of the data from the temporary directory into the 'analog.ADC' directory in the Perforce IPLM workspace - 'cp -r /path/to/temp/source/dir /path/to/temp/pi/dir/analog.ADC'.
      2. Run 'p4 reconcile' to add all these copied files to Perforce. These files will go into the location specified in the IP (//depot/common/analog/ADC/TRUNK)
      3. Run 'p4 submit -d "Some Message"' to submit these files to the repository
      4. Run 'pi release -d "Release Message" to create a new Perforce IPLM release that now contains the files from the source DM. This action creates 'analog.ADC@1.TRUNK'. 
   2. If specific tags have to be migrated
      1. Update the temporary workspace created in step 1 to the first Tag that needs to be moved
      2. Follow steps to copy, add/update and release the files in Perforce IPLM. Each of these new releases will correspond to the tag from which the release was created. Users might want to copy the tag's description or notes/comments as the release message.
      3. Optionally, if the tag has a specific pattern, for example 'REVX' where X is an integer denoting some internal revision or milestone, that can also be migrated to Perforce IPLM in the form of an ALIAS. Each release in Perforce IPLM can be aliased to a text string. We recommend that these aliases should be kept short - only the variable part of the alias pattern should be used as much as possible. Setting an alias can be achieved by running 'pi ip aliasset analog.ADC@VER.TRUNK REV1', where VER is the release version just generated, and 'REV1' is an example of a tag's variable component.

Hierarchy migration

Once individual blocks have been migrated into Perforce IPLM + P4, these blocks need to be assembled into project and subsystem hierarchies by recreating the existing hierarchy using the Perforce IPLM 'resources' field. See the IP Hierarchy page for details.