Migrating from P4 Server based data management systems

This document presents a methodology for migrating an existing Perforce database from a 3rd party tool to a new database for Perforce IPLM managed by Perforce. Migration of existing databases from 3rd party tools managed by Perforce to native Perforce is a straightforward operation because the revision control system remains the same - Perforce. Copying and splitting depots is a common situation for which there are tools available. The recommended Perforce methodology for this is to use p4 fetch.

In addition to the Perforce depot, there is a second element which also has to be migrated and retained - the release history. The release history has to be transferred over from the database of the 3rd party tool to the database of Perforce IPLM for all projects. A workspace is built for each release in the 3rd party tool, the same workspace is recreated in the native Perforce configuration and a new release is created in Perforce IPLM for each native Perforce workspace.

Migration Methodology

Assumptions

The migration methodology described in this document is based on the following assumptions:

The 3rd party tool uses single container with multiple IP blocks placed with no hierarchy.
1. Current 3rd party tools, which use Perforce as a backbone revision control system, fit in this requirement - they use a single container with multiple IP blocks placed with no hierarchy.
3rd party tools’ block releases are in the form of a static, read-only branches
1. This assumption is also valid for current 3rd party tools: releases are represented as static, read-only branches in the Perforce database.
Projects are migrated incrementally
1. Due to the large number of projects to move, customers typically prefer to migrate projects incrementally. Migration of the whole database at once is not a prefered choice because of its complexity - the database is large and constantly changing.
Reference 3rd party tools’ libraries are migrated last and are set up as filesystem IP blocks for use by the Perforce projects
1. Any reference 3rd party tools’ libraries are to be migrated last due to the references they contain: if reference libraries are migrated before the data, the references will be broken in the sense they will not point to the right location of the data.
Reference libraries are set up as filesystem IP blocks because filesystem IP blocks can be easily updated in the filesystem during the migration process.
1. If reference libraries are imported before the end of the migration, each change will result in reimporting the data in as new IP versions. When all projects have been imported, the reference libraries can be imported last.

Reference libraries are migrated using the following steps:

reference libraries are copied from the depot to the client workspace in unix directory on disk.

p4 sync //depot/proj1/reference_library/refliba /mdx/reflibs/refliba/1

A filesystem IP block is created which points at the reference Library on the disk
Periodically update the reference Library
The reference Library is to be updated periodically to keep its contents up-to-date with the depot (p4sync).
- Perforce triggers can also be used to update the reference Library whenever a Perforce update occurred in the depot.

Migration flow

1) Extract the first project to be migrated from the 3rd party tool server to the Perforce IPLM server using p4 fetch.

To make sure that a complete dataset is gathered for migration from a live instance, it is necessary to prevent users from making changes to the path(s) which are to be extracted. With superuser access rights, this can be done by simply restricting write access to this path. One way to disallow write access is by using ‘p4 protect’ command. This command opens the protections table which defines read-write access rights for users and groups. Normally, only super users can edit the protections table. In ‘Protections’ section of the table, define read access for the depo path which is to be extracted (you may also need to delete lines granting ‘write’ access for that depo path; further information about protections can be found here

If the 3rd party tool server has files integrated in from other depots, ‘p4 snap’ command may be used to undo lazy copies and copy the lazy integrated files to their physical lbr location.

 p4 fetch -r <source p4d server>

We also recommend using the verbose flag which lists all files that were extracted and other debugging information. Three verbose levels are supported - 1,3,5.

Further information about p4 fetch is available at https://www.perforce.com/perforce/doc.current/manuals/cmdref/Content/CmdRef/p4_fetch.html.

2) Create IPs in Perforce IPLM and build an initial workspace

Empty IP blocks are to be created in Perforce IPLM corresponding to each of the IP blocks being migrated from the 3rd party tool.

A top level container hosts all empty IP blocks. The empty IP blocks are listed in the 'resources' section of the container. There are two kinds of IPs in the container's resource list:

- IPs that are in development, and need to stay on the top-of-tree. In Perforce IPLM, these IPs are called @HEAD. When adding IPs to the container, select all IPs that need to stay on the top-of-tree as 'LIB.IP@HEAD'
- IPs that are on fixed releses. These IPs can simply be called out by 'LIB.IP' in the resource list.

The empty IP blocks must also reside in a Library which is to be created first (‘pi lib add’ command). When the Library has been created, the IPs are created by ‘pi ip add’ command. An initial workspace is built using ‘pi ip load’ command. The top level container is created last by ‘pi ip add’.

Add a new Library if needed. A Library is like a folder and is a handy way to organize IPs. It is not necessary to create a new Library for every migration, existing Libraries can be used as needed.

This command opens the Library creation file. The Library name appears in the ‘name’ field. Save the file and the Library is created.

> pi lib add <lib_name>

To add IPs to Perforce IPLM:

> pi ip add <lib_name>.<ip_name>

This command opens the IP creation file. This form requires an IP name, the Library that the IP belongs to, the DM type (Perforce) and the repo_path, which is the path that the IP files reside in inside the P4 repository.

Create a top level container IP that contains all these newly added IPs. The top level container is of type 'CONT', has no 'repo_path', and has a list of other IPs as resources. Sample container input is shown below:

# Name is required
name = proj1

# Library is required
library = analog

# For add, line is always TRUNK

# DM type is required
# One of: svn p4 container
dm_type = CONT

# Required for p4, svn, dss, dssm and FS; not used for container IP
repo_path = 

# Required
version_message = First release of the top level container

# Description is optional
description = Proj1 top level container. Contains all the IPs that are part of this project

# Labels are optional
# One label per line, format: <label_type>.<label_name>labels = 

# Resources are optional
# One resource per line (newlines require a leading space)
resources = analog.adc1@HEAD
            analog.dac1@HEAD
            digital.mux
            common.gates
...

3) Create a workspace

For container IPs, ‘dm_type’ is ‘Container’, and ‘repo_path’ is not required. All the newly created IPs should be defined as resources of this container IP. Note that container IPs also need to belong to a Library, like any other IP. Note also that you will need to add a pre-release hook to generate hierarchical releases of all the resources in this container. Refer to section 3.2.7 for more information on this hook.

Load the newly created container IP into a workspace:

> pi ip load <lib_name>.<container> <workspace>

This creates a workspace in <workspace> using <lib_name>.<ip_name> data.

Make the IPs in the workspace ‘local’

> pi ip local -a

By default a workspace is created with IP blocks in refer mode. Since later we will need the IP blocks in local mode (when we sync them), we put all IPs (by using wildcard) in local mode using ‘pi ip local’ command.

4) Create a workspace for the first release of the 3rd party tool.

A workspace is to be created for the first release of the 3rd party tool. The same workspace will later be reused (updated) for other releases. All releases will provide the configuration per project.

5) View the 3rd party tool client to get list of branches

Look at the .p4config file for the p4 client name.

The p4 config file contains information like:

P4PORT=host:port
P4USER=user
P4CLIENT=myWorkspace

The value of P4CLIENT is the one of interest

View the client workspace specification and its views:

> p4 client -o <name_of_client>

List the filelist for each branch

In order to list files and revisions to the client workspace (the filelist for each branch), we need to run ‘p4 have’ command from each IP directory:

> cd /path/to/3rd/party/workspace
> p4 have ... > /tmp/filelist

5) Bring each IP into sync in the Perforce IPLM workspace

Each of the IPs in the Perforce IPLM workspace is to be synced against the filelist returned by the ‘p4 have’ command in the previous paragraph. For this purpose, we first ‘cd’ in the IP directory and cat the filelist, remove the second column from the output (‘cut’ command) and perform the sync (‘sync’ command)

> cd ip_dir
> cat /tmp/filelist | cut -d '-' -f 1 | xargs p4 sync

6) Make a hierarchical release of the top level container

As a result of the hierarchical release, all IPs will have a new incremental version, for instance LIB.IP@2. This version constitutes the release of Perforce IPLM matching the release of the 3rd party tool.

> pi ip edit <container_name>

In order to release the top level container, we have to ‘edit’ (pi ip edit) it and change the version of the resources to ‘LATEST’ as shown below. Then run the release command with the '–hier' option'.

7) Update the workspace to another branch of the 3rd party tool and make hierarchical release for that branch

This procedure is to be repeated for each release of the third party tool which is to be migrated to Perforce IPLM. Update the workspace to another branch of the 3rd party too.

Each of the IPs in the Perforce IPLM workspace is to be synced against the filelist for the branch as performed in step 5.