Migrating from Base ClearCase to UCM

Jirong Hu
December 15, 2008

1 Base ClearCase and UCM

The main objective of this article is to help readers do the actually migration, not to help them decide if UCM is more suitable for their environment. However, I think it is a good idea to have a bit comparison between Base ClearCase and UCM.
Let’s start with some good definitions about Base ClearCase and UCM I found from the Internet:
Base ClearCase consists of a collection of tools to establish an environment in which developers can work in parallel on a shared set of files, and project managers can define policies that govern how developers work together.

UCM is an out-of-the-box implementation of a development process layered upon base ClearCase. Adopting UCM makes it possible to manage your projects with ClearCase more quickly in an efficient manner without needing to master the details of base ClearCase.

UCM offers the convenience of a readily available management process; base ClearCase offers the flexibility to implement virtually any configuration management process that you deem appropriate for your environment.
To me, UCM is a later technology and more Object-Oriented. During implementation, this characteristic makes UCM easier for a ClearCase architect to map the ClearCase structure into modern programming methods such as C++ and Java, both conceptually and practically.
However, there are limitations in UCM, and also in many circumstances Base ClearCase is more suitable. We are not going to elaborate this topic in this article. Interested readers can find some good discussions in ClearCase Forum in developerWorks: http://www.ibm.com/developerworks/forums/thread.jspa?threadID=227980. The fact is, most likely we will see both Base ClearCase and UCM co-exist for a long time. And occasionally, we may need to migrate some projects from Base ClearCase to UCM or vice versa.
A real SCM environment in a large organization can be very complicated, e.g. multisited globally, ClearQuest or other products integrated, etc., to achieve a heavy weight process. Migrating ClearCase VOBs in such an environment is no longer a simple ClearCase job. In this article, we will not trying to cover all these complexities, but to give a general idea about what can be expected during such a migration from perspective of ClearCase. In the rest of sections, we will use a simply but real project as an example. This is a C/C++ project in telecommunication industry called HWC.

2 Business Requirements

Below is a list of Base ClearCase VOBs to be migrated into UCM in project HWC. VOB BMcore contains the project’s in-house developed source code, and the rest are mostly 3^rd party libraries.
* /vobs/projects/builds /data/storage/vobs1/builds.vbs
* /vobs/projects/BlueCat4_0 /data/storage/vobs1/BlueCat4_0.vob
* /vobs/projects/linuxIXAedu /data/storage/vobs1/linuxIXAedu.vob
* /vobs/projects/enp_cygmon /data/storage/vobs1/enp_cygmon.vob
* /vobs/projects/BMcore /data/storage/vobs1/BMcore.vob
* /vobs/projects/NPE_Firmware /data/storage/vobs1/NPE_Firmware.vob
* /vobs/projects/Doc /data/storage/vobs6/Doc.vbs
* /vobs/projects/HWC_3rd_Party_Lib /data/storage/vobs6/HWC_3rd_Party_Lib.vbs

2.1 Performance

This project has been run for a long time and the main source code VOB, BMcore, has been grown to more than 60G. The size of this VOB makes some of maintenance work extremely hard in our environment, so one purpose of this migration is to shrink this VOB.

2.2 Data Integrity

Except VOB BMcore, all history data has to be kept in other VOBs.

2.3 No Breaking Anything

An important requirement of this project is that the new ClearCase structure shouldn’t break any existing build/release/processes. Due to the time constrain and many hard-coded directory path in the build scripts, we can’t re-write the build scripts, although I think it’s a perfect time to re-structures it. In terms of ClearCase migration, we will not be able to redefine the VOB/Component structure, in order to keep the path name unchanged.

2.4 Live Branches Need to be Migrated

This is a live project and has many customers out there using a few major releases. These releases are currently maintained in a few branches in base ClearCase, and patches are released constantly. These branches therefore also need to be migrated into the new UCM structure.

2.5 Minimum Downtime

Usually such a migration will require minimum or even no downtime.

3 Solution

Based on the above requirements, we come out the following solution for this migration:

1. Define a UCM PVOB/VOB structure for this project and lay out a good foundation for further development.
2. We can not keep the in-house source code VOB BMcore in the new UCM environment. The solution is to create a new UCM VOB and copy certain code base to it.
3. For the rest of Base ClearCase VOBs, we will convert them into UCM component VOBs.
4. A method has to be developed to maintain the old build path so that the existing build script will still work.
5. Establish a good process to manage all new UCM projects.
6. Create new UCM projects/streams to replace live branches in Base ClearCase.

4 A New UCM Structure for HWC

4.1 Differences between UCM and Base ClearCase

If you are new to UCM, actually there are quite some differences between UCM and Base ClearCase. Here is my understanding:

1. UCM is OO based. Most of its terminologies are abstract concepts, just as in OO design. For example, whatever you see in the Project Explorer are OO objects. This is very important.
2. You are going to use more GUI based interface tools than in Base ClearCase. They are helpful.
3. In UCM, there are additional layers on top of VOBs in Base ClearCase. You use PVOB, projects and components to manage the assets. The Project in UCM can be mapped to the real project team, and component mapped to the sub-systems.
4. While in Base ClearCase, branches are mostly created implicated by a view config spec, you have to create streams in UCM explicitly.
5. A view is a super view in Base ClearCase, and you can use that to look into any VOBs by just changing config specs. This is not the case in UCM. You’ll have to create different views into the streams you want to look into. You don’t manage config spec directly. UCM does it for you.
6. There is no label in UCM, instead, you use baselines. Although there are still in the system, label is a Base ClearCase concept.

4.2 Naming Standards

A good naming convention can help you organize a clearly structured projects and streams and make users easy to read. Below is the naming convention I always advocate for most of organizations.

4.2.1 Admin VOB: <OrganizationName_Admin>

This is the organization wise Admin VOB for all VOBs, to share global types.

4.2.2 Line of Business Name: <LoB>

During to process, security and other reasons, projects are organized into different groups, different PVOBs in terms of ClearCase UCM implementation. The organization structure is a good place to find criteria in this case. This name will usually be mapped to PVOB names.

4.2.3 Official Project Name: <ProjectName>

Every project should have an official name defined by the Project Management Office. Strictly use this name with the exact case and spelling for easy communication and understanding among different groups across the organization. Use short term if possible.
The same should be applied for the <LoB> and <application_name>.

4.2.4 PVOB Name: <LoB>_PVOB

Projects in one organization group will likely share the same process. In this case, we put all components and projects under HWC_PVOB.

4.2.5 Component VOB Name: <LoB>_cvob

We group all applications in a few component VOBs for easy management and better performance.

4.2.6 UCM Component Name: <LoB>_<ApplicationName>_<ComponentName>_<ComponentType>

With <LoB> name attached, it’s easy to identify the component, especially in a multisited environment.
<component_type> can be one of these: src, docs, bin, model, etc. to identify the type of content stored inside the component.

4.2.7 UCM Project Name: <ProjectName>_mainline, <ProjectName>_<PhasesName>

With a naming standard, it’s a lot easy to find projects and components in ClearCase project explorer where all UCM objects are sorted by names.

4.2.8 UCM Project Folders: <BusinessFunction>

When there are too many projects in a PVOB, we can group them into different folders according to the business functions they belong to.

4.2.9 UCM Composite Baseline Name: <ProjectName>_CBL

CBL stands for “Composite Baseline”. This is the rootless component storing the project’s composite baseline. Every project will have only one, and shared by the mainline and all phase projects.

4.3 HWC UCM Design

As a result, we come out such a simple design in UCM as shown below:

1. HWC_PVOB will be the project and admin VOB for this project.
2. Create a new component VOB called BMnew, and a new component inside this VOB called BMcore to migrate certain code base from the old BMcore VOB.
3. Convert all the rest of Base ClearCase VOBs into UCM component VOBs.
4. Create a mainline project, hwc_mainline, as the bootstrap project.
5. Create another project, others_mainline, to manage all 3^rd party libraries.
6. Create new development project for each working branch in the old BMcore.

We will explain the details afterwards.

Figure 1: A New UCM Structure for HWC

4.3.1 Creating HWC_PVOB

To set up the above UCM structure, we have to create the project VOB first. Launch the VOB Creation Wizard and following the steps as shown below to create the new PVOB:

Figure 2: Create a New PVOB

4.3.2 Convert Base ClearCase VOBs to UCM Components

The next step is to convert VOBs in Base ClearCase to the new UCM environment. There are many ways you can do this step based on different requirements. In this project, because the existing build script doesn’t allow us to have any changes in the directories structure, so we will simply convert the existing VOBs in Base ClearCase to a component VOB in UCM. Other scenarios you may have are:

1. Converting multiple VOBs into a single component
2. Converting multiple VOBs into separate components in a single VOB
3. Converting VOB directories to separate components in a single VOB

You can find more information in Frank’s wiki on developerWorks:
http://www.ibm.com/developerworks/wikis/display/cm/How+to+migrate+from+ClearCase+VOBs+to+UCM+components
The following steps show how to convert all 3^rd party Base ClearCase VOBs into UCM component VOBs (single component VOB) in HWC_PVOB.

Figure 3: Convert Base ClearCase VOB to UCM Component

Figure 4: Choose the VOB List You Want to Convert

4.3.3 BMcore

For BMcore, we don’t want to simply convert the whole VOB into a UCM component as other VOBs, because we want to reduce the size of the VOB. What we will do is copy a few released code base into the new UCM environment and leave all the rest of history in the old Base ClearCase VOB BMcore.
We will create a new component BMcore@HWC_PVOB in a newly created component VOB BMnew, and copy certain released baselines over to the new mainline project. Afterwards, new phase projects will be created off these baselines.
There are a few things required:

1. Create a new multi-component VOB called BMnew.
2. Create a new component called BMcore inside the above VOB.

The rest of sections will describe how we set up the project in the new UCM environment.

4.3.4 Mainline Approach

We will adopt the mainline approach for the branching strategy in the new UCM structure. Please refer to Reference section for the difference between a mainline and cascading approach, and how you can implement a mainline approach in UCM.
Basically, each project will have a mainline project called <ProjectName>_mainline, and all subsequent phase projects will be created directly off the mainline, no cascading is allowed for any development projects.
A standard application/project will have the following set of UCM projects:

4.3.4.1 Mainline Project: <ProjectName>_mainline

The only purpose of the mainline is as the bootstrap project/stream for all other phase projects. In our case, the mainline for HWC will be called hwc_mainline.

4.3.4.2 Phase/Sibling Projects: <ProjectName>_<PhaseName>

For example, the version 5 release 1 project will be called hwc_v5r3. In this example, they are simply called v5r3.

4.3.5 HWC UCM Projects and Streams

A few new UCM projects will be created to manage all these applications. Official application/project names will be used as the UCM project name.

4.3.5.1 Mainline Project

This is the mainline project which will have the BMcore as the modifiable component and other components as read-only. It will be the boostrap project for other development projects.

Figure 5: Modifiable and R/O Components in Mainline
A composite baseline will be used to manage the projects:

4.3.5.2 Build Project

There will be a build project for each phase/development project. All components in this project should be read-only, and used only by build engineers. As an exception, this project can be created off each phase project since no development will happen here.
For example, the build project for release 5 project will be called v5r3_build.

4.3.5.3 others_mainline

We can manage the rest of converted component VOBs in one project. Let’s call it others_mainline in this example. This should be a single-stream project with all the rest of components as modifiable. All updated to these 3^rd party libraries can be performed in this project.

4.3.6 Build Root

This section is just to show you how symlinks can help when you are in a similar situation.
In order not to break the existing build scripts, we’ll have to keep the original file structure. Due to the additional VOB path of the new component for BMcore (now it’s /BMnew/BMcore in your view vs. /BMcore in Base ClearCase), we’ll have to find a workaround to keep the original file structure.
We can create another new component called buildroot inside BMnew and below that have a set of symlinks pointing to all converted components required by hwc_mainline.
Buildroot will be the root directory where developers and build engineer find there code as shown below. Ignore the rest they see in the view. Only this buildroot is required to be loaded into the snapshot view.

Figure 6: Use symlink to "relocate" components
The rest can be hided in their development view by not loading the other components as shown below (the screen below still loading the code):

Figure 7: Load only "buildroot"

Figure 8: Only buildroot is seen in a view

5 Processes

The purpose of this section is to provide a consistent process to manage the ClearCase projects’ lifecycles to ensure that:
v All software development groups have a common understanding.
v ClearCase UCM projects’ lifecycle fits into the software development process.
v This process is required when:
a. A system is ready to release to production.
b. A new development phase to be started.
c. A ECR fix starts
d. Changes need to be merged between running projects.

5.1 Process Flow Diagram

The diagram below takes the HWC project as an example to describe some common scenarios for a parallel software development environment.

5.2 Roles and Responsibilities

The table below lists the major responsibilities of each role in this process. The actual person who fit in the roles can be from either CM team or a project team.

Role	Responsibilities
Project Team	- Analysis project requirements - Raise change requests
ClearCase Administrator	- Analysis project requests - Propose/Test/Implement Rational solutions - Post-implementation actions
CM Team	- Review and approve the proposed Rational solution internally

5.3 Process Details

This section discusses each step in details.

5.3.1 Step1: Start Project

When a new software project/program launches, we always create a new ClearCase mainline project as the project’s foundation, with initial code base loaded onto mainline if there is any. In this case, the new project will be hwc_mainline.

The Project Team will request to CM team before the project launches.

5.3.2 Step2: Start v6

After that, the v6 project is created with that initial code. In the above diagram, v6 project was created with an initial baseline on hwc_mainline.
The Project Team will request to CM team to create the new project.

5.3.3 Release v6 to Production

Code to be released to production can be taken from a stable baseline from the development project. For example, baseline Release 6.1 from the project v6 will be released directly to production.
After production release, a new development project will be created for the following phase so that the existing project can be used for later ECR fix.
Project Team will request CM team to promote the production baseline to promotion level PRODUCTION after release.

5.3.4 Merge v6 back to mainline

The released baseline in v6 will be merged back to hwc_mainline, to be ready to start v7.
Project Team will request to CM team for a proper merge solution.

5.3.5 Start a v7 Project

Development work can continue with the v6 project as the maintenance project after the first release to production. In case a new v7 development project is required, the v7 project with be created with the merged baseline in mainline. In the above diagram, v7 will be created from the baseline Release1.1 in hwc_mainline.
Project Team will identify a baseline to start the new development project and CM team will help in find a proper solution to start the new phase project.

5.3.6 Deliver changes from v6 to v7

Changes should be delivered directly from v6 to v7 if required, not going though mainline.

Project Team will request to CM team for a proper merge solution.

5.3.7 Merge v7 to mainline

Code in v7 will also be merged back to mainline when required.

Project Team will request to CM team for a proper merge solution.

5.3.8 Group or feature streams

A group of developers can work in the same development stream to minimize the deliver/rebase overhead. There will be no deeper level streams created of this type of group stream.
A feature stream can be created to isolate some long-lived tasks that can’t be merged into the active integration stream in a short time. Different from a group stream, developers will join a feature stream to create sub-streams. Avoid feature streams depends on each other.

Project Team will request to CM team for a proper merge solution

5.3.9 Merge feature streams to v6

Feature steams should only be merged back to v6. Defer the integration of feature streams could make merge more difficult.

Project Team will request to CM team for a proper merge solution.

6 Migrating Branches

The project we are migrating is a running project and there are customers out there using a few versions of this product. This requires us to migrate a few running branches into the new UCM structure, so developers can do the maintained work for these released versions out there. This section describes how we can migrate these branches into UCM.

6.1 Cannot Compare Two Imported Baselines

There is this particular issue you need to be aware of during such a migration: two imported baselines can not be compared. The consequence is that later on they can not be used in an inter-project deliver/rebase or any action comparing two baselines are required.

Figure 9: Import Label as Baseline

Figure 10: Both Baselines Are Imported
The following error message will pop-up if you try to compare the above two imported baselines:

Figure 11: Error Message

6.2 Migration Approach for BMcore

Because we want to reduce the size of old BMcore VOB in base ClearCase, so we don’t want to directly convert this VOB into a UCM component in the new structure. Instead, we will just copy the code from each branch to the new corresponding UCM projects.
In order to make future inter-project deliver work in the new UCM structure, all sibling projects to be created in the new UCM structure have to be inherited from the same foundation baseline in the mainline so that they will have a common ancestor.
Here are live branches in base ClearCase we need to migrate. They all used to develop or maintain early releases.

1. v4_maint
2. v4r2_maint
3. v5_int
4. v5_maint
5. v5r1_int
6. v5r1_maint

There are labels in each branch, e.g.

1. “AS_V4R1_0_24”
2. “AS_V4R1_1_10”
3. “AS_V5R0_10033_0”
4. “AS_V5R0_10034_0”

For each live branch, we decide only to migrate the latest label and LATEST code to its new corresponding UCM projects in the new structure.
Based on the process described in Section 4, the following approach will be used to create all new projects:

1. clearfsimport the latest label on v4_maint branch, “AS_V4R1_0_10” , into BMcore component on mainline, create a new baseline, “AS_V4R1_0_10_Import”. This is the first baseline on the new mainline.
2. Use the above baseline as the foundation, create a new project, v4_maint, to be used for v4 release maintenance. This project is corresponding to the old v4_maint branch.
3. clearfsimport the LATEST code from branch v4_maint onto the above project, so now the code base on the new project is synchronized with the old branch, and developers can start further coding from there.
4. clearfsimport the latest label on next branch v4r2_maint on top of baseline “AS_V4R1_0_10_Import” on mainline and create a new baseline, v4r2_maint_Import.
5. Create a new project v4r2_maint. This project will replace the old v4r2_maint branch.
6. clearfsimport the LATEST code from branch v4r2_maint onto the new project, so now the code base on the new project is synchronized with the old branch, and developers can start further coding from there.
7. Repeat 4,5,6 for the rest of branches.

Because we always use clearfsimport to import the code, therefore all created baseline on mainline are comparable. The earliest baseline “AS_V4R1_0_10_Import” is their command ancestor.
Another approach is to create all new projects from the same foundation baseline on mainline:

1. clearfsimport the latest label on v4_maint,“AS_V4R1_0_10”, into BMcore component on mainline, create a new baseline, “AS_V4R1_0_10_Import”.
2. Create all new UCM projects for all branches from the above baseline, so that all these projects start from this same baseline. The new projects are:

1. v4_maint
2. v4r2_maint
3. v5_int
4. v5_maint
5. v5r1_int
6. v5r1_maint

1. Then clearfsimport latest label and LATEST onto each new project from their corresponding branch in base ClearCase, and make a new baseline.

The key to make the above approach work is always using clearfsimport to move the code, so that all baselines created in the process has a relationship.
Use the first imported baseline as the foundation for the mainline.

Figure 12: The Foundation Baseline for Mainline

6.2.1 Import label AS_V4R1_0_24

To import a label, create two views, a base ClearCase view to the base ClearCase and set the config spec as follows, and a dynamic view to the newly created UCM project.

Here is an example of clearfsimport command:
clearfsimport -recurse -nsetevent -rmname M:\jhu_view_mism122b\BMcore\* M:\jhu_hwc_mainline_int\BMnew\BMcore > C:\temp\import\import_AS_V4R1_0_24.out

6.3 Migration Approach for Other Projects

The import part is easier for the rest of 3^rd party component VOBs, because we can keep these VOBs. The process will be as follows:

1. Convert the Base ClearCase VOBs into components.
2. Import the latest label as a baseline in each component. Please note this is an imported baseline.
3. Set up the new project, others_mainline, as described in Section 3, using the above imported baseline as the foundation for the new project.
4. Import the Latest code from each branch in Base ClearCase to these new components, and create a new baseline.
5. Further update to these 3^rd party libraries will be performed in this new others_mainline project.

7 Reference

· Using mainline projects and composite baselines to manage large-scale J2EE development with IBM Rational ClearCase
· What is a mainline approach?