Import Trac bug database

Hi Rafiu,

Please can you send me the XML file, or a link to it, so I can do a test import locally? If that goes well, I'll get it imported into our staging instance so you can check it over before we run the import in production.

I don't think that a Trac export contains sensitive information, but if you're worried, send it (or a link to it) to me, PGP encrypted with my C3038BC9 key (see my Launchpad profile page to verify).

Also, please confirm that you want the bugs imported into the Openfiler project <https://edge.launchpad.net/openfiler>?

Thanks, Gavin.

Hey Gavin,

Yes https://edge.launchpad.net/openfiler is the import target.

Thanks for looking into this. Download link for the XML is here: https://download.openfiler.com/tracexport.xml.tgz

Appreciate it.

R.

Cool, this imports correctly on a test instance of Launchpad. There was only one thing missing from the export, the namespace on the root element:

  <launchpad-bugs xmlns="https://launchpad.net/xmlns/2006/bugs">

It's a trivial thing to fix, but I was wondering if you used a 3rd party tool to generate the export, or did you use the Trac to Launchpad Migrator*? I'll see if I can fix it or at least file a bug.

[* <https://edge.launchpad.net/trac-launchpad-migrator>]

In any case, I'll get this imported into staging now, then you can take a look and confirm it's sane before we push it into production. I'll let you know when it's ready.

Gavin.

Yes https://edge.launchpad.net/trac-launchpad-migrator (0.1.2) was the tool we used. It was choking on one attachment and there were some "no such table: session_attribute()" log messages, but aside from that, nothing untoward.

Thanks.

R.

When we import this into https://staging.launchpad.net/openfiler,
there are many conflicts of bug nicknames. In order to import your
bugs we need to change the nickname. I have two suggestions:

 1. Add a prefix to all nicknames.

    All of your bug nicknames are of the form: elisa-[0-9]+

    We could add a prefix of, say, "openfiler-" to these, so that the
    new form is openfiler-elisa-[0-9]+. This probably won't clash with
    any other bugs already in Launchpad.

    Of course, we could just as easily change "elisa" to "openfiler",
    so the form would be: openfiler-[0-9]+

 2. Use a different nickname, and add the current nickname as a tag.

    The following link might be really useful to you:

        https://bugs.launchpad.net/bugs/<nickname>

    So, if you were to have the nickname form:
        openfiler-<bug number in trac>

    you could add an Apache rewrite rule (or something similar) to
    redirect seamlessly from your site to Launchpad. Suppose the
    nickname were as above. You could add a redirect from:

        https://project.openfiler.com/tracker/ticket/<bug#>

    to:

        https://bugs.launchpad.net/bugs/openfiler-<bug#>

    and Launchpad would take you to the correct bug after import,
    which might be bug #423102 or something; we can't tell in advance.

    Then, if the elisa-# information is useful, we could instead add
    this as a tag to the bug. You could then see this in each bug page
    and search for it on the advanced search page:

        https://bugs.launchpad.net/openfiler/+bugs?advanced=1

I suggest (2), but (1) is probably easier to do. I can take care of
modifying the export if you'd like, but obviously the redirection from
projects.openfiler.com would be your task.

Let me know what you'd prefer.

Also, are people still editing your existing bug tracker? If so,
you'll have to do a final export just before we load it into Launchpad
production, but you're probably well aware of that :)

Thanks, Gavin.

Hi Gavin,

So this is taking you more effort than I'd envisaged - so thanks a lot for the work and the suggestions.

We'll go with your recommendation (i.e option 2). Are you ready for the final XML export? Also I noticed the total number of bugs in https://staging.launchpad.net/openfiler was 24. Was it a full import you did or just a subset for testing?

Thanks,

R.

Hi Rafiu,

No worries about the effort. Maybe it'll make us automate it some day! And sorry to you that this is taking a while.

The reason only 24 bugs appears is because the import of the rest failed because of nickname collisions. I'll modify the export file you gave me before to have new nicknames, and then check the remaining bugs import correctly into staging. Once that's confirmed I'll ask for the final export.

Once I have the final export I'll try and make sure it gets imported into Launchpad asap so your work is not interrupted too much, but it could take an hour or more to coordinate.

I'll go and write a script to modify your export file as in (2) above, and get the remaining bugs loaded now.

Gavin.

Hi Rafiu,

I discovered another problem with the export data: several of the descriptions are too long to put straight into Launchpad. We have a hard limit of 50000 characters for a description. What I've done in these instances is changed the description to a maximum of 30 lines, followed by a message explaining where to find the full description. For example, the description of bug 146 is changed to:

    Traceback (most recent call last):
      File "/usr/lib/anaconda/gui.py", line 943, in handleRenderCallback
        self.currentWindow.renderCallback()
      File "/usr/lib/anaconda/iw/progress_gui.py", line 156, in renderCallback
        self.intf.icw.nextClicked()
      File "/usr/lib/anaconda/gui.py", line 768, in nextClicked
        self.dispatch.gotoNext()
      File "/usr/lib/anaconda/dispatch.py", line 157, in gotoNext
        self.moveStep()
      File "/usr/lib/anaconda/dispatch.py", line 225, in moveStep
        rc = apply(func, self.bindArgs(args))
      File "/usr/lib/anaconda/packages.py", line 75, in writeConfiguration
        id.write(instPath)
      File "/usr/lib/anaconda/instdata.py", line 119, in write
        self.rootPassword.write (instPath, self.auth)
      File "/usr/lib/anaconda/users.py", line 93, in write
        setPassword(instPath, "root", pure, auth.useMD5)
      File "/usr/lib/anaconda/users.py", line 129, in setPassword
        os.chown("/mnt/sysimage/etc/shadow", 0, 94)
    OSError: [Errno 2] No such file or directory: '/mnt/sysimage/etc/shadow'

    Local variables in innermost frame:
    account: root
    alreadyCrypted: 0
    useMD5: 1
    devnull: 43
    password: $1$7u8OWKyy$3FBrZf2Hz2EeAzoFVmSRI1
    argv: ['/usr/sbin/usermod', '-p', '$1$7u8OWKyy$3FBrZf2Hz2EeAzoFVmSRI1', 'root']
    instPath: /mnt/sysimage...

    [Truncated; see "Full description" attachment]

And, sure enough, there is a "Full description" attachment which contains the full description, utf-8 encoded. Get back to me if this isn't okay. There are only about 12 bugs that need changing like this, and I think they seem to be where people have pasted logs straight into the description. My script reports:

    Bug 146's description is too long (175656 chars).
    Bug 147's description is too long (175656 chars).
    Bug 192's description is too long (182928 chars).
    Bug 197's description is too long (195427 chars).
    Bug 198's description is too long (187305 chars).
    Bug 207's description is too long (176053 chars).
    Bug 241's description is too long (184328 chars).
    Bug 250's description is too long (182061 chars).
    Bug 269's description is too long (182004 chars).
    Bug 300's description is too long (176071 chars).
    Bug 301's description is too long (179016 chars).
    Bug 317's description is too long (174090 chars).

I wanted to get the results of my script loaded into staging today so you could see it, but staging is not fully available right now. I'll keep you posted.

Gavin.

Hi Gavin,

Those long descriptions are superfluous anyway. Your solution works fine for us. Please feel free to do what you think is best in order to get the tickets imported.

Thanks.

R.
Sent from my BlackBerry® wireless device

-----Original Message-----
From: Gavin Panella <email address hidden>

Date: Thu, 06 Aug 2009 16:40:01
To: <email address hidden>
Subject: Re: [Question #78754]: Import Trac bug database

Your question #78754 on Launchpad Bugs changed:
https://answers.edge.launchpad.net/malone/+question/78754

Gavin Panella requested for more information:
Hi Rafiu,

I discovered another problem with the export data: several of the
descriptions are too long to put straight into Launchpad. We have a hard
limit of 50000 characters for a description. What I've done in these
instances is changed the description to a maximum of 30 lines, followed
by a message explaining where to find the full description. For example,
the description of bug 146 is changed to:

    Traceback (most recent call last):
      File "/usr/lib/anaconda/gui.py", line 943, in handleRenderCallback
        self.currentWindow.renderCallback()
      File "/usr/lib/anaconda/iw/progress_gui.py", line 156, in renderCallback
        self.intf.icw.nextClicked()
      File "/usr/lib/anaconda/gui.py", line 768, in nextClicked
        self.dispatch.gotoNext()
      File "/usr/lib/anaconda/dispatch.py", line 157, in gotoNext
        self.moveStep()
      File "/usr/lib/anaconda/dispatch.py", line 225, in moveStep
        rc = apply(func, self.bindArgs(args))
      File "/usr/lib/anaconda/packages.py", line 75, in writeConfiguration
        id.write(instPath)
      File "/usr/lib/anaconda/instdata.py", line 119, in write
        self.rootPassword.write (instPath, self.auth)
      File "/usr/lib/anaconda/users.py", line 93, in write
        setPassword(instPath, "root", pure, auth.useMD5)
      File "/usr/lib/anaconda/users.py", line 129, in setPassword
        os.chown("/mnt/sysimage/etc/shadow", 0, 94)
    OSError: [Errno 2] No such file or directory: '/mnt/sysimage/etc/shadow'

    Local variables in innermost frame:
    account: root
    alreadyCrypted: 0
    useMD5: 1
    devnull: 43
    password: $1$7u8OWKyy$3FBrZf2Hz2EeAzoFVmSRI1
    argv: ['/usr/sbin/usermod', '-p', '$1$7u8OWKyy$3FBrZf2Hz2EeAzoFVmSRI1', 'root']
    instPath: /mnt/sysimage...

    [Truncated; see "Full description" attachment]

And, sure enough, there is a "Full description" attachment which
contains the full description, utf-8 encoded. Get back to me if this
isn't okay. There are only about 12 bugs that need changing like this,
and I think they seem to be where people have pasted logs straight into
the description. My script reports:

    Bug 146's description is too long (175656 chars).
    Bug 147's description is too long (175656 chars).
    Bug 192's description is too long (182928 chars).
    Bug 197's description is too long (195427 chars).
    Bug 198's description is too long (187305 chars).
    Bug 207's description is too long (176053 chars).
    Bug 241's description is too long (184328 chars).
    Bug 250's description is too long (182061 chars).
    Bug 269's description is too long (182004 chars).
    Bug 300's description is too long (176071 chars).
    Bug 301's description is too long (179016 chars).
    Bug 317's description is too long (174090 chars).

I wanted to get the results of my script loaded into staging today so
you could see it, but staging is not fully available right now. I'll
keep you posted.

Gavin.

--
To answer this request for more information, you can either reply to
this email or enter your reply at the following page:
https://answers.edge.launchpad.net/malone/+question/78754

You received this question notification because you are a direct
subscriber of the question.

Hi Rafiu,

I'm sorry this is taking so long. I am working on this right now, and it is my priority. There is one outstanding bug <https://bugs.edge.launchpad.net/malone/+bug/412940> that I am in the middle of preparing a fix for.

Gavin.

Hi Rafiu,

I've fixed bug 412940 and tested importing your data into staging without a hitch. Shall we schedule a time to do a final export from Trac and import into Launchpad? Right now I'm normally at work between ~0800 and ~1700 UTC, and I'm not planning any holiday until the 8th August. Choose your moment :)

Gavin.

All of the bugs in the final export you provided have now been imported into the production system, including those 13 which had previously failed. Welcome to Launchpad!

Marking solved.