Unable to push initial branch - permission denied (publickey)

Have you done 'bzr launchpad-login' yet? You might need to do `bzr launchpad-login jaraco`

Yes. I had run the login command. It returns my username. Yet still I'm given permission denied.

jaraco@livid:~/projects/beautifulsoup$ bzr launchpad-login
jaraco
jaraco@livid:~/projects/beautifulsoup$ bzr push lp:~jaraco/beautifulsoup/easy-installable
Permission denied (publickey).
bzr: ERROR: Connection closed: Unexpected end of message. Please check connectivity and permissions, and report a bug if problems persist.

Do you have the SSH key that Launchpad knows about on the machine that you are pushing from?

Have you overridden the SSH key for launchpad.net in the ~/.ssh/config?

The SSH key I'm using is a DSA key (it says "ssh-dss" at the beginning of the public key). It's either 1024 bit or 2048 bit. This is a key that I also use on a number of other Linux, BSD, and Windows servers for public-key authentication. I've uploaded the public key into launchpad under the jaraco account.

Also, when I use "ssh -v bazaar.launchpad.net", it appears to authenticate, but refuses to give me a shell (which I assume is by design).

I have no special configuration for my local ~/.ssh/config for launchpad. In fact, I have no config file at all. Is it possible that OpenSSH knows where to find the identity file (~/.ssh/identity) but bazaar does not?

Indeed, that appears to be the case. I copied ~/.ssh/identity to ~/.ssh/id_dsa, and the push worked.

Is it a bazaar limitation or a launchpad/bazaar limitation that it doesn't recognize ~/.ssh/identity as a private key (while other popular SSH clients do)?

bzr invokes ssh, so I'm pretty sure its not a bazaar issue at all. You could strace bzr and see what precise ssh command line its running.

I did an strace, and it appears the cause may be that it's not in fact invoking ssh, but is using ssh.py. I don't know if that's a different implementation, but I wouldn't be surprised if it has a somewhat different resolution of identity files.

Here are the lines from the strace

jaraco@livid:~/projects/beautifulsoup$ grep ssh strace_bzr_fails
stat("/usr/lib/python2.6/dist-packages/bzrlib/transport/ssh", 0x7fff670ee030) = -1 ENOENT (No such file or directory)
open("/usr/lib/python2.6/dist-packages/bzrlib/transport/ssh.so", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/lib/python2.6/dist-packages/bzrlib/transport/sshmodule.so", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/lib/python2.6/dist-packages/bzrlib/transport/ssh.py", O_RDONLY) = 4
open("/usr/lib/python2.6/dist-packages/bzrlib/transport/ssh.pyc", O_RDONLY) = 5
stat("/usr/lib/pymodules/python2.6/paramiko/ssh_exception", 0x7fff670df910) = -1 ENOENT (No such file or directory)
open("/usr/lib/pymodules/python2.6/paramiko/ssh_exception.so", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/lib/pymodules/python2.6/paramiko/ssh_exceptionmodule.so", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/lib/pymodules/python2.6/paramiko/ssh_exception.py", O_RDONLY) = 8
open("/usr/lib/pymodules/python2.6/paramiko/ssh_exception.pyc", O_RDONLY) = 9
write(3, "1.498 ssh implementation is Ope"..., 37) = 37

Do you agree with my assessment (that openssh is not being used, but ssh.py is)? Unfortunately, this doesn't indicate much more to be about how the SSH connection is being established or which key was being attempted.

Looks like there is something in .bzr.log - can you see what it says there ?

It looks like a generic error message to me:

Sun 2010-04-11 19:15:07 -0400
0.132 bzr arguments: [u'push', u'lp:~jaraco/beautifulsoup/easy-installablex']
0.163 looking for plugins in /home/jaraco/.bazaar/plugins
0.163 looking for plugins in /usr/lib/python2.6/dist-packages/bzrlib/plugins
0.335 encoding stdout as sys.stdout encoding 'UTF-8'
0.421 opening working tree '/home/jaraco/projects/beautifulsoup'
1.498 ssh implementation is OpenSSH
2.397 Traceback (most recent call last):
  File "/usr/lib/python2.6/dist-packages/bzrlib/commands.py", line 842, in exception_to_return_code
    return the_callable(*args, **kwargs)
  File "/usr/lib/python2.6/dist-packages/bzrlib/commands.py", line 1037, in run_bzr
    ret = run(*run_argv)
  File "/usr/lib/python2.6/dist-packages/bzrlib/commands.py", line 654, in run_argv_aliases
    return self.run(**all_cmd_args)
  File "/usr/lib/python2.6/dist-packages/bzrlib/builtins.py", line 1156, in run
    use_existing_dir=use_existing_dir)
  File "/usr/lib/python2.6/dist-packages/bzrlib/push.py", line 83, in _show_push_branch
    dir_to = bzrdir.BzrDir.open_from_transport(to_transport)
  File "/usr/lib/python2.6/dist-packages/bzrlib/bzrdir.py", line 879, in open_from_transport
    return format.open(transport, _found=True)
  File "/usr/lib/python2.6/dist-packages/bzrlib/bzrdir.py", line 2065, in open
    return self._open(transport)
  File "/usr/lib/python2.6/dist-packages/bzrlib/bzrdir.py", line 3294, in _open
    return remote.RemoteBzrDir(transport, self)
  File "/usr/lib/python2.6/dist-packages/bzrlib/remote.py", line 114, in __init__
    response = self._call('BzrDir.open', path)
  File "/usr/lib/python2.6/dist-packages/bzrlib/remote.py", line 52, in _call
    return self._client.call(method, *args)
  File "/usr/lib/python2.6/dist-packages/bzrlib/smart/client.py", line 129, in call
    result, protocol = self.call_expecting_body(method, *args)
  File "/usr/lib/python2.6/dist-packages/bzrlib/smart/client.py", line 142, in call_expecting_body
    method, args, expect_response_body=True)
  File "/usr/lib/python2.6/dist-packages/bzrlib/smart/client.py", line 90, in _call_and_read_response
    expect_body=expect_response_body)
  File "/usr/lib/python2.6/dist-packages/bzrlib/smart/message.py", line 299, in read_response_tuple
    self._wait_for_response_args()
  File "/usr/lib/python2.6/dist-packages/bzrlib/smart/message.py", line 264, in _wait_for_response_args
    self._read_more()
  File "/usr/lib/python2.6/dist-packages/bzrlib/smart/message.py", line 286, in _read_more
    "Unexpected end of message. "
ConnectionReset: Connection closed: Unexpected end of message. Please check connectivity and permissions, and report a bug if problems persist.

2.398 return code 3

So, it thinks its using openssh - that log message is output by our ssh.py (which is the code that brokers different ssh implementations and tunnelling to bzr across ssh).

 In your strace you should have an 'exec' call somewhere - can you paste that ?

I didn't find it in the strace.

jaraco@livid:~/projects/beautifulsoup$ grep exec strace_bzr_fails
execve("/usr/bin/bzr", ["bzr", "push", "lp:~jaraco/beautifulsoup/easy-in"...], [/* 38 vars */]) = 0

Should I have supplied more parameters to the strace? I just ran it using only -o to save the output.

-f will be needed

I ran strace with -f (Thanks!).

I think I found the relevant lines

jaraco@livid:~/projects/beautifulsoup$ grep /usr/bin/ssh strace_bzr_fails
3868 execve("/usr/bin/ssh", ["ssh", "-V"], [/* 38 vars */]) = 0
3869 execve("/usr/bin/ssh", ["ssh", "-oForwardX11=no", "-oForwardAgent=no", "-oClearAllForwardings=yes", "-oProtocol=2", "-oNoHostAuthenticationForLocalho"..., "-l", "jaraco", "bazaar.launchpad.net", "bzr", "serve", "--inet", "--directory=/", "--allow-writes"], [/* 38 vars */]) = 0

This led me to the fundamental cause of the trouble.

The call by bazaar to ssh forces the protocol to protocol 2, which apparently then excludes ~/.ssh/identity as a default key file (but still includes ~/.ssh/id_dsa, which is why it works when the private key is copied to that name). In fact, I can now reproduce the bazaar behavior with this simple test:

$ ssh -2 bazaar.launchpad.net
Permission denied (publickey).

So it appears that I should probably just use id_dsa as my private key file name.

It could be proposed that bzr consider not forcing protocol 2 at the client and instead allow the server administrators to force a protocol version as appropriate. This would provide more consistent behavior with the default SSH behavior. However, I would be more of the opinion that if id_dsa is the canonical name for the private key for SSH-2, and if users want to name their SSH-2 keys with the SSH-1 canonical name, 'identity', then it's reasonable for them to expect the key to be required to be referenced explicitly in the .ssh/config file (or otherwise specify the key).

Thanks for all the help.

Thanks Robert Collins, that solved my question.

On Mon, 12 Apr 2010 10:53:55 you wrote:
> Question #107036 on Launchpad Bazaar Integration changed:
> https://answers.edge.launchpad.net/launchpad-code/+question/107036
>
> Jason R. Coombs gave more information on the question:
> Indeed, that appears to be the case. I copied ~/.ssh/identity to
> ~/.ssh/id_dsa, and the push worked.
>
> Is it a bazaar limitation or a launchpad/bazaar limitation that it
> doesn't recognize ~/.ssh/identity as a private key (while other popular
> SSH clients do)?

It is probably the way that bazaar is invoking ssh.

Yes, bzrlib/transport/ssh.py forces Protocol=2.

There is no comment as to why.

Protocol 1 is pretty old and deprecated now, and has some security
problems. Protocol 2 came out in May 2000. So, generally speaking,
relying on a key that only works with protocol 1 is not a great idea.
However, this is a very confusing way to fail.

The openssh manpage says it defaults to protocol 2 (at least in lucid)
so I'm not sure what we would hope to gain by forcing it on the
command line.

--
Martin <http://launchpad.net/~mbp/>

To be clear, my key is a protocol 2 key - it was just named with the default
name for a protocol 1 key. When the connection was established, it was
established using protocol 2.