New crypt function test fails with system python2.6 on OSX

[craigcitro]

/cc @dhermes

i didn't dig too far, but it looks like the system python doesn't behave as expected:

======================================================================
FAIL: test_succeeds (tests.test_crypt.Test_pkcs12_key_as_pem)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/craigcitro/gh/oauth2client/tests/test_crypt.py", line 57, in test_succeeds
    self.assertEqual(pem_contents, pkcs12_key_as_pem)
AssertionError: '-----BEGIN RSA PRIVATE KEY-----\nMIIEpAIBAAKCAQEA4ej0p7bQ7L/r4rVGUz9RN4VQWoej1Bg1mYWIDYslvKrk1gpj\n7wZgkdmM7oVK2OfgrSj/FCTkInKPqaCR0gD7K80q+mLBrN3PUkDrJQZpvRZIff3/\nxmVU1WeruQLFJjnFb2dqu0s/FY/2kWiJtBCakXvXEOb7zfbINuayL+MSsCGSdVYs\nSliS5qQpgyDap+8b5fpXZVJkq92hrcNtbkg7hCYUJczt8n9hcCTJCfUpApvaFQ18\npe+zpyl4+WzkP66I28hniMQyUlA1hBiskT7qiouq0m8IOodhv2fagSZKjOTTU2xk\nSBc//fy3ZpsL7WqgsZS7Q+0VRK8gKfqkxg5OYQIDAQABAoIBAQDGGHzQxGKX+ANk\nnQi53v/c6632dJKYXVJC+PDAz4+bzU800Y+n/bOYsWf/kCp94XcG4Lgsdd0Gx+Zq\nHD9CI1IcqqBRR2AFscsmmX6YzPLTuEKBGMW8twaYy3utlFxElMwoUEsrSWRcCA1y\nnHSDzTt871c7nxCXHxuZ6Nm/XCL7Bg8uidRTSC1sQrQyKgTPhtQdYrPQ4WZ1A4J9\nIisyDYmZodSNZe5P+LTJ6M1SCgH8KH9ZGIxv3diMwzNNpk3kxJc9yCnja4mjiGE2\nYCNusSycU5IhZwVeCTlhQGcNeV/skfg64xkiJE34c2y2ttFbdwBTPixStGaF09nU\nZ422D40BAoGBAPvVyRRsC3BF+qZdaSMFwI1yiXY7vQw5+JZh01tD28NuYdRFzjcJ\nvzT2n8LFpj5ZfZFvSMLMVEFVMgQvWnN0O6xdXvGov6qlRUSGaH9u+TCPNnIldjMP\nB8+xTwFMqI7uQr54wBB+Poq7dVRP+0oHb0NYAwUBXoEuvYo3c/nDoRcZAoGBAOWl\naLHjMv4CJbArzT8sPfic/8waSiLV9Ixs3Re5YREUTtnLq7LoymqB57UXJB3BNz/2\neCueuW71avlWlRtE/wXASj5jx6y5mIrlV4nZbVuyYff0QlcG+fgb6pcJQuO9DxMI\naqFGrWP3zye+LK87a6iR76dS9vRU+bHZpSVvGMKJAoGAFGt3TIKeQtJJyqeUWNSk\nklORNdcOMymYMIlqG+JatXQD1rR6ThgqOt8sgRyJqFCVT++YFMOAqXOBBLnaObZZ\nCFbh1fJ66BlSjoXff0W+SuOx5HuJJAa5+WtFHrPajwxeuRcNa8jwxUsB7n41wADu\nUqWWSRedVBg4Ijbw3nWwYDECgYB0pLew4z4bVuvdt+HgnJA9n0EuYowVdadpTEJg\nsoBjNHV4msLzdNqbjrAqgz6M/n8Ztg8D2PNHMNDNJPVHjJwcR7duSTA6w2p/4k28\nbvvk/45Ta3XmzlxZcZSOct3O31Cw0i2XDVc018IY5be8qendDYM08icNo7vQYkRH\n504kQQKBgQDjx60zpz8ozvm1XAj0wVhi7GwXe+5lTxiLi9Fxq721WDxPMiHDW2XL\nYXfFVy/9/GIMvEiGYdmarK1NW+VhWl1DC5xhDg0kvMfxplt4tynoq1uTsQTY31Mx\nBeF5CT/JuNYk3bEBF0H/Q3VGO1/ggVS+YezdFbLWIRoMnLj6XCFEGg==\n-----END RSA PRIVATE KEY-----\n' != '-----BEGIN PRIVATE KEY-----\nMIIEvgIBADANBgkqhkiG9w0BAQEFAASCBKgwggSkAgEAAoIBAQDh6PSnttDsv+vi\ntUZTP1E3hVBah6PUGDWZhYgNiyW8quTWCmPvBmCR2YzuhUrY5+CtKP8UJOQico+p\noJHSAPsrzSr6YsGs3c9SQOslBmm9Fkh9/f/GZVTVZ6u5AsUmOcVvZ2q7Sz8Vj/aR\naIm0EJqRe9cQ5vvN9sg25rIv4xKwIZJ1VixKWJLmpCmDINqn7xvl+ldlUmSr3aGt\nw21uSDuEJhQlzO3yf2FwJMkJ9SkCm9oVDXyl77OnKXj5bOQ/rojbyGeIxDJSUDWE\nGKyRPuqKi6rSbwg6h2G/Z9qBJkqM5NNTbGRIFz/9/LdmmwvtaqCxlLtD7RVEryAp\n+qTGDk5hAgMBAAECggEBAMYYfNDEYpf4A2SdCLne/9zrrfZ0kphdUkL48MDPj5vN\nTzTRj6f9s5ixZ/+QKn3hdwbguCx13QbH5mocP0IjUhyqoFFHYAWxyyaZfpjM8tO4\nQoEYxby3BpjLe62UXESUzChQSytJZFwIDXKcdIPNO3zvVzufEJcfG5no2b9cIvsG\nDy6J1FNILWxCtDIqBM+G1B1is9DhZnUDgn0iKzINiZmh1I1l7k/4tMnozVIKAfwo\nf1kYjG/d2IzDM02mTeTElz3IKeNriaOIYTZgI26xLJxTkiFnBV4JOWFAZw15X+yR\n+DrjGSIkTfhzbLa20Vt3AFM+LFK0ZoXT2dRnjbYPjQECgYEA+9XJFGwLcEX6pl1p\nIwXAjXKJdju9DDn4lmHTW0Pbw25h1EXONwm/NPafwsWmPll9kW9IwsxUQVUyBC9a\nc3Q7rF1e8ai/qqVFRIZof275MI82ciV2Mw8Hz7FPAUyoju5CvnjAEH4+irt1VE/7\nSgdvQ1gDBQFegS69ijdz+cOhFxkCgYEA5aVoseMy/gIlsCvNPyw9+Jz/zBpKItX0\njGzdF7lhERRO2cursujKaoHntRckHcE3P/Z4K565bvVq+VaVG0T/BcBKPmPHrLmY\niuVXidltW7Jh9/RCVwb5+BvqlwlC470PEwhqoUatY/fPJ74srztrqJHvp1L29FT5\nsdmlJW8YwokCgYAUa3dMgp5C0knKp5RY1KSSU5E11w4zKZgwiWob4lq1dAPWtHpO\nGCo63yyBHImoUJVP75gUw4Cpc4EEudo5tlkIVuHV8nroGVKOhd9/Rb5K47Hke4kk\nBrn5a0Ues9qPDF65Fw1ryPDFSwHufjXAAO5SpZZJF51UGDgiNvDedbBgMQKBgHSk\nt7DjPhtW69234eCckD2fQS5ijBV1p2lMQmCygGM0dXiawvN02puOsCqDPoz+fxm2\nDwPY80cw0M0k9UeMnBxHt25JMDrDan/iTbxu++T/jlNrdebOXFlxlI5y3c7fULDS\nLZcNVzTXwhjlt7yp6d0NgzTyJw2ju9BiREfnTiRBAoGBAOPHrTOnPyjO+bVcCPTB\nWGLsbBd77mVPGIuL0XGrvbVYPE8yIcNbZcthd8VXL/38Ygy8SIZh2ZqsrU1b5WFa\nXUMLnGEODSS8x/GmW3i3KeirW5OxBNjfUzEF4XkJP8m41iTdsQEXQf9DdUY7X+CB\nVL5h7N0VstYhGgycuPpcIUQa\n-----END PRIVATE KEY-----\n'

The issue is (likely?) a difference in some crypto dependency, but we should check that things work (or fail appropriately).

[dhermes]

Did you run via tox or other?

[craigcitro]

both -- fails in tox and directly using /usr/bin/python2.6.

[dhermes]

I'm trying to find documentation, but doesn't Apple ship a fake version of openssl or their own custom substitute not even named openssl?

[dhermes]

On my machines:

$ # Linux
$ openssl version
OpenSSL 1.0.1f 6 Jan 2014

$ # Mac OS X
$ openssl version
OpenSSL 0.9.8za 5 Jun 2014

[dhermes]

So, regarding googleapis/google-cloud-python#537, I tested with this and it works even though OpenSSL 0.9.8za spits out a different PEM key than OpenSSL 1.0.1f. (Like I actually visited the signed URL, gave an expiration of 60 seconds and I could view the content for the whole 60 seconds on both OSes.)

It still means the tests fails on OS X. Can you get one of your auth contacts to weigh in?

[dhermes]

More context:

OS X puts stuff in /System/... and then puts /System/... ahead of standard location (i.e. the one pip uses) on the import path.

SRSLY APPLE? Thanks but no thanks.

On the flip-side: Heartbleed never impacted OS X since they use a custom OpenSSL (at C level, not just Python).

[craigcitro]

@anthmgoogle Anthony, an auth question for you: we're seeing that some of the auth libraries we use for signing behave slightly differently. what OS/openssl version(s) do we want to support? "as many as possible" is my guess, but want to confirm.

@dhermes same command as you on the machine where it fails:

$ which openssl
/usr/bin/openssl
$ openssl version
OpenSSL 0.9.8zc 15 Oct 2014

maybe za and zc differ in some small way?

[dhermes]

@craigcitro We still support Mac OS X. The test just fails because the signed content is different so comparing to pem_from_pkcs12.pem fails.

As I mentioned above, using the signed content still works, i.e. it's still valid.

[craigcitro]

ah, maybe i misread that above -- should we just loosen the test to check for either key?

@anthmgoogle if everything seems to function properly, do you have any strong feelings from the auth side?

[dhermes]

Yeah we could just create the equivalent for the openssl that Apple uses and then use that file in the test if sys.platform == 'darwin'.

EDIT: This will cause failures if people ditch Apple's openssl and install the actual one via Homebrew. This would also require removing the version of PyCrypto that is shipped in /System/....

[craigcitro]

i was thinking something more lowbrow -- the input is fixed, and we have a small finite number of possible outputs. just check that it's one of them and call it a day?

[dhermes]

SGTM

[googlebot]

Thanks for your pull request.

It looks like this may be your first contribution to a Google open source project, in which case you'll need to sign a Contributor License Agreement (CLA) at https://cla.developers.google.com/.

If you've already signed a CLA, it's possible we don't have your GitHub username or you're using a different email address. Check the information on your CLA or see this help article on setting the email on your git commits.

Once you've done that, please reply here to let us know. If you signed the CLA as a corporation, please let us know the company's name.

[googlebot]

CLAs look good, thanks!

[craigcitro]

PTAL @dhermes

[dhermes]

Why don't you name _alternate Mac OS X or something? Other than the history, the test gives no indication of why more than one file is needed.

[craigcitro]

(test failure is because i cancelled the second run when i pushed to correct author; passing version is here: https://travis-ci.org/google/oauth2client/builds/49809411

as to the filename -- i don't think we're going to encode enough into it to nail down the failing version, so it seems misleading. (for instance, it works for you on OSX -- someone else in the same boat might say "oh maybe we can delete this now".)

[dhermes]

I was just suggesting a comment as to the origin of the file. Do as you wish.

[anthmgoogle]

No concerns with checking for either key type.

On Fri, Feb 6, 2015 at 3:54 PM, Craig Citro [email protected]
wrote:

Merged #117 #117.

—
Reply to this email directly or view it on GitHub
#117 (comment).