Skip to content

Java: Simple support for Ratpack HTTP Framework #4991

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 29 commits into from
Oct 27, 2021

Conversation

JLLeitschuh
Copy link
Contributor

Adds some early & very simple support for the Ratpack HTTP framework.

https://ratpack.io/

@JLLeitschuh JLLeitschuh requested a review from a team as a code owner January 20, 2021 20:25
@github-actions github-actions bot added the Java label Jan 20, 2021
Copy link
Contributor

@smowton smowton left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add some tests

@JLLeitschuh
Copy link
Contributor Author

Please add some tests

I was looking into doing this, but there didn't look like there were any sample/examples for spring, nor apache. Do you have a good suggestion to base these tests upon?

@smowton
Copy link
Contributor

smowton commented Jan 22, 2021

You can identify existing tests using a particularly library by looking for the options files used by the test framework that include the stubs of the relevant library. For example, here are the tests for CWE-090 using the Spring framework stubs: https://github.com/github/codeql/blob/main/java/ql/test/query-tests/security/CWE-090/options

All existing library stubs can be found here: https://github.com/github/codeql/tree/main/java/ql/test/stubs

New ones are created by stripping function bodies, essentially producing Java files that document the library's public interface but not method definitions.

@smowton
Copy link
Contributor

smowton commented Apr 12, 2021

@JLLeitschuh do you intend to pursue this?

@JLLeitschuh
Copy link
Contributor Author

I do, I just haven't had the time yet

@JLLeitschuh JLLeitschuh force-pushed the feat/JLL/early_ratpack_support branch from 295e866 to e516537 Compare April 29, 2021 16:16
@JLLeitschuh JLLeitschuh marked this pull request as draft April 30, 2021 00:45
@JLLeitschuh JLLeitschuh marked this pull request as ready for review May 3, 2021 17:03
}

/**
* Methods on `ratpack.http.TypedData` that return user tainted data.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Check: you're sure TypedData, which sounds pretty generic, can always be assumed to come from an untrusted source?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

According to @ldaley, one of the Ratpack authors, almost always.

The actual source of the data would be Request::getBody, where TypedData is where this input gets pulled from the body. However, there are cases in user libraries where UploadedFile (a supertype of TypedData) is instantiated inside a Jackson deserialized object, which isn't otherwise easily traceable.

Now that you're asking, it would probably be better to use TypedData as a taint passthrough and make sure that getBody is captured as a source instead. 🤔

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I refactored this to use getBody as the remote flow source and calls on TypedData to just be taint flow.

@JLLeitschuh JLLeitschuh force-pushed the feat/JLL/early_ratpack_support branch from 106ce24 to eb27f40 Compare May 10, 2021 16:06
smowton
smowton previously approved these changes May 11, 2021
Copy link
Contributor

@smowton smowton left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me; running tests and passing to @aschackmull for final review as this targets the main query pack.

Given that, it will also need a change note @JLLeitschuh

@smowton
Copy link
Contributor

smowton commented May 11, 2021

Check failures:

File "ql/java/ql/test/stubs/ratpack-1.9.x/ratpack/core/handling/Context.java" contains a non-ASCII character at the location marked with | in:

al to the sample given for the {@link #parse(Parse)} variant|

File "ql/java/ql/test/stubs/ratpack-1.9.x/ratpack/core/handling/Handler.java" contains a non-ASCII character at the location marked with | in:

us.
 * That is, there is no expectation that the handler is |

File "ql/java/ql/test/stubs/ratpack-1.9.x/ratpack/exec/Promise.java" contains a non-ASCII character at the location marked with | in:

tMap(Function)}, {@link #cache()} etc.) allow a pipeline of |

File "ql/java/ql/test/stubs/ratpack-1.9.x/ratpack/exec/registry/Registry.java" contains a non-ASCII character at the location marked with | in:

cedence.
   * This means that child entries are effectively |

File "ql/java/ql/test/stubs/ratpack-1.9.x/ratpack/func/Predicate.java" contains a non-ASCII character at the location marked with | in:

 and from the JDK type.
 *
 * @param <T> the type of object |

File "ql/java/ql/test/stubs/ratpack-1.9.x/ratpack/func/Function.java" contains a non-ASCII character at the location marked with | in:

nal(Function, Action)} alternatively to specify a different |

@smowton
Copy link
Contributor

smowton commented May 11, 2021

QLDoc checks fail because file semmle/code/java/frameworks/ratpack/RatpackExec.qll lacks file-level qldoc

@smowton
Copy link
Contributor

smowton commented May 11, 2021

Also the use of type-variables in signatures is apparently going away soon; use the variable's lower bound (usually Object, but String for T extends String) instead

@aschackmull
Copy link
Contributor

Also the use of type-variables in signatures is apparently going away soon; use the variable's lower bound (usually Object, but String for T extends String) instead

Huh? You mean upper bound, right? A lower bound of Object doesn't make sense.
Any way, use the erasure of the type.

@smowton
Copy link
Contributor

smowton commented May 11, 2021

Apparently so, yes. It appears I've been visualising the Java type hierarchy the wrong way up for about 5 years.

@JLLeitschuh
Copy link
Contributor Author

Currently this test fails, I'm wondering if anyone has any insights into why and how I can fix it.

void test9() {
    String tainted = taint();
    Promise
        .value(tainted)
        .apply(Resource::identity)
        .then(value -> {
            sink(value); //$hasTaintFlow // passes
        });
    Promise
        .value("potato")
        .apply(Resource::identity)
        .then(value -> {
            sink(value); // no taints flow // fails here
        });
}

public static Promise<String> identity(Promise<String> input) {
    return input.map(i -> i);
}

@aschackmull
Copy link
Contributor

Currently this test fails, I'm wondering if anyone has any insights into why and how I can fix it.

A quick guess is that it has to do with call contexts getting thrown away (and that's something we have to do due to how you are modelling lambda flow, and part of the reason why we want to do it differently).

@aschackmull
Copy link
Contributor

... call contexts getting thrown away ...

To elaborate: Every time you make an additional taint step that jumps from one callable to another, then the call context needs to be discarded (as it doesn't make sense for the new callable).

@JLLeitschuh
Copy link
Contributor Author

I think that this "stuck" tainted method issue is probably intractable with my current approach. I think for now my best approach is to simply drop support of methods like flatMap.

@JLLeitschuh JLLeitschuh force-pushed the feat/JLL/early_ratpack_support branch from 332d23f to 5a2bdc9 Compare October 18, 2021 16:21
@JLLeitschuh JLLeitschuh marked this pull request as ready for review October 18, 2021 18:28
@JLLeitschuh
Copy link
Contributor Author

@aschackmull @smowton this is now ready for another review round. 🙂

@JLLeitschuh JLLeitschuh force-pushed the feat/JLL/early_ratpack_support branch from 4ba9d75 to db2892b Compare October 18, 2021 18:30
Copy link
Contributor

@aschackmull aschackmull left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some minor comments, otherwise looks good to me.

@JLLeitschuh
Copy link
Contributor Author

Should be resolved. 🙂

aschackmull
aschackmull previously approved these changes Oct 20, 2021
@aschackmull
Copy link
Contributor

aschackmull commented Oct 20, 2021

File "ql/java/ql/test/stubs/ratpack-1.9.x/ratpack/exec/Promise.java" contains a non-ASCII character at the location marked with `|` in:
  ```
  tMap(Function)}, {@link #cache()} etc.) allow a pipeline of |
  ```
  
  File "ql/java/ql/test/stubs/ratpack-1.9.x/ratpack/core/handling/Handler.java" contains a non-ASCII character at the location marked with `|` in:
  ```
  us.
   * That is, there is no expectation that the handler is |
  ```
  
  ASCII check failed!

@JLLeitschuh
Copy link
Contributor Author

Resolved the non-ASCII character issue

@aschackmull
Copy link
Contributor

There are more non-ascii chars. In particular, the two mentioned in the error message I quoted are still there.

@JLLeitschuh JLLeitschuh force-pushed the feat/JLL/early_ratpack_support branch from 00c6257 to 5eb2839 Compare October 22, 2021 14:53
@github-actions
Copy link
Contributor

⚠️ The head of this PR and the base branch were compared for differences in the framework coverage reports. The generated reports are available in the artifacts of this workflow run. The differences will be picked up by the nightly job after the PR gets merged.

Click to show differences in coverage

java

Generated file changes for java

  • Changes to framework-coverage-java.rst:
-    Others,"``cn.hutool.core.codec``, ``com.esotericsoftware.kryo.io``, ``com.esotericsoftware.kryo5.io``, ``com.fasterxml.jackson.core``, ``com.fasterxml.jackson.databind``, ``com.opensymphony.xwork2.ognl``, ``com.unboundid.ldap.sdk``, ``flexjson``, ``groovy.lang``, ``groovy.util``, ``jodd.json``, ``net.sf.saxon.s9api``, ``ognl``, ``org.apache.commons.codec``, ``org.apache.commons.jexl2``, ``org.apache.commons.jexl3``, ``org.apache.commons.ognl``, ``org.apache.directory.ldap.client.api``, ``org.apache.ibatis.jdbc``, ``org.apache.shiro.codec``, ``org.apache.shiro.jndi``, ``org.codehaus.groovy.control``, ``org.dom4j``, ``org.hibernate``, ``org.jooq``, ``org.mvel2``, ``org.xml.sax``, ``org.xmlpull.v1``, ``play.mvc``",7,28,151,,,,14,18,,
+    Others,"``cn.hutool.core.codec``, ``com.esotericsoftware.kryo.io``, ``com.esotericsoftware.kryo5.io``, ``com.fasterxml.jackson.core``, ``com.fasterxml.jackson.databind``, ``com.opensymphony.xwork2.ognl``, ``com.unboundid.ldap.sdk``, ``flexjson``, ``groovy.lang``, ``groovy.util``, ``jodd.json``, ``net.sf.saxon.s9api``, ``ognl``, ``org.apache.commons.codec``, ``org.apache.commons.jexl2``, ``org.apache.commons.jexl3``, ``org.apache.commons.ognl``, ``org.apache.directory.ldap.client.api``, ``org.apache.ibatis.jdbc``, ``org.apache.shiro.codec``, ``org.apache.shiro.jndi``, ``org.codehaus.groovy.control``, ``org.dom4j``, ``org.hibernate``, ``org.jooq``, ``org.mvel2``, ``org.xml.sax``, ``org.xmlpull.v1``, ``play.mvc``, ``ratpack.core.form``, ``ratpack.core.handling``, ``ratpack.core.http``, ``ratpack.exec``, ``ratpack.form``, ``ratpack.func``, ``ratpack.handling``, ``ratpack.http``, ``ratpack.util``",39,99,151,,,,14,18,,
-    Totals,,143,5261,408,13,6,10,107,33,1,66
+    Totals,,175,5332,408,13,6,10,107,33,1,66
  • Changes to framework-coverage-java.csv:
- com.fasterxml.jackson.databind,,,5,,,,,,,,,,,,,,,,,,,,,5,
+ com.fasterxml.jackson.databind,,,6,,,,,,,,,,,,,,,,,,,,,6,
+ ratpack.core.form,,,3,,,,,,,,,,,,,,,,,,,,,3,
+ ratpack.core.handling,,6,4,,,,,,,,,,,,,,,,,,,,6,4,
+ ratpack.core.http,,10,10,,,,,,,,,,,,,,,,,,,,10,10,
+ ratpack.exec,,,26,,,,,,,,,,,,,,,,,,,,,,26
+ ratpack.form,,,3,,,,,,,,,,,,,,,,,,,,,3,
+ ratpack.func,,,5,,,,,,,,,,,,,,,,,,,,,,5
+ ratpack.handling,,6,4,,,,,,,,,,,,,,,,,,,,6,4,
+ ratpack.http,,10,10,,,,,,,,,,,,,,,,,,,,10,10,
+ ratpack.util,,,5,,,,,,,,,,,,,,,,,,,,,,5

@JLLeitschuh
Copy link
Contributor Author

Oof... I cleaned up the character, but I didn't see the weird quote character. Should be cleaned up now.

@aschackmull
Copy link
Contributor

There are some more non-ascii quotes in Promise.java.

@github-actions
Copy link
Contributor

⚠️ The head of this PR and the base branch were compared for differences in the framework coverage reports. The generated reports are available in the artifacts of this workflow run. The differences will be picked up by the nightly job after the PR gets merged.

Click to show differences in coverage

java

Generated file changes for java

  • Changes to framework-coverage-java.rst:
-    Others,"``cn.hutool.core.codec``, ``com.esotericsoftware.kryo.io``, ``com.esotericsoftware.kryo5.io``, ``com.fasterxml.jackson.core``, ``com.fasterxml.jackson.databind``, ``com.opensymphony.xwork2.ognl``, ``com.unboundid.ldap.sdk``, ``flexjson``, ``groovy.lang``, ``groovy.util``, ``jodd.json``, ``net.sf.saxon.s9api``, ``ognl``, ``org.apache.commons.codec``, ``org.apache.commons.jexl2``, ``org.apache.commons.jexl3``, ``org.apache.commons.ognl``, ``org.apache.directory.ldap.client.api``, ``org.apache.ibatis.jdbc``, ``org.apache.shiro.codec``, ``org.apache.shiro.jndi``, ``org.codehaus.groovy.control``, ``org.dom4j``, ``org.hibernate``, ``org.jooq``, ``org.mvel2``, ``org.xml.sax``, ``org.xmlpull.v1``, ``play.mvc``",7,28,151,,,,14,18,,
+    Others,"``cn.hutool.core.codec``, ``com.esotericsoftware.kryo.io``, ``com.esotericsoftware.kryo5.io``, ``com.fasterxml.jackson.core``, ``com.fasterxml.jackson.databind``, ``com.opensymphony.xwork2.ognl``, ``com.unboundid.ldap.sdk``, ``flexjson``, ``groovy.lang``, ``groovy.util``, ``jodd.json``, ``net.sf.saxon.s9api``, ``ognl``, ``org.apache.commons.codec``, ``org.apache.commons.jexl2``, ``org.apache.commons.jexl3``, ``org.apache.commons.ognl``, ``org.apache.directory.ldap.client.api``, ``org.apache.ibatis.jdbc``, ``org.apache.shiro.codec``, ``org.apache.shiro.jndi``, ``org.codehaus.groovy.control``, ``org.dom4j``, ``org.hibernate``, ``org.jooq``, ``org.mvel2``, ``org.xml.sax``, ``org.xmlpull.v1``, ``play.mvc``, ``ratpack.core.form``, ``ratpack.core.handling``, ``ratpack.core.http``, ``ratpack.exec``, ``ratpack.form``, ``ratpack.func``, ``ratpack.handling``, ``ratpack.http``, ``ratpack.util``",39,99,151,,,,14,18,,
-    Totals,,143,5261,408,13,6,10,107,33,1,66
+    Totals,,175,5332,408,13,6,10,107,33,1,66
  • Changes to framework-coverage-java.csv:
- com.fasterxml.jackson.databind,,,5,,,,,,,,,,,,,,,,,,,,,5,
+ com.fasterxml.jackson.databind,,,6,,,,,,,,,,,,,,,,,,,,,6,
+ ratpack.core.form,,,3,,,,,,,,,,,,,,,,,,,,,3,
+ ratpack.core.handling,,6,4,,,,,,,,,,,,,,,,,,,,6,4,
+ ratpack.core.http,,10,10,,,,,,,,,,,,,,,,,,,,10,10,
+ ratpack.exec,,,26,,,,,,,,,,,,,,,,,,,,,,26
+ ratpack.form,,,3,,,,,,,,,,,,,,,,,,,,,3,
+ ratpack.func,,,5,,,,,,,,,,,,,,,,,,,,,,5
+ ratpack.handling,,6,4,,,,,,,,,,,,,,,,,,,,6,4,
+ ratpack.http,,10,10,,,,,,,,,,,,,,,,,,,,10,10,
+ ratpack.util,,,5,,,,,,,,,,,,,,,,,,,,,,5

@aschackmull
Copy link
Contributor

Looks like there's at least one more in Promise.java (on the same line as the one you just removed).

@JLLeitschuh
Copy link
Contributor Author

I'm truly pulling my hair out over these. The VS-code "find" feature isn't doing a good job finding all of the non-ascii quotes. I think I got the last one finally. 😫

@github-actions
Copy link
Contributor

⚠️ The head of this PR and the base branch were compared for differences in the framework coverage reports. The generated reports are available in the artifacts of this workflow run. The differences will be picked up by the nightly job after the PR gets merged.

Click to show differences in coverage

java

Generated file changes for java

  • Changes to framework-coverage-java.rst:
-    Others,"``cn.hutool.core.codec``, ``com.esotericsoftware.kryo.io``, ``com.esotericsoftware.kryo5.io``, ``com.fasterxml.jackson.core``, ``com.fasterxml.jackson.databind``, ``com.opensymphony.xwork2.ognl``, ``com.unboundid.ldap.sdk``, ``flexjson``, ``groovy.lang``, ``groovy.util``, ``jodd.json``, ``net.sf.saxon.s9api``, ``ognl``, ``org.apache.commons.codec``, ``org.apache.commons.jexl2``, ``org.apache.commons.jexl3``, ``org.apache.commons.ognl``, ``org.apache.directory.ldap.client.api``, ``org.apache.ibatis.jdbc``, ``org.apache.shiro.codec``, ``org.apache.shiro.jndi``, ``org.codehaus.groovy.control``, ``org.dom4j``, ``org.hibernate``, ``org.jooq``, ``org.mvel2``, ``org.xml.sax``, ``org.xmlpull.v1``, ``play.mvc``",7,28,151,,,,14,18,,
+    Others,"``cn.hutool.core.codec``, ``com.esotericsoftware.kryo.io``, ``com.esotericsoftware.kryo5.io``, ``com.fasterxml.jackson.core``, ``com.fasterxml.jackson.databind``, ``com.opensymphony.xwork2.ognl``, ``com.unboundid.ldap.sdk``, ``flexjson``, ``groovy.lang``, ``groovy.util``, ``jodd.json``, ``net.sf.saxon.s9api``, ``ognl``, ``org.apache.commons.codec``, ``org.apache.commons.jexl2``, ``org.apache.commons.jexl3``, ``org.apache.commons.ognl``, ``org.apache.directory.ldap.client.api``, ``org.apache.ibatis.jdbc``, ``org.apache.shiro.codec``, ``org.apache.shiro.jndi``, ``org.codehaus.groovy.control``, ``org.dom4j``, ``org.hibernate``, ``org.jooq``, ``org.mvel2``, ``org.xml.sax``, ``org.xmlpull.v1``, ``play.mvc``, ``ratpack.core.form``, ``ratpack.core.handling``, ``ratpack.core.http``, ``ratpack.exec``, ``ratpack.form``, ``ratpack.func``, ``ratpack.handling``, ``ratpack.http``, ``ratpack.util``",39,99,151,,,,14,18,,
-    Totals,,143,5261,408,13,6,10,107,33,1,66
+    Totals,,175,5332,408,13,6,10,107,33,1,66
  • Changes to framework-coverage-java.csv:
- com.fasterxml.jackson.databind,,,5,,,,,,,,,,,,,,,,,,,,,5,
+ com.fasterxml.jackson.databind,,,6,,,,,,,,,,,,,,,,,,,,,6,
+ ratpack.core.form,,,3,,,,,,,,,,,,,,,,,,,,,3,
+ ratpack.core.handling,,6,4,,,,,,,,,,,,,,,,,,,,6,4,
+ ratpack.core.http,,10,10,,,,,,,,,,,,,,,,,,,,10,10,
+ ratpack.exec,,,26,,,,,,,,,,,,,,,,,,,,,,26
+ ratpack.form,,,3,,,,,,,,,,,,,,,,,,,,,3,
+ ratpack.func,,,5,,,,,,,,,,,,,,,,,,,,,,5
+ ratpack.handling,,6,4,,,,,,,,,,,,,,,,,,,,6,4,
+ ratpack.http,,10,10,,,,,,,,,,,,,,,,,,,,10,10,
+ ratpack.util,,,5,,,,,,,,,,,,,,,,,,,,,,5

Copy link
Contributor

@aschackmull aschackmull left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎉

@aschackmull aschackmull merged commit 4a67ac5 into github:main Oct 27, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants