java java.net.http.HttpClient returns http status 403 while curl works fine

Question

I'm trying to fetch a json string from a (GraphQL ?) server. The input is a json string + POST request and the ouput is a json document.

When using curl, it works (well , let's be honest it works if you don't call the server too many times).

 curl -v  'https://afb.ukbiobank.ac.uk/api' -X POST -H 'User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:137.0) Gecko/20100101 Firefox/137.0' -H 'Accept: application/json, text/plain, */*' -H 'Accept-Language: en-US,en;q=0.5'  -H 'Content-Type: application/json' -H 'Origin: https://afb.ukbiobank.ac.uk' -H 'Connection: keep-alive' -H 'Referer: https://afb.ukbiobank.ac.uk/variant/chr3-38435047-A-G'  --data-raw '{"query":"\nquery VariantSplitPopulationQuery($chrom: String!, $pos: Int!, $ref: String!, $alt: String!) {\n    variant(chrom: $chrom, pos: $pos, ref: $ref, alt: $alt) {\n      Chrom\n      Pos\n      Ref\n      Alt\n      alleleCount\n      alleleNum\n      alleleFreq\n      nHomozygotes\n      maxImpact\n      maxConsequence\n      HGVSp\n      geneSymbol\n      populationDetails {\n        population\n        alleleCount\n        alleleNum\n        alleleFreq\n        nHomozygotes\n        nHemiAlt\n      }\n    }\n  } \n","variables":{"chrom":"chr3","pos":38435047,"ref":"A","alt":"G"}}'
(...)
> POST /api HTTP/2
> Host: afb.ukbiobank.ac.uk
> User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:137.0) Gecko/20100101 Firefox/137.0
> Accept: application/json, text/plain, */*
> Accept-Language: en-US,en;q=0.5
> Content-Type: application/json
> Origin: https://afb.ukbiobank.ac.uk
> Connection: keep-alive
> Referer: https://afb.ukbiobank.ac.uk/variant/chr3-38435047-A-G
> Content-Length: 596
(..)
< 
{"data":{"variant":{"Chrom":"chr3","Pos":38435047,"Ref":"A","Alt":"G","alleleCount":587506,"alleleNum":980494,"alleleFreq":0.5991938757401881,"nHomozygotes":178020,"maxImpact":"LOWEST","maxConsequence":"intergenic_variant","HGVSp":".","geneSymbol":"XYLB","populationDetails":[{"population":"African","alleleCount":3779,"alleleNum":18454,"alleleFreq":0.20477945160940716,"nHomozygotes":438,"nHemiAlt":null},{"population":"Ashkenazi Jewish","alleleCount":3032,"alleleNum":5714,"alleleFreq":0.5306265313265663,"nHomozygotes":796,"nHemiAlt":null},{"population":"East Asian","alleleCount":2101,"alleleNum":4484,"alleleFreq":0.4685548617305977,"nHomozygotes":507,"nHemiAlt":null},{"population":"Non-Finnish European","alleleCount":558994,"alleleNum":917198,"alleleFreq":0.6094583721290278,"nHomozygotes":170620,"nHemiAlt":null},{"population":"Other","alleleCount":8312,"alleleNum":15340,"alleleFreq":0.5418513689700131,"nHomozygotes":2309,"nHemiAlt":null},{"population":"South Asian","alleleCount":11288,"alleleNum":19304,"alleleFreq":0.5847492747617075,"nHomozygotes":3350,"nHemiAlt":null}]}}}

but when I want to use a java client, it always returns a code 403 (and a HTML page instead of a json). Here is a minimal example which use the query above.


import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;

public class Test {

    public static void main(final String[] args) {
        try {
            final String query="{\n" +
                    "    \"query\": \"\nquery VariantSplitPopulationQuery($chrom: String!, $pos: Int!, $ref: String!, $alt: String!) {\n    variant(chrom: $chrom, pos: $pos, ref: $ref, alt: $alt) {\n      Chrom\n      Pos\n      Ref\n      Alt\n      alleleCount\n      alleleNum\n      alleleFreq\n      nHomozygotes\n      maxImpact\n      maxConsequence\n      HGVSp\n      geneSymbol\n      populationDetails {\n        population\n        alleleCount\n        alleleNum\n        alleleFreq\n        nHomozygotes\n        nHemiAlt\n      }\n    }\n  } \n\",\n" +
                    "    \"variables\": {\n" +
                    "        \"chrom\": \"chr3\",\n" +
                    "        \"pos\": 38435047,\n" +
                    "        \"ref\": \"A\",\n" +
                    "        \"alt\": \"G\"\n" +
                    "    }\n" +
                    "}\n";
            HttpClient httpClient = HttpClient.newHttpClient();
            
        
            HttpRequest.Builder hb= HttpRequest.newBuilder();
            
            hb.header("User-Agent","Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:137.0) Gecko/20100101 Firefox/137.0");
            hb.header("Accept","application/json, text/plain, */*");
            hb.header("Accept-Language","en-US,en;q=0.5");
            hb.header("Content-Type","application/json");
            hb.header("Origin","https://afb.ukbiobank.ac.uk");
            hb.header("Referer","https://afb.ukbiobank.ac.uk/variant/chr3-38435047-A-G");
            

            hb.uri(URI.create("https://afb.ukbiobank.ac.uk/api"));
            hb.POST(HttpRequest.BodyPublishers.ofString(query));
            HttpRequest request = hb.build();
            
            
            HttpResponse<String> resp= httpClient.send(request, HttpResponse.BodyHandlers.ofString());
            System.err.println(resp.body());
            System.err.println(resp.statusCode());
                
        }
        catch(Throwable err) {
            err.printStackTrace();
        }
    }
}

output is:

<!DOCTYPE html><h(...)
403

How can I make the java program works ? Thanks !

EDIT: I also tested with '\n' instead of '\n' in the json string but got the Same result:

(...)
"    \"query\": \"\\nquery VariantSplitPopulationQuery($chrom: String!, $pos: Int!, $ref: String!, $alt: String!) {\\n    variant(chrom: $chrom, pos: $pos, ref: $ref, alt: $alt) {\\n      Chrom\\n      Pos\\n      Ref\\n      Alt\\n      alleleCount\\n      alleleNum\\n      alleleFreq\\n      nHomozygotes\\n      maxImpact\\n      maxConsequence\\n      HGVSp\\n      geneSymbol\\n      populationDetails {\\n        population\\n        alleleCount\\n        alleleNum\\n        alleleFreq\\n        nHomozygotes\\n        nHemiAlt\\n      }\\n    }\\n  } \\n\",\n" +

(...)

Don’t repeat the header name in a header value. This is wrong: hb.header("Content-Type","Content-Type: application/json"); This is correct: hb.header("Content-Type", "application/json"); Notice that “Content-Type” is in the first string argument, but not in the second argument. — VGR, Commented Apr 23 at 17:31
@VGR thanks ! I'll update the code above but I still have the same error . — Pierre, Commented Apr 23 at 18:15
I see a typo in your request body: \"pos\": :38435047\", But I would expect that to produce an HTTP 400, not an HTTP 403. You also have a string value in your JSON which contains newlines. I don’t think JSON allows that, but it does allow a two-character escape. So, every \n that follows \"query\": should be changed from \n (a literal linefeed character in a JSON string) to \\n (a two-character escape sequence in a JSON string). — VGR, Commented Apr 23 at 18:35
Rhha, thanks again, I did those errors when I reduced my original code to a minimal example. In my original code the json string is validated with the gson library. But I'll try with the escaped '\n'... — Pierre, Commented Apr 23 at 18:47

Arno · Accepted Answer · 2025-04-23 19:50:09Z

Just copied your body from curl to a one line Java String.

final String query= "{\"query\":\"\\nquery VariantSplitPopulationQuery($chrom: String!, $pos: Int!, $ref: String!, $alt: String!) {\\n    variant(chrom: $chrom, pos: $pos, ref: $ref, alt: $alt) {\\n      Chrom\\n      Pos\\n      Ref\\n      Alt\\n      alleleCount\\n      alleleNum\\n      alleleFreq\\n      nHomozygotes\\n      maxImpact\\n      maxConsequence\\n      HGVSp\\n      geneSymbol\\n      populationDetails {\\n        population\\n        alleleCount\\n        alleleNum\\n        alleleFreq\\n        nHomozygotes\\n        nHemiAlt\\n      }\\n    }\\n  } \\n\",\"variables\":{\"chrom\":\"chr3\",\"pos\":38435047,\"ref\":\"A\",\"alt\":\"G\"}}";

Works like charme

btw. you should use HttpClient.newHttpClient() in a try block for using autoclose:

HttpResponse<String> resp;
final String query = "{\"query\":\"\\nquery VariantSplitPopulationQuery($chrom: String!, $pos: Int!, $ref: String!, $alt: String!) {\\n    variant(chrom: $chrom, pos: $pos, ref: $ref, alt: $alt) {\\n      Chrom\\n      Pos\\n      Ref\\n      Alt\\n      alleleCount\\n      alleleNum\\n      alleleFreq\\n      nHomozygotes\\n      maxImpact\\n      maxConsequence\\n      HGVSp\\n      geneSymbol\\n      populationDetails {\\n        population\\n        alleleCount\\n        alleleNum\\n        alleleFreq\\n        nHomozygotes\\n        nHemiAlt\\n      }\\n    }\\n  } \\n\",\"variables\":{\"chrom\":\"chr3\",\"pos\":38435047,\"ref\":\"A\",\"alt\":\"G\"}}";
try (HttpClient httpClient = HttpClient.newHttpClient()) {
  HttpRequest.Builder hb = HttpRequest.newBuilder();

  hb.header("User-Agent", "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:137.0) Gecko/20100101 Firefox/137.0");
  hb.header("Accept", "application/json, text/plain, */*");
  hb.header("Accept-Language", "en-US,en;q=0.5");
  hb.header("Content-Type", "application/json");
  hb.header("Origin", "https://afb.ukbiobank.ac.uk");
  hb.header("Referer", "https://afb.ukbiobank.ac.uk/variant/chr3-38435047-A-G");

  hb.uri(URI.create("https://afb.ukbiobank.ac.uk/api"));
  hb.POST(HttpRequest.BodyPublishers.ofString(query));
  HttpRequest request = hb.build();

  resp = httpClient.send(request, HttpResponse.BodyHandlers.ofString());
  System.err.println(resp.body());
  System.err.println(resp.statusCode());

} catch (Throwable err) {
  err.printStackTrace();
}

Thanks. I'm using jdk17. HttpClient which doesn't implement AutoCloseable docs.oracle.com/en/java/javase/17/docs/api/java.net.http/java/… (but 22 do). On my side , the code above, without the try-with-resource , doesn't work. I still have a 403 while curl is still working. — Pierre, Commented Apr 23 at 20:26
I just tested with jdk23, , on another server, same error AND the try-with-resource and it works ! hum... puzzling... — Pierre, Commented Apr 23 at 20:30
and now it works with the original code I'm lost. I'm going to validate your answer but there is something I don't understand. — Pierre, Commented Apr 23 at 20:33
AH ! I added hb.version(Version.HTTP_2); and it now works with jdk17 ! — Pierre, Commented Apr 23 at 20:38

Collectives™ on Stack Overflow

java java.net.http.HttpClient returns http status 403 while curl works fine

1 Answer 1

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Related