Handling gRPC Proto Requests with SSL Certificate dependency via JMeter

Sharing a challenging & interesting problem statement resolution cracked to execute Load Testing on gRPC Proto based request bind with SSL certificate.

Need for Speed :)

We have been supporting Load Testing of REST API calls and things are pretty smooth with that in our Organization. Over the period of time, gradually for few services we shifted towards gRPC Proto from REST based architecture and it became a requisite for us to support Load Testing of such services too.

Striking Challenges

1. To understand & handle required Proto structure support via JMeter.
2. To get some knowledge transfer from internal engineering team in understanding the way they are testing proto requests.
3. To pass SSL certificate required to authenticate server via JMeter
4. To understand the certificate types & its structure before passing .
5. To understand the different error types occurrence while handling Proto & SSL Certificate.

Navigation Solution

1. A 3rd party JMeter- gRPC- request plugin helped out in seeking the right path towards the required direction.
2. Bloomrpc client ( open source) tool also helped in understanding the easy way to test gRPC requests. It’s simple & clean UI is a perfect base to trigger requests.
3. JMeter accepts certificate types either JKS or PKCS12 and have to pass through system.properties file of JMeter.
4. Original certificate to authenticate app server got shared as crt file. So have to convert it to .jks ( Java Keystore) via KeyStore Explorer ( open source tool ) as stated above — JMeter allows only passing of JKS or PKCS12 extension certificates. This tool offers a simple UI to understand the certificate types & its keys ( public / private) structure. Keytool command line is another alternative here. Appreciating the support of Felix in helping this out.
5. Understanding on errors related to gRPC based Proto requests can be done via this site.

Assembling Components

1. Added grpc jar inside JMeter to add grpc request in JMeter test suite.

JMeter gRPC Request Plugin

2. Received proto from Engineering Team having service method required to execute and configured bloomrpc to test that. Thanks to Pankaj Khandelwal for helping out for proto understanding & setup.

BloomRPC Client

3. Converted .crt ( app server authentication) certificate to .jks and configured against truststore entries in system.properties JMeter file.

4.KeyStore Explorer ( as mentioned above) can be used to identify the certificate type & convert it to required format ( .jks / PKCS12) for JMeter

KeyStore Explorer

5. Here keytool command also can be used to identify certificate type — truststore or keystore which is very important while configuring JMeter. System.properties file.

Keystore vs Truststore

Problems Faced

1. Error while executing request via version JMeter 4.0 java.lang.NoSuchMethodError: com.google.common.net.HostAndPort.getHost()Ljava/lang/String;

Solution: Update with guava 23.0 jar file in JMeter 4.0 OR switch to updated JMeter 5.4.1 version to avoid updating jar file.

2. Error -> SSLManager: Problem loading keystore: No key(s) found
java.io.IOException: No key(s) found.

Solution: Configure certificate in JMeter against Truststore entry instead of Keystore. This depends on certificate type identification as explained above.

3. Error -> Response code: 500 Response message: Exception: io.grpc.StatusRuntimeException: UNKNOWN Caught exception while waiting for rpc

Solution : This problem consumed much time to get cracked as very less information is available over internet related to this specific problem related to JMeter. Hence after multiple hit & trials on live server with app server certificate, we further decided to test this out on local to filter out if this error is related to any further required certificate dependency or any problem with 3rd party JMeter gRPC plugin.

So here Aman Garg done set up on his local and created certificate locally for testing using openssl. Thereafter We replicated test cases with & without certificate dependency on local and concluded that there was NO ISSUE with either certificate or JMeter plugin. So what further debugging pointed out issue was with the format of passing Auth token in metadata i.e. shouldn’t get passed in string format. And post giving token in required format, finally we got success response. Thanks to Tarun Wadhwa for his suggestions & feedback on certificate & server access problem.

JMeter Proto Request Success Response — 200 OK

TakeAway & Credits

This Project has boosted Knowledge & Confidence in —

>> Load test gRPC proto requests smoothly & learning Proto structuring.

>>Understanding SSL Certificate & it’s conversion to pass in JMeter request.

>> Learning Truststore vs Keystore & their usage per certificate type.

Many thanks to @Felix Schumacher ( JMeter community ), Pankaj Khandelwal , Aman Garg, Tarun Wadhwa, Om Vikram Thapa for their real time contribution & support in making this project success and live whilst solving typical hurdles.

Cheers for Victory ✌️