S-J2D2 stands for Systemd-Java-JSON-Dynamic-Data. It is a way to build web services with several benefits that can be summarized as:
Read more about these below. But, before that, let's get into the technical details.
HTTP is a complex protocol. An implementation of all its aspects can be very complicated.
In this approach, it is only treated as a wrapper for a JSON message. Thus, only the POST request type is used, and the only other headers that really matter are the Content-Length header, which gives the length of the JSON message, and the Host header, which gives which virtual host that is requested. The request is always expected to be JSON and the datatype application/json.
Also, the respons is basically only code 200 and a JSON message.
An implementation of more than this must deliver a huge set of functionality for the declarative language of HTTP headers. For example, different HTTP versions, the different methods, paths, query parameters, status codes, accepts, cache control, encodings, keep-alives, etags and much, much more. For example, with the cache headers, the web server is expected to provide caching as expressed in the header...
In the approach described here, the web server gets a JSON message and decides to do more if needed.
When we get JSON from the wild on a web service, we need to verify that it is correct. We can read in the JSON into a static data structure using a library that uses reflection to directly insert data into the data structure. The drawback of this is the limitations of Java classes to specify constraints on the data.
Here are some examples:
In general, the approach described here is highly imperative. The program is set up to create what is needed imperiatively instead of declaratively: HTTP is used as a message wrapper, builds are compiled Java files with imperative code, deployments are simple installations of imperative programs.
The only thing we need to run a web service using this technique, is a plain install of Ubuntu + the headless OpenJDK JRE:
# apt install openjdk-17-jre-headless
We can then use ssh, scp and systemctl to deploy and start our service.
A basic webservice takes a string as input and gives a string as output. We thus needs to implement the following function. Notice that this does not involve HTTP headers or other request types than POST. It is, however, a general form of webservice taking data as input and returning data as output.
static public void CommandHandler(StringBuilder response, String request)
We can create the required HTTP wrapper for this function using the built-in Java web server, found in com.sun.net.httpserver. The complete source code is below.
Change the port 8000 to listen to another port. Set the IP to 0.0.0.0 to listen on all IPs, or a particular one to only listen to it. The last parameter, 32, to create, is the backlog of requests the server should support.
import java.io.*;
import java.net.InetSocketAddress;
import java.nio.charset.StandardCharsets;
import java.util.concurrent.Executors;
import com.sun.net.httpserver.HttpExchange;
import com.sun.net.httpserver.HttpHandler;
import com.sun.net.httpserver.HttpServer;
public class WebService {
public static void main(String[] args) throws Exception {
HttpServer server;
server = HttpServer.create(new InetSocketAddress("127.0.0.1", 8000), 32);
server.createContext("/", new MyHandler());
server.setExecutor(Executors.newFixedThreadPool(32));
server.start();
}
static class MyHandler implements HttpHandler {
@Override
public void handle(HttpExchange t) throws IOException {
InputStream is = t.getRequestBody();
String request = GetStringFromInputStream(is);
StringBuilder responseSB = new StringBuilder();
CommandHandler(responseSB, request);
String response = responseSB.toString();
t.sendResponseHeaders(200, response.length());
OutputStream os = t.getResponseBody();
os.write(response.getBytes());
os.close();
}
}
public static String GetStringFromInputStream(InputStream is) throws IOException {
BufferedInputStream bis = new BufferedInputStream(is);
ByteArrayOutputStream buf = new ByteArrayOutputStream();
for (int result = bis.read(); result != -1; result = bis.read()) {
buf.write((byte) result);
}
return buf.toString(StandardCharsets.UTF_8);
}
static public void CommandHandler(StringBuilder response, String request){
System.out.println("Got: " + request + "\n");
response.append("Got: " + request + "\n");
}
}
We can build this easily:
javac src/WebService.java -d out/
jar cfe webservice.jar WebService -C out/ .
Then run it:
java -jar webservice.jar
We can try this out using curl.
$ curl http://localhost:8000/ --data '{}'
Got: {}
We now have a web service running.
This server is single threaded by default. This is a useful default for many cases.
server.setExecutor(null);
If multi-threading is needed, we can add an import:
import java.util.concurrent.Executors;
And change one line. The number given in the parameter is how many concurrent threads will be handling requests.
server.setExecutor(Executors.newFixedThreadPool(2));
You can test running it with systemd as follows.
$ systemd-run --user --unit webservice java -jar <directory>/webserivce.jar
Check the logs using
$ journalctl --user -u webservice
To stop it, run:
$ systemctl --user stop webservice
$ systemctl --user reset-failed webservice
To install the service permanently, create the following file
/etc/systemd/system/webservice.service
and, fill it with:
[Unit]
Description=Web Service
Documentation=https://progsbase.com
After=network.target
# Disable restart limit
StartLimitIntervalSec=0
[Service]
WorkingDirectory=/opt/webservice/
Type=simple
User=webservice
ExecStart=/usr/bin/java -jar webservice.jar
Restart=always
[Install]
WantedBy=multi-user.target
You need to place webservice.jar in /opt/webservice/ and create the user webservice.
Start the service using:
# systemctl start webservice
Enable it at startup using:
# systemctl enable webservice
Check the logs using:
# journalctl -u webservice
JSON provides a good way to serialize and deserialize data. We have developed a library where we can easily construct dynamic data structures in Java.
We can change the signature of our central function to take a dynamic data structure as input and return a dynamic data structure as output.
void CommandHandler(DataReference response, Data request);
We need a file src/info.json.
{
"name": "WebService",
"version": "0.1.0-SNAPSHOT",
"organization namespace": "com.progsbase.sj2d2",
"scientific namespace": "other.other.other",
"imports": [
["", "no.inductive.libraries", "DataStructures", "0.1.5"],
["", "no.inductive.idea10.programs", "JSON", "0.5.10"]
],
"development imports": [],
"ownerCustomerId": "Inductive AS"
}
And then import the libraries:
$ progsbase importdeps Java IDE
We also need to modify our wrapper:
import DataStructures.Array.Structures.Data;
import DataStructures.Array.Structures.DataReference;
import com.sun.net.httpserver.HttpExchange;
import com.sun.net.httpserver.HttpHandler;
import com.sun.net.httpserver.HttpServer;
import references.references.StringReference;
import java.io.*;
import java.net.InetSocketAddress;
import java.nio.charset.StandardCharsets;
import java.util.concurrent.Executors;
import static DataStructures.Array.Structures.Structures.AddStringToStruct;
import static DataStructures.Array.Structures.Structures.CreateNewStructData;
import static JSON.Parser.Parser.ReadJSON;
import static JSON.Writer.Writer.WriteJSON;
public class WebServiceDD {
public static void main(String[] args) throws Exception {
HttpServer server;
server = HttpServer.create(new InetSocketAddress("127.0.0.1", 8000), 32);
server.createContext("/", new MyHandler());
server.setExecutor(Executors.newFixedThreadPool(32));
server.start();
}
static class MyHandler implements HttpHandler {
@Override
public void handle(HttpExchange t) throws IOException {
boolean success;
int code;
StringReference message = new StringReference();
InputStream is = t.getRequestBody();
String request = GetStringFromInputStream(is);
DataReference requestData = new DataReference();
success = ReadJSON(request.toCharArray(), requestData, message);
String response;
if(success) {
DataReference responseData = new DataReference();
CommandHandler(responseData, requestData.data);
t.getResponseHeaders().add("Content-Type", "application/json");
response = new String(WriteJSON(responseData.data));
code = 200;
}else{
response = new String(message.string);
code = 400;
}
t.sendResponseHeaders(code, response.length());
OutputStream os = t.getResponseBody();
os.write(response.getBytes());
os.close();
}
}
public static String GetStringFromInputStream(InputStream is) throws IOException {
BufferedInputStream bis = new BufferedInputStream(is);
ByteArrayOutputStream buf = new ByteArrayOutputStream();
for (int result = bis.read(); result != -1; result = bis.read()) {
buf.write((byte) result);
}
return buf.toString(StandardCharsets.UTF_8);
}
static public void CommandHandler(DataReference response, Data request){
response.data = CreateNewStructData();
AddStringToStruct(response.data.structure, "key".toCharArray(), "value".toCharArray());
System.out.println("Got: " + new String(WriteJSON(request)) + "\n");
}
}
And, the build requires the imports on the class path:
$ javac -cp imports src/WebServiceDD.java -d out/
$ jar cfe webservice.jar WebService -C out/ .
We can now send and receive JSON from our web server.
$ curl -s http://localhost:8000/ --data '{}' | jq
{
"key": "value"
}
We can easily serialize and deserialize JSON. If we have JSON in a string, simply run:
char [] json = ...;
StringReference message = new StringReference();
DataReference dataRef = new DataReference();
boolean success = ReadJSON(json, dataRef, message);
if(success){
Data data = dataRef.data;
...
}else{
// message.string containts the error message.
}
If we want JSON from a dynamic data structure, run:
Data data = ...;
char [] json = WriteJSON(data);
Data is either a number, a boolean, a string, a structure or an array. If we have a structure or an array, we can write it directly.
Structure struct = ...;
char [] json = WriteJSONFromStruct(struct);
and
Array array = ...;
char [] json = WriteJSONFromArray(array) {
Structure struct = CreateStructure();
AddStringToStruct(struct, "a".toCharArray(), "val".toCharArray());
AddNumberToStruct(struct, "b".toCharArray(), 1);
AddBooleanToStruct(struct, "c".toCharArray(), true);
Gives
{
"a": "val",
"b": 1,
"c": true
}
Array array = CreateArray();
AddStringToArray(array, "val".toCharArray());
AddNumberToArray(array, 1);
AddBooleanToArray(array, true);
Gives
["val", 1, true]
Structure struct = ...
double n = GetNumberFromString(struct, "b".toCharArray());
Array array = ...
char [] str = IndexArrayString(array, 1);
To deploy a new version, build the JAR, copy it to the server using scp and restart the service using systemd.
HTTPS should be put on a gateway server and, for example, be handled by Apache2 + mod_ssl. The requests can be forwarded to the Java webservice running HTTP with mod_proxy. This gateway server can also have mod_security for added security features.
Here is an example of using mod_proxy with mod_ssl.
<VirtualHost *:443>
ServerName example.com
...
SSLEngine on
...
ProxyPass "/" "http://localhost:8000/"
ProxyPassReverse "/" "http://localhost:8000/"
...
</VirtualHost>
If we are calling other services developed in the same way, we can set up SSH tunnels and tunnel HTTP requests between the servers. Thus, we can set up our own Public Key Infrastructure (PKI), and we don't need third party services for encryption.
Logging can be done to stdout and stderr using the built-in print functions in Java. This can be done wherever the developer likes.
It is accessible using systemd's journalctl. It is also possible to log to files in the /var/log/ folder and configure logrotate to rotate the logs.
A useful convention is to return a structure with two required elements: "success" and "message".
For example, a success:
{
"success": true,
"message": "",
"data": [
...
]
}
or a failure:
{
"success": false,
"message": "The key \"x\" did not contain one of the valid values."
}
HTTP will return code:
For requests, a useful convention is to take a structure with one required key "command". This tells the service which operation to invoke. For example,
{
"command": "CreateOrder",
...
}
When calling another JSON webservice, there are many things that the call can result in:
We can unify all this error handling as follows. We set up a JSON call that takes a dynamic data structure and returns a dynamic data structure:
static public void JSONCall(String url, DataReference response, Data request)
This call will set the response entries "success" and "message" if the call fails with cases 1, 2, 3 and 4. In case of 5 and 6, the call can use "success" and "message" from the result.
Thus, the caller only needs to check a single variable, "success", for whather the call succeeded or not. The error message is always available in "message".
This is much simpler than the common checks:
Let's go through each of the benefits mentioned at the start, and see how S-J2D2 is a good way to set up a web service.
We should be able to set up a web service using a plain Linux distro and Java. These are stable and standard systems available everywhere for free. Spinning up a new cloud initialized version of, say, Ubuntu or Debian, is quick, and we can apt install headless OpenJDK JRE in a minute.
This makes it flexible with regards to which cloud provider we can use. As long as the cloud provider provides a plain Linux VPS, we can easily deploy and run our web services.
Having the webserver embedded in the application, means we can run our webservice locally or remotely and expect it to behave the same. Having the whole service wrapped up in a JAR generally give these benefits.
If we use container systems such as Docker or Kubernetes, or frameworks such as Spring, we need our developers to understand how these systems work. This, in order to secure it's operational stability and to debug failures when they occur.
Having an executable with a main function means we are in full control over what the application is doing. Simply decide which IP and port to listen to. Set up the HTTPS separately on the gateway. If we need logging, then simply write the log statements where needed. Need more threads, simply run some.
We can create high quality components by carefully validating the data we receive from the clients and give them clear responses in return.
Web services that communicate using JSON on ports compose well. Need more workers? Add more web servers and do round robbing between them. Need to queue requests? Create a queue and process them in order.
Security problems are bugs where a client gains access to something he should not have access to. Having an understandable system means developers can understand what is going on. There is a simple entry to the web service, taking JSON as input and returning JSON. No other complicating issue. Once the input is in, it should be carefully validated. First, it should be validated as valid JSON, then the data should be validated. This gives a good starting point for a secure application.
Builds can be complex. However, if a build is simply a bunch of Java files compiled and placed into a JAR, then it is simple. The JAR is executable, meaning one of the classes has a main method. This is where execution starts and everything is set out there instead of in declarative specifications.
Deploying a server can be complicated. It might require setting up and configuring a bounch of software on the server. The deployment described here is simply uploading a JAR file and restarting the systemd service.
We would be more than happy to help you. Our opening hours are 9–15 (CET).
📞 (+47) 93 68 22 77
Nils Bays vei 50, 0876 Oslo, Norway