HTTP/2 JSON Deployment Debugging Guide

Practical debugging guide for Axis2/C HTTP/2 JSON services running under Apache httpd on Debian/Ubuntu systems. Based on real deployment issues encountered on Ubuntu 22.04 with the packaged Apache (apache2 package).

Quick Reference: Log Locations

Log	Path	What it tells you
Apache error log	/var/log/apache2/error.log	Module load failures, SSL errors, child crashes
Apache access log	/var/log/apache2/access.log	Request routing (empty = requests not reaching vhost)
Apache other vhosts	/var/log/apache2/other_vhosts_access.log	Requests hitting wrong vhost
Axis2/C log	/var/log/axis2c/axis2.log	Service loading, transport init, worker creation

Quick Reference: Diagnostic Commands

# Is Apache running and on which ports?
systemctl status apache2
ss -tlnp | grep apache2

# Which vhosts are active?
apache2ctl -S

# Which modules are loaded?
apache2ctl -M | grep -E 'ssl|http2|axis2'

# Test config syntax
apache2ctl configtest

# Recent core dumps from Apache
coredumpctl list apache2 | tail -5

---

Problem 1: Apache Starts but Nothing Listens / Curl Returns 000

Symptoms:

• systemctl status apache2 shows active
• curl -k https://localhost/ returns empty or 000
• access.log is empty

Diagnosis: Check if the SSL vhost is actually loaded.

apache2ctl -S

If only the port 80 vhost appears and the port 443 vhost is missing:

# Is the site config file present?
ls /etc/apache2/sites-available/axis2-services.conf

# Is it enabled (symlinked)?
ls /etc/apache2/sites-enabled/ | grep axis2

# Enable it
a2ensite axis2-services
apache2ctl configtest && systemctl restart apache2

Root cause: The site config was never copied to sites-available/ or a2ensite was never run. The deploy script copies from the source tree — if the file isn‘t committed to git, it won’t exist on the target machine.

---

Problem 2: TLS Handshake Bytes in Port 80 Log

Symptom: other_vhosts_access.log shows:

penguin.alphatheory.local:80 127.0.0.1 - - [...] "\x16\x03\x01\x02" 400 499

What it means: \x16\x03\x01 is a TLS ClientHello hitting the plain HTTP port 80 vhost. The client is sending HTTPS but there is no port 443 listener, so the connection fails and may fall through to port 80.

Fix: Ensure mod_ssl is loaded and the 443 vhost is active:

a2enmod ssl
apache2ctl -S   # should show *:443 vhost

---

Problem 3: OCSP Stapling Errors with Self-Signed Certificates

Symptom in error.log:

AH02217: ssl_stapling_init_cert: can't retrieve issuer certificate!
AH02604: Unable to configure certificate for stapling

What it means: SSLUseStapling on requires a CA-signed certificate with an OCSP responder URL. Self-signed certificates have no issuer to query for stapling.

Fix: Disable OCSP stapling in the vhost config:

# SSLUseStapling on

Re-enable when switching to a CA-signed certificate (e.g., Let's Encrypt).

---

Problem 4: “Transport sender is NULL” — Worker Creation Fails

Symptom in axis2.log:

Loading shared library libaxis2_http_sender.so Failed. No such file or directory
Transport sender is NULL for transport http, unable to continue

Symptom in error.log:

[Axis2] Error creating mod_axis2 apache2 worker
Child returned a Fatal error... Apache is exiting!

What it means: axis2.xml references axis2_http_sender and axis2_http_receiver, which map to shared libraries. In HTTP/2 mode these libraries are intentionally not built — Apache httpd handles all transport via mod_axis2 (see docs/HTTP2_CONDITIONAL_COMPILATION.md).

Fix: Comment out the <transportSender> and <transportReceiver> blocks in axis2.xml:

<!--transportSender name="http" class="axis2_http_sender">
    ...
</transportSender-->

<!--transportReceiver name="http" class="axis2_http_receiver">
    ...
</transportReceiver-->

The source-controlled copy is samples/server/axis2.xml. After editing, run make install to propagate to $(prefix)/share/axis2c/axis2.xml.

See also: docs/HTTP2_AXIS2_DOT_XML.md

---

Problem 5: Child Process Segfault (SIGSEGV) Loop

Symptom in error.log:

child pid NNNNN exit signal Segmentation fault (11), possible coredump in /etc/apache2

The parent process stays alive, keeps spawning children that immediately crash — Apache appears “running” but serves no requests.

Getting a Backtrace

On systemd systems, core dumps are captured by systemd-coredump:

# List recent Apache crashes
coredumpctl list apache2 | tail -10

# Show crash info including stack trace
coredumpctl info apache2 | head -80

# Full GDB backtrace (if debug symbols available)
coredumpctl debug apache2 --debugger=gdb \
    --debugger-arguments="-batch -ex 'bt full' -ex 'quit'"

If coredumpctl is not available, enable manual core dumps:

# Set core dump pattern and size limit
echo '/tmp/core.%e.%p' > /proc/sys/kernel/core_pattern
ulimit -c unlimited

# Note: systemctl-started processes inherit systemd's limits, not the shell's.
# For systemd services, edit the unit override:
systemctl edit apache2
# Add:
#   [Service]
#   LimitCORE=infinity

systemctl restart apache2
sleep 3
ls /tmp/core.apache2.*

# Analyze with GDB
gdb /usr/sbin/apache2 /tmp/core.apache2.* -batch -ex 'bt full' -ex 'quit'

Common Segfault: axis2_get_instance Signature Mismatch

Stack trace pattern:

#0  axis2_svc_create (libaxis2_engine.so)
#1  axis2_<servicename>_create (lib<servicename>.so)
#2  axutil_class_loader_create_dll (libaxutil.so)
#3  axis2_msg_recv_load_and_init_svc_impl (libaxis2_engine.so)
#4  axis2_apache2_worker_create (libmod_axis2.so)

Root cause: The service's axis2_get_instance function has the wrong signature. The class loader (axutil_class_loader_create_dll) calls it as:

// Expected signature (CREATE_FUNCT from axutil_dll_desc.h):
int axis2_get_instance(void **inst, const axutil_env_t *env);

// Called as:
create_funct(&obj, env);

If the service instead exports:

// WRONG — causes segfault:
axis2_svc_t* axis2_get_instance(const axutil_env_t *env);

Then &obj (a void**) is misinterpreted as env, and the real env goes to an unused parameter slot. Any dereference of env inside the service (e.g., env->allocator in axis2_svc_create) segfaults.

Fix: Match the CREATE_FUNCT / DELETE_FUNCT typedefs exactly:

AXIS2_EXTERN int AXIS2_CALL
axis2_get_instance(void **inst, const axutil_env_t *env)
{
    *inst = axis2_my_service_create(env);
    if (!(*inst)) return AXIS2_FAILURE;
    return AXIS2_SUCCESS;
}

AXIS2_EXTERN int AXIS2_CALL
axis2_remove_instance(void *inst, const axutil_env_t *env)
{
    axis2_svc_t *svc = (axis2_svc_t *)inst;
    if (svc) axis2_svc_free(svc, env);
    return AXIS2_SUCCESS;
}

Alternative approach: Some services (e.g., bigdata-h2-service, login-service, testws-service) avoid this trap entirely by not exporting axis2_get_instance. They rely solely on the ServiceClass parameter in services.xml and the _invoke_json dlsym lookup path in the JsonRpcMessageReceiver. This sidesteps the CREATE_FUNCT signature requirement altogether. Only export axis2_get_instance / axis2_remove_instance if your service needs custom initialization during worker startup.

---

Problem 6: Apache Module Not Found at Install Time

Symptom during make install:

Neither mod_axis2.so nor libmod_axis2.so found at /usr/local/apache2/modules/

What it means: The Makefile.am apachemoddir is set to a path that doesn't exist. Source-built Apache uses /usr/local/apache2/modules/; packaged Apache uses system paths.

Fix: Set apachemoddir = @prefix@/lib in src/core/transport/http/server/apache2/Makefile.am, then autoreconf -fiv and rebuild. The axis2-services.conf loads the module from /usr/local/axis2c/lib/libmod_axis2.so which matches --prefix=/usr/local/axis2c.

---

Problem 7: json-c Version Incompatibility

Symptom during make:

error: 'json_tokener_error_memory' undeclared

What it means: The code uses json_tokener_error_memory which was added in json-c 0.17, but the version guard checks for >= 0.14. Ubuntu 22.04 ships json-c 0.15 which passes the guard but lacks the enum.

Fix: In src/core/transport/http/util/axis2_json_reader.c, change the version guard for json_tokener_error_memory and json_tokener_error_size from (14 << 8) to (17 << 8):

#if defined(JSON_C_VERSION_NUM) && JSON_C_VERSION_NUM >= ((0 << 16) | (17 << 8))
    case json_tokener_error_memory:
        return "error_memory";
    case json_tokener_error_size:
        return "error_size";
#endif

---

Problem 8: Post-Restart Health Check Passes but Service Crashes

Symptom: The deploy script reports “Apache is running” but verification curls all return 000.

What it means: Apache's parent process starts successfully (passes systemctl is-active check), but child processes crash during worker initialization. With mpm_event, the parent keeps trying to spawn children, creating a crash loop visible in error.log.

Diagnosis:

# The parent is alive but children are dying
tail -20 /var/log/apache2/error.log   # look for "Child returned a Fatal error"
tail -20 /var/log/axis2c/axis2.log    # look for loading errors before the crash

Common causes (in order of likelihood):

1. Missing transport libs in axis2.xml (Problem 4)
2. Service DLL signature mismatch (Problem 5)
3. Missing service shared libraries
4. Corrupt axis2.xml (syntax errors)

---

Problem 9: Request-Path Segfault — NULL Transport Descriptor

Symptom: Worker init succeeds (axis2.log shows “Axis2 worker created”) but Apache child crashes on the first incoming request.

Stack trace pattern (from coredumpctl info):

#0  axis2_transport_in_desc_get_recv (libaxis2_engine.so)
#1  axis2_apache2_worker_process_request (libmod_axis2.so)
#2  axis2_handler (libmod_axis2.so)
#3  ap_run_handler (apache2)

Root cause: In apache2_worker.c, the request processing path calls:

in_desc = axis2_conf_get_transport_in(conf, env, AXIS2_TRANSPORT_ENUM_HTTP);
// in_desc is NULL because transportReceiver is commented out in axis2.xml
receiver = axis2_transport_in_desc_get_recv(in_desc, env);  // SEGFAULT

When transportReceiver is absent from axis2.xml (correct for HTTP/2 mode), axis2_conf_get_transport_in() returns NULL. The code then passes NULL to axis2_transport_in_desc_get_recv() which dereferences it.

Fix: NULL-guard the transport descriptor before use:

if(in_desc)
{
    receiver = axis2_transport_in_desc_get_recv(in_desc, env);
    if(receiver)
        axis2_transport_receiver_set_server_ip(receiver, env, ...);
}

Key insight: There are two separate crash points when removing transport config from axis2.xml — one during init (Problem 4: “Transport sender is NULL”) and one during request handling (this problem). Fixing Problem 4 alone is not sufficient; the request path must also tolerate NULL transport descriptors.

---

Problem 10: “Operation Not Found” for Valid Service

Symptom in axis2.log:

Operation Not found. Endpoint reference is : /services/MyService/someOperation

Symptom in response:

{"error":{"message":"Service processing failed","code":500}}

What it means: The service is loaded and running, but the operation name in the URL path doesn't match any <operation name="..."> in services.xml.

Diagnosis: Check what operations are actually registered:

grep '<operation name=' /usr/local/axis2c/services/MyService/services.xml

Common causes:

• URL uses a different operation name than what's in services.xml
• The test/deploy script uses a placeholder operation name (e.g., benchmark) that was never defined
• Case sensitivity mismatch

---

Packaged Apache vs Source-Built Apache

Many issues stem from path differences between these two environments:

Component	Packaged (Debian/Ubuntu)	Source-built
Binary	/usr/sbin/apache2	/usr/local/apache2/bin/httpd
Modules dir	/usr/lib/apache2/modules/	/usr/local/apache2/modules/
Config dir	/etc/apache2/	/usr/local/apache2/conf/
Site management	a2ensite / a2dissite	Manual Include directives
Module management	a2enmod / a2dismod	LoadModule in httpd.conf
mod_http2 name	mod_http2.so (a2enmod http2)	mod_h2.so
Service management	systemctl restart apache2	httpd -k restart

The deploy-localhost.sh script and axis2-services.conf are designed for the packaged Apache. The individual service build scripts (build_json_service.sh, etc.) detect source-built Apache at /usr/local/apache2 and configure it automatically; otherwise they skip module installation and defer to deploy-localhost.sh.

---

Related Documentation

• docs/HTTP2_AXIS2_DOT_XML.md — Why axis2.xml doesn't need HTTP/2 transport config
• docs/HTTP2_CONDITIONAL_COMPILATION.md — What's built/excluded in HTTP/2 mode
• docs/HTTP2_LIBRARY_LOADING.md — How Axis2/C loads service shared libraries
• docs/HTTP2_APACHE2_WORKER.md — Apache worker initialization details
• docs/userguide/json-httpd-h2-userguide.md — Full deployment guide