sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

We are getting below exception “sun.security.provider.certpath.SunCertPathBuilderException” error on JBoss instance.

Error log:

*********************************************************************************
sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:387)
at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:292)
at sun.security.validator.Validator.validate(Validator.java:260)
at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324)
at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:229)
at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:124)
at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1351)
at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:156)
at sun.security.ssl.Handshaker.processLoop(Handshaker.java:925)
at sun.security.ssl.Handshaker.process_record(Handshaker.java:860)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1043)
at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1343)
at sun.security.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:728)

Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:145)
at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:131)
at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280)
at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:382)
*********************************************************************************

OBSERVATION:

While java makes a connection to another application services over HTTPS connections, to make this connection successful another party certificate should be imported to the installed JAVA keystore ( $JAVA_HOME/jre/lib/security/cacerts ) file which is known as truststore.

OR 

We can create a keystore and import the trusted certificates and provide its location to standalone.conf.bat or standalone .conf file.

SOLUTION:

Method 1: Import the signed or trust store certificate to JDK   ( $JAVA_HOME/jre/lib/security/cacerts ) file.

Command:
keytool   -import  -trustcacerts   -alias "alias_name"  -file   certificate.cer   -keystore cacerts

OR

keytool   -import    -alias  server      -file  server.cer    -keystore  cacerts

METHOD 2:  Create a new keystore and import certificates and provide the location to application server standalone.conf.bat  OR standalone.conf file.  and provide the password of keystore file.


Step 1: command to create new trsutstore (.jks) file.

Command :  
keytool   -import  -trustcacerts   -alias "alias_name"  -file   certificate.cer   -keystore truststorecert.jks

OR


keytool   -import    -alias  server      -file  server.cer    -keystore  truststorecert.jks


Step 2: Make configuration changes on JBoss (standalone.conf.bat).


set "JAVA_OPTS=%JAVA_OPTS%  -Djavax.net.ssl.trustStore=D:/path/truststorecert.jks                      -Djavax.net.ssl.trustStoreType=jks    -Djavax.net.ssl.trustStorePassword=123password"

Step 3: Restart the application server to reflect the changes.


Thanks :-)

java.sql.SQLRecoverableException: Closed Connection

We are getting below exception “java.sql.SQLRecoverableException: Closed Connection” error on JBoss instance.

Error log:

*********************************************************************************

 ERROR [org.apache.catalina.core.ContainerBase.[jboss.web].[default-host].[/WebApp].[WebApp]] (http--0.0.0.0-8080-1)

Servlet.service() for servlet WebApp threw exception: java.sql.SQLRecoverableException: Closed Connection

at oracle.jdbc.driver.PhysicalConnection.commit(PhysicalConnection.java:3817) [ojdbc6.jar:11.2.0.2.0]

at oracle.jdbc.driver.PhysicalConnection.commit(PhysicalConnection.java:3857) [ojdbc6.jar:11.2.0.2.0]

at org.jboss.jca.adapters.jdbc.BaseWrapperManagedConnection.jdbcCommit(BaseWrapperManagedConnection.java:986)

at org.jboss.jca.adapters.jdbc.WrappedConnection.commit(WrappedConnection.java:757)

at org.hibernate.engine.transaction.internal.jdbc.JdbcTransaction.doCommit(JdbcTransaction.java:112) [hibernate-core-4.0.1.Final.jar:4.0.1.Final]

at org.hibernate.engine.transaction.spi.AbstractTransactionImpl.commit(AbstractTransactionImpl.java:178) [hibernate-core-4.0.1.Final.jar:4.0.1.Final]

at org.springframework.orm.hibernate4.HibernateTransactionManager.doCommit(HibernateTransactionManager.java:554) [spring-orm-3.2.5.RELEASE.jar:3.2.5.RELEASE]

at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:755) [spring-tx-3.2.5.RELEASE.jar:3.2.5.RELEASE]

at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:724) [spring-tx-3.2.5.RELEASE.jar:3.2.5.RELEASE]

at org.springframework.transaction.interceptor.TransactionAspectSupport.commitTransactionAfterReturning(TransactionAspectSupport.java:475) [spring-tx-3.2.5.RELEASE.jar:3.2.5.RELEASE]

at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:270) [spring-tx-3.2.5.RELEASE.jar:3.2.5.RELEASE]

at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:94) [spring-tx-3.2.5.RELEASE.jar:3.2.5.RELEASE]

at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) [spring-aop-3.2.5.RELEASE.jar:3.2.5.RELEASE]

at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:631) [spring-aop-3.2.5.RELEASE.jar:3.2.5.RELEASE]

at  

*********************************************************************************

What is java.sql.SQLRecoverableException: Closed Connection?

Above exception is due to one of these reasons this may due to the lower version of Oracle [ojdbc6.jar], network issue, firewall timeout configuration or database timeout.

The timeout should properly check or configured in JBoss, the value must be greater than 0 but less than timeout has been used in a database, network firewall, etc.

But as per my case, Oracle [ojdbc6.jar] is lower version as per database server required.

How to check ODJBC.jar version using the command line?

Command:   java  -jar  ojdbc6.jar  -version

Solution: We should use the same Oracle database JDBC driver version as per the database version.

Thanks :-)

java.lang.VirtualMachineError: out of space in CodeCache for adapters

Code Cache issue:

We are getting below exception “java.lang.VirtualMachineError: out of space in CodeCache” error on JBoss instance

Error log:

*********************************************************************************

 11:19:08,460 ERROR [org.apache.catalina.core.ContainerBase.[jboss.web].[default-host].[/WebApplication].[default]] (ajp-/192.168.100.171:8009-1) Servlet.service() for servlet default threw exception: java.lang.VirtualMachineError: out of space in CodeCache for adapters

at java.lang.Class.getDeclaredConstructors0(Native Method) [rt.jar:1.6.0_35]

at java.lang.Class.privateGetDeclaredConstructors(Class.java:2389) [rt.jar:1.6.0_35]

at java.lang.Class.getDeclaredConstructors(Class.java:1836) [rt.jar:1.6.0_35]

at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.determineCandidateConstructors(AutowiredAnnotationBeanPostProcessor.java:229) [spring-beans.jar:3.1.1.RELEASE]

at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.determineConstructorsFromBeanPostProcessors(AbstractAutowireCapableBeanFactory.java:962) [spring-beans.jar:3.1.1.RELEASE]

at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:935) [spring-beans.jar:3.1.1.RELEASE]

at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:485) [spring-beans.jar:3.1.1.RELEASE]

at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:456) [spring-beans.jar:3.1.1.RELEASE]

at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:313) [spring-beans.jar:3.1.1.RELEASE]

at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:193) [spring-beans.jar:3.1.1.RELEASE]

at 

*********************************************************************************

What is CodeCache ??

  

The Java Virtual Machine (JVM) generates native (compiled) code and stores it in a memory area called the codecache. The code cache is where jar definitions are stored.

The default maximum size of the CodeCache on most of the platforms is 48M. If any application needs to compile a large number of methods resulting in a huge amount of compiled code then this CodeCache may become full. 

You can set the code cache size by setting the property -XX:ReservedCodeCacheSize and monitor it by tools like JConsole. When the code cache becomes full, the Java VM flushes and sweeps it. In Java 7 and higher, code cache flushing is enabled by default.

Codecache Size Options:-

Option	Default	Description
InitialCodeCacheSize	160K (varies)	Initial code cache size (in bytes)
ReservedCodeCacheSize	32M/48M	Reserved code cache size (in bytes) - maximum code cache size
CodeCacheExpansionSize	32K/64K	Code cache expansion size (in bytes)

Steps to check CodeCache Size in JBOSS:

Step 1: Locate to $JBOSS_Home/bin

Step 2: Command to connect JBoss instance :

            Command:   jboss-cli.bat -c --controller=localhost:9999

Step 3: Command to check CodeCache size in JVM.

            Command:    /core-service=platform-mbean/type=memory-pool/name=Code_Cache/:read-resource(recursive=true,proxies=true,include-runtime=true,include-defaults=true)

Step 4: Output as follows:

{

    "outcome" => "success",

    "result" => {

        "name" => "Code_Cache",

        "type" => "NON_HEAP",

        "valid" => true,

        "memory-manager-names" => ["CodeCacheManager"],

        "usage-threshold-supported" => true,

        "collection-usage-threshold-supported" => false,

        "usage-threshold" => 0L,

        "collection-usage-threshold" => undefined,

        "usage" => {

            "init" => 2555904L,

            "used" => 99788096L,

            "committed" => 100663296L,

            "max" => 100663296L

        },

        "peak-usage" => {

            "init" => 2555904L,

            "used" => 99797824L,

            "committed" => 100663296L,

            "max" => 100663296L

        },

        "usage-threshold-exceeded" => true,

        "usage-threshold-count" => 0L,

        "collection-usage-threshold-exceeded" => undefined,

        "collection-usage-threshold-count" => undefined,

        "collection-usage" => undefined

    }

}

Step 5: Increase a CodeCache size by “-XX:ReservedCodeCacheSize=128M” in  JBOSS JVM  argument.

Reference URL :

https://docs.oracle.com/javase/8/embedded

https://www.atlassian.com/blog/archives/codecache

    

Apache Rewrite Rules

• Apache provides a REWRITE RULE function to redirect the requests from one URL/Page to another URL/Page.
• To use this functionality we have to enable the mod_rewrite module in apache web server.
• mod_rewrite provides a flexible and powerful way to manipulate URLs using an unlimited number of rules. 
• mod_rewrite operates on the full URL path, including the path-info section. A rewrite rule can be invoked in httpd.conf or in .htaccess file.

Enable module:

• LoadModule rewrite_module modules/mod_rewrite.so

NOTE: We should not always use .htaccess file for large number of rewrite rule redirection, since which may leads to the slowness of Apache server.

#####Forward/Rewrite , main domain towards application context.

RewriteEngine on

RewriteRule   ^/$     /CONTEXT/ [R]

RewriteRule Basics

A RewriteRule consists of three arguments separated by spaces. The arguments are

• Pattern: which incoming URLs should be affected by the rule;
• Substitution: where should the matching requests be sent;
• [flags]: options affecting the rewritten request.

The Substitution can itself be one of three things:

A full filesystem path to a resource

• RewriteRule "^/games"  "/usr/local/games/web"

This maps a request to an arbitrary location on your filesystem, much like the Alias directive.

A web-path to a resource

• RewriteRule "^/foo$"  "/bar"

If DocumentRoot is set to /usr/local/apache2/htdocs, then this directive would map requests for http://example.com/foo to the path /usr/local/apache2/htdocs/bar.

An absolute URL

• RewriteRule "^/product/view$"  "http://site2.example.com/seeproduct.html" [R]

Rewrite Conditions

For example, to send all requests from a particular IP range to a different server, you could use:

RewriteCond  "%{REMOTE_ADDR}"  "^10\.2\."

RewriteRule  "(.*)"  http://intranet.example.com%241/

Examples:

Redirect All Website Pages

# Redirect all pages from olddomain.com # to newdomain.com

Options +FollowSymLinks

RewriteEngine on

RewriteCond %{HTTP_HOST} ^www.olddomain.com$ [OR]

RewriteCond %{HTTP_HOST} ^olddomain.com$

RewriteRule ^(.*)$ http://www.newdomain.com/$1 [R=301,L]

Meta Refreshes:

This method uses a special Meta tag in the HTML source to control the redirect. In the early days of the Internet, this was the main method of generating redirects.

The meta refresh should appear within the head section of the HTML source.

An example is:

<meta http-equiv="Refresh" content="0; url=http://www.example.com/" />

The number (0 in this example) refers the the number of seconds to wait before performing the redirect. A zero second delay results in an instant redirect and is the equivalent of doing a 301 permanent redirect.

Reference link:

http://redirectdetective.com/creating-redirects.html

Redirect rule for old page to new page:

RewriteEngine On

Redirect permanent   /page1.html   http://example.com/newpage1.html

Redirect permanent   /folder/page2.html  http://example.com/newpage2.html

Reference link:

http://redirectdetective.com/creating-redirects.html

 Redirect a portion of your site to HTTPS, you might do the following:

<If "%{SERVER_PROTOCOL} != 'HTTPS'">

    Redirect "/admin/" "https://www.example.com/admin/"

</If>

Redirect all requests from the www to the non-www version of the domain, or vice-versa:

RewriteEngine On

RewriteCond %{HTTP_HOST} ^www.example.com$

RewriteRule ^(.*)$ "http://example.com/$1" [R=301,L]

Reference link:

http://redirectdetective.com/creating-redirects.html

PHP Redirects :

If you want to do a PHP 301 permanent redirect, you just need to add the redirect code, ie:

<?php

Header('Location: http://example.com/newpage',TRUE,301);

?>

Reference link:

http://redirectdetective.com/creating-redirects.html

Rewriting from Old to New (external):

Solution:

We force a HTTP redirect to the new URL which leads to a change of the browsers and thus the users view:

RewriteEngine  on

RewriteRule    "^/foo\.html$"  "bar.html"  [R]

Reference link:

https://httpd.apache.org/docs/current/rewrite/remapping.html

Redirect all pages from Http to Https:

<VirtualHost *:80>

  ServerName www.example.com

  ServerAlias example.com www.new-example.com

  Redirect "/" "https://www.example.com/"

</VirtualHost>

 OR

RewriteEngine On

RewriteRule ^/$  https://%{HTTP_HOST}/ [R,L]

RewriteRule ^(.*)$ https://%{HTTP_HOST}$1 [R,L]

 

 OR

RewriteEngine On

RewriteCond %{HTTPS}  !=on

RewriteRule ^/?(.*) https://%{SERVER_NAME}/$1 [R,L

Redirect all pages from Https to Http:

    <IfModule mod_rewrite.c>

        RewriteEngine on

        RewriteCond %{SERVER_PORT} ^443$ [OR]

        RewriteCond %{HTTPS} =on

        RewriteCond %{HTTP_HOST} !^vhost2.test.com$

        RewriteRule ^/(.*) http://%{HTTP_HOST}/$1 [L,R]

    </IfModule>

Redirect a URL to www.example.com:

RewriteEngine On

RewriteCond %{HTTP_HOST} ^example.com$

RewriteRule (.*) http://www.example.com$1 [R=301,L]

Redirect request from url https://10.20.40.50 to https://www.domain.example.com.

<VirtualHost _default_:443>

ServerName www.domain.example.com

RewriteEngine on

RewriteCond %{SERVER_PORT} ^443$ [OR]

RewriteCond %{HTTPS} =on

RewriteCond %{HTTP_HOST} !^www.domain.example.com$

RewriteRule ^/(.*) https://www.domain.example.com/$1 [L,R]

</VirtualHost> 

Alias & AliasMatch  Directive:

The Alias directive allows documents to be stored in the local filesystem other than under the DocumentRoot.

Alias "/image" "/ftp/pub/image"

                &

Alias "/icons/" "/usr/local/apache/icons/"

Note that if you include a trailing / on the URL-path then the server will require a trailing / in order to expand the alias

A request for http://example.com/image/foo.gif would cause the server to return the file /ftp/pub/image/foo.gif.

Reference link:

https://httpd.apache.org/docs/2.4/mod/mod_alias.html

Logging:

mod_rewrite offers detailed logging of its actions at the trace1 to trace8 log levels.

Example:

LogLevel alert rewrite:trace3

To check the logs :

tail -f error_log|fgrep '[rewrite:'

Redirect Detective is a free redirect checker

http://redirectdetective.com/
http://redirectcheck.com/

Reference links:

https://httpd.apache.org/docs/2.4/mod/mod_alias.html

https://httpd.apache.org/docs/current/mod/mod_rewrite.html

https://httpd.apache.org/docs/current/rewrite/intro.html

https://support.hostgator.com/

http://redirectdetective.com/creating-redirects.html

https://httpd.apache.org/docs/current/rewrite/

Thanks :-)

How to configure Apache MPM.

     

                   Apache uses one of following MPM(Multi-Processing Module) for handling incoming requests and processes them. It modifies basic functionality of the Apache server related to multi-thread and multi-processes style of working. Only one MPM can be loaded into the server at any time.

Types of MPM’s :-

1. Prefork MPM
2. Worker MPM
3. Event MPM

Command to check MPM:   httpd  –V      

Number of connection calculation as per core  CPU.

The MaxClients is optimal at 200 * the count of CPU cores for the prefork mpm  &

 300 * the count of CPU cores for the workermpm.

Below is the snapshot of "httpd-mpm.conf" file.

 <IfModule mpm_worker_module>

                ThreadLimit                       30

                ServerLimit                        10

                StartServers                       3

                MinSpareThreads             5

                MaxSpareThreads            20

                MaxClients                          300

                ThreadsPerChild               30

                MaxRequestsPerChild     0

</IfModule>

Let’s understand the above MPM_worker _module configuration.

1.    Server will going to start with “3” child processes ( i.e StartServers = 3)

Which is called as number of child processes during start.

2.    Where each child processes will start handling “30” threads/Requests.

 (i.e ThreadsPerChild = 30)

3.     So total number of concurrent connection/clients = “90”

      (i.e StartServers * ThreadsPerChild  = concurrent connection/clients at start [3*30= 90])

Total = 90 concurrent connection/clients

4.     If more concurrent users where to come then another child process will go to start/add.

5.    As already “3” child processes where consumed/busy , so another child processes where going to start as per “ServerLimit”   ( i.e we have ServerLimit  =  10).

6.    So total number of  max Connections/clients can be serve = 300

( i.e ServerLimit * ThreadsPerChild , 10*30 = 300 MaxClients  )

7.    A “ServerLimit” can be calculated,   As  ServerLimit =  MaxClients  / ThreadsPerChild

 (i.e ServerLimit= 300/30)

8.     A “ThreadLimit”   where stands for Number of threads on per child processes.

9.     “MinSpareThreads” where stands for minimum number of worker threads which are kept spare, Sets the desired minimum number of idle [free] child server processes)

10.   If  MaxRequestsPerChild / MaxConnectionsPerChild is 0, then the process will never expire.

The recommended value of maxThreads & MaxClients is 200 per CPU, so here we assume the server is a single core machine. 
If it had been quad core, we could push that value to 800 or more depending on RAM and other machine specs. 
The total threads is an aggregate value. If Apache and JBoss are on the same server, and that server has four cores, then you would halve the maxThreads and MaxClients to 400 each.
 Always ensure the total threads possible doesn't exceed 200 times the number of CPU cores.

Reference links: 

http://httpd.apache.org/docs/2.2/mpm.html

http://httpd.apache.org/docs/current/mod

https://www.woktron.com/secure/knowledgebase/

http://dasunhegoda.com/configure-apache-MPM

https://www.middlewarebox.com/2018/05/apache-http-server

Thanks :-)