An I/o Error Has Occurred While Writing a Response Message Entity to the Container Output Stream.

Symptom Action Reference

List of common errors that can be ignored while troubleshooting Smart IT outage situations:

File: <tomcat>\logs\catalina.<date>.log

.. SEVERE [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [smartit] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@683d467e]) and a value of type [com.bmc.arsys.apitransport.session.ApiThreadLocalStorageBlock] (value [com.bmc.arsys.apitransport.session.ApiThreadLocalStorageBlock@4cb33167]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

.. SEVERE [http-nio-9000-exec-5] org.glassfish.jersey.server.ServerRuntime$Responder.writeResponse An I/O error has occurred while writing a response message entity to the container output stream.

File: <tomcat>\logs\localhost.<DATE>.log

.. SEVERE [localhost-startStop-1] org.apache.tomcat.util.descriptor.web.SecurityConstraint.findUncoveredHttpMethods For security constraints with URL pattern [/*] only the HTTP methods [TRACE OPTIONS] are covered. All other methods are uncovered.

File: ..\Smart_IT\Logs\smartit.log

.. | ERROR | c.b.b.m.s.r.SmartReportingService | Error generating URL for cross launch

.. | ERROR | c.b.b.m.s.p.a.APNSServiceConfig | Failed iOS certificate verification

.. | ERROR | c.b.b.m.s.n.NotificationService | Notification service could not be initialized for tenant 000000000000001

File: ..\Smart_IT\Logs\smartit-metadata.log

.. | WARN | c.b.b.m.s.ARMetaDataService | Fail to load fields from data store <arServer>. Cause - ERROR (303): Form does not exist on server; SHR:UnionOverview_MasterConsole

.. | WARN | c.b.b.m.s.ARMetaDataService | Fail to load views which are available for schema SHR:UnionOverview_MasterConsole. Cause - ERROR (303): Form does not exist on server; SHR:UnionOverview_MasterConsole

.. | WARN | c.b.b.m.s.ARMetaDataService | Fail to load fields from data store <arServer>. Cause - ERROR (303): Form does not exist on server; SB:ServiceRequestStub

.. | WARN | c.b.b.m.s.ARMetaDataService | Fail to load views which are available for schema SB:ServiceRequestStub. Cause - ERROR (303): Form does not exist on server; SB:ServiceRequestStub

.. | WARN | c.b.b.m.s.ARMetaDataService | Fail to load fields from data store <arServer>. Cause - ERROR (303): Form does not exist on server; CTM:Support Group Assoc LookUp People Join

.. | WARN | c.b.b.m.s.ARMetaDataService | Fail to load views which are available for schema CTM:Support Group Assoc LookUp People Join. Cause - ERROR (303): Form does not exist on server; CTM:Support Group Assoc LookUp People Join

F12 Network log (.har)

HTTP 404 error for the URL: http(s)://<smartITServer:Port>/tenant-custom-res/000000000000001/style-smart-it.css

No action required. The errors listed in this row can be ignored while troubleshooting Smart IT outage situations.

File: <tomcat>\logs\commons-daemon.<date>.log

[date_time] [error] [ 8024] Failed creating Java 'C:\Program Files\Java\jdk-12.0.2\bin\server\jvm.dll'.
[date_time] [error] [ 8024] The system cannot find the path specified.

Specify the correct JVM path for Tomcat.

File: <tomcat>\logs\catalina.<date>.log

.. SEVERE [localhost-startStop-1] org.apache.tomcat.jdbc.pool.ConnectionPool.init Unable to create initial connections of pool.

[Detailed_SQL_Exception_>>_Examples]

.. SEVERE [localhost-startStop-1] org.apache.catalina.core.StandardContext.startInternal One or more listeners failed to start. Full details will be found in the appropriate container log file
.. SEVERE [localhost-startStop-1] org.apache.catalina.core.StandardContext.startInternal Context [/smartit] startup failed due to previous errors

Examples of detailed SQL Exceptions:

com.microsoft.sqlserver.jdbc.SQLServerException: The TCP/IP connection to the host <smartITDBHost>, port <smartITDBPort> has failed.

com.microsoft.sqlserver.jdbc.SQLServerException: Cannot open database "<smartITDBName>" requested by the login. The login failed.

com.microsoft.sqlserver.jdbc.SQLServerException: Login failed for user '<smartITSystemUserName|smartITBusinessUserName>'

java.sql.SQLException: ORA-01017: invalid username/password; logon denied

Files: <tomcat>\logs\localhost.<DATE>.log and ..\Smart_IT\Logs\bootstrap.log

org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'persistenceUnitManager' defined in class path resource [app-context-jpa.xml]
..
Caused by: javax.naming.NameNotFoundException: Name [<resourceName>] is not bound in this Context. Unable to find [jdbc].

This error indicates that the connection to the Smart IT Database for the SmartIT_System and SmartIT_Business resource is failing. The database connection details are stored in the following XML file on the Smart IT Server:

..<tomcat>\conf\Catalina\localhost\smartit.xml

Take corrective steps based on the detailed SQL error message captured in the catalina log.

Parameters of interest in the smartit.xml file (Each parameter is present twice, once for the SmartIT_System resource and once for the SmartIT_Business resource):

• username (username used to connect to the Smart IT database)
• password (plain text password, do not encrypt)
• url (contains Smart IT Database host, port and database details)

Confirm that the Smart IT database can be reached from the Smart IT Server and (have a DBA) check the configured credentials directly against the database.

File: <tomcat>\logs\localhost.<DATE>.log

.. SEVERE [localhost-startStop-1] org.apache.catalina.core.StandardContext.listenerStart Exception sending context initialized event to listener instance of class [org.springframework.web.context.ContextLoaderListener]
..
..
Internal Exception: com.microsoft.sqlserver.jdbc.SQLServerException: Invalid object name 'CONFIGURATION_PARAMETER'.

OR

File: ..\Smart_IT\Logs\smartit.log

.. | ERROR | c.b.b.m.s.s.i.<className> | DB could not be indexed
..
Internal Exception: com.microsoft.sqlserver.jdbc.SQLServerException: Invalid object name 'SUGGESTED'.

These errors suggests Smart IT was able to connect to the database, but an expected table is not found. Check in the Smart IT Database if a table with the indicated object name exists. The diagram below shows the table names and their schema for a default installation.

If the table exists, check the connection details in this file on the Smart IT Server:

..<tomcat>\conf\Catalina\localhost\smartit.xml

• Check if the 'url' parameter is pointing to the expected database.
• Check if the usernames configured for the SmartIT_Business and SmartIT_System resources are set up with the expected default schema.

For information about pre-creating the Smart IT database for the expected user setup, see Creating the database.

File: <tomcat>\logs\rsso-agent.0.log

javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
sun.security.ssl.Alert.createSSLException(Unknown Source)

Correct the certificate setup.

File: ..\Smart_IT\Logs\bootstrap.log

.. | ERROR | c.b.a.c.c.n.i.NotificationControllerImpl | GSI call failed for message broker
com.bmc.arsys.api.ARException: ERROR (623): Authentication failed; Remedy Application Service

or

..| ERROR | c.b.b.m.s.c.CCSPropertiesService | Error while checking application upgrade status.
com.bmc.arsys.api.ARException: ERROR (623): Authentication failed; Remedy Application Service

or

.. | DEBUG | c.b.b.m.s.ZDTAppsUpgradeService | Error while fetching the Version from SHARE:Application_Properties
com.bmc.arsys.api.ARException: ERROR (623): Authentication failed; Remedy Application Service

The Remedy Application Service password defined in CONFIGURATION_PARAMETER table (Smart IT System schema) is incorrect. Update the password.

Important: The password must be stored in the table in an encrypted format (plain text will not work). In versions 18.0x and 19.0x, use the Smart IT Maintenance Tool to encrypt the password.

For Smart IT 20.02 and higher, use the Password Encryption utility.

File: ..\Smart_IT\Logs\bootstrap.log

.. | ERROR | c.b.b.m.s.c.CCSPropertiesService | Error while initializing CCSPropertiesService
javax.crypto.IllegalBlockSizeException: Input length must be multiple of 16 when decrypting with padded cipher
at com.sun.crypto.provider.CipherCore.doFinal(CipherCore.java:913)

The Remedy Application Service password defined in the CONFIGURATION_PARAMETER table in the Smart IT System schema is not properly encrypted. Encrypt the password.

File: ..\Smart_IT\Logs\bootstrap.log

.. | DEBUG | c.b.b.m.s.ZDTAppsUpgradeService | Error while fetching the Version from SHARE:Application_Properties
com.bmc.arsys.api.ARException: ERROR (103): The length of the name parameter (or name field in a parameter) is longer than the maximum allowed length.
..
.. | ERROR | c.b.b.m.s.ZDTAppsUpgradeService | Error while checking application upgrade status.
java.lang.NullPointerException: null

The Remedy Application Service password defined in the CONFIGURATION_PARAMETER table in the Smart IT System schema is not properly encrypted. Encrypt the password.

This error can be seen on Smart IT 20.02 when the password is encrypted with the Maintenance Tool of a lower version, while the new 20.02 password encryption utility should be used.

File: ..\Smart_IT\Logs\smartit-metadata.log

.. | ERROR | c.b.b.m.c.ConfigurationManager | Error loading ITSM Version:

com.bmc.arsys.api.ARException: ERROR (623): Authentication failed; Remedy Application Service
at com.bmc.arsys.apitransport.ApiProxyJRpcBase.arCall(ApiProxyJRpcBase.java:300)

The Remedy Application Service password defined in the CONFIGURATION_PARAMETER table in the Smart IT System schema might be entered in plain text. Encrypt the password and update it in the table.

File: ..\Smart_IT\Logs\bootstrap.log

.. | ERROR | c.b.a.c.c.n.i.NotificationControllerImpl | GSI call failed for message broker
com.bmc.arsys.api.ARException: ERROR (90): Cannot establish a network connection to the AR System server; Connection refused: connect <arServer>:<port>

or

| ERROR | c.b.b.m.s.c.CCSPropertiesService | Error while checking application upgrade status.
com.bmc.arsys.api.ARException: ERROR (90): Cannot establish a network connection to the AR System server; Connection refused: connect <arServer>:<port>

or

| DEBUG | c.b.b.m.s.ZDTAppsUpgradeService | Error while fetching the Version from SHARE:Application_Properties
com.bmc.arsys.api.ARException: ERROR (90): Cannot establish a network connection to the AR System server; Connection refused: connect <arServer>:<port>

Smart IT is unable to connect to the AR Server with the connection information stored in the CONFIGURATION_PARAMETER table in the Smart IT System schema.

Verify if the AR port defined in this table is correct and that Smart IT can connect to AR Server on the specified port.

File: ..\Smart_IT\Logs\bootstrap.log

org.xml.sax.SAXParseException: schema_reference.4: Failed to read schema document 'http://www.springframework.org/schema/context/spring-context-3.0.xsd', because 1) could not find the document; 2) the document could not be read; 3) the root element of the document is not <xsd:schema>.
at org.apache.xerces.util.ErrorHandlerWrapper.createSAXParseException(Unknown Source)

File: <tomcat>\logs\localhost.<current_date>.log

.. SEVERE [localhost-startStop-1] org.apache.catalina.core.StandardContext.filterStart Exception starting filter [Agent]
java.lang.NullPointerException
at com.bmc.atrium.sso.agents.web.SSOFilter.init(SSOFilter.java:47)

File: ..\Smart_IT\Logs\bootstrap.log

.. | ERROR | c.b.bsm.myit.util.FilesCopyUtil | File sso-sdk.properties from /apps/ars/Smart_IT/Smart_IT/smartit/WEB-INF/classes/sso-sdk.properties not copied to /apps/apache/tomcat8.5/external-conf properly, try to restart application server and verify properties files
java.io.FileNotFoundException: /apps/ars/Smart_IT/Smart_IT/smartit/WEB-INF/classes/sso-sdk.properties (Permission denied)

Add the correct file system permission on the Smart IT Server.

File: <tomcat>\logs\localhost.<current_date>.log

.. SEVERE [main] org.apache.catalina.core.StandardContext.filterStart Exception starting filter [Agent]
java.lang.NoClassDefFoundError: io/jsonwebtoken/ExpiredJwtException

Important: The example error is related to a class from jjwt-api-<version>.jar. The actual class name referenced in the error might be different depending on the .jar file that is affected. General format of the error is:

.. SEVERE [main] org.apache.catalina.core.StandardContext.filterStart Exception starting filter [Agent]
java.lang.NoClassDefFoundError: <classname>

Check the presence (and size) of the required .jar files for the RSSO Agent in the <tomcat>\external-conf\lib folder on the Smart IT Server. Integrating BMC Remedy SSO with Smart IT

F12 Network log (.har)

{"error":"MOBILITY_ERROR_LOGIN","errorCode":1006,"defaultMessage":"Unable to check user status","additionalMessage":"ServiceException: Unable to check user status","detailedMessage":"Unable to check user status","arconnectionProblem":false}

File: ..\Smart_IT\Logs\smartit.log

.. | ERROR | c.b.b.m.service.UsersServiceImpl | Unable to check user status
com.bmc.bsm.myit.providers.ServiceNotSupportedException: com.bmc.bsm.myit.providers.userdetails.UserDetailsService

Confirm if the User Details Pluggable provider is enabled and the .jar file for this provider (name: itsm-user-details-<version>-BUILD-SNAPSHOT.jar) is present in the ..\Smart_IT\smartit\WEB-INF\classes\provider\itsm-user-details folder.

F12 Network log (.har)

{"error":"MOBILITY_ERROR_LOGIN","errorCode":1006,"defaultMessage":"Login Failed: An authentication error occurred in the data server.","additionalMessage":"Your password has expired. Please, login to AR server and change it","detailMessage":"ERROR (623): Authentication failed; Password has expired","ARConnectionProblem":false}

Open the record of the affected user in AR System User form, and check if the Force Password Change On Login field is set. Have the user change their password to clear this field.

F12 Network log (.har):

{"error":"MOBILITY_ERROR_INTERNAL_SERVER_ERROR","errorCode":1013,"defaultMessage":"Missing People Info.","additionalMessage":"UnauthorizedException: Missing People Info.","detailedMessage":"HTTP 403 Forbidden","arconnectionProblem":false}

File: ..\Smart_IT\Logs\smartit.log

..| ERROR | c.b.b.m.assembler.EntryAssembler | Failed to fill(queryParameters=[{name=loginId, value=<LoginID>}], user:<LoginID>, host: <arServer>)

com.bmc.arsys.api.ARException: ERROR (623): Authentication failed; Remedy Application Service

Validate the connection settings for the MyIT-ITSM Persistent Data Store of Smart IT. Make sure that they are correct and that there are no connectivity issues between Smart IT and AR System.

The specific error here suggests that the Remedy Application Service password stored in the Persistent Data Store of Smart IT is incorrect.

Important: This error can also be seen in the network logs when attempting to log in to Smart IT with a user that has no people record (for instance, if you attempt to log in with Remedy's classic 'Demo' administrator user). Make sure the user ID used to log in has been set up with the correct Smart IT Roles and Permissions.

Troubleshooting integration issues

Long running queries against the SMT:Social_WorklogView form together with high CPU usage on the AR System database. Check the resources in the Reference column for resolution.

An I/o Error Has Occurred While Writing a Response Message Entity to the Container Output Stream.

Source: https://docs.bmc.com/docs/smartit1805/troubleshooting-smart-it-availability-or-stability-issues-1004423521.html

Share this post