Find a resolution
ibm.com/redbooksInternational Technical Support Organization
Determination for Distributed Platforms
November 2005
SG24-6798-00 Note: Before using this information and the product it supports, read the information in "Notices" on page ix.
Note to U.S. Government Users Restricted Rights -- Use, duplication or disclosure restricted by GSA ADPSchedule Contract with IBM Corp.Contents
Notices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix Trademarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . x
1.1 Introduction to problem determination. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 1.1.1 Causes of problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 1.1.2 Types of problem symptoms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 1.2 Preparing for and preventing problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 1.2.1 Applying WebSphere maintenance . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 1.2.2 Checking the prerequisites . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 1.2.3 Testing the application . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 1.2.4 Setting up a test environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 1.2.5 Establishing safe operational procedures . . . . . . . . . . . . . . . . . . . . . 12 1.2.6 High availability and failover . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 1.2.7 Monitoring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 1.2.8 System documentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 1.2.9 Diagnostic data collection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 1.3 What to do when a problem occurs. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 1.3.1 Revert to safe conditions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 1.3.2 Identify problem symptoms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 1.3.3 Investigate and research the problem . . . . . . . . . . . . . . . . . . . . . . . . 27 1.3.4 Problem determination strategies . . . . . . . . . . . . . . . . . . . . . . . . . . . 29 1.4 Classify the problem and determine the root cause . . . . . . . . . . . . . . . . . 31 1.4.1 Installation or migration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34 1.4.2 Application packaging and deployment . . . . . . . . . . . . . . . . . . . . 35 1.4.3 System management and configuration . . . . . . . . . . . . . . . . . . . . . . 37 1.4.4 Runtime. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 1.5 Contacting IBM for support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70 1.5.1 IBM support structure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70 1.5.2 Research the problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72 1.5.3 Collect MustGather files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72 1.5.4 Determine the severity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73 1.5.5 Create a PMR . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74
1.5.6 Send data to IBM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76
2.1 JVM logs (SystemOut and SystemErr) . . . . . . . . . . . . . . . . . . . . . . . . . . . 78 2.2 Tracing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82 2.3 Collector tool . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87 2.4 First Failure Data Capture (FFDC) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87 2.5 Other logs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88 2.5.1 Process (native) logs. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88 2.5.2 Service log (activity.log) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89 2.5.3 Installation logs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90 2.6 IBM HTTP Server and plug-in logs and traces . . . . . . . . . . . . . . . . . . . . . 90 2.6.1 IBM HTTP Server logs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90 2.6.2 Web server plug-in logs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90 2.6.3 Web server plug-in trace . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91 2.6.4 Network trace . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91 2.7 System management logs. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92 2.7.1 Output from wsadmin . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92 2.7.2 Management scripts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92 2.7.3 Profile management logs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92 2.8 WebSphere Rapid Deployment logs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93 2.9 Summary of logs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93
3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98 3.2 Work the problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100 3.2.1 Symptom: Launchpad or installation wizard will not start or fails . . 101 3.2.2 Symptom: Installation wizard hangs . . . . . . . . . . . . . . . . . . . . . . . . 103 3.2.3 Symptom: Profile creation failure . . . . . . . . . . . . . . . . . . . . . . . . . . 105 3.2.4 Symptom: IVT fails . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108 3.3 Analyzing problem areas. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110 3.3.1 Web browser requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110 3.3.2 Application server startup problems . . . . . . . . . . . . . . . . . . . . . . . . 111 3.3.3 Profile creation problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112 3.4 The next step. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113
4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116 4.1.1 Collecting data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122 4.2 Work the problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125 4.2.1 High-level symptom analysis. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125 4.3 Analyzing problem areas. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127 4.3.1 Problem: Unable to access the administrative console. . . . . . . . . . 127 4.3.2 Problem: wsadmin or management scripts can't access server . . . 130
4.3.3 Problem: Unable to stop a server process . . . . . . . . . . . . . . . . . . . 133 4.3.4 Problem: Unable to start a server process . . . . . . . . . . . . . . . . . . . 135 4.3.5 Problem: Unable to access a node agent . . . . . . . . . . . . . . . . . . . . 137 4.3.6 Problem: Unable to manage a Web server . . . . . . . . . . . . . . . . . . . 138 4.3.7 Problem: Unable to manage applications . . . . . . . . . . . . . . . . . . . . 141 4.3.8 Problem: Failure adding a node to a deployment manager . . . . . . 143 4.3.9 Problem: Repository synchronization . . . . . . . . . . . . . . . . . . . . . . . 145 4.3.10 Problem: Save conflicts in the administrative console . . . . . . . . . 150 4.3.11 Problem: enterprise applications missing . . . . . . . . . . . . . . . . . . . 151 4.3.12 Problem: Invalid or expired certificates . . . . . . . . . . . . . . . . . . . . . 153 4.3.13 Problem: WebSphere Rapid Deployment . . . . . . . . . . . . . . . . . . . 1584.4 The next step. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 162
5.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 166 5.1.1 Initial symptoms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1685.2 Work the problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 169 5.2.1 Data to collect . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 169 5.2.2 High-level symptom analysis. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 170 5.2.3 Symptom: HTTP 404 error - The page cannot be displayed. . . . . . 172 5.2.4 Symptom: HTTP 404 error - Failed to find resource . . . . . . . . . . . . 175 5.2.5 Symptom: HTTP 404 error - WebGroup/virtual host not defined . . 177 5.2.6 Symptom: HTTP 500 error - JSP processing error . . . . . . . . . . . . . 179 5.2.7 Symptom: HTTP 500 error - IllegalStateException . . . . . . . . . . . . . 1815.3 Analyzing problem areas. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 186 5.3.1 Application URL specification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 186 5.3.2 Static resources not displayed . . . . . . . . . . . . . . . . . . . . . . . . . . . . 190 5.3.3 Web resources not reloading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 192 5.3.4 Encoding and internationalization issues . . . . . . . . . . . . . . . . . . . . 195 5.3.5 HTTP session management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2025.4 The next step. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 209
6.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 212 6.1.1 JCA technical overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2136.2 Work the problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 216 6.2.1 Symptom: A JDBC call returns incorrect data . . . . . . . . . . . . . . . . . 217 6.2.2 Symptom: Failure to connect to a new data source . . . . . . . . . . . . 222 6.2.3 Symptom: Failure to connect to an existing data source . . . . . . . . 224 6.2.4 Symptom: Failure to access a resource through JDBC . . . . . . . . . 225 6.2.5 Symptom: Failure to access a non-relational resource . . . . . . . . . . 2266.3 Analyzing problem areas. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 227 6.3.1 Configuration problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 228
7.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 252 7.2 Work the problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 256 7.2.1 Collect the data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 257 7.2.2 Analyze the high-level symptoms . . . . . . . . . . . . . . . . . . . . . . . . . . 259 7.3 Analyzing problem areas. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 261 7.3.1 Problem: Web server will not start . . . . . . . . . . . . . . . . . . . . . . . . . 261 7.3.2 Problem: Failure between the Web server and plug-in . . . . . . . . . . 263 7.3.3 Problem: Sessions are being lost . . . . . . . . . . . . . . . . . . . . . . . . . . 272 7.3.4 Problem: The application works intermittently. . . . . . . . . . . . . . . . . 277 7.3.5 Problem: Application load is not being evenly distributed . . . . . . . . 280 7.4 The next step. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 283
8.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 286 8.2 Work the problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 286 8.2.1 High-level symptom analysis. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 287 8.2.2 Data to collect . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 288 8.2.3 Analyze the data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 290 8.2.4 Analyze the javacore file . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 291 8.2.5 Finding a workaround for JIT problems. . . . . . . . . . . . . . . . . . . . . . 293 8.3 Analyzing problem areas. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 295 8.3.1 Stack overflow. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 295 8.3.2 Out of memory error . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 298 8.4 The next step. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 299 8.4.1 Sun Solaris . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 300 8.4.2 HP-UX . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 300
9.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 302 9.1.1 Problem categories . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 305 9.2 Work the problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 305 9.2.1 Collect the data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 306 9.2.2 Analyze the high-level symptoms . . . . . . . . . . . . . . . . . . . . . . . . . . 307 9.2.3 What to do if your symptom is not listed here . . . . . . . . . . . . . . . . . 310
9.3 Analyzing problem areas. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 310 9.3.1 Messaging engine startup problems . . . . . . . . . . . . . . . . . . . . . . . . 310 9.3.2 Message flow problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 324 9.3.3 Application configuration and resource problems . . . . . . . . . . . . . . 344 9.3.4 Product errors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3499.4 The next step. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 350
IBM Redbooks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 353Online resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 353How to get IBM Redbooks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 356Help from IBM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 356
Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 357
Contents viiviii WebSphere Application Server V6 Problem Determination for Distributed PlatformsNotices
This information was developed for products and services offered in the U.S.A.
Any references in this information to non-IBM Web sites are provided for convenience only and do not in anymanner serve as an endorsement of those Web sites. The materials at those Web sites are not part of thematerials for this IBM product and use of those Web sites is at your own risk.
IBM may use or distribute any of the information you supply in any way it believes appropriate withoutincurring any obligation to you.
TrademarksThe following terms are trademarks of the International Business Machines Corporation in the United States,other countries, or both:
Eserver® DB2 Connect™ OS/400® Redbooks (logo) ™ DB2® Rational® developerWorks® Informix® Redbooks™ z/OS® IBM® SecureWay® AIX® MQSeries® Tivoli® ClearCase® Netfinity Manager™ WebSphere® Cloudscape™ Netfinity® CICS® OS/2®
iPlanet, Enterprise JavaBeans, EJB, Java, Java Naming and Directory Interface, JavaBeans, JavaScript,JavaServer, JavaServer Pages, JDBC, JDK, JMX, JSP, JVM, J2EE, Solaris, Sun, and all Java-basedtrademarks are trademarks of Sun Microsystems, Inc. in the United States, other countries, or both.
Microsoft, Windows, Win32, and the Windows logo are trademarks of Microsoft Corporation in the UnitedStates, other countries, or both.
UNIX is a registered trademark of The Open Group in the United States and other countries.
Linux is a trademark of Linus Torvalds in the United States, other countries, or both.
Other company, product, and service names may be trademarks or service marks of others.
identifying and resolving problems in WebSphere Application Server V6 on distributed platforms. Previously published as a series of IBM Redpapers, each chapter addresses a specific aspect of the problem determination process. Links to the original papers are provided throughout the book.
The subsequent papers address selected topics that have been identified as some of the more common problem types that customers usually need to address.
This redbook was produced by a team of specialists from around the world working at the International Technical Support Organization, Raleigh Center.
Previously he was an independent/free-lance programmer in the UK. He has over 20 years experience working in the IT field. Has been permanently employed by IBM since 2000 working in the WebSphere MQSeries® service area specializing in Java and JMS.
Software Labs, India. He has over seven years of experience in IT industry in the ares of development, support and test of operating systems, systems management software and e-business solutions. He holds a masters degree in
Structural Engineering from Regional Engineering College (REC) Warangal, India. His areas of expertise include IBM OS/2®, Windows® 2K, Netfinity® Manager™, IBM Director, Healthcare domain solutions of HIPAA(Health Insurance Portability and Accountability Act) and HCN(Healthcare Collaborative Network).
as a team leader for IBM WebSphere Application Server L2 support group in RTP, North Carolina.
where she spent the last thirteen years providing technical support and consulting in the fields of databases, development tools and application servers. Thu-Giang also had four years of experience as a software engineer involved in the development of an application code generator and various financial applications. Before joining IBM in RTP five years ago, Thu-Giang worked for DataDirect Technology (formerly known as MERANT, Intersolv), Oracle and Software AG of Canada. Thu-Giang holds a Bachelor of Mathematics degree, joint honors in the fields of Computer Science, Combinatorics and Optimization from the University of Waterloo in Waterloo, Ontario, Canada.
experience in IT and over five years experience in WebSphere Application Server. He holds a degree in Computer Science from the University of Canberra. His areas of expertise includes WebSphere Application Server, WebSphere Edge Server, IBM DB2® and IBM Content Manager.
based in Raleigh, North Carolina. He joined IBM in 2001 and has been in his current position for four years. In this position, he works directly with clients to resolve WebSphere Application Server problems. He specializes in problems relating to connection pooling, messaging, EJBs, transactions, Web services, and more. He is an IBM Certified Advanced System Administrator for WebSphere Application Server V5. He holds a degree in Computer Science from the Pennsylvania State University.
helping customers to design, implement, and tune J2EE™ applications. For thepast four years, he has been working as project leader for a J2EE developmentframework based on J2EE patterns in the Clinical Engineering R&D Center,Argentina. He holds a degree in Information Technology Engineering from theU.T.N. Córdoba University, Argentina. His areas of expertise include J2EEarchitecture design, WebSphere Application Server consulting services andlegacy mainframe systems integration.
With special thanks to Ron Verbruggen for his guidance in designing anddeveloping the content of this book.
Gustavo BustosGrupo Leviminond
Your efforts will help increase product acceptance and customer satisfaction. As a bonus, you'll develop a network of contacts in IBM development labs, and increase your productivity and marketability.
Find out more about the residency program, browse the residency index, and apply online at: ibm.com/redbooks/residencies.html
Comments welcome Your comments are important to us!
about this or other Redbooks in one of the following ways: Use the online Contact us review redbook form found at: ibm.com/redbooks Send your comments in an email to: redbook@us.ibm.com Mail your comments to: IBM Corporation, International Technical Support Organization Dept. HZ8 Building 662 P.O. Box 12195 Research Triangle Park, NC 27709-2195
determination This paper introduces problem determination strategies for WebSphere Application Server V6. It discusses how to prevent problems, how to plan and prepare for problems that can occur, and what to do when a problem does occur so that it is resolved as quickly as possible. It then guides you to the more detailed information that can help you diagnose the cause of the type of problem that you are experiencing.
1.1 Introduction to problem determination Keeping your enterprise applications highly available to your customers is crucial in today's on demand business environment. WebSphere Application Server V6 has many new features and tools that are designed to minimize problem occurrences. However, if a problem does occur that might negatively impact your business, you need to be able to respond quickly and effectively.
IBM Education Assistant: WebSphere Application Server problem determination http://www-1.ibm.com/support/docview.wss?rs=180&uid=swg27005460
When a problem occurs, your first inclination might be to call IBM Support so that they can provide a fix that resolves the problem. In some cases, contacting IBM is necessary. However, the experience of the WebSphere Application Server Support team has shown that a small percentage of client-reported problems are actually due to defects with WebSphere Application Server code. Most problems are caused by configuration issues, environment issues, application code defects, or a misunderstanding of WebSphere Application Server. Many of these problems can be resolved easily without calling IBM Support to open a problem management record (PMR). In addition, many issues can be resolved by following the problem determination procedures that are discussed in this book.
When a user of an application that is running on WebSphere Application Server first notices a problem, a problem symptom is observed. Sometimes the problem symptom provides clues about the cause of the problem. Other times, a significant amount of problem determination is needed to determine the problem's root cause.
An application does not respond to incoming requests. An application produces unexpected results (possibly errors or exceptions). An application cannot connect to an external system or resource. An application performs slowly or its performance degrades over time.
In any enterprise computing system, you can expect that some problems - large or small - will occur at times. In a best case scenario, the problems that you encounter will not be severe and will not result in critical business impact. However, it is a best practice to prepare and plan for the worst.
Although many problems are caused by factors other than WebSphere Application Server code defects, the WebSphere Application Server Support team does find product defects when working through PMRs with clients. When a defect is found, the support team opens an authorized program analysis report (APAR). Each APAR has a unique identifier, a string that contains two letters (either PQ or PK) and five numbers. You can search for a particular APAR or a problem symptom reported in an APAR on the WebSphere Application Server Support site (Figure 1-1 on page 5). The site is available at: http://www-306.ibm.com/software/webservers/appserv/was/support
Figure 1-1 Searching for an APAR on the WebSphere Application Server Support site
Fix packs do not contain upgrades to the Java Software Development Kit (SDK).They are tested with the latest Java SDK service release, but the upgrades to the
Java SDK are delivered as separate fixes. You can also download the SDK fixes from the WebSphere Application Server Support site.
way to prevent problems from occurring. When you install a fix pack, you can be assured that you will not encounter any of the WebSphere Application Server code defects that are fixed in the fix pack. This saves the time and frustration of seeing one of these problems occur on your system.
Strategy, you can review the Update Strategy document, which is available on the Support site: http://www-1.ibm.com/support/docview.wss?rs=180&uid=swg21191989
Figure 1-3 Checking the refresh pack and fix pack level from the administrative console
Another strategy for preventing problems is to ensure that all software and hardware in your environment meets the prerequisites of WebSphere Application Server V6. WebSphere Application Server has been tested with specific software and hardware configurations. It is known to work successfully in these configurations and to integrate well with the products with which it has been tested. You can find the software and hardware with which WebSphere Application Server has been tested and that it supports at: http://www-306.ibm.com/software/webservers/appserv/doc/latest/prereq.html
The best strategy to prevent problems from occurring when running WebSphere Application Server in production is thorough testing. You should develop a detailed testing strategy for your application and make sure that the strategy is followed every time that you install a new version of the application or that you upgrade WebSphere Application Server.
There are countless methods of software testing that you can employ, and we could fill an entire book discussing every method. Instead, we discuss the basic types of tests that every client should perform. In addition, we provide links to other documentation that might be useful in developing your testing strategy. Testing methods include the following: Unit testing Unit testing ensures that each method in each class of the application provides the expected output for all possible input. There are several unit testing frameworks that can be used to make unit testing easier. One popular framework is JUnit, which is open-source software. Functional testing Functional testing ensures that the application performs as the user expects it to. It tests the entire application to make sure that every component works together correctly. Functional testing is sometimes called integration testing. There are several functional testing frameworks available. JFunc is an extension to JUnit for functional testing. Performance testing Performance testing ensures that the application's performance is acceptable to users. WebSphere Application Server V6 includes an enhanced Tivoli Performance Viewer, which is accessible within the administrative console. In Version 5 and 5.1, the Tivoli Performance Viewer is run in a separate graphical user interface (GUI). To use the Tivoli Performance Viewer, enable the Performance Monitoring Infrastructure (PMI) metrics that you want to view in the administrative console. Log onto the administrative console, select your application server, and then select Performance Monitoring Infrastructure (PMI). PMI monitoring can be done in a production environment using the Basic set (default configuration) or the Extended set with minimal impact. After selecting a statistic set, you can then use the Tivoli Performance Viewer to monitor the current performance of the system, log the performance data, or view data that had been previously logged. You can access the Tivoli Performance Viewer by expanding Monitoring and Tuning in the administrative console, then expanding Performance Viewer, and selecting to view the current activity or previously logged performance data. The charts and graphics in the Tivoli Performance Viewer can give the administrator clues about where performance bottlenecks exist. They can then tune the appropriate WebSphere Application Server properties to relieve the problem. Another feature included in WebSphere Application Server V6 is the Tivoli Performance Advisor. The Advisor analyzes the performance data from your system and provides suggestions on which WebSphere configuration
properties to change to improve the performance. To enable the Advisor, select your application server in the administrative console and then select Runtime Performance Advisor Configuration. For more information about the Tivoli Performance Viewer and Advisor, review the following WebSphere Information Center sections: - Why use Tivoli Performance Viewer? http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/ com.ibm.websphere.base.doc/info/aes/ae/cprf_tpv.html - Monitoring performance with Tivoli Performance Viewer (TPV) http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/ com.ibm.websphere.base.doc/info/aes/ae/tprf_tpvmonitor.html - PMI data organization http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/ com.ibm.websphere.base.doc/info/aes/ae/rprf_datacounter6.html In addition to the Tivoli Performance Viewer and Advisor, there are several other tools, both open-source and proprietary, that are available to make performance testing easier. You can read an overview of the performance tools that are available in the technical article Comment lines from Ruth Willenborg: Selecting WebSphere performance tools, which is available at: http://www-128.ibm.com/developerworks/websphere/techjournal/ 0410_col_willenborg/0410_col_willenborg.html Load and scalability testing Load testing involves testing your application with a simulated workload that corresponds to the amount of load that you expect your application to be able to handle in production. Scalability testing involves testing your application with increasingly higher amounts of load to determine if the application is scalable for future growth. There are several popular tools for simulating load during load and scalability testing. One such tool is Apache JMeter. You should coordinate your load and scalability testing with the tools that you use for performance testing so that you can tune WebSphere Application Server to provide better performance with higher levels of load.
within Rational® Application Developer, Version 6 to develop and test all of yourapplication code before installing it on your WebSphere Application Serverenvironment.
As you design your tests, consider the following: Your test scenarios should focus on the most used code path, but you should comprehensively test all possible code paths. Testing should be done with multiple users (not just the same one, over and over). Tests should be done with multiple functions in parallel. In production, several functions are executed at the same time. Some problems might only occur when the functions are run in parallel. It is important to test your application's functions together and not in isolation.
We strongly recommend that you maintain a test environment that is configured exactly the same as your production environment. The WebSphere Application Server maintenance level (including refresh packs and fix packs), the versions of your applications, and your configuration should all be the same on both systems. There are many benefits to doing this: When you successfully test an application in your test environment, it gives you an accurate reflection of how the application will perform in production. When you need to make a change to an application or apply maintenance to WebSphere Application Server, you can fully test these changes in the test environment to be sure that there are no problems before you make any changes to your production environment. When a problem does occur in production, you can reproduce the problem in the test environment and perform problem determination in the test environment. This ensures that your users will not experience down time with the production environment. You can collect any diagnostic data that is needed to determine the root cause of a problem in the test environment. Because some diagnostic methods can impact performance or necessitate restarting one or more WebSphere Application Server processes, it is usually beneficial to collect the data in your test environment.
The key to this strategy is ensuring that the test environment and productionenvironment are configured exactly the same in every way. This includes: The hardware and network configuration. The operating system level and operating system patches. The other software that is used in conjunction with WebSphere Application Server. This might include Web servers, databases, and messaging systems. The WebSphere Application Server level, including refresh packs, fix packs, Java SDK fixes, and any individual APAR fixes that you might have downloaded or obtained from the WebSphere Application Server Support team. The versions of all applications that are installed. The WebSphere Application Server edition (Base, Network Deployment, or Express). The WebSphere Application Server configuration. WebSphere Application Server V6 includes functionality to create configuration archive files with a .car extension. You can export the configuration from one machine to a .car file and then import that .car file to another system. Any configuration information that is specific to one system (for example, the host name) is removed in the configuration archive. It is a good practice to use configuration archive files to replicate the same WebSphere Application Server configuration on your test environment and production environment. Configuration archives are exported and imported with the wsadmin tool. To export the configuration of a WebSphere Application Server V6 profile or application server, use these wsadmin commands: $AdminTask exportWasprofile {-archive c:\myDirectory\myCell.car} $AdminTask exportServer {-archive c:\myDirectory\myServer.car -nodeName node1 -serverName server1} Use the target directory, node name, and server name that is appropriate for your system. To import the configuration of a profile or application server into a WebSphere Application Server V6 environment, use these commands: $AdminTask importWasprofile {-archive c:\myDirectory\myCell.car} $AdminTask importServer {-archive c:\myDirectory\myServer.car [-nodeInArchive node1][-serverInArchive server1][-nodeName node1][-serverName server1]} Again, use the target directory, node names, and server names that are appropriate for your system.
You can also automate changes to your test and production environments withscripts that run in wsadmin. Automation is advantageous if changes must be
done during non-peak hours (typically late nights or on weekends). The scripts can be scheduled to run at those times so that no one has to run the scripts manually during those hours. You might also want to run scripts to update the configuration of other software on your system at the same time. You can also use the scripts as part of your change log (see "Establishing safe operational procedures" on page 12).
Any time you make a change to either the test environment or the production environment, you should synchronize the two environments so that they remain identical.
that has been tested successfully and found to be stable. When you make any type of change, you can test it. If the change is successful and does not cause any problems, you can then add the change to the baseline. If the change does cause a new problem, you can revert to the safe baseline configuration.
Another important facet of problem prevention and preparation is establishing a set of safe operational procedures for your organization. These procedures should outline the proper processes for making any types of changes to your test and production environments. The WebSphere Application Server Support team has found that many problems occur as a result of configuration or code changes made by one person in an organization of which other people in the organization were not aware. A strategy to eliminate these occurrences greatly reduces the chance of unexpected problems.
ensure that only people who are authorized or part of the appropriate role can make changes to your configuration, upgrade your software, install applications, or do anything else that could potentially introduce problems. WebSphere Application Server V6 provides a comprehensive security infrastructure that you can use to define roles, authenticate, and authorize users. You should also take advantage of the security features of your operating system. This book does not discuss security. However, you can learn more about security in the security section of the WebSphere Information Center at: http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com.ibm. websphere.base.doc/info/aes/ae/welc_concepts_csec.html
In addition to restricting the configuration of your environment, you should alsoimplement a change control system for your application code. There are severalchange control software products available. However, we recommend RationalClearCase®, which integrates seamlessly with Rational Application Developer.You can get more information about Rational ClearCase at its product Web site:http://www-306.ibm.com/software/awdtools/clearcase
vitally important to document any and all changes that are made to yourproduction and test environments. We call this documentation a change log.Many problems surface after an application has been running successfully inproduction for a long time. When you carefully document all changes that weremade to your environment in one location, such as the change log, it is mucheasier to determine why a problem might have occurred at a certain time or date.Ensure that each person in your organization is aware of the process forupdating the change log when any change is made to the environment and thatthey follow the process. They should include the precise time and date of anychanges when they update the change log. Maintaining the change log andensuring strict compliance with this process can save many hours ofinvestigation and frustration when a problem occurs.
process. A good set of procedures that are agreed upon and acted upon byeveryone in your organization can prevent problems from occurring and makeyou more prepared when problems do occur.
1.2.6 High availability and failover As discussed earlier, even the most sophisticated problem prevention techniques cannot guarantee that problems will never occur in your WebSphere Application Server V6 environment. Given this, it is logical to develop a contingency plan for when a WebSphere Application Server process or service becomes unavailable. Fortunately, new features in the area of high availability and failover are included with Version 6.
to provide failover for all applications and WebSphere Application Server services. It is configured automatically when you install WebSphere Application Server Network Deployment Version 6. The high availability manager runs critical WebSphere services (such as WLM and the transaction manager) on any available WebSphere Application Server processes.
available in Version 6, you must store the transaction logs (located by default in the /profiles//tranlog directory) on a network attached storage (NAS) system that is accessible to all of the WebSphere processes to which your transactions could failover (members of the core group). You also must select Enable high availability for persistent services for your cluster in the administrative console. When this is done, and a cluster member fails, its in-flight transactions are recovered on another cluster member.
(deployment managers, node agents, application servers, and cluster members) within a cell that can participate in high availability together. That is, a service running on one member of the core group can failover to another member of the core group. By default, all processes in a cell are part of one core group (called DefaultCoreGroup). This is also the recommended scenario for most production environments, although it is possible to have multiple core groups within one cell.
A process can only be a member of one core group. System services, such as the WLM service, PMI, and the high availability coordinator itself, can failover to any member of the core group. Other services, such as the transaction manager and messaging engine, must failover another cluster member in the same cluster in which they were running when they failed. This is because the transaction manager and messaging engine need the same applications to be installed on the process that they failover to in order to recover.
comparable to Version 5 and 5.1. The addition of the high availability manager significantly enhances the failover capabilities in Version 6, making it no longer necessary to run third-party high availability tools with WebSphere Application Server.
and possibly dissatisfied with your company when these types of situations occur.
A good monitoring strategy can help you to identify problems before your customers experience them.
Application Server V6 includes a wide variety of system messages that are designed to provide you with information, warnings, and notifications of errors. You also find many system messages in WebSphere Application Server traces. Each system message has a unique message identifier that is nine characters in length and is in the form CCCC1234X. The first four characters (CCCC) indicate the WebSphere Application Server component that issued the message. The next four characters (1234) indicate the specific message that is being issued by the component. The last character (X) indicates the severity of the message. Its value is either I (informational), W (warning), or E (error).
SECJ0231I: The Security component's FFDC Diagnostic Module com.ibm.ws.security.core.SecurityDM registered successfully: true. ADMN0001W: The service is unable to parse the MBean descriptor file com/ibm/ws/management/descriptor/xml/mbeans.xml. SRVE0068E: Could not invoke the service() method on servlet /com.ibm.ws.console.probdetermination/loggingSettingsGroups.jsp. Exception thrown : java.lang.NullPointerException
messages usually are not indicative of a problem. However, if an informational message is unexpected, it might alert you to an unusual occurrence that can result in a problem. Warnings and error messages are definitely signs that a problem has occurred.
The SystemErr log does not contain WebSphere system messages, but it doesshow exceptions that are thrown by WebSphere Application Server or by anapplication. It is a good idea to monitor the SystemErr log entries in addition tothe system messages in the SystemOut log.
By monitoring the SystemOut and SystemErr logs and adding your ownapplication logging, you can discover proactively a majority of the problems thatare covered in this book.
You can view errors, warnings, and informational messages directly in theadministrative console (Figure 1-4).
Figure 1-4 Viewing Runtime Events in the administrative console
You can select each message that you see to get more details about the message, the source of the message, and the reason why it occurred (Figure 1-5 on page 18).
Viewer, discussed in "Testing the application" on page 7. You can observe
performance metrics, such as average response time, number of requests, thread pool sizes, connection pool sizes, JVM memory, CPU, I/O, and system paging, to monitor the health of WebSphere Application Server and your applications. You can log the performance data from a time when everything is running normally and then compare that data to the current performance data to see if there are major discrepancies. Unexpected discrepancies are an indication of a problem. When you see signs that a problem might be occurring, you can perform the specific problem diagnostic steps that are described within this book.
availability and failover strategy, should enable you to diagnose and resolve problems that occur, possibly before your users and customers know about them.
In the event that a problem occurs in your environment, it is possible that you will need to enlist the help of other people, either internal or external to your organization, to determine the root cause of the problem. When that happens, you will want everyone involved to understand thoroughly the details of the systems that are involved in your environment.
"Establishing safe operational procedures" on page 12, we discussed documenting all of the changes that have been made to your environment. In addition to that, you should maintain a high-level description of your basic topology. We call this system documentation. System document is useful in the following circumstances: A problem occurs and you need to get assistance from others who might not be as familiar with your application and topology as you are. The system documentation allows you to bring them up to speed as quickly as possible. A problem occurs and you want to identify from which parts of your environment you should collect diagnostic data or monitor. Your system documentation shows the software components that are involved and the flow of your application, that is how different software components are used when your application processes a request.
Your system documentation should consist of written documents and diagrams. Which information is included in the written documents and which is included in diagrams is a matter of preference. Overall, the information should be detailed and should show the specific versions and maintenance levels of the operating system and all software products involved, the hardware and network configuration, and specific host names and IP addresses of the systems that are involved.
diagram. It gives a quick overview of your system topology and application flow. Figure 1-6 on page 20 illustrates an example.
host5.mycompany.com host2.mycompany.com 1.12.34.60 1.12.34.57
Web
Firewall Server Cluster MQ Member 2
specific software and hardware levels that are involved.
Detailed system documentation is an integral part of your problem planning strategy that should not be overlooked.
Finally, to prepare for a problem occurrence, you should plan what diagnostic data to collect for various problem scenarios. In "Types of problem symptoms" on page 3, we discussed several broad categories of problem symptoms. In "What to do when a problem occurs" on page 24 and other papers in this series, we elaborate on how to determine the cause of a problem for each of the problem symptoms. We also include information about what data to collect for different types of problems. It is a good idea to identify the most common problems that have occurred in the past in your environment and those that you believe might occur the most in the future. Then, you can form a diagnostic data collection plan so that you are prepared to collect the necessary data if a problem does occur.
several times to collect the necessary data. To avoid this situation, it is recommended that you have between 2 GB and 5 GB of extra disk space available on each system. After you resolve a problem, either delete or archive the diagnostic data that you collected for the problem. This will prevent the old diagnostic data from being confused with the new diagnostic data the next time that a problem occurs. Configure the thread monitor for hang detection. WebSphere Application Server V6 includes a thread monitor feature. This is also included in Version 5.1.1. The thread monitor is notified when the Web container, ORB, or asynchronous bean thread pools give work to a thread. By default, the thread monitor checks the status of all active threads every three minutes. If it finds a thread that has been active for more than ten minutes, it outputs a warning to the SystemOut log, similar to the following: WSVR0605W: Thread threadname has been active for hangtime and may be hung. There are totalthreads threads in total in the server that may be hung. The thread monitor makes it easier to determine that a problem has occurred. If you see a WSVR0605W warning in your SystemOut log, you out know that a thread has stopped responding. You can then perform further diagnostic steps to determine the cause of the hung thread. The thread monitor does not take any action to fix the problem beyond notifying you of the problem. In preparing for diagnostic data collection, you might want to change the default thread monitor behavior. You can change the interval that the thread monitor checks the status of threads (this is three minutes by default) and the amount of time that a thread can be active before it is reported by the thread monitor (this is ten minutes by default). To alter these properties: a. Log onto the administrative console. b. Select your application server. c. Under Server Infrastructure, select Administration. d. Select Custom Properties and then click New. e. Create these properties and provide the desired values for them: * com.ibm.websphere.threadmonitor.interval (the interval that the thread monitor checks the status of threads) * com.ibm.websphere.threadmonitor.threshold (the amount of time that a thread can be active before it is reported by the thread monitor) For more information about the thread monitor, see the following sections in the WebSphere Information Center:
- Detecting hung threads in J2EE applications http://publib.boulder.ibm.com/infocenter/ws60help/index.jsp?topic=/ com.ibm.websphere.base.doc/info/aes/ae/ctrb_hangdetection.html - Configuring the hang detection policy http://publib.boulder.ibm.com/infocenter/ws60help/index.jsp?topic=/ com.ibm.websphere.base.doc/info/aes/ae/ttrb_confighangdet.html Consider enabling verbose garbage collection on each application server. The performance impact of enabling verbose garbage collection is minimal, and the data is often useful when performance problems occur. To enable verbose garbage collection: a. Log onto the administrative console. b. Select your application server. c. Under Server Infrastructure, expand Java and Process Management, and then select Process Definition. d. On the resulting screen, select Java Virtual Machine under Additional Properties. e. Select Verbose garbage collection. When verbose garbage collection is enabled, the output appears in the native_stderr.log file for your application server. Become familiar with the WebSphere Application Server collector tool. The collector tool is run as an executable in the /bin directory. It produces a Java archive (jar) file that contains all of the logs and XML configuration files from your system. The resulting jar file is very useful to the WebSphere Application Server Support team and any others who are involved in the problem determination process. It allows them to view quickly your WebSphere Application Server configuration and to see any errors or exceptions that have occurred. You can find more information about the collector tool in WebSphere Application Server V6: Diagnostic Data at: http://www.redbooks.ibm.com/redpapers/pdfs/redp4085.pdf
comprehensive list of MustGather documents for different types of WebSphereApplication Server problems. For more information, see IBM - MustGather: Readfirst for all WebSphere Application Server products at:http://www-1.ibm.com/support/docview.wss?rs=180&uid=swg21145599
The files listed in the MustGather documents are useful in determining the causeof your problem. You can use these documents to as part of your diagnostic datacollection plan.
Consider all the types of problems that you might be likely encounter in your environment while reading this book. Based on the types of problems that have occurred in your environment in the past, you might be able to predict what types of problems can occur in the future. Take note of the diagnostic data that is needed to determine the root cause of each problem type. Make sure that this is documented so that everyone in your organization knows what data to collect. This will make the problem determination process simpler and less stressful.
You have prepared, and you have planned. Your application is now running in your production environment, and it is being used successfully by your customers. Just as you are ready to celebrate, you get a call that a problem has occurred. What do you do now?
When any problem occurs, your first action should be to consider the business impact of the problem. Depending on the business impact, it might be necessary to take steps to limit the business impact before beginning your problem determination efforts. If the problem occurred in your test environment, the business impact is relatively low when compared to the potential impact of a production outage. In that case, making an effort to work around the problem while looking for a permanent solution is probably not necessary. You can use the test environment to execute all of the necessary problem determination steps.
consider how to quickly alleviate the problem symptoms so that customers and users will experience the least possible negative effects. In this section, we refer to this process as reverting to safe conditions. You will want to do this in parallel with your problem determination efforts.
Making your test environment or another similar environment where the problem is not occurring your temporary production environment. Configure
your systems so that incoming requests are processed by an environment where the problem does not occur. Installing an older version of application code where the problem does not occur. If the problem started to occur after an application code change was introduced, it might be a good idea to go back to an earlier working version of the application. Changing any recent configuration changes back to your baseline configuration. As discussed in "Setting up a test environment" on page 10, the baseline configuration should be a configuration that is fully tested and is known to be stable. Removing any WebSphere Application Server maintenance that was recently installed before the problem occurred. This would temporarily resolve the problem if it is caused by a WebSphere Application Server code defect introduced by the latest maintenance package. Making the application function that produced the problem inaccessible to customers and users. You can post a notification on your Web site that the function is temporarily unavailable or under maintenance. You can provide an estimated time for when it will be available again.
When you are first notified about the problem, you might only receive a vague, non-specific set of problem symptoms. You might be told that users cannot access your application at all or possibly that a specific action taken by your users is resulting in an error message.
How do you know when this particular problem occurs? Is there a something specific to watch for in order to recognize if the problem occurs again? How would you know that the problem was resolved? Would it be that an error message no longer occurred? Would the application behave differently? What specifically would confirm a problem resolution? Where did the problem occur? Did the problem occur only in your test environment, only in your production environment, or on both? Did it only occur on one system in your environment? Did it occur on multiple systems? Did it occur on every cluster member or only one? When did the problem occur? What was the time stamp of the error or unexpected behavior? Did the problem occur only once or many times? How often did the problem occur? Did it occur at certain intervals, or did it seem to occur at random times? Was there some event that occurred that might have triggered the problem? For example, did the user attempt a specific application function? Why might the problem have occurred? You might not be able to answer this right away. Was this the first time something specific was tried? Was there a recent change to the application code or the configuration of your environment? Does it happen in all of your environments or only one? If it only happens in one environment, how is this environment different from the others? Has the diagnostic data that is identified in your diagnostic data collection plan been collected? Does the data provide any other details about the problem or offer immediate clues as to why the problem occurred?
On the other hand, the answer might require more investigation. If this is the case, you have developed a solid and complete problem description to use as the basis for more problem determination efforts.
While investigating the problem, you should form a list of all of the problem symptoms that occurred within your problem log. Sometimes, there is only one symptom. If this is the case, your job is easier. However, often several symptoms occur, and it can be difficult to determine which symptoms characterize the problem and which symptoms are simply the result of the problem. It is a good idea to organize the symptoms into a time line. You can also include details about what was happening in your environment in your time line. For example, your time line might look similar to the following: 10:00 - Peak workload reached 10:07:53 - ConnectionWaitTimeoutException in SystemOut.log for server1 10:08:24 - ConnectionWaitTimeoutException in SystemOut.log for server1 10:14:09 - ConnectionWaitTimeoutException in SystemOut.log for server2 10:20:46 - Users cannot log onto Web site, application is not responding
observed, it is a good starting point for your investigation. However, it is possible that your investigation might reveal that a symptom is actually a different problem that is unrelated to the problem that you are investigating.
Informational messages and warnings provide context. They show you what was occurring immediately before or immediately after a problem. Error messages are significant problem symptoms. The message identifier for an error message is useful for entering into search engines.
When you choose a symptom to investigate, you can begin researching it. A good place to start is your organization's internal documentation. It is possible
that someone else has already seen the same problem or a similar problem in your environment. If they have, it is likely that the root cause of the problem is the same. If this is the first time that a problem similar to this has been seen in your organization, the next place to research is online documentation that is available from IBM. We recommend using the following resources to research your symptoms: WebSphere Information Center http://publib.boulder.ibm.com/infocenter/ws60help/index.jsp The WebSphere Information Center includes detailed information about the features and configuration of the product. It also includes descriptions of every WebSphere Application Server system message identifier, and it contains a troubleshooting guide that might aid your problem determination efforts: http://publib.boulder.ibm.com/infocenter/ws60help/index.jsp?topic=/com.ibm. websphere.base.doc/info/aes/ae/welc_concepts_ctrb.html WebSphere Application Server Support site http://www-306.ibm.com/software/webservers/appserv/was/support You can search the support site for APARs and technotes that might be related to your problem symptom. As discussed in "Applying WebSphere maintenance" on page 4, APARs are reports of known WebSphere Application Server code defects. If an APAR description matches your problem symptom, you should install the WebSphere Application Server fix pack that contains the fix for the APAR. The APAR description explains which fix pack includes the fix. Technotes are documents that are created and maintained by the WebSphere Application Server Support team and knowledge engineering team. They document known problems and the solutions to those problems. Technotes are usually created for problems that are solved by a configuration change rather than a code change. WebSphere developerWorks® http://www-130.ibm.com/developerworks/websphere The developerWorks site contains many articles written by IBM developers and other technical staff members. Many articles discuss application programming best practices and provide tips for avoiding and resolving problems with WebSphere Application Server. IBM Support Assistant http://www-1.ibm.com/support/docview.wss?rs=180&uid=swg21192593 The IBM Support Assistant is a downloadable tool that is used to simplify the problem determination process for many IBM software products. There are several product plug-ins for the Support Assistant, including one for WebSphere Application Server V6. It includes a federated search interface
that allows you to search for your problem symptoms at multiple IBM Web sites simultaneously. It also provides easy access to product education modules, including the IBM Education Assistant. If you need to open a PMR with IBM Support, the tool collects pertinent information that is needed by the support team. You can even open a PMR within the tool.
This book also provides valuable information as you research your problem.
At this point, you have identified specific and detailed problem symptoms, reverted to safe conditions so that you have minimized the business impact of the problem, and begun your investigation of the problem. The next step is to start determining the root cause of the problem, which is the main focus of this book.
For any problem that occurs, there are two major strategies that you can use to determine the root cause: the analysis strategy and the isolation strategy.
The analysis strategy involves analyzing the diagnostic data, possibly through several iterations, until the cause of the problem is found, as illustrated in Figure 1-7 on page 30. To be successful with this strategy, you must have a good understanding of the diagnostic data. This is the strategy used most often by the WebSphere Application Server Support team. There are many diagnostic tools that are available to help you. For example, the ThreadAnalyzer helps you to analyze Java thread dumps, and the HeapRoots tool is useful in analyzing out of memory issues. You can review and download the most current diagnostic tools at: WebSphere Application Server Support site http://www-306.ibm.com/software/webservers/appserv/was/support WebSphere developerWorks http://www-130.ibm.com/developerworks/websphere
Analyze data
Collect diagnostic data
the SystemOut or SystemErr log for the application server, you see the Java stack trace of the exception. The stack trace is read from the bottom to the top. At the bottom, you see the thread that is allocated by the thread pool. Reading the stack trace upwards, you see every method that is called (including WebSphere Application Server code, application code, and possibly other third-party utilities) until the exception is thrown. You can use the stack trace to identify the different variables in the problem scenario and then remove them.
It might also help to insert print statements into the application to print debugging information as the application executes.
Similar to the analysis strategy, the isolation strategy is an iterative process. You keep removing variables until you have isolated the variable that is causing the problem.
Analyze results
Reproduce the problem
Now that we have discussed the general strategies for problem determination, we focus on the specific strategies for each problem classification.
The next step in the problem determination process, and a very important one, is classifying the problem. This book refers to many different classifications of problems. You need to identify which classification the problem fits into.
performance monitoring. See "System management and configuration" on page 37 for more information. An application or WebSphere Application server process (for example, an application server, node agent, or deployment manager) is unable to start. When running the startServer, startNode, or startManager commands to start a WebSphere Application Server process or when using the administrative console or wsadmin to start a process or an application, a problem occurs that causes the process or application to fail to start. See "Application or WebSphere process is unable to start" on page 42 for more information. An application does not respond to incoming requests. This could occur due to a Web server, Edge component, or Web server plug-in problem, application server crash, hang, out of memory condition, or 100% CPU utilization condition. See "Application does not respond to incoming requests" on page 43 for more information about determining the root cause of the problem. An application produces unexpected results (possibly errors or exceptions). An error is seen or an exception is thrown when certain application code is executed or when certain conditions (for example, heavy load) are met. The application could also behave differently than expected, but produce no error or exception. There are several components of WebSphere Application Server in which this type of problem could occur. See "Unexpected results from an application" on page 49 for more information. An application cannot connect to an external system or resource. The application might need to access an external system or resource. This could be a database, a messaging system, an enterprise information system accessed through the Java Connector Architecture (JCA), an Enterprise JavaBeans™ (EJB™) running on a remote system, or a Web service. There might be a problem establishing a connection to the external system or resource, or an error might occur when the application interacts with it. See "Application cannot connect to an external system or resource" on page 63 for more information. An application performs slowly or its performance degrades over time. Although performance problems fall outside the scope of this book, we do provide some external resources for performance problems. In general, performance problems can be corrected by tuning WebSphere Application Server, the other software products with which it interacts, and the operating system. Thorough performance testing and tuning should be completed before an application is put into production. See "Application is slow or its performance degrades over time" on page 69 for more information.
The WebSphere Application Server Support team has found that the majority ofproblems encountered by clients fit into one of these problem classifications. Inthis book, we provide comprehensive information about problem determinationstrategies for some of these common problem areas. For the problemclassifications that are not covered in this book, we provide links to externalresources that discuss problem determination for each problem type.
If you have already examined the WebSphere logs and have determined that aparticular error message holds the key to the problem, use Table 1-1 as a quickreference to find information based on the message prefix.
Message prefixes Topic
WSVRT (during IVT)
WACS, APPR, ASYN, OBPL, SCHD, STUP, "Program model extensions" on page 59 ACWA
Message prefixes Topic
character set" on page 61
CHFW, DCSV, HTPC, SSLC, TCPC, WSSC, "Transport channel service" on page 62 and XMEM
CWSIF, CWSIH, CWSII, CWSIJ, CWSIK, CWSIL, CWSIM, CWSIN, CWSIO, CWSIP, CWSIQ, CWSIR, CWSIS, CWSIT, CWSIU, CWSIV, CWSIW, CWSIX, CWSIY, CWSIZ, CWSJA, CWSJB, CWSJC, CWSJD, CWSJO, CWSJQ, CWSJR, CWSJU, CWSJW, CWSWS, WMSG
This section addresses problems related to the installation of WebSphere Application Server or the migration from previous releases.
Note: For problem determination strategies for installation problems, see WebSphere Application Server V6: Installation Problem Determination at: http://www.redbooks.ibm.com/redpapers/pdfs/redp4068.pdf
Symptom: You are having problems deploying an application to WebSphere Application Server using the administrative console or wsadmin. You might receive WebSphere system messages that begin with ADMA.
problems, review the following resources: WebSphere Information Center: Developing and deploying applications http://publib.boulder.ibm.com/infocenter/ws60help/index.jsp?topic=/com.ibm. websphere.base.doc/info/aes/ae/welc6topdeveloping.html
WebSphere Information Center: Troubleshooting deployment http://publib.boulder.ibm.com/infocenter/ws60help/index.jsp?topic=/com.ibm. websphere.base.doc/info/aes/ae/ttrb_deploy.html WebSphere Information Center: Explanation of ADMA system messages http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.messages.doc/doc/ADMA.html MustGather: Problems during Deployment of EAR/JAR/WAR files http://www-1.ibm.com/support/docview.wss?rs=180&uid=swg21199344 MustGather: Enhanced EAR file problems for V6.0 http://www-1.ibm.com/support/docview.wss?rs=180&uid=swg21199181
Symptom: You are trying to use WebSphere Rapid Deployment to develop and are trying to test an application. You cannot connect to an application server or WebSphere Rapid Deployment does not create or update the applications.
Application Server V6: System Management Problem Determination at: http://www.redbooks.ibm.com/redpapers/pdfs/redp4067.pdf
Symptom: You have a problem when using the Application Server Toolkit (AST) to assemble your applications. This encompasses all possible problems with the AST, including starting it, any problems that you experience with application assembly, and any errors that occur.
1.4.3 System management and configuration Symptom: You experience difficulties in WebSphere Application Server system management or configuration. This includes problems with the following system management tools or functions: Configuration and management using administration tools Security JMX clients PMI and Tivoli Performance Viewer
Symptom: You have a problem accessing or using the administrative console, wsadmin scripting tool, or command line scripts. This would include the following symptoms: You are not able to access the administrative console. You cannot access server processes using wsadmin or the management scripts such as stopServer. You are getting errors performing system management functions, for example managing application servers, node agents, Web servers, or applications. You cannot federate a node with a deployment manager. You are getting save conflict messages in the administrative console. Your enterprise applications no longer appear in the administrative console.
problems, see WebSphere Application Server V6: System Management Problem Determination at: http://www.redbooks.ibm.com/redpapers/pdfs/redp4067.pdf
You can find problem determination strategies for issues with administrative scripting tools in the following resources: WebSphere Information Center: Using Ant to automate tasks http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.base.doc/info/aes/ae/tovr_ant.html WebSphere Information Center: Using command line tools http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.base.doc/info/aes/ae/txml_command.html
WebSphere Information Center: Troubleshooting administration http://publib.boulder.ibm.com/infocenter/ws60help/index.jsp?topic=/com.ibm. websphere.base.doc/info/aes/ae/ttrb_admin.html WebSphere Information Center: Explanation of WASX system messages http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.messages.doc/doc/WASX.html MustGather: ws_ant problems on V6.0 http://www-1.ibm.com/support/docview.wss?rs=180&uid=swg21196231 MustGather: WebSphere configuration archive functionality http://www-1.ibm.com/support/docview.wss?rs=180&uid=swg21200348
You can find problem determination strategies for other system management issues in the following resources: WebSphere Information Center: Setting up the administrative architecture http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.nd.doc/info/ae/ae/tagt_admin.html WebSphere Information Center: Administering application servers http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.nd.doc/info/ae/ae/trun_svr_conf.html WebSphere Information Center: Troubleshooting administration http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.base.doc/info/aes/ae/ttrb_admin.html WebSphere Information Center: Explanations of ADFS, ADMB, ADMC, ADMD, ADME, ADMF, ADMG, ADMK, ADML, ADMN, ADMR, ADMS, ADMU, BNDE, CHKC, CHKP, CHKS, CHKW, ECNS, ODCF, PROC, WACT, and WSVM system messages http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.express.doc/info/exp/ae/welc_ref_trb_msg.html MustGather: System management functionality for V5.0, V5.1 and V6.0 http://www-1.ibm.com/support/docview.wss?rs=180&uid=swg21199596 MustGather: Synchronization problems in V6.0 http://www-1.ibm.com/support/docview.wss?rs=180&uid=swg21196219 MustGather: Federation or Removal of a Node Issues for Version V6.0 http://www-1.ibm.com/support/docview.wss?rs=180&uid=swg21196227 MustGather: Profile Creation/Removal Issues for V6.0 http://www-1.ibm.com/support/docview.wss?rs=180&uid=swg21196228
MustGather: Usage and creation of templates fail on V6.0 http://www-1.ibm.com/support/docview.wss?rs=180&uid=swg21195439 MustGather: Node agent and Deployment Manager discovery problems for all releases and editions of V6.0 http://www-1.ibm.com/support/docview.wss?rs=180&uid=swg21196220 MustGather: Port Management for V6.0 http://www-1.ibm.com/support/docview.wss?rs=180&uid=swg21196226 WebSphere Application Server V6 System Management & Configuration Handbook, SG24-6451.
Note: For information about problems with SSL configuration see WebSphere Application Server V6: System Management Problem Determination at: http://www.redbooks.ibm.com/redpapers/pdfs/redp4067.pdf
WebSphere Information Center: Explanation of SECG system messages http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.messages.doc/doc/SECG.html WebSphere Information Center: Explanation of JSAS system messages http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.messages.doc/doc/JSAS.html WebSphere Information Center: Explanation of JSSL system messages http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.messages.doc/doc/JSSL.html WebSphere Information Center: Explanation of WSEC system messages http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.messages.doc/doc/WSEC.html WebSphere Information Center: Explanation of WSSK system messages http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.messages.doc/doc/WSSK.html MustGather: Problems using Global Security for all Releases of V6.0 http://www-1.ibm.com/support/docview.wss?rs=180&uid=swg21199336 MustGather: Problems using JAAS Web logins for all Releases of V6.0 http://www-1.ibm.com/support/docview.wss?rs=180&uid=swg21199332 MustGather: Problems using Web services Security for all Releases of V6.0 http://www-1.ibm.com/support/docview.wss?rs=180&uid=swg21199335 MustGather: Java Secure Socket Extension (JSSE), SSL or Java Cryptography Extensions (JCE) problems http://www-1.ibm.com/support/docview.wss?rs=180&uid=swg21162961 WebSphere Application Server V6 Security Handbook, SG24-6316.
WebSphere Information Center: Troubleshooting administration http://publib.boulder.ibm.com/infocenter/ws60help/index.jsp?topic=/com.ibm. websphere.base.doc/info/aes/ae/ttrb_admin.html MustGather: JMX API client for V6.0 http://www-1.ibm.com/support/docview.wss?rs=180&uid=swg21196218
Symptom: You have a problem with the Performance Monitoring Infrastructure (PMI) or the Tivoli Performance Viewer. You might receive WebSphere system messages that begin with PMON. You might also have problems configuring PMI or performing any operation with the Tivoli Performance Viewer in the administrative console.
performance through several counters and statistics. You can view the performance data using the Tivoli Performance Viewer within the administrative console. For information about these features, see "Testing the application" on page 7.
Unexpected results from an application Application cannot connect to an external system or resource Application is slow or its performance degrades over time
Symptom: An application or WebSphere Application Server process (for example an application server, node agent, or deployment manager) is unable to start.
startManager, and stopManager commands, see WebSphere Application Server V6: System Management Problem Determination at: http://www.redbooks.ibm.com/redpapers/pdfs/redp4067.pdf
Application does not respond to incoming requestsSymptom: An application does not respond to incoming requests. The fault couldlie with any of the following components or conditions: IBM HTTP Server Edge components Web server plug-in Application server crash Application server hang 100% CPU utilization Out of memory
Symptom: An application server does not respond to incoming requests andstatic HTML pages are not being served by the IBM HTTP Server.
IBM HTTP Server, Version 6 Information Center http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.ihs.doc/info/welcome_ihs.html IBM HTTP Server, Version 6 Information Center, Troubleshooting IBM HTTP Server http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.ihs.doc/info/aes/ae/welc_troub.html MustGather: Read first for IBM HTTP Server http://www-1.ibm.com/support/docview.wss?rs=177&uid=swg21192683
Load Balancer, Version 6 Troubleshooting http://www-306.ibm.com/software/webservers/appserv/doc/v60/ec/infocenter/ edge/LBguide.htm#HDRTRB
The Web server plug-in enables the Web server to send requests for dynamiccontent to Web applications (servlets and JSPs) that are installed in WebSphereApplication Server.
responds when accessed directly through the application server but not whenaccessed through the Web server. High-level symptoms of a Web server plug-inproblem include: Users cannot access an application through the Web server Load balancing and failover not working properly Session data is being lost Slow or intermittent application response The Web server will not start after plug-in installation or configuration
When your application fails to respond to requests, first check to see if your Webserver is responding to requests for static HTML pages. If it is, you can narrowyour focus to the plug-in and the application server.
the application server, bypassing the plug-in. You can do this by using theapplication server's HTTP transport port (port 9080 by default) in the URL toaccess the servlet or JSP. For example, you can try to access the snoop servlet,which is one of the WebSphere Application Server samples:http://localhost:9080/snoop
Application server crash Symptom: An application is not responding to incoming requests. The application server process is no longer running.
Symptom: An application is not responding to incoming requests but the application server process is still running.
Detecting hung threads in J2EE applications http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.base.doc/info/aes/ae/ctrb_hangdetection.html Web module or application server dies or hangs http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.base.doc/info/aes/ae/rtrb_appdies.html IBM Education Assistant: WebSphere Application Server problem determination http://www-1.ibm.com/support/docview.wss?rs=180&uid=swg27005460 You can find a list of MustGather documents for hangs with the following search argument http://www-1.ibm.com/support/search.wss?rs=180&tc=SSEQTP&tc1=SSCMPB9&q=Must GatherDocument
Symptom: You observe that a WebSphere Application Server process (anapplication server, cluster member, node agent, or deployment manager)reaches 100% or an unusually high percentage of CPU utilization.
You might notice this by using your operating system utilities to check the CPUuse of each process on the system, or you might have an application server hangcaused by the high CPU use.
You can find problem determination strategies for 100% CPU usage issues in thefollowing resources: MustGather: 100% CPU Usage on AIX® Platforms http://www-1.ibm.com/support/docview.wss?rs=180&uid=swg21116458 MustGather: 100% CPU Usage on HP-UX http://www-1.ibm.com/support/docview.wss?rs=180&uid=swg21166017 MustGather: 100% CPU usage on Linux® http://www-1.ibm.com/support/docview.wss?rs=180&uid=swg21132699 MustGather: 100% CPU Usage on Solaris™ platforms http://www-1.ibm.com/support/docview.wss?rs=180&uid=swg21115625 MustGather: 100% CPU Usage on Windows Platforms http://www-1.ibm.com/support/docview.wss?rs=180&uid=swg21137447
Out of memorySymptom: You observe that a WebSphere Application Server process (anapplication server, cluster member, node agent, or deployment manager)
consumes all of the available memory on your system, or you see a java.lang.OutOfMemoryError in the SystemOut or SystemErr log for the process.
You might notice this by checking the memory usage that is using your operating system utilities, by checking the logs, or by having an application server hang that is caused by an out of memory problem.
Unexpected results from an applicationAn application produces unexpected results (possibly errors or exceptions). Thisis a broad category that includes problems that can occur in any of the followingruntime components or services: Just-In-Time (JIT) compiler Web container EJB container Classloader Session management Dynamic cache Transaction manager Workload management High availability manager Data replication service Program model extensions Internationalization/Double byte character set Transport channel service
Symptom: You receive intermittent unexpected results from an application.Disabling the JIT compiler resolves the problem.
For more information about mixed mode threshold, see: http://www.ibm.com/developerworks/java/jdk/diagnosis
If the problem does still occur when JIT is disabled, review the other problem classifications described here, determine into which classification the problem fits, and follow the problem determination steps for that classification.
Note: For problem determination strategies for Web container problems, see WebSphere Application Server V6: Web Container Problem Determination at: http://www.redbooks.ibm.com/redpapers/pdfs/redp4058.pdf
EJB containerThe EJB container is the runtime environment for EJBs in WebSphereApplication Server and it handles all EJB requests from clients.
the problem appears to originate with an EJB (entity beans, session beans, andmessage-driven beans). High-level symptoms can include any of the following: Unexpected behavior when an EJB runs Problems with the EJB life cycle and caching provided by the EJB container Any errors or exceptions that occur when running an EJB Error or warning messages with the following prefixes: CNTR, PMGR, and ACIN
WebSphere Information Center: Explanation of PMGR system messages http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.messages.doc/doc/PMGR.html WebSphere Information Center: Explanation of ACIN system messages http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.messages.doc/doc/ACIN.html MustGather: EJB container for releases of V4.0, V5.0, V5.1 and V6 http://www-1.ibm.com/support/docview.wss?rs=180&uid=swg21153218 MustGather: Persistence Manager http://www-1.ibm.com/support/docview.wss?rs=180&uid=swg21200344
You can find problem determination strategies for classloader problems in the following resources: WebSphere Information Center: Class loading http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.base.doc/info/aes/ae/trun_classload.html WebSphere Information Center: Class loading: Resources for learning http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.base.doc/info/aes/ae/rrun_classload_rlinks.html WebSphere Information Center: Troubleshooting class loaders http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.nd.doc/info/ae/ae/ttrb_classload_viewer.html MustGather: Classloader Issues in V6.0 http://www-1.ibm.com/support/docview.wss?rs=180&uid=swg21196187
Session management A session (sometimes called an HTTP session) is a series of HTTP requests to a servlet from the same user using the same Web browser. WebSphere
Application Server provides session management functionality to keep track ofeach user and enable applications to provide personalized content.
sessions by WebSphere Application Server. This can include any of thefollowing: Unexpected session behavior Session time outs Problems with session storage (database persistence or memory-to-memory sessions) Session data being lost Errors or exceptions from the WebSphere Application Server session manager Problems with personalized Web content Error or warning messages with the prefix SESN
Application Server V6: Web Container Problem Determination at: http://www.redbooks.ibm.com/redpapers/pdfs/redp4058.pdf
environment, there might be an underlying Web server plug-in problem. For more information about this, see WebSphere Application Server V6: Web Server Plug-in Problem Determination at: http://www.redbooks.ibm.com/redpapers/pdfs/redp4045.pdf
WebSphere Information Center: Problems creating or using HTTP sessions http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.base.doc/info/aes/ae/rtrb_httpsessprobs.html WebSphere Information Center: Managing HTTP sessions: Resources for learning http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.base.doc/info/aes/ae/rprs_r4ln.html WebSphere Information Center: Explanation of SESN system messages http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.messages.doc/doc/SESN.html MustGather: Sessions and session management problems in V6.0 http://www-1.ibm.com/support/docview.wss?rs=180&uid=swg21192604
Note that problems with the data replication service can cause session management problems (see "Data replication service" on page 58).
Dynamic cache The dynamic cache feature caches the output of servlets, JSPs, and external Web services that are called by clients within WebSphere Application Server.
WebSphere Application Server. Potential problems can include any of the following: Unexpected or incorrect cache behavior Errors and exceptions from the dynamic cache component Dynamic cache configuration issues Slow performance Error or warning messages with the prefix DYNA
WebSphere Information Center: Troubleshooting tips for the dynamic cache service http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.base.doc/info/aes/ae/rdyn_trb.html WebSphere Information Center: Explanation of DYNA system messages http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.messages.doc/doc/DYNA.html MustGather: Dynamic cache problems in V6.0 http://www-1.ibm.com/support/docview.wss?rs=180&uid=swg21193837
Note that problems with the data replication service can cause dynamic cacheproblems (see "Data replication service" on page 58).
by WebSphere Application Server. This includes any of the following symptoms: Unexpected transaction behavior Transaction time outs Unexpected transaction rollbacks Problems with transaction recovery Any errors or exceptions from the transaction manager Error or warning messages that have the prefixes WTRN or WLTC
enterprise information system connection problems, which are described in"Application cannot connect to an external system or resource" on page 63.
WebSphere Information Center: Tips for troubleshooting transactions http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.base.doc/info/aes/ae/rjta_prob0.html WebSphere Information Center: Transaction service exceptions http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.base.doc/info/aes/ae/rjta_except.html WebSphere Information Center: Explanation of WTRN system messages http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.messages.doc/doc/WTRN.html WebSphere Information Center: Explanation of WLTC system messages http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.messages.doc/doc/WLTC.html MustGather: Java Transaction Service (JTS) problems in V4.0, V5.0, V5.1 and V6.0 http://www-1.ibm.com/support/docview.wss?rs=180&uid=swg21153216
Application Server, including load balancing and failover of an EJB request or HTTP request. Potential problems include: Workload between cluster members not being balanced correctly A cluster member that is down does not failover Workload is not routed to a cluster member that is running An EJB client cannot reach any cluster members (a CORBA NO_IMPLEMENT error is issued) Any errors or exceptions relating to WLM Error or warning messages with the prefix WWLM
You can find problem determination strategies for problems with WLM of EJBrequests in the following resources: WebSphere Information Center: Workload management (WLM) for distributed platforms http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.nd.doc/info/ae/ae/crun_wlm.html WebSphere Information Center: Workload management component troubleshooting tips http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.nd.doc/info/ae/ae/rtrb_wlmcomp.html WebSphere Information Center: Workload is not getting distributed http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/topic/com.ibm.websphe re.nd.doc/info/ae/ae/rtrb_wlmprobs.html WebSphere Information Center: Workload management run-time exceptions http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.nd.doc/info/ae/ae/rrun_wlm_exceptions.html WebSphere Information Center: Clustering and workload management: Resources for learning http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.nd.doc/info/ae/ae/rrun_wlm_rlinks.html WebSphere Information Center: Explanation of WWLM system messages http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.messages.doc/doc/WWLM.html MustGather: Enterprise JavaBeans (EJB) workload management http://www-1.ibm.com/support/docview.wss?rs=180&uid=swg21052165
The high availability manager in WebSphere Application Server V6 eliminatessingle points of failure and provides failover for all applications and WebSphereApplication Server services. It manages the availability of your applications andservices. It is discussed in detail in "High availability and failover" on page 14.
WebSphere Application Server, including One process in your core group fails and its services are not started on another process as expected. Errors or exceptions from the high availability manager. Error or warning messages that begin with HMGR, CWRCB, or CWWCW.
You can find problem determination strategies for high availability manager problems in the following resources: WebSphere Information Center: Setting up a high availability environment http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.nd.doc/info/ae/ae/trun_ha_environment.html WebSphere Information Center: High availability manager http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.nd.doc/info/ae/ae/crun_ha_hamanager.html WebSphere Information Center: Troubleshooting high availability environment problems http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.nd.doc/info/ae/ae/rtrb_ha_env_trbl.html WebSphere Information Center: Learning about high availability http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.pmc.nd.doc/tasks/tjt0014_.html WebSphere Information Center: Explanation of HMGR system messages http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.messages.doc/doc/HMGR.html WebSphere Information Center: Explanation of CWRCB system messages http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.messages.doc/doc/CWRCB.html WebSphere Information Center: Explanation of CWWCW system messages http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.messages.doc/doc/CWWCW.html MustGather: High Availability and the High Availability (HA) Manager http://www-1.ibm.com/support/docview.wss?rs=180&uid=swg21201016
The data replication service transfers information between cluster members, enabling memory-to-memory session replication, dynamic cache, and stateful session bean failover.
WebSphere Application Server. Potential problems include: Unexpected data replication behavior (might appear as workload management, dynamic cache, or session management problems). Any errors or exceptions from the data replication service. Error or warning messages that begin with CWWDR.
You can find problem determination strategies for data replication serviceproblems in the following resources: WebSphere Information Center: Replicating data across application servers in a cluster http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.nd.doc/info/ae/ae/trun_drs_replication.html WebSphere Information Center: Replication http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.nd.doc/info/ae/ae/crun_drs_replication.html WebSphere Information Center: Explanation of CWWDR system messages http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.messages.doc/doc/CWWDR.html
Programming model extensions (PMEs) enhance application capability andperformance, and make programming and deployment faster and moreproductive.
WebSphere Information Center: Troubleshooting ActivitySessions http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.base.doc/info/aes/ass/tasks/tas_probd.html WebSphere Information Center: ActivitySession service: Resources for learning http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.base.doc/info/aes/ass/ref/ras_rlinks.html WebSphere Information Center: Explanation of WACS system messages http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.messages.doc/doc/WACS.html WebSphere Information Center: Application profiling http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.base.doc/info/aes/ae/welc6tech_appprof.html WebSphere Information Center: Explanation of APPR system messages http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.messages.doc/doc/APPR.html WebSphere Information Center: Asynchronous beans http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.base.doc/info/aes/ae/welc6tech_asb.html WebSphere Information Center: Explanation of ASYN system messages http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.messages.doc/doc/ASYN.html WebSphere Information Center: Object pools http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.base.doc/info/aes/ae/welc6tech_objp.html WebSphere Information Center: Object pools: Resources for learning http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.base.doc/info/aes/asyncbns/ref/rasb_objpoolsrlinks.html WebSphere Information Center: Explanation of OBPL system messages http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.messages.doc/doc/OBPL.html WebSphere Information Center: Scheduler service http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.base.doc/info/aes/ae/welc6tech_sch.html WebSphere Information Center: Explanation of SCHD system messages http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.messages.doc/doc/SCHD.html
WebSphere Information Center: Startup beans http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.base.doc/info/aes/ae/welc6tech_sub.html WebSphere Information Center: Explanation of STUP system messages http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.messages.doc/doc/STUP.html WebSphere Information Center: Work area http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.base.doc/info/aes/ae/welc6tech_wa.html WebSphere Information Center: Explanation of ACWA system messages http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.messages.doc/doc/ACWA.html
Symptom: You have any type of problem with character encoding or theinternationalization of your application (producing output in the language, timezone, currency, and cultural conventions for different regions or locales). Thiscan include: Problems displaying double byte characters for certain languages. Problems that only occur with internationalized applications. Error or warning messages that begin with I18N and LTXT.
Server V6: Web Container Problem Determination at: http://www.redbooks.ibm.com/redpapers/pdfs/redp4058.pdf
problems in the following resources: WebSphere Information Center: Internationalization service http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.base.doc/info/aes/ae/welc6tech_in.html WebSphere Information Center: Internationalization service errors http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.base.doc/info/aes/i18n/ref/rin_troubleshoot.html WebSphere Information Center: Internationalization: Resources for learning http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.base.doc/info/aes/ae/rin_resources.html
WebSphere Information Center: Explanation of I18N messages http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.messages.doc/doc/I18N.html WebSphere Information Center: Explanation of LTXT system messages http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.messages.doc/doc/LTXT.html MustGather: i18n (Internationalization)/Double Byte Character Set (DBCS) http://www-1.ibm.com/support/docview.wss?rs=180&uid=swg21141732
The transport channel service is a new feature in WebSphere Application Server V6. It manages client connections and I/O processing for HTTP and JMS requests based on the new non-blocking I/O features in Java 1.4.
WebSphere Information Center: Explanation of HTPC system messages http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.messages.doc/doc/HTPC.html WebSphere Information Center: Explanation of SSLC system messages http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.messages.doc/doc/SSLC.html WebSphere Information Center: Explanation of TCPC system messages http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.messages.doc/doc/TCPC.html WebSphere Information Center: Explanation of WSSC system messages http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.messages.doc/doc/WSSC.html WebSphere Information Center: Explanation of XMEM system messages http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.messages.doc/doc/XMEM.html
These problems include: JCA connection manager Database connections Messaging JNDI naming ORB Web services
Symptom: You have any problem connecting to an external resource through theJava Connector Architecture (JCA) connection manager. This includes datasources (which are used to obtain JDBC™ database connections), JMSconnection factories, and connections to enterprise information systems with aninstalled JCA resource adapter. The JCA connection manager pools andmanages connections to these systems.
issues, problems establishing a connection, problems with activationspecifications that get messages from your back-end systems. JCA connectionsymptoms are observed through WebSphere error messages with prefixes DSRA,WSCL, J2CA, WTRN, CONM, SQLException, or database error codes.
can appear as one or more of the following initial symptoms:
A connection hangs or incorrectly returns data to application An application connect to or access a database or EIS
problems, see WebSphere Application Server V6: JCA Connection Problem Determination at: http://www.redbooks.ibm.com/redpapers/pdfs/redp4080.pdf
problems, see WebSphere Application Server V6: JCA Connection Problem Determination at: http://www.redbooks.ibm.com/redpapers/pdfs/redp4080.pdf
CWSIA, CWSIB, CWSIC, CWSID, CWSIE, CWSIF, CWSIH, CWSII, CWSIJ, CWSIK, CWSIL, CWSIM, CWSIN, CWSIO, CWSIP, CWSIQ, CWSIR, CWSIS, CWSIT, CWSIU, CWSIV, CWSIW, CWSIX, CWSIY, CWSIZ, CWSJA, CWSJB, CWSJC, CWSJD, CWSJO, CWSJQ, CWSJR, CWSJU, CWSJW, and CWSWS. Note that these messages contain ten characters instead of nine. Problems with the WebSphere MQ JMS provider or a generic JMS provider might include JMS connection factory or destination configuration issues, message listener service and message-driven bean problems, and WebSphere system messages that begin with WMSG.
WebSphere Application Server V6: Default Messaging Provider Problem Determination at: http://www.redbooks.ibm.com/redpapers/pdfs/redp4076.pdf
through a JNDI lookup. Indications of a JNDI problem include: JNDI naming error or exception in the SystemOut or SystemErr log. Problems using the dumpNameSpace tool. Error or warning messages that begin with NMSV.
WebSphere Information Center: Naming and directories: Resources for learning http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/topic/com.ibm.websphe re.base.doc/info/aes/ae/rnam_r4ln.html WebSphere Information Center: Explanation of NMSV system messages http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.messages.doc/doc/NMSV.html MustGather: Java Naming and Directory Interface (JNDI) and naming problems for all releases http://www-1.ibm.com/support/docview.wss?rs=180&uid=swg21143296
through the Object Request Broker (ORB). Indications of an ORB problem include: ORB errors or exceptions in SystemOut or SystemErr log. Most likely, these would be CORBA error codes (such as a CORBA COMM_FAILURE). Error or warning messages that begin with ORBX.
WebSphere Information Center: Explanation of ORBX system messages http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.messages.doc/doc/ORBX.html MustGather: Object Request Broker (ORB) for all releases http://www-1.ibm.com/support/docview.wss?rs=180&uid=swg21157005
services client to a remote Web service or when acting as a Web service that isaccessed by external clients. This would include: Web services errors and exceptions that appear in the SystemOut and SystemErr logs Unexpected Web services behavior Configuration issues Problems with Web services tooling, such as the Java2WSDL and WSDL2Java scripts Problems that occur when configuring Web services or Web services gateway instances with the service integration bus. Error or warning messages that begin with WSWS, SOAP, WSIF (for Web services), or CWWSG (for Web services gateway)
You can find problem determination strategies for Web services problems in thefollowing resources: WebSphere Information Center: Web services http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.base.doc/info/aes/ae/welc6tech_wbs.html WebSphere Information Center: Troubleshooting Web services http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.base.doc/info/aes/ae/twbs_troubleshootwbs.html WebSphere Information Center: Troubleshooting Web services command-line tools http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.base.doc/info/aes/ae/rwbs_trbcommand.html
WebSphere Information Center: Troubleshooting Web services compiled bindings http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.base.doc/info/aes/ae/rwbs_trbjavacompiler.html WebSphere Information Center: Troubleshooting the runtime for a Web services client http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.base.doc/info/aes/ae/rwbs_trbclientruntime.html WebSphere Information Center: Troubleshooting serialization and deserialization in Web services http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.base.doc/info/aes/ae/rwbs_trbserialize.html WebSphere Information Center: Troubleshooting the Web Services Invocation Framework http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.base.doc/info/aes/ae/twsf_trouble.html WebSphere Information Center: UDDI Registry troubleshooting http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.base.doc/info/aes/ae/twsu_probdet.html WebSphere Information Center: Universal Discovery, Description, and Integration, Web Service, and SOAP component troubleshooting tips http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.base.doc/info/aes/ae/rtrb_svsccomp.html WebSphere Information Center: Errors returned to a client sending a SOAP request http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.base.doc/info/aes/ae/rtrb_soapprobs.html WebSphere Information Center: Tracing Web services http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.base.doc/info/aes/ae/twbs_tracewbscomp.html WebSphere Information Center: Tracing SOAP messages with tcpmon http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.base.doc/info/aes/ae/twbs_tracewbs.html WebSphere Information Center: Frequently asked questions about Web services http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.base.doc/info/aes/ae/rwbs_faq.html
WebSphere Information Center: Web services: Resources for learning http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.base.doc/info/aes/ae/rwbs_resourceslearning2.html WebSphere Information Center: Explanation of WSWS system messages http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.messages.doc/doc/WSWS.html WebSphere Information Center: Explanation of SOAP system messages http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.messages.doc/doc/SOAP.html WebSphere Information Center: Explanation of WSIF system messages http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.messages.doc/doc/WSIF.html WebSphere Information Center: Explanation of CWWSG system messages http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.messages.doc/doc/CWWSG.html MustGather: Web Services Engine problems for all releases and editions of WebSphere Application Server V5.0.2, V5.1 and V6.0 http://www-1.ibm.com/support/docview.wss?rs=180&uid=swg21198363 MustGather: Problems with the Web services gateway http://www-1.ibm.com/support/docview.wss?rs=180&uid=swg21159216 MustGather: Problems with the Web Services Invocation Framework (WSIF) http://www-1.ibm.com/support/docview.wss?rs=180&uid=swg21159155 MustGather: Problems with UDDI http://www-1.ibm.com/support/docview.wss?rs=180&uid=swg21164764
This broad classification covers all problems with slow or degradingperformance.
WebSphere Information Center: Performance: Resources for learning http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com. ibm.websphere.base.doc/info/aes/ae/rprf_resourceslearning.html WebSphere Application Server Performance information http://www-306.ibm.com/software/webservers/appserv/was/performance.html WebSphere Application Server V6 Scalability and Performance Handbook, SG24-6392.
As you might have realized by now, many potential problems with WebSphere Application Server V6 can be resolved without the help of the WebSphere Application Server Support team. By carefully preparing for and preventing problems and by following your problem determination strategy, you have the ability to resolve most problems on your own. However, there might be instances where invoking the Support team will be necessary, especially if a WebSphere Application Server code defect is the cause of a problem. This section explains the IBM support process and how you can work with the Support team most effectively.
In order to get the most out of the IBM support process, it is important to understand how the support organization is structured. There are several groups of support personnel with whom you might work for each problem:
Front office (Problem entry) team If you report a problem by phone, the first person you speak with is a member of the front office (also called problem entry) team. The people on this team work with you in your national language. They are usually located in the same region of the world from where you are calling. The main goals of the front office team are to get a detailed problem description from you, confirm your contact information and availability, open a problem management record (PMR), and then route your PMR to the appropriate product support team. Front end (Level 1) team The next step in the process is for the front end team (called the Level 1 support team in the United States) to work on your PMR. If you open your PMR electronically, this would be the first team with whom you would work. This team also works with you in your national language, and they are usually located in the same region of the world as you. The people on the front end team in many countries have broad skills in many IBM software products, and they work PMRs for several products. The Level 1 team in the United States only works WebSphere Application Server PMRs. They have broad skills in WebSphere Application Server, and they support the entire product. The support analysts obtain more details about the problem. They are likely to request files or logs that aid in determining the cause of the problem. They then conduct an investigation into the problem and provide possible solutions. If they are unable to provide a solution to the problem, they escalate the PMR to the next level of support. Back end (Level 2) team Your PMR is escalated to the back end (Level 2) support team if it cannot be resolved by the front end or Level 1 team. The Level 2 team for WebSphere Application Server is located only in the United States, and they work only in English. The front end team asks you if you are able to work with the Level 2 team in English. If you would prefer to work in your own language, the front end team can translate the communication from the Level 2 team for you. The Level 2 team is composed of a group of specialized teams. Each team works only on certain areas of the product, and they develop expertise in those areas. They conduct a more extensive analysis of the problem and the files that you have sent, and they obtain more information about the problem if it is required. The Level 2 team might also ask for more detailed diagnostic data, such as traces for a specific component, to help in their investigation. After analyzing all of the information and conducting research, the Level 2 team either provides a solution to the problem or works with the next level of support if they believe that the problem is caused by a WebSphere
Application Server code defect. In that case, an authorized program analysis report (APAR) is opened. Change team (Level 3) If the Level 2 team's analysis shows that the problem is caused by a defect with WebSphere Application Server code, they send your PMR to the change team, also known as the Level 3 team. The change team is part of the WebSphere Application Server development team that is specifically focused on fixing defects. The members of the Level 3 team work only on one specific component of WebSphere Application Server. They have very deep knowledge of that particular component. The Level 3 team does not communicate directly with clients. Instead, they provide updates to the Level 2 team, and the Level 2 team communicates updates to the client.
Before you open a PMR, we recommend that you will follow all of the problem determination strategies discussed in this book. The most important thing to do is to research the problem. The WebSphere Application Server Support team has found that a large percentage of problems have already been reported by other clients. In these cases, you can save a great deal of time and effort by reviewing IBM documentation to make sure that the problem is not a known problem before you open a PMR. You can review the list of Web sites in "Investigate and research the problem" on page 27 to conduct your research.
We first mentioned MustGather documents in "Diagnostic data collection" on page 21. These documents are useful guides when developing your diagnostic data collection plan, and they are also important when you open a PMR with the WebSphere Application Server Support team. The support team needs certain diagnostic data to investigate each type of WebSphere Application Server problem. In "Classify the problem and determine the root cause" on page 31, we listed each problem classification, and we included the links to the MustGather documents for each classification. You can review the specific document for the type of problem that you are experiencing, or you can start with the general MustGather document for all types of problems, IBM - MustGather: Read first for all WebSphere Application Server products, at: http://www-1.ibm.com/support/docview.wss?rs=180&uid=swg21145599
1.5.4 Determine the severity When you open a PMR, you provide a severity level for the PMR. The severity levels are in the range of 1 to 4, with 1 being the most severe and 4 being the least severe. The severity level reflects the business impact and, therefore, the urgency of the problem. Although it is tempting to open all PMRs at the Severity 1 level, it is very important to determine an accurate severity level for the situation. IBM support prioritizes PMRs according to the severity level. When you have a true Severity 1 issue, you do not want the support team to have to prioritize it at the same level as other issues that are not as critical. Table 1-2 provides the official descriptions of each severity level from the IBM Software Support Handbook.
Note: Although IBM support analysts might sometimes ask you if a severity level can be changed, it is ultimately your decision (or the decision of others in your company) as to what the severity level is.
Severity level Description
to use the program, resulting in a critical impact on operations. This condition requires an immediate solution.
program is usable but is severely limited.
usable with less significant features (not critical to operations) unavailable.
causes little impact on operations, or a reasonable circumvention to the problem has been implemented.
Severity 1 PMRs are typically situations where the availability of your production environment is affected. In a case where your production system is down, the support team's first priority is to help you restore your production functionality, as discussed in "Revert to safe conditions" on page 24. It can be any situation where there is a high level of business impact. For example, it could be a problem that causes a loss of revenue or productivity for your organization. For Severity 1
PMRs, there is no work around or solution that is currently available. The support team works around the clock on your PMR, as long as they are able to contact you at any time of day, until the problem is resolved. Severity 2 PMRs are the most common levels of PMRs. In this case, the problem has a significant impact, but the application is still running in your production environment. It can also include situations where your application is not yet in production. These include situations where you have a deadline for resolving the problem, and situations where the problem delays when your application goes into production. Severity 2 PMRs are considered severe and are prioritized highly by the support team but the problems are not considered as critical as Severity 1 problems. The support team works with you during the normal business hours in your time zone. Severity 3 PMRs indicate that a problem is not severely impacting your business. Almost all Severity 3 problems occur in your test environment and do not affect production. If it does impact production, it has only a minor impact, and you might have a work around for the problem. Although you expect the problem to be resolved in a timely manner, you do not have a deadline for resolving the problem and the problem is not delaying the production date for your application. The support team works with you during the normal business hours in your time zone. Severity 4 PMRs indicate that a problem has almost no business impact. This includes technical questions and requests to update documentation. It can also include problems that occur only in your test environment where a work around has already been provided. The severity is sometimes lowered to Severity 4 when a solution is provided, but you need some time to test the solution before closing the PMR. It is expected that Severity 4 PMRs are not impacting your production environment. You do not have a deadline for resolving the problem and you are able to wait a longer period of time to receive a solution. The support team will work with you during the normal business hours in your time zone and updates by the support team might not be frequent.
Now that you are familiar with the IBM support structure, you have researched the problem, collected the MustGather data, and determined the severity of the problem, it is time to open a PMR.
You can currently open a PMR in two ways, electronically or by phone. We havefound that opening PMRs electronically is more convenient for most clients. Werecommend that you use that method. However, phone support is still available.
To open a PMR electronically on the Web, you can use the Electronic ServiceRequest (ESR) problem submission tool at:http://www-306.ibm.com/software/support/probsub.html
Your other option is to open a PMR by phone. The phone number that you call toopen a PMR varies by country and changes from time to time. To find the phonenumber for your country, check this link from the IBM Software SupportHandbook:http://techsupport.services.ibm.com/guides/contacts.html
1.5.6 Send data to IBM In the course of working a PMR, it might be necessary to send many different types of data to the WebSphere Application Server Support team. As discussed in "Create a PMR" on page 74, using the ESR tool to attach files to your PMR makes this process easy. However, if you do not use the ESR tool, you can still send data through e-mail or FTP. As a general rule, you should use e-mail if the total size of the data is less than 10 MB, and you should use FTP if the data is more than 10 MB in size. Also, when you FTP data, your PMR is updated automatically so that the support team is aware that you sent the data and they can begin analyzing it right away. However, if you send an e-mail, be sure to either update your PMR electronically or call the IBM support phone number for your country and ask to have the PMR updated, so that the support team knows that you sent the data.
weblev2@us.ibm.com. This is a shared e-mail address. The entire WebSphere Application Server Level 1 and Level 2 teams have access to this e-mail address. If you only send an e-mail to a specific support analyst, only that analyst has access to your e-mail. This can be disadvantageous, especially if the analyst is out of the office or if you need help outside of normal business hours.
This paper contains information about the diagnostic data that is available in WebSphere Application Server V6. It contains information about the location of the data, how it is collected, and configuration options.
JVM logs (SystemOut and SystemErr) Tracing Collector tool First Failure Data Capture (FFDC) Process (native) logs Service log (activity.log) Installation logs IBM HTTP Server and plug-in logs and traces System management logs WebSphere Rapid Deployment logs
2.1 JVM logs (SystemOut and SystemErr) SystemOut and SystemErr logs are created for every WebSphere Application Server process (application server, cluster member, node agent, and deployment manager). These logs are known as JVM logs. The System.Out and System.Err streams for each JVM are redirected to the SystemOut and SystemErr logs. WebSphere Application Server writes to these logs. Your applications can also write to them by using the print(), println(), and printStackTrace() methods.
/profiles//logs/
In the Configuration tab, you can edit the following: The file name (and the directory) of the SystemOut and SystemErr logs The file formatting We recommend that you leave this at the default value of Basic to make the logs easier to read. Log file rotation The SystemOut and SystemErr logs are self-managing. They write to the specified file until either the maximum file size or a certain time is reached. When that happens, the current log file is renamed as the current file name plus the current time stamp. Then a new SystemOut or SystemErr file is created for further logging. The older log files are called historical log files. For example, after this occurs you might have the following SystemOut files in the /profiles//logs/ directory: - SystemOut.log, the current log file - SystemOut_05.06.07_10.28.48.log, the historical log file Depending on your needs, you can choose to have the log files rotate (roll over) when they reach a specified size, a certain time interval, or both. If you choose a time, we recommend that you specify 24 hours as the Repeat Time. You can also set the Start Time to specify the time at which the logs rotate. If you specify a file size, we recommend that you increase the Maximum Size above its default of 1 MB. You will want to coordinate the value of Maximum Size with the Maximum Number of Historical Log Files, based on the available disk space on your system. With either method, make sure that the amount of log data that is saved is enough so that the relevant log data is there when you identify that a problem has occurred. Maximum Number of Historical Log Files The value that is entered here is the number of historical log files that are kept. If the value is reached and another historical log file needs to be created, the oldest one is removed from your system. Installed Application Output These properties affect how print and println statements from your applications are output. There are two options: - Show application print statements. This is enabled by default. If you deselect it, application print and println statements are not logged to the SystemOut and SystemErr log files. - Format print statements. This is also enabled by default. You can deselect it if you do not want your application print and println statements to be
formatted similar to WebSphere Application Server messages in the log files.
All of these properties can be changed for both the SystemOut and the SystemErr logs. You can choose to use the same properties for both logs, which we recommend, or use different properties for them.
The entries in the output of the SystemOut.log are in the following format: [7/12/05 14:46:00:264 EDT] 0000001a ApplicationMg A WSVR0221I: Application started: adminconsole
Time stamp In the example, the time stamp is [7/12/05 14:46:00:264 EDT]. The time stamp is formatted using the locale of the process where it is formatted. It includes a fully qualified date (for example MM/DD/YY), 24-hour time with millisecond precision, and a time zone. Thread ID In the example, the thread ID is 0000001a. The thread ID is an eight-character hexadecimal value that is generated from the hash code of the thread that issued the message. Short name In the example, the short name is ApplicationMg. The short name is the abbreviated name of the component that issued the message. This name is typically the class name of a WebSphere Application Server component and would be some other identifier for applications.
Event type In the example, the event type is A. The event type is a one character field that indicates the type of the message. The possible values are: - F - fatal message - E - error message - W - warning message - A - audit message - I - informational message - C - configuration message - D - detail message - O - message that was written directly to System.out by an application or internal components - R - message that was written directly to System.err by the user application or internal components - Z - a placeholder to indicate that the type was not recognized Message Identifier In the example, the message identifier is WSVR0221I. The message identifier is a string that is nine characters in length and is in the form CCCC1234X. The first four characters (CCCC) indicate the WebSphere Application Server component that issued the message. The next four characters (1234) indicate the specific message that the component is issuing. The last character (X) indicates the severity of the message. Its value is either I (informational), W (warning), or E (error). You can find descriptions of all WebSphere Application Server message identifiers in the WebSphere Information Center item Troubleshooter reference: Messages at: http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/ com.ibm.websphere.express.doc/info/exp/ae/welc_ref_trb_msg.html Message In the example, the message is Application started: adminconsole. The message is the data that is logged to the SystemOut.log by the component. It is meant to provide useful output for informational purposes, debugging, and troubleshooting.
2.2 Tracing Trace logs can also be configured in a manner similar to the JVM logs. Traces must be explicitly enabled. They are disabled by default. Trace output gives very detailed information about the execution of WebSphere Application Server code. It provides time stamps, details about which WebSphere methods were called, and special diagnostic data that is included to make troubleshooting easier.
1. Select Troubleshooting → Logs and Trace. 2. Select the process whose trace logs you want to configure. 3. Click Diagnostic Trace. The General Properties window opens, as shown in Figure 2-2.
Trace files cannot be rolled over based on time. You must specify a MaximumFile Size in conjunction with the Maximum Number of Historical Files. You shouldset these values appropriately, depending on how long it might take to reproducethe problem with trace enabled and how much disk space is available. Withthese properties set, the trace files roll over in the same manner as the JVM logs.You can also specify a File Name and specify a directory for the trace files.
As with the JVM logs, we strongly recommend selecting Basic (the default value)for the Trace Output Format. This makes the trace easier to read, and it is thepreferred format of the WebSphere Application Server support team.
Figure 2-3 Changing the log detail level in the administrative console
Groups are predefined sets of packages and classes that are useful for troubleshooting a particular component.
The MustGather documents for WebSphere Application Server componentsdiscuss which specific log detail level to set for different types of problems. It is agood idea to record the log detail level for different types of problems in yourdiagnostic data collection plan. When setting the log detail level, you should setthe level to all in almost all cases. You should also include *=info in thebeginning of the log detail level so that informational logging is enabled forcomponents that are not being traced. For example, in order to enable a trace forthe J2C connection manager component, you would set the log detail level to:*=info:WAS.j2c=all
Time stamp In the example, the time stamp is [7/12/05 16:13:10:379 EDT]. The time stamp is formatted using the locale of the process where it is formatted. It includes a fully qualified date (for example MM/DD/YY), 24-hour time with millisecond precision, and a time zone. Thread ID In the example, the thread ID is 00000032. The thread ID is an eight-character hexadecimal value generated from the hash code of the thread that issued the trace event.
Short name In the example, the short name is DSConfigurati. The short name is the abbreviated name of the component that issued the trace event. This is typically the class name of a WebSphere Application Server component, and would be some other identifier for applications. Event type In the example, the event type is a greater than symbol (>). The event type is a one character field that indicates the type of the trace event. The possible values are: - > - indicates the entry of the specified method name - < - indicates the exit of the specified method name - 1 - a trace entry of type fine or event - 2 - a trace entry of type finer - 3 - a trace entry of type finest, debug, or dump - Z - a placeholder to indicate that the trace type was not recognized The example indicates that the getPooledConnection method is entered. Class name The class name is an optional part of the trace entry. It indicates the class that generated the trace event. In the example, the class name does not appear. Method name The method name is another optional part of the trace entry. It indicates the method that generated the trace event. In the example, the method name is getPooledConnection. Text message In the example, the message is Entry.
2.3 Collector tool The WebSphere Application Server collector tool is a script that can be found in the /bin directory (collector.bat or collector.sh). Running the script produces a Java archive (jar) file that contains all of the logs and XML configuration files from your WebSphere Application Server installation, as well as operating system information, Java information, and data on whether the software prerequisites were met and their levels.
WebSphere Application Server V6 includes a feature called First Failure Data Capture (FFDC). The FFDC feature runs in the background and collects events and errors that occur during WebSphere Application Server runtime. The information that it collects are written to log files in the /profiles//logs/ffdc directory.
FFDC does not affect the performance of WebSphere Application Server and should not be disabled. The FFDC logs will not, most likely, be useful in your
problem determination efforts. However, they might be useful to the WebSphere Application Server support team if you open a PMR.
directory. The only file that you should modify is the ffdcRun.properties file. You can add the ExceptionFileMaximumAge property to the file. This property specifies the number of days that an FFDC log remains in the /profiles//logs/ffdc directory before it is deleted. As part of your diagnostic data collection plan, you might want to modify the ExceptionFileMaximumAge property to ensure that the FFDC files remain on your system for a certain time period. You should not modify any other properties unless you are asked to do so by the WebSphere Application Server support team.
The following logs are not always useful for problem determination, but you might find that on occasion they will be required.
Native code running in a WebSphere Application Server process can write data to the process logs (also called native logs). Native code is non-Java code, typically found in files with .dll, .exe, and .so extensions. The process logs are named native_stdout.log and native_stderr.log. They are located in the /profiles//logs/ directory.
You can find more information about the process logs in the WebSphere Information Center item Process logs at: http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com.ibm. websphere.base.doc/info/aes/ae/ctrb_stdlogs.html
The service log is more commonly known as the activity.log and is found in the /profiles//logs directory. There is only one activity.log for each node. WebSphere Application Server runtime events are logged to the activity.log. It is written in binary format, so it cannot be viewed in a text editor. The main purpose of the activity.log is that it can be viewed with the Log Analyzer tool, is a graphical user interface that displays the events from the activity.log and uses a symptom database to analyze the events and diagnose problems.
It is also possible to view the events in the activity.log outside of the Log Analyzer by using the showlog script in the /bin directory.
1. Select Troubleshooting → Logs and Trace. 2. Select the WebSphere Application Server process. 3. Select IBM Service Logs. You can select whether to enable or disable the activity.log, choose the directory location and file name, set the maximum file size, and select what types of messages will be logged.
2.5.3 Installation logs When you have a problem installing WebSphere Application Server V6, you might need to view the following logs to determine the failure causes. /logs/log.txt This log file records the installation status /profiles//logs/pctlog.txt This log file records the profile creation status /profiles//logs/ivtClient.log This log file records the events of install verification test
When you have a problem relating to the IBM HTTP Server or the Web server plug-in, you might need to view the logs or enable a trace. This section discusses the details about these logs and traces.
The IBM HTTP Server writes two log files: an access log that contains details of all accesses to the Web server and an error log that contains details of any errors. The default location of the logs is as follows: Windows - Access log: \logs\access.log - Error log: \logs\error.log UNIX® - Access log: /logs/access_log - Error log: /logs/error_log
The plug-in also writes its own log, which you can find in the Web server plug-in install directory path. The log file that you are looking for is under another directory structure, named for the logical Web server as defined in the WebSphere configuration.
You can find the location of the log file by first looking at the Web server configuration. This refers to the plug-in configuration file as shown in Example 2-1 on page 91. The plug-in configuration file then tells you where the
log file is as shown in Example 2-1. This example also shows you where you set the amount of detail that is logged.
The default setting for LogLevel is Error, but you can set it to Trace to collect significantly more information. Should you need to raise this problem with IBM Support, they will request a plug-in trace.
To get an effective trace, you need to enable as much logging as possible on the Web server. For example, you can set the logging level to capture verbose output in the IBM HTTP Server by modifying the LogLevel directive in the configuration file as shown: LogLevel debug
You need to restart the IBM HTTP Server for this to take effect.
You do not need to restart the IBM HTTP Server for this change to take effect.
Tip: The plug-in trace generates significant amounts of data. Make your test as specific as possible, and run it in isolation to reduce the number of lines generated.
In rare cases, you might need to use a network protocol analyzer that allows you to capture an iptrace. This tool can help you to determine where the problem lies. WebSphere Application Server does not supply such a tool. However, there are third-party tools available (for example, Ethereal from http://www.ethereal.com/).
2.7 System management logs When you have a system management problem, you might need to view certain logs or enable a trace. This section discusses the details about these logs and traces.
Messages from wsadmin are written to the wsadmin.traceout log file: /profiles//logs/wsadmin.traceout
You can also increase the amount of data that is logged to this file by tracing the wsadmin utility. To do so, update the following file: /properties/wsadmin.properties
com.ibm.ws.scripting.traceString=com.ibm.*=all=enabled
Note that the information that is logged is of limited use because wsadmin calls MBeans in the application server that is running the administrative console application. So, you usually need to trace that application server as well.
You can manage WebSphere Application Server services using the supplied management scripts. For example, each WebSphere Application Server installation has a script to start an application server, a script to stop an application server, and a script to show you the status of all application servers defined in a profile. Each of these scripts writes its own log file into the server's logs directory. For example, the stopServer script writes stopServer.log into the logs directory: /profiles//logs//stopServer.log
The profile creation and management tool wasprofile writes messages to the profile independent logs directory, that is: /logs/wasprofile/.log
The Java graphical interface that is used to create a profile simply calls the wasprofile command after collecting the information needed. By default, it does not write a log, but you can pass it a log parameter as shown: pctWindows -is:log c:\temp\pct.log
The WebSphere Rapid Deployment tool works on a directory that you create and pass to WebSphere Rapid Deployment in the WORKSPACE environment variable. It logs Eclipse messages into two separate files within this directory: /.metadata/.log /project/.metadata/.log
Rapid Deployment calls MBeans on the application server. The application server logs can help you resolve a problem with WebSphere Rapid Deployment. There is no way to trace the WebSphere Rapid Deployment utility. However, you can trace the application server as described in "Tracing" on page 82.
Table 2-1 on page 94 shows a summary of the WebSphere logs.
represents the installation root for WebSphere Application Server, for example: c:\WebSphere\Appserver represents the root directory for a specific WebSphere Application Server profile, for example: /profiles/ represents the installation directory for the IBM HTTP Server, for example: c:\IBM HTTP Server
Table 2-1 Log file summary Tasks Logs Format / tools
Installation tasks
ihsv6_install.log gskitInstall.log
installation: DefaultApplication defaultapp_config.log ivtApp defaultapp_deploy.log query ivt_config.log PlantsByWebSphere query_config.log SamplesGallery samples_config.log samples_install.log
installation: filetransfer filetransfer_config.log filetransferSecured mejb.log ManagementEJB scheduler.cal_config.log SchedulerCalenders webui_config.log adminconsole
Profile tasks
/logs/waspro XML file/wasprofile_create_.l og.
SystemOut.log1 JVM System.out and System.err SystemErr.log1 administrative console: streams Troubleshooting → Logs and Trace →→JVM Logs 1 Configurable 2 can represent the location of the profile for an application server, node agent, or deployment manager. If the profile is for a node agent,is "nodeagent". If the profile is for a deployment manager,is "dmgr"
Tasks Logs Format / tools
2 /logs//: to view with a text editor) native_stderr.log1 native_stdout.log1 administrative console: Troubleshooting → Logs and Trace →→Process Logs
Operational tasks
SystemOut.log1 SystemErr.log1
-trace, see /logs//: startServer.log stopServer.log
manager SystemOut.log1 SystemErr.log1
/logs/dmgr/: startServer.log stopServer.log
/logs/nodeagent/ : startServer.log stopServer.log1 Configurable2 can represent the location of the profile for an application server, node agent, ordeployment manager. If the profile is for a node agent,is "nodeagent". If the profile is for adeployment manager,is "dmgr"
Tasks Logs Format / tools
that action is taken on each server. The logging is the same as though you started or stopped each server.
Configuration tasks
addNode.log runAddNode.log 1 Configurable 2 can represent the location of the profile for an application server, node agent, or deployment manager. If the profile is for a node agent,is "nodeagent". If the profile is for a deployment manager,is "dmgr"
determination This paper discusses how to diagnose issues associated with WebSphere Application Server V6 installation. It covers actions and resolutions related to the following installation issues: Launchpad or installation wizard will not start Installation failure or hang Profile creation failure Install Verification Test (IVT) failures
The installation routine for other products, such as IBM HTTP Server and Web Server plug-ins, is separate from the application server products. This paper only covers issues that are related to WebSphere Application Server V6 installation.
by reading Approach to Problem Determination in WebSphere Application Server V6 at http://www.redbooks.ibm.com/redpapers/pdfs/redp4073.pdf.
3.1 Introduction The installation process for WebSphere Application Server V6 has changed from previous versions. The main change is that only one copy of the core product files is installed. Profiles are used to define multiple server runtime environments.
The entry point into WebSphere Application Server installation is the launchpad (Figure 3-1). From here, you can choose to launch the installation wizard for WebSphere Application Server.
Figure 3-2 provides an overview of the installation events and common issuesthat are associated with installation.
LaunchPad Displays tasks to perform
Common Problems No supported browser, Browser configuration Insufficient disk-space
Common problems Low resources like disk space, virtual memory Problems with JVM
Create a new application server, deployment manager, custom profile
Verify the installation
Note: In Figure 3-2 on page 99, the Installation wizard and Profile creation wizard are shown as different entities. This is what you will see if you are installing the Network Deployment version. In a WebSphere Application Server base or Express installation, profile creation for the default profile occurs in the Installation wizard itself.
You begin the problem determination process by evaluating the high-level symptoms to determine if one of these symptoms describes your problem. If it does not, you need to collect the appropriate data that is required to diagnose the problem.
If you do not find your symptom listed here, go to "The next step" on page 113.
hangs, you might need to uninstall WebSphere Application Server manually before you retry the process. The WebSphere Information Center has detailed, platform-specific instructions for uninstalling at: http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/ com.ibm.websphere.nd.doc/info/ae/ae/tins_uninstman.html
3.2.1 Symptom: Launchpad or installation wizard will not start or fails This symptom covers the following situations: Executing the launchpad.bat command in windows or the launchpad.sh script in a UNIX environment does not start the launchpad. Selecting Launch the installation wizard for WebSphere Application Server from the launchpad fails to start the installation wizard or the wizard exits with or without an error message.
Possible causes for type of problem include Web browser requirements and disk space or permission requirements.
If the installation wizard starts and runs for a while but appears to hang, go to "Symptom: Installation wizard hangs" on page 103.
If this file does not exist, then run the installer from a command window using -log option to create a log of all events. - UNIX ./install -log !logfile @ALL - Windows install -log !logfile @ALL Where logfile is a fully qualified file name for writing the log events.
As you collect the log files, copy them to a place where you can view them. If you need to recreate the problem in order to collect this documentation, be sure to prepare (schedule off-shift hours, review prerequisites, and so forth).
Problems starting the launchpad or installation wizard can usually be traced back to missing prerequisite system or application levels. The first step in diagnosing this problem is to ensure that you have the proper prerequisites for installation. You can find this information at: http://www-306.ibm.com/software/webservers/appserv/doc/latest/prereq.html
If the problem you are having is in starting the launchpad, look at the IBM_WebSphere_LaunchPad_log.txt log. This log normally contains minimal messages that indicate when the launchpad was run. In the event of an error during launch, it might also contain error messages. If the launchpad does not start at all, then no log file is created.
Installation events are installed in log.txt. If the installation wizard starts but fails during the installation process, look in log.txt for messages that contain INSTCONFSUCCESS, INSTCONFPARTIALSUCCESS, or INSTCONFFAILED, which might indicate the current status of the installation.
should be error or warning messages preceding them (for example, messages that indicate problems with resources, such as not enough disk space, exceptions in the JVM, segmentation faults, and so forth).
An error that resembles the following indicates a problem with disk space: A suitable JVM could not be found. Please run the program again using the option -is:javahome < JAVA HOME DIR> No space left on device
This error indicates there is not enough free space for the installer to run on. You can get this error even if enough space exists where you plan to install WebSphere Application Server (for example, drive D: or /usr) .
UNIX has enough free space for the installer to run. You need to check the install document to determine the exact amount of temporary disk space required, typically a minimum of 100 MB.
Another option is to use -is:tempdir with the installation wizard, where tempdir is the location of a temporary directory on a partition with enough free space.
If you meet the installation prerequisites and you still have not resolved your problem, go to "The next step" on page 113 for information about how to search for known problems with installation.
During installation, a progress indicator is displayed to show you how far the install has progressed. If there is no change in the progress indicator for a very log time, the installation process could be hung.
installation, the installation wizard creates a default server profile. The installer first starts the file copy process and then the profile creation process. In WebSphere Application Server Network Deployment, this is a two-step process. The install completes, then you are asked if you want to create a profile.
Data to collect If the installation hangs, check the following: /logs/log.txt If the installer fails at a very early stage, then this log file might not be created or it might be exist in the system temporary area, which is %TEMP%\log.txt in Windows or /tmp/log.txt in UNIX. To ensure that you get a full listing of messages in this log, run the installer from a command window using the -log option to create a log of all events. - UNIX ./install -log !logfile @ALL - Windows install -log !logfile @ALL Where logfile is a fully qualified file name for writing the log events. /profiles//logs/pctLog.txt This log file is created only when the profile creation wizard is executed. This log is not created when using the wasprofile command directly or during installation of the product.
If you think the install process is hung, check the log.txt file periodically to see if progress is being made. Messages such as INSTCONFSUCCESS, INSTCONFPARTIALSUCCESS, or INSTCONFFAILED in log.txt indicate the current status of the installation. If you see the INSTCONFPARTIALSUCCESS or INSTCONFFAILED messages, then there should be error or warning messages preceding them.
Also, check other system activities, such as CPU utilization, hard disk usage, or any network activity (if installing remotely), to make sure there are not external factors that are affecting the install.
If the installation does appear to be hung, look for the last recorded message in the log file. This message gives you an idea of what the installer was doing before it hung.
Determine if errors occurred in the file copy process Look in log.txt for an entry such as the following: (), Install, com.ibm.ws.install.ni.ismp.actions.ISMPConfigManagerLaunchAction, msg1, INSTCONFSUCCESS: Post-installation configuration is successful
If you see this message, the file copy process has completed successfully. If not, inspect the messages in the log for an indication of the error.
If the file copy was completed successfully, any error messages after this indicate problems in profile creation or with other subsequent steps, including sample application and administrative console application deployment. Look for the following entry in the log (log.txt for WebSphere Application Server and WebSphere Application Server - Express, pctLog.txt for WebSphere Application Server Network Deployment): (), Install, com.ibm.ws.install.ni.ismp.actions.ISMPWSProfileLaunchAction, msg1, INSTCONFSUCCESS: Post-installation configuration is successful
If you do not see this message, then there are problems with profile creation. To diagnose problems with profile creation, see "Symptom: Profile creation failure" on page 105.
On the other hand, if you see this message, the profile creation was successful, and it is time to re-evaluate your symptoms. Go to "The next step" on page 113.
Profiles allow you to define multiple runtime environments, each with its own administrative interface while sharing the same code base. Problems with profile creation might be due to long directory paths, file permissions, problems with the host name, and so forth.
WebSphere Application Server or WebSphere Application Server - Express installation. The Network Deployment installation wizard gives you the option of creating a profile. Profiles can also be created at any time after installation.
Data to collect If the profile creation fails, check the following: /logs/wasprofile/wasprofile_create_.log This log file is created when the installation phase has completed the file copy process and starts creating a default profile. This log file is also created whenever the profile creation wizard or wasprofile command is executed. This log file traces all the events that occur during profile creation. It is an XML log file and is best viewed by a viewer that can format XML, for example a Web browser or WordPad in Windows. The entries in this log file consist ofentries. A sample log entry that indicates an error would look something similar to the following:2005-07-19T14:18:53 1121762933500 2984 com.ibm.ws.install.configmanager.ConfigManager WARNING com.ibm.ws.install.configmanager.ConfigManager executeAllActionsFound 11 Fatal configuration action failed: com.ibm.ws.install.configmanager.actionengine.ANTAction -C:\IBM\WAS\profileTemplates\managed\actions\ executeManagedProfileSetup.ant /logs/log.txt
The following log files are created during profile creation. They are located in the /profiles//logs directory. Each log might or might not exist depending on the type of profile that is created. pctLog.txt Created only when the profile creation wizard is executed. This log is not created when using the wasprofile command directly or during installation of the product. amjrte_config.log Tivoli Access Manager configuration log for its Java Runtime Environment. collect_metadata.log Collects metadata information about managed objects in the system to evaluate and prevent potential installation conflicts.
createDefaultServer.log A log from wsadmin recording the creation of the server1 process in the default profile. createshortcutforprofile.log Windows tool log for creating menu entries and shortcuts. defaultapp_config.log JACL script log from configuring default application resources. Service.log Start and stop events for server1. Application installation and configuration logs for system and sample applications: - filetransfer_config.log for the file transfer application - ivt_config.log for the ivtAPP application - mejb_config.log for the ManagementEJB application - hamanager_config.log for the high availability application - query_config.log for the Query application - samples_config.log for the PlantsByWebSphere sample application - samples_install.log for the SamplesGallery and PlantsByWebSphere sample applications - scheduler.cal_config.log for the SchedulerCalendars application - webui_config.log for the administrative console application - defaultapp_deploy.log for the DefaultApplication application SIBDefineChains.log Creation log for service integration bus endpoints, inbound channels and channel chains, outbound thread pool, and outbound channel and channel chains. SIBDeployRA.log Deployment log for the service integration bus function. winservice_config.log Service log for the Windows service created for server1. addNode.log Log for federating a node to a cell (Custom profile).
Messages such as INSTCONFSUCCESS, INSTCONFPARTIALSUCCESS, orINSTCONFFAILED in log.txt or pctLog.txt indicate the current status of the
installation. If you see the INSTCONFPARTIALSUCCESS or INSTCONFFAILED messages, then there should be error or warning messages preceding them.
(), Install, com.ibm.ws.install.ni.ismp.actions.ISMPWSProfileLaunchAction, err, INSTCONFFAILED: Cannot complete required configuration actions after installation. The configuration failed. The installation is not successful. Refer to \install_root\logs\wasprofile\wasprofile_create_profilename.log for more details
determine what task was being performed when the profile creation failed. Most tasks, such as system or sample application installation, are logged to individual log files in the /profiles//logs directory. If you can determine which task the profile creation was doing, collect the file for that task. For example, if you had problems creating a custom node then look at the log file /profiles//logs/addNode.log for any errors.
The IVT looks at the profile configuration for the server and looks for a server running on the server port number. Note that if a server is up and running on that port, the IVT runs against that server, even if it is not the one you just installed. If no server is running on that port, the IVT attempts to start the server. When the server has started successfully, the IVT accesses the server and runs various tests, including servlet engine verification, JSP verification, EJB verification, and so forth.
The option to run the IVT is displayed in the First Steps console after the installation completes and after every profile creation, with the exception of a custom profile creation because no server is actually created.
You should run the IVT before making any configuration changes to WebSphere Application Server. This acts as a checkpoint to see if any problems exist as the result of the installation. If the IVT runs clean and problems show up later, they
are most likely due to configuration changes done after the installation. It isrecommended that all instances of WebSphere Application Server be stoppedbefore running the IVT.
A failure in the IVT is usually because the application server fails to start.Common reasons for this include port conflicts, not enough memory, and soforth.
The first file to look at in case IVT has failed is ivtClient.log. Messages with IVTLand ADMU prefixes provide information about what the IVT application is doingand the status of each action.
ADMUXXXXX: Server servername open for e-business; process id is xxxx
If you find this message, then the server has started successfully, and the IVTfailed while executing one of the tests. Examine the error messages inivtClient.log to locate the failing process.
If the server does not start, look at the Server Port number is: entry. This entrycontains the server port number of the profile instance. Make sure that this port isnot in use (see "Application server startup problems" on page 111).
If the port does not seem to be the problem, look at startServer.log, SystemErr.log, and SystemOut.log for information related to server startup, such as the following: Look for any error messages starting with WSVR (server runtime) or ADMU (management utility) in the startServer.log and SystemOut.log. Look for error messages in SystemErr.log. These messages will start with: [Date and time stamp] 0000000a SystemErr
If you cannot identify the problem, go to "The next step" on page 113.
Your analysis of the data that you gathered will most likely lead you to one of the following areas. If not, see "The next step" on page 113.
If you are having problems in starting the launchpad, it is possible that you do not have a supported Web browser installed or configured properly. If this is the case: Verify that the latest version of a supported Web browser is installed. Mozilla and Internet Explorer are supported for configuration and installation activities. On UNIX platforms, ensure that the location of the supported browser is exported. For example if the Mozilla executable is located in the /usr/bin directory, then export its location as follows: export BROWSER=/usr/bin/mozilla Ensure that JavaScript™ is enabled in the browser options or preferences. For example: - In Mozilla, select Edit → Preferences → Advanced → Scripts & Plugins - Enable JavaScript for: Navigator, Allow Scripts to: (select all boxes) - In Internet Explorer, select Tools → Internet Options → Security → Custom Level for Internet Scripting → Active scripting → Enable
The following link gives specific information about system readiness for installing WebSphere Application Server on RedHat Linux Enterprise Edition 4: http://www-1.ibm.com/support/docview.wss?uid=swg21201306
3.3.2 Application server startup problems During profile creation, it is important that the port numbers that are specified for each of the WebSphere Application Server services are unique. The Profile creation wizard, by default, tries to define port numbers that do not conflict with other profiles in the installation. However, it is also possible that there are other applications using the same port numbers (for example, older versions of WebSphere Application Server in a co-existence environment or the unit test environment of Rational Application Developer).
If an application server does not start during the IVT process, or later using the startServer command, look for the following message in startServer.log: ADMYUXXXXX: an instance of the server may already be running: server1
This message indicate that another instance of the server is already running or another process is using the same port number.
Use the serverStatus command to determine if the server instance that you are trying to start is already started. Review the ports that are required by the application server process. You can see these in: /profiles//config/cells//nodes// serverindex.xml List the ports that are currently being used by the system. One way of doing this is by issuing a netstat -a command in a command prompt window. If you have a port conflict, update the serverindex.xml file to use a non-conflicting port number.
3.3.3 Profile creation problems The following are common problems with profile creation.
Windows 2000 has a length restriction of 258 characters for a command. A problem can occur that prevents the successful creation of a profile when a path is too long. The maximum length for thedirectory is 60 characters. The maximum length for the profiles installation root directory is 80 characters.
If your log files have errors similar to Input line is too long then the length of the file path and node name in the command string has caused the entire command to exceed the operating system limit for command length.
This error can show up in a message box during the wizard or if using wasprofile, in /profiles//logs/collect_metadata.log.
Create the profile again using shorter directory paths and node names. If you are still in the installation process, consider re-installing using a shorter path for the installation root.
If you see errors similar to localhost is not a valid host name for remote access, then you have entered localhost as the value in the host name field of the Profile creation wizard. Other machines in the network cannot reach your node using localhost, so you must provide a host name that can be resolved by other systems to your system's IP address.
If in a UNIX environment you get an error similar to the following then verify the profile templates location and permissions are correct: Incoming command line is: { "-create" ,"-help" ,"-templatesPath" ,"/opt/WebSphere/AppServer/crso/profileTemplates/managed" } Could not resolve templatePath from command line
If an error occurred creating a custom profile, look at the addNode.log file for any additional errors. Look for an error message similar to the following: [Date and Time stamp] 0000000a AbstractNodeC E ADMU0006E: Exception creating Deployment Manager connection: com.ibm.websphere.management.exception.ConnectorException: ADMC0016E: The system cannot create a SOAP connector to connect to host hostname at port 8879
If you see this message, it is likely that the deployment manager profile is not running. Ensure that the deployment manager profile is created and running on the specified port and host before creation of custom profile.
If neither of these messages appears and you do not see any other obvious problems, go to "The next step" on page 113.
The symptoms and problem areas included in this paper are some that you are more likely to experience. However, there are other things that can go wrong during installation.
Review the problem classifications to see if there are any other components that might be causing the problem.
resources for diagnosing and fixing issues: http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/ com.ibm.websphere.base.doc/info/aes/ae/rtrb_allrfl.html
114 WebSphere Application Server V6 Problem Determination for Distributed Platforms 4
problem determination This paper discusses techniques for determining the cause of problems with WebSphere Application Server system management. The symptoms that this paper discusses are: You are not able to access the administrative console. You cannot access server processes using wsadmin or the management scripts such as stopServer. You are getting errors performing system management functions (for example, managing application servers, node agents, Web servers, or applications). You are not able to federate a node with a deployment manager. You are getting save conflict messages in the administrative console. Your enterprise applications no longer appear in the administrative console. You have problems with WebSphere Rapid Deployment. You are having trouble communicating using SSL and are getting messages that indicate your certificates have expired.
by reading Approach to Problem Determination in WebSphere Application Server V6 at http://www.redbooks.ibm.com/redpapers/pdfs/redp4073.pdf.
4.1 Introduction In this paper, we look at resolving problems that occur during system management activities. WebSphere Application Server configuration is maintained in XML files. The XML files are maintained in a directory structure that reflects the topology of your WebSphere Application Server installation (Figure 4-1).
config repository
Admin console
wsadmin
WebSphere WebSphere server server
administration tool read, maintain, and modify the contents of the XML files.
basis. You can either back up the repository manually via a file system backup, or you can use the WebSphere Application Server supplied tool backupConfig. This script creates a zipped file that contains all the configuration files that you might need to restore the WebSphere Application Server configuration.
Tip: Editing the XML files in the repository directly is not supported and can lead to unexpected results.
At the most basic level, that is a stand-alone installation of a single WebSphereApplication Server, the config directory is as shown in Figure 4-2. The XML filesshown contain the data that gets formatted by the administrative console anddisplayed in the various boxes and text fields in the browser.
The administrative console and other management tools call management beans(MBeans) in the WebSphere Application Server process. MBeans are used byWebSphere Application Server to perform system management functions.
master repository for all of the WebSphere Application Server nodes and serversthat it manages in the cell. Copies of the files that each node needs arereplicated to that node by a process known as synchronization.
D e p lo ym e n t M a s te r M anager R e p o s ito ry
C e ll
Node01 Node02
Node01 Node02 R e p o s ito ry R e p o s ito ry
automatic synchronization every minute (the default interval), this will have a minimal performance impact because the node agent will connect to the deployment manager and be told that nothing has changed.
selecting System administration → Nodes, putting a check by the node that you wish to synchronize, and then clicking either Synchronize or Full Resynchronize.
console is aware of using the normal synchronization mechanism. Choosing Full Resynchronize synchronizes all changes that have been made to the master repository by whatever means. For example, if you were to edit the master copy of the server.xml file with a text editor, clicking Synchronize does not replicate this file to the node. If you click Full Resynchronize, then your updates are replicated.
You can also perform synchronization from the node agent using thesyncNode.bat|sh script. You must stop the node agent to use this tool. You canpass parameters to the script to log messages to a specified file or to trace thesynchronization.
Note: You can also perform synchronization using the wsadmin tool. See the WebSphere Information Center for further details.
Tip: Synchronization does not protect against losing your entire replicated repository. The node agent configuration files must exist for synchronization to work. It is not a replacement for a comprehensive backup policy.
application and then remove that profile when testing is complete.
Profiles are created either by the graphical profile creation wizard or thewasprofile command and are managed via the wasprofile command.
A user registry contains the names, groups memberships, roles, and passwordsof users in an organization. At the most basic level, most operating systems
contain a user registry so that users can logon with a user name and password and have access to operating system resources. At the more complex level, a large distributed organization might maintain a complete list of users including their passwords, personal information, group memberships, and job roles in a distributed LDAP repository.
servers that you need to secure, then you need to use an LDAP registry so that the same registry is available to all servers.
You can encode the passwords in these files to prevent them from being read easily by using the password encoder utility: /bin/PropFilePasswordEncoder.bat|sh
communication from the browser to the administrative console and for other administrative communications. SSL communication relies on a key database file and password. WebSphere Application Server comes with a dummy key database file that is secured with the password WebAS. We highly recommend
that you create your own key database file for production servers. Refer to theWebSphere Information Center for details on customizing the key database:http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com.ibm.websphere.base.doc/info/aes/ae/tsec_preparetruststorefile.html
WebSphere Rapid Deployment is a new feature in WebSphere ApplicationServer V6 that simplifies and speeds up the process of deploying enterpriseapplications or their components for development purposes. WebSphere RapidDeployment is a cut-down version of the Eclipse IDE and runs in batch mode.
directory. If the developer copies an enterprise application EAR file to thatlocation, WebSphere Rapid Deployment detects the new EAR and automaticallydeploys it to the WebSphere Application Server. Figure 4-4 shows theWebSphere Rapid Deployment components.
WebSphere Rapid Deployment is only intended for development environments to simplify the process of testing updates to applications. It is not intended for production environments where you do not want production code updated easily. For that reason, the WebSphere Rapid Deployment process only runs in the foreground.
You need to collect the following log files and data for system management problem determination: WebSphere Application Server logs wsadmin output Management script logs Profile management logs WebSphere Rapid Development logs
"The next step" on page 162 for a complete list of documentation to collect before continuing. In particular, you should review the MustGather documents for a complete list of documentation that is required by IBM support.
You need to look at the WebSphere Application Server log files when diagnosing system management problems.
Application serverWith some problem types, you might also need to look at the logs for theapplication server that you are trying to manage. These log files are:/profiles//logs//SystemOut.log/profiles//logs//SystemErr.log
Most system management problems can be traced by setting the following tracestring:Admin=all
can set tracing by updating the server.xml file for the application server that youneed to trace. You can find the server.xml file under the server's entry in theconfiguration file directory structure:/profiles//config/cells//nodes//servers//server.xml
Open this file in a text editor and update the services tag, changing the startupTraceSpecification to the trace string that you require, as shown in Example 4-1.
wsadmin output Messages from wsadmin are written to the wsadmin.traceout log file: /profiles//logs/wsadmin.traceout
You can also increase the amount of data that is logged to this file by tracing the wsadmin utility. Update the following file: /properties/wsadmin.properties
com.ibm.ws.scripting.traceString=com.ibm.*=all=enabled
Note that the information that is logged is of limited use because wsadmin calls MBeans in the application server that is running the administrative console application. You usually need to trace the application server as well.
WebSphere Application Server services can be managed using the supplied management scripts. For example, each WebSphere Application Server installation has a script to start an application server, a script to stop an application server, and a script to show you the status of all application servers that are defined in a profile. Each of these scripts writes its own log file into the server's logs directory. For example, the stopServer script writes stopServer.log into the logs directory:
/profiles//logs//stopServer.log
The profile creation and management tool wasprofile writes messages to the profile independent logs directory, that is: /logs/wasprofile/.log
The Java graphical interface that is used to create a profile simply calls the wasprofile command after collecting the information that is needed. By default, it does not write a log. However, you can pass it a log parameter as shown: pctWindows -is:log c:\temp\pct.log
The WebSphere Rapid Deployment tool works on a directory that you create and pass to WebSphere Rapid Deployment in the WORKSPACE environment variable. It logs Eclipse messages into two separate files within this directory: /.metadata/.log /project/.metadata/.log
Deployment calls MBeans on the application server. The application server logs can help you resolve a problem with WebSphere Rapid Deployment. There is no way to trace the WebSphere Rapid Deployment utility. However, you can trace the application server as described in "WebSphere Application Server logs" on page 122.
Note: In a UNIX environment, files and directories starting with a period (.) are hidden. You need to pass the -a parameter to the ls command to see them.
You begin the problem determination process by evaluating the high level symptoms to determine if one of these describes your problem. If it does not, you collect the appropriate data that is required to diagnose the problem.
Let's start working the problem: If you are not able to access the WebSphere administrative console, go to "Problem: Unable to access the administrative console" on page 127. If you are using wsadmin or the management scripts such as stopServer, but these admin tools cannot access the server process, go to "Problem: wsadmin or management scripts can't access server" on page 130.
If you are trying to perform administrative tasks against WebSphere Application Server processes (such as starting or stopping a server), consider the following situations: - If you are unable to stop an application server, go to "Problem: Unable to stop a server process" on page 133. - If you are unable to start an application server, go to "Problem: Unable to start a server process" on page 135. - If the deployment manager is not communicating with a node agent, go to "Problem: Unable to access a node agent" on page 137. - If you are unable to manage a Web server from the WebSphere administrative tools, go to "Problem: Unable to manage a Web server" on page 138. - If you are unable to manage (start, stop, install) an application, go to "Problem: Unable to manage applications" on page 141. If you are unable to federate a node with a deployment manager, go to "Problem: Failure adding a node to a deployment manager" on page 143. If you are making changes in the administrative console and getting messages warning you of a save conflict, go to "Problem: Save conflicts in the administrative console" on page 150. If enterprise applications no longer appear in the administrative console go to "Problem: enterprise applications missing" on page 151. If you are having a problem communicating with your system via SSL or are receiving messages that your certificates have expired, go to "Problem: Invalid or expired certificates" on page 153 If you are having problems with WebSphere Rapid Deployment, go to "Problem: WebSphere Rapid Deployment" on page 158. If you cannot create, delete, or update WebSphere Application Server profiles, see WebSphere Application Server V6: Installation Problem Determination at: http://www.redbooks.ibm.com/redpapers/pdfs/redp4068.pdf
Where to go from here: If, after finding your symptom and going to the appropriate section, or if none of these symptoms resemble your problem, go to "The next step" on page 162.
4.3 Analyzing problem areas Your analysis of the data that you gathered will most likely lead you to one of the following problem areas. If not, see "The next step" on page 162.
You are trying to access the WebSphere administrative console and the login page is not being displayed. Instead, you see a blank page or an HTTP 404 error, page not found error.
Server, after enabling security, after adding a node to the deployment manager, or after restarting the WebSphere Application Server process.
manager. In a stand-alone server environment, the process will run in the server (server1).
You might also need to take an admin trace and a security trace, depending on the problem. See "Collecting data" on page 122 for details.
This problem can occur because the server that hosts the administrative console is not running, or you are trying to connect to the wrong port number. It can also occur if the administrative console files have been accidently deleted or security is incorrectly configured.
By default, access to the administrative console is via the following URL: http://:9060/ibm/console
However, the port is configurable, and if multiple profiles have been installed on a machine, the port might be different from the default.
administrative console has started. You will see messages similar to those in Example 4-2 on page 128, showing you that the administrative console has started and on which port the administrative console is listening.
Example 4-2 Administrative console startup messages [6/22/05 9:58:31:342 EDT] 0000001b ApplicationMg A WSVR0221I: Application started: adminconsole [6/22/05 9:58:31:492 EDT] 0000000a TCPChannel A TCPC0001I: TCP Channel TCP_1 is listening on host * (IPv4) port 9060.
Ensure that you are connecting to the correct server on the correct port number, for example: http://localhost:9060/ibm/console
If the administrative console fails to start, you will see errors similar to those shown in Example 4-3.
[7/3/05 15:10:29:359 EDT] 0000000a DeployedAppli E WSVR0100W: An error occurred initializing,adminconsolejava.io.FileNotFoundException: C:\IBM\WAS6\AppServer\systemApps\adminconsole.ear\deployment.xml(The system cannot find the path specified)at java.io.FileInputStream.open(Native Method)
Note: In WebSphere Application Server V6, the adminconsole.ear and filetransfer.ear applications no longer appear as installed applications and cannot be managed in the same way as other applications. This is to prevent them from being uninstalled accidently.
You might not be able to access the administrative console application if securityis enabled due to a problem with the digital certificates that are used to setup anSSL communication between your browser and WebSphere Application Server.If this is the case, you would be able to access other applications on theapplication server that did not use SSL without any problems. Refer to "Problem:Invalid or expired certificates" on page 153 for details.
If your problem is not resolved by fixing invalid certificates, you can confirm that itis related to security by turning security off and retesting.
You can find the security.xml file in the configuration directory structure under: /profiles//config/cells//security.xml
You need to restart the server for this change to take effect.
4.3.2 Problem: wsadmin or management scripts can't access server You are trying to use wsadmin to manage your system, but you are unable to connect to an application server process or are getting no response from wsadmin.
You have this problem when trying to use the supplied management scripts, such as stopServer.bat or serverStatus.bat.
Server, after enabling security, after adding a node to the deployment manager, or after restarting the WebSphere Application Server server.
stopServer.bat, then you need the log file from that script (stopServer.log) and possibly a trace of that script.
This problem can occur because the application server to which you are trying to connect is not running, or you are trying to connect to the wrong port number.
environment or deployment manager in a distributed environment. The following error shows that no connection could be established: WASX7023E: Error creating "SOAP" connection to host "hostname"
Make sure that the application server is running and that you are using the correct port number to connect to the application server or deployment manager.
You can determine to which server wsadmin is trying to connect and on what port by looking at the wsadmin.properties file. This file is in the profile directory structure: /profiles//properties/wsadmin.properties
Example 4-4 shows the host name and port number that wsadmin uses bydefault.
com.ibm.ws.scripting.host=localhostcom.ibm.ws.scripting.port=8879
You can compare this to the entry in the SystemOut.log from the applicationserver that tells you on which port the application server is actually listening forSOAP connections, as shown in Example 4-5.
[7/8/05 14:19:55:445 EDT] 0000000a JMXSoapAdapte A ADMC0013I: The SOAPconnector is available at port 8880
Alternatively, you can pass the port number to wsadmin with the -portparameter:wsadmin -host kll6571 -port 8880
deployment manager. Again, the wsadmin.properties file contains the host nameand port number of the deployment manager process. Using the -host and -portparameters, you can use wsadmin to connect directly to a process other than thedeployment manager. For example you could connect directly to a node agent orremote application server. Example 4-6 shows a connection to a node agent andthen issuing a stopServer command.
$ ./wsadmin.bat -host m23vnx60.itso.ral.ibm.com -port 8881WASX7209I: Connected to process "nodeagent" on node m23vnx60Node01 using SOAPconnector; The type of process is: NodeAgentWASX7029I: For help, enter: "$Help help"wsadmin>$AdminControl stopServer nodeagentWASX7337I: Invoked stop for server "nodeagent" Waiting for stop completion.
Note: In Example 4-6, you will not receive any further response because the server shuts down. Use Ctrl+c to exit the utility.
You can use the netstat command to ensure that a process is listening on the port. Example 4-7 shows the output of the netstat command that has been filtered with the grep tool to only show lines containing the string 8880.
C:\IBM\WAS6\AppServer\profiles\StandAlone\bin>netstat -an|grep 8880 TCP 0.0.0.0:8880 0.0.0.0:0 LISTENING TCP 9.42.171.145:8880 9.42.171.145:1114 CLOSE_WAIT TCP 127.0.0.1:8880 127.0.0.1:1122 CLOSE_WAIT TCP 127.0.0.1:8880 127.0.0.1:1161 CLOSE_WAIT
Note: The netstat command is available on both UNIX and Windows. The grep command is available on UNIX by default, and you can download free copies of grep for Windows.
You can also use the telnet command to try and get a connection to the host name and port as shown in Example 4-8.
C:\IBM\WAS6\AppServer\profiles\StandAlone\bin>telnet localhost 8880 Connecting To localhost...Could not open a connection to host on port 8880 : Connect failed
Note that even if you can connect, you will not be able to do anything because WebSphere Application Server does not respond to telnet commands. You can close your telnet session by pressing Ctrl+c.
If telnet is not able to connect, then the server is not started or not accepting any form of connection. Try restarting the application server. You probably need to kill the Java process because it will not accept the stopServer command.
Verify the security configuration If security is enabled, then you need to pass a user name and password to wsadmin in order to connect. If you do not pass a user name or password, or get them wrong, you get an error connecting, as shown in Example 4-9
WASX7246E: Cannot establish "SOAP" connection to host "localhost" because of an authentication failure. Ensure that user and password are correct on the command line or in a properties file. Exception message (if any): "ADMN0022E: Access is denied for the getProcessType operation on Server MBean because of insufficient or empty credentials."
manager if security is enabled, due to a problem with the digital certificates that are used to setup an SSL communication. See "Problem: Invalid or expired certificates" on page 153 for details.
You are trying to stop an application server through the administrative console but are getting failure messages.
stopServer.bat, then you need the log file from that script (stopServer.log) and possibly a trace of that script.
What to look for Figure 4-6 shows the result that you would see in the administrative console if you tried to stop an application server that had hung. It takes some time for the stop request to time out.
In this case, the SystemOut.log for the deployment manager will most likely not contain any relevant messages. However, in the SystemOut.log for the node agent, you might see messages similar to those that are shown in Example 4-10.
[7/11/05 13:51:54:207 EDT] 0000000d ThreadMonitor W WSVR0605W: Thread"SoapConnectorThreadPool : 6" (00000032) has been active for 607027 milliseconds and may behung. There is/are 1 thread(s) in total in the server that may be hung.
In this example, the problem is that the application server is hung and cannot respond to requests. You might also see a situation in which the node agent is hung, as shown in Example 4-11.
[7/11/05 14:19:00:918 EDT] 0000000f ThreadMonitor W WSVR0605W: Thread "WebContainer : 1"(00000038) has been active for 619290 milliseconds and may be hung.There is/are 1 thread(s) in total in the server that may be hung.
In the case where the node agent is hung, the administrative console will eventually time out after at least 10 minutes. You will see a message in the deployment manager's SystemOut.log before the time out.
Note: In the Windows Task Manager, the process ID is not shown by default. However you can add the process ID (PID) using the View → Select Columns option.
Stopping a hung process is simply a work-around. The real problem is the hang.
If you are unable to start an application server, the problem is likely to be either a configuration or an application error. Both the administrative console and the startServer script output will return an error telling you where to look.
Note: The steps outlined here are also true for a deployment manager or node agent.
Data to collect If you are trying to start the server through the administrative console, open a command window and try to start the server using the startServer command.
If these logs do not make clear the cause of the problem, you might also need to trace the script.
When a server fails to start, you need to review the output from the startServer command and the logs for that application server for messages that tell you at what point the server failed.
Example 4-12 shows the output from a startServer command that indicates why a server will not start.
C:\IBM\WAS6\AppServer\profiles\StandAlone\bin>startserver server1 ADMU7701I: Because server1 is registered to run as a Windows Service, the request to start this server will be completed by starting the associated Windows Service. ADMU0116I: Tool information is being logged in file
the server server1 is already running b) some other process is using port 8880 ADMU3027E: An instance of the server may already be running: server1 ADMU0111E: Program exiting with error: com.ibm.websphere.management.exception.AdminException: ADMU3027E: An instance of the server may already be running: server1 ADMU1211I: To obtain a full trace of the failure, use the -trace option. ADMU0211I: Error details may be seen in the file:
C:\IBM\WAS6\AppServer/profiles/StandAlone\logs\server1\startServer.log
In Example 4-12, the output tells us that some other process is using one of the ports that this server needs in order to start. You can confirm that the port is in use with the netstat command.
Example 4-13 shows that some process is listening on port 8880. In UNIX, you can determine what process is listening on that port using the lsof command.
TCP 9.42.171.145:8880 9.42.171.145:2698 TIME_WAIT TCP 9.42.171.145:8880 9.42.171.145:2739 TIME_WAIT
Example 4-14 shows the output from the lsof command. This output tells you that a Java process is listening on TCP port 8880. The process ID is 12968.
[root@m23vnx60 root]# lsof -i|grep 8880 java 12968 wasadmin 135u IPv4 655014 TCP *:8880 (LISTEN)
Note: There are Windows equivalents of the lsof command available for download from the Internet.
By checking the server logs, you can determine if the server has actually started.
C:\IBM\WAS6\AppServer\profiles\StandAlone\bin>startserver server1 ADMU7701I: Because server1 is registered to run as a Windows Service, the request to start this server will be completed by starting the associated Windows Service. ADMU0116I: Tool information is being logged in file
You might be able to resolve this by setting the SOAP timeout parameter in the following file: /profiles//properties/soap.client.props
You have recently restarted the deployment manager and discovered that one of your node agents does not appear to be available.
However, depending on the cause of the problem, you might or might not see any helpful messages in the logs.
Check the node agent to ensure that it is actually running by looking for its process and checking its SystemOut.log. You could also ensure that it is running by trying to connect to the node agent directly with wsadmin as shown in Example 4-6 on page 131.
If the node agent is running and the deployment manager is not reporting any errors but is still saying that the node agent is down, it is possible that a DNS error occurred as the deployment manager was starting.
WebSphere Application Server V6 has the ability to manage the IBM HTTP Server V6 from within the administrative console.
You are having problems starting, stopping, or getting the status of the Web server.
You can start and stop any of the supported Web servers using the WebSphere administrative console. The Web server must be installed on a managed node. A WebSphere Application Server node agent executes the appropriate start or stop command.
When you are using IBM HTTP Server V6.0, you do not need to install the Webserver on a managed node because administration can be performed throughthe IBM HTTP Administration Server. In addition to starting and stopping theIBM HTTP Server V6.0, you can also manage its configuration file and view theWeb server logs.
If you cannot administer the IBM HTTP Server from the WebSphereadministrative console, verify that the IBM HTTP Server administration serverhas been properly set up.
New password: ****** Re-type new password: ****** Adding password for user webadmin
When you are managing an IBM HTTP Server using the WebSphere administrative console, you must ensure that the following conditions are met: Verify that the IBM HTTP Server administration server is running. Verify that the Web server host name and port that are defined in the WebSphere administrative console match the IBM HTTP Server administration host name and port. Verify that the firewall is not preventing you from accessing the IBM HTTP Server administration server from the WebSphere administrative console. If you are trying to administer the IBM HTTP Server over a secure SSL connection, verify that you have exported the IBM HTTP Server administration server key database personal certificate and imported it into the WebSphere key database as a signer certificate. The key database will be identified in the com.ibm.ssl.trustStore parameter in the sas.client.props file. Verify the IBM HTTP Server admin_error.log file and the WebSphere Application Server logs (trace.log) do not contain any errors.
When the IBM HTTP Server on a managed node is started using WebSphere administration tools. The node agent issues the same command that you would use from a command line, that is: /bin/apachectl
The deployment manager does not log any messages when you try to start a Web server. The node agent logs a simple message to tell you that it is trying to start the Web server as shown in Example 4-16.
If the Web server fails to start, then you get a failure message in the administrative console and a message in the node agent log, as shown in Example 4-16.
[7/12/05 14:56:15:658 EDT] 000000a6 AdminHelper A ADMN1001I: An attempt is made to launchwebserver1 on node m23vnx60Node01.[7/12/05 14:56:15:661 EDT] 000000a6 NodeAgent W ADML0065W: A sync operation prior tostarting an application server failed.
[7/12/05 14:56:22:843 EDT] 000000a6 NodeAgent W ADML0040E: The process timed out waitingfor server "webserver1" initialization: 1200 seconds
You need to review the Web server logs to determine what went wrong.
Also, you can trace both the deployment manager and node agent process, as described in "Collecting data" on page 122, to gather further information about the source of a failure.
You are trying to manage applications, for example starting, stopping, or installing an application, and are having problems.
application, or after a server restart.
When an error occurs when you are trying to manage an application, the following are things you can do: Verify that there are no resource or classpath errors Verify that the application was installed correctly Verify that the configuration repository is not corrupted
The application server log files are the first place to look as any problems with the application itself, such as classpath issues or resource issues, are reported there. The error message from the administrative console tells you which node and server to check, as shown in Figure 4-7 on page 142. Should you find such an error, work with your application developers to resolve the application coding or resource problem.
If the application fails to start on a server after it has been installed, the installation might not have completed successfully. The installation might have
appeared to be successful, but when you try to start the application, it fails with a message in the administrative console as shown in Figure 4-7 on page 142.
Figure 4-7 Message from administrative console when application fails to start
shown in Example 4-17.
Example 4-18.
Example 4-18 Message in application server log when application fails to start[7/12/05 9:44:07:379 EDT] 00000039 ApplicationMg W WSVR0215W: Starting application, HelloApp,failed. The application is not installed.
Corruptions in the configuration repository can also cause enterprise applications to not start. For example, a node's serverindex.xml file contains a listing of all servers and ports in a cell with which the node might need to communicate. It also lists applications that are deployed on the servers in those nodes.
If this file becomes corrupt and the list of applications is lost, then the applications will not be started on the server in this node. When the server process starts, it thinks that the application is not installed and does not try and start it. If you try and start the application from the administrative console, you see the message that is shown in Figure 4-7 on page 142. The application server logs tell you that the application is not installed, as shown in Example 4-18 on page 142.
Nodes. Select the node and choose Full Resynchronize, as shown in Figure 4-8.
A WebSphere Application Server node is federated to a deployment manager using the addNode command. This can sometimes fail.
Data to collect The following logs can be helpful in determining why you cannot add a node to a deployment manager: Management script log file: addNode.log Deployment manager logs
Depending on the cause of the failure, you might also need to look at the logs on the deployment manager.
Example 4-19 shows that the connection from the addNode script to the deployment manager timed out before the work completed.
ADMU0116I: Tool information is being logged in file /opt/IBM/WAS6/AppServer/profiles/Craig02/logs/addNode.log ADMU0128I: Starting tool with the Craig02 profile ADMU0001I: Begin federation of node m23vnx60Node02 with Deployment Manager at kll6571:8879. ADMU0009I: Successfully connected to Deployment Manager Server: kll6571:8879 ADMU0507I: No servers found in configuration under:
ADMU0211I: Error details may be seen in the file: /opt/IBM/WAS6/AppServer/profiles/Craig02/logs/addNode.log
You can get more detailed information from the addNode log as shown in Example 4-20 on page 145.
[7/12/05 15:20:02:155 EDT] 0000000a AdminTool A ADMU0113E: Program exiting with error: com.ibm.websphere.management.exception.ConnectorException: ADMC0009E: The system failed to make the SOAP RPC call: invoke at com.ibm.ws.management.connector.soap.SOAPConnectorClient.invokeTemplate(SOAPCon nectorClient.java:642) ... resulting from: [SOAPException: faultCode=SOAP-ENV:Client; msg=Read timed out; targetException=java.net.SocketTimeoutException: Read timed out] at org.apache.soap.transport.http.SOAPHTTPConnection.send(Unknown Source)
You are making changes to your configuration in a Network Deployment environment and while you can save the changes, the synchronization request to a remote node is failing.
What to look for Review SystemOut.log from the deployment manager and from the node agent on the server that is not synchronizing. The administrative console message will tell you which node is failing to synchronize as shown in Figure 4-9.
deployment manager.
[7/3/05 16:01:17:845 EDT] 00000026 CellSync E ADMS0104I: The system is unable to invokea synchronization request on nodeWebSphere:platform=common,cell=IBM-99TVXRDCell01,version=6.0.1.2,name=nodeSync,mbeanIdentifier=nodeSync,type=NodeSync,node=IBM-99TVXRDNode01,process=nodeagent.javax.management.InstanceNotFoundException: MBeanServer cannot find MBean with ObjectNameWebSphere:platform=common,cell=IBM-99TVXRDCell01,version=6.0.1.2,name=nodeSync,mbeanIdentifier=nodeSync,type=NodeSync,node=IBM-99TVXRDNode01,process=nodeagent
The replicated copy of the file being modified might be corrupted or inaccessible on the node where the change is being made. For example, making a change to the Java heap size parameter updates the server.xml file in the repository. The change is made to the master copy of the file, and the administrative console returns a message that the change was saved and that the synchronization was successful, as shown in Figure 4-10.
successful: [7/3/05 16:11:13:343 EDT] 00000085 DeploymentMan A ADMS0208I: The configuration synchronization complete for cell.
However, the node agent's SystemOut.log shows you that the node agent is not able to open the file due to permissions, as shown in Example 4-22.
[7/3/05 16:11:13:902 EDT] 00000040 FileDocument E ADMR0104E: The system is unable to readdocument cells/IBM-99TVXRDCell01/nodes/IBM-99TVXRDNode01/servers/server1/server.xml:java.io.FileNotFoundException:C:\IBM\WAS6\AppServer\profiles\AppSrv01\config\cells\IBM-99TVXRDCell01\nodes\IBM-99TVXRDNode01\servers\server1\server.xml (Access is denied)[7/3/05 16:11:13:912 EDT] 00000040 NodeSyncTask A ADMS0016I: The configurationsynchronization failed.
In this case, the solution is to fix the file permissions and retry the synchronization.
In other cases, such as with a corrupt or empty file, the solution is to delete the file and allow automatic synchronization to replace the file with the good copy from the master repository.
the node either from the administrative console or from the node itself. Figure 4-11 shows the file synchronization settings for the node agent. In this instance, automatic synchronization is enabled.
If you are unable to determine why synchronization is failing from the messages in the logs, try a command line synchronization initiated from the node that you cannot synchronize. You need to stop the node agent. From a command line, issue the syncNode command, as shown in Figure 4-23. You need to pass it the host name and SOAP port of the deployment manager. You can also specify the -trace option.
[wasadmin@m23vnx60 bin]$ ./syncNode.sh kll6571 8879 -trace ADMU0115I: Trace mode is on. ADMU0116I: Tool information is being logged in file /opt/IBM/WAS6/AppServer/profiles/Node01/logs/syncNode.log ADMU0401I: Begin syncNode operation for node m23vnx60Node01 with Deployment Manager kll6571: 8879 ADMU0016I: Synchronizing configuration between node and cell. ADMU0111E: Program exiting with error: com.ibm.websphere.management.exception.AdminException: ADMU0005E: Error synchronizing repositories ADMU0211I: Error details may be seen in the file: /opt/IBM/WAS6/AppServer/profiles/Craig01/logs/syncNode.log
Note: You can stop a node agent without having to stop the application servers. The operational impact of stopping the node agent is that it will not monitor the application servers to make sure they are up.
Reviewing the log file shows the problem as shown in Example 4-24.
[7/11/05 12:14:00:448 EDT] 0000000c FileDocument E ADMR0105E: The system is unable to writedocument cells/kll6571Cell01/nodes/m23vnx60Node01/servers/server02/server.xml:java.io.FileNotFoundException:/opt/IBM/WAS6/AppServer/profiles/Node01/config/cells/kll6571Cell01/nodes/m23vnx60Craig01/servers/server02/server.xml (Permission denied)
In this example, it was not necessary to run the trace because the error message would have appeared anyway. A trace is useful for finding the problem when there are no obvious errors.
Example 4-25 Benign FileNotFound exceptionjava.io.FileNotFoundException:/opt/IBM/WAS6/AppServer/profiles/Node01/config/cells/kll6571Cell01/nodes/m23vnx60Node01/servers/nodeagent/variables.xml (No such file or directory)
You are seeing messages telling you that there is a save conflict in the administrative repository when you login to the administrative console.
After you made changes to the WebSphere Application Server configuration in the administrative console and try and save those changes, you see a message telling you there is a save conflict as shown in Figure 4-12.
This error tells you that there is a conflict in updates to one or more files in the configuration repository. The save conflict is not reported in the SystemOut.log unless you choose to overwrite the conflicting change or have enabled tracing.
A save conflict can happen for a number of reasons: More than one administrator has made changes to the configuration at the same time. Changes have been made to the configuration repository files directly. You are deploying applications using WebSphere Rapid Deployment.
[7/11/05 11:20:03:348 EDT] 00000037 FileRepositor A ADMR0016I: User WTRNTDM/config modifieddocument cells/kll6571Node01Cell/nodes/kll6571StandAlone/servers/server1/server.xml.[7/11/05 11:20:10:569 EDT] 0000003f ServletWrappe A SRVE0242I:[/secure/tiles/syncconflict.jsp]: Initialization successful.[7/11/05 11:20:23:808 EDT] 00000038 FileRepositor W ADMR0114W: The system is overwritingdocument cells/kll6571Node01Cell/nodes/kll6571StandAlone/servers/server1/server.xml by request.[7/11/05 11:20:23:848 EDT] 00000038 FileRepositor A ADMR0016I: User WTRNTDM/wasadmin modifieddocument cells/kll6571Node01Cell/nodes/kll6571StandAlone/servers/server1/server.xml.
the same time and not have problems with save conflicts when those administrators are working with different files and objects in the repository. Unless you constantly check the SystemOut.log, you cannot be sure that the changes you are making are not being overwritten. Thus, we recommend that you only have one administrator making changes at a time.
You have logged into the administrative console and are trying to manage enterprise applications but none are displayed. The applications might be working fine operationally, you just cannot manage them. This problem is most likely to happen after a server restart.
Data to collect The following logs can be helpful in determining why applications are missing: Deployment manager log Node agent log Application server log
From the administrative console, you should be able to see a list of installed applications under Applications → Enterprise Applications. If your applications are no longer listed here, start by checking the deployment manager log. This problem can be caused by a corruption to the configuration repository or problems with permissions, as shown in Example 4-27.
[7/13/05 14:09:48:993 EDT] 0000003a FileDocument E ADMR0104E: The system is unable to readdocument cells/m23vnx60Cell01/applications/HelloApp.ear/deployments/IBMUTC/deployment.xml:java.io.IOException: No such file or directory
Reviewing the node agent's log would show you a similar message, possibly referring to a different file, as shown in Example 4-28.
[7/13/05 14:12:03:979 EDT] 00000035 FileDocument E ADMR0109E: An error occurred restoringdocumentcells/m23vnx60Cell01/applications/HelloApp.ear/deployments/HelloApp/META-INF/was.policy:java.io.FileNotFoundException:/opt/IBM/WAS6/AppServer/profiles/AppSrv01/config/cells/m23vnx60Cell01/applications/HelloApp.ear/deployments/HelloApp/META-INF/was.policy (Permission denied)
The application server log also shows that the problem is caused by file permissions as shown in Example 4-29.
[7/13/05 13:55:26:517 EDT] 0000001b FileDocument E ADMR0104E: The system is unable to readdocument cells/m23vnx60Cell01/nodes/m23vnx60Node01/perftuners.xml: java.io.IOException:Permission denied
In all three cases, the messages refer to the different files and different locations in the file system. Checking the file permissions for those files listed shows that the user who is running the processes does not have write access to those files.
In this example, the servers run on a UNIX platform and are configured to run as a non-root user under normal conditions. At some point, the servers were
restarted by the root user and the HelloApp application was deployed. Then the servers were restarted as the normal non-root user and this problem occurred. The non-root user was not able to write to the files that were created when the server was running as root. The problem is resolved by resetting the file permissions.
If your problems seem to be specific to applications that require SSL access, it is a good probability that the problem is due to invalid or expired certificates. This can cause a variety of system management problems. Problems with expired certificates could occur at any time if you are not managing your certificates. You could also run into this problem if you have recently enabled security.
You might also need to take an Admin trace and a security trace, depending on the problem. See "Collecting data" on page 122 for details.
WebSphere Application Server writes a warning message (Example 4-30) into the SystemOut.log before a certificate is due to expire. This gives you the opportunity to renew the certificates before the actual expiration dates and before applications stop working. However, at present, WebSphere Application Server V6 does not notify you that a certificate has expired.
7/6/05 11:37:15:860 EDT] 0000000a SASRas W JSAS0456W: WARNING in sasOutboundSSLConfig:The certificate with alias expires 07/07 from keyStoreC:\IBM\WAS6\AppServer/profiles/StandAlone/etc/ExpiresKeyFile.jks will be expired in 1 days.
Tracing the application server with the following tracestring shows you if the problem is with certificates, as shown in Example 4-31: traceString=ORBRas=all
[7/7/05 10:23:17:553 EDT] 00000020 ORBRas 1 com.ibm.rmi.transport.ListenerThread run:259LT=3:P=897141:O=0:port=9402 The following exception was loggedjavax.net.ssl.SSLException: No available certificate corresponds to the SSL cipher suites whichare enabled.
If you see messages that indicate a certificate has expired or has a problem, you should verify the certificate using the IBM Key Management tool (ikeyman).
/profiles//config/cells//security.xml
In security.xml, look for theXML tag as shown in Example 4-32.
You will need to know the password on the key database file. If you have not replaced the supplied dummy key file databases, the default password is WebAS.
Figure 4-13 shows the DummyServerKeyFile key database open in the ikeymanutility. Click on the certificate name, websphere dummy server in this example,and then click View/Edit. If the certificate has expired, the utility displays amessage box telling you this. Then, it displays the certificate details.
The Validity field also shows if the certificate has expired. In Figure 4-14 onpage 156, the certificate is only valid up to July 7, 2005. Click OK to close thewindow.
Figure 4-14 Expired certificate details
If the certificate has expired, you can resolve this problem by creating a new valid certificate in the existing key database file.
server, the client can trust the contents of a certificate that is verified by a trusted third party. A certificate authority acts as a trusted third party. The key management utility allows you to request a certificate from a certificate authority. The certificate authority will charge you for this service.
Alternatively, you can create a self-signed certificate in the dummy key file database. This is a certificate that can be used to set up an SSL connection if the client chooses to accept the certificate even though it has not been verified by a certificate authority. In a production environment, you should not use self-signed certificates for securing SSL connections.
Figure 4-13 on page 155.
Figure 4-15 Creating a new self signed certificate
Enter the information for the self signed certificate. The maximum validity periodyou can set is 7300 days or approximately 20 years.
After you have created your new certificates, restart WebSphere ApplicationServer and retest.
Where to go from here: If you have exhausted all the possibilities in this section and are still having problems with SSL and certificates, refer to "The next step" on page 162.
4.3.13 Problem: WebSphere Rapid Deployment You are trying to use WebSphere Rapid Deployment to develop and test an application, and you are not able to connect to an application server or WebSphere Rapid Deployment will not create or update the applications.
If you have problems with WebSphere Rapid Deployment, you can take the following actions: Verify the WORKSPACE variable Verify the WebSphere Rapid Deployment configuration Verify the application is being built Verify the deployment manager and application server are running Look for Java coding errors
You tell WebSphere Rapid Deployment what directory to monitor by setting the WORKSPACE environment variable. What directory you use is up to you. Ensure that you are putting the files that you want updated in the location that is specified by the WORKSPACE environment variable. For example: WORKSPACE=C:\IBM\WRD
Rapid Deployment starts but it does not build or deploy your applications because it is monitoring the wrong directory.
Ensure that you have run wrd-config to set up your environment for the WebSphere Rapid Deployment project that you are working on. You can check your project settings by looking in the project configuration file at: //.wrdconfig.xml
name of the application server that will run the application. The serverJMXHostand serverJMXPort should refer to the host name and SOAP port of thedeployment manager.
You can check the WebSphere Rapid Deployment log (Example 4-34) to ensurethe environment has been initialized and that changes to the workspace arebeing recognized. The messages also shows if the changes are being handledproperly.
!MESSAGE Product org.eclipse.platform.ide could not be found.
!MESSAGE Creating a new project 'HelloApp'.
!MESSAGE Could not load library: libcore_2_1_0b.so. This library provides platform-specific optimizations for certain file system operations. This library is not present on all platforms, so this may not be an error. The resources plug-in will safely fall back to using java.io.File functionality.
!MESSAGE Recording active directory ID to the workspace root.
!MESSAGE Configuring style 'WebSphere Free Form Project' for project 'HelloApp'.
!MESSAGE Recording the current style and project properties...
!MESSAGE Configuring build output location to : '/HelloApp/bin'
!MESSAGE Configuring source folder to '/HelloApp'
!MESSAGE Configuring source folder to '/HelloApp/gen/src'
!MESSAGE Configuring new build output location to '/HelloApp/bin'
If WebSphere Application Server is not available when you start WebSphere Rapid Deployment, you see the following message in the WebSphere Rapid Deployment console: ERROR! Failed to make connection to WebSphere Application Server
Error occurred during download. : publishrecord.remote [01:23:40 PM] Exporting Ear File. [01:23:41 PM] ERROR! [SOAPException: faultCode=SOAP-ENV:Client; msg=Error opening socket: java.net.ConnectException: Connection refused: connect;
targetException=java.lang.IllegalArgumentException: Error opening socket:java.net.ConnectException: Connection refused: connect]
deployment manager. However, the application is run on a separate applicationserver. If the deployment manager is not running, you see the message that isshown in Example 4-35 in the console. If the target application server is notrunning, the application publishes but does not start. You see a message in theconsole as shown in Example 4-36.
[12:08:01 PM] ERROR! Application Failed to Start. HelloApp[12:08:01 PM] ERROR! MBeanServer cannot find MBean with ObjectNameWebSphere:platform=dynamicproxy,cell=m23vnx60Cell01,version=6.0.1.2,name=ApplicationManager,mbeanIdentifier=ApplicationManager,type=ApplicationManager,node=m23vnx60Node01,process=server1[12:08:01 PM] ERROR! Please see server logs for more details.
After you have restarted the process, you need to make a change to one of theapplication files so that WebSphere Rapid Deployment detects the update andpublishes the application again.
administrative console if you or another administrator are logged in. This is another reason why you should use WebSphere Rapid Deployment only for development.
If your application code has errors in it, WebSphere Rapid Deployment showsthe coding error in the console, for example:[01:28:05 PM] 'Syntax error, insert ";" to complete Statement' in resource'HelloServlet.java' on line number 37
Where to go from here: If you have checked all of the problems described thus far and yet still cannot build and deploy an application with WebSphere Rapid Deployment, check the logs for the deployment manager if appropriate and the application server for any messages that are related to the application deployment.
If you have exhausted all the possibilities that are described in this section and are still not able to use WebSphere Rapid Deployment, refer to the next section.
The symptoms and problem areas included in this paper are some that you are more likely to experience. However, there are other things that can go wrong, or the cause of the problem might be related to something other than system management components.
Review the problem classifications to see if there are any other components that might be causing the problem.
MustGather: Profile Creation/Removal Issues for V6.0 http://www-1.ibm.com/support/docview.wss?rs=180&uid=swg21196228 MustGather: Federation or Removal of a Node Issues for Version V6.0 http://www-1.ibm.com/support/docview.wss?rs=180&uid=swg21196227 MustGather: Port Management for V6.0 http://www-1.ibm.com/support/docview.wss?rs=180&uid=swg21196226 MustGather: Node agent and Deployment Manager discovery problems for all releases and editions of V6.0 http://www-1.ibm.com/support/docview.wss?rs=180&uid=swg21196220 MustGather: Usage and creation of templates fail on V6.0 http://www-1.ibm.com/support/docview.wss?rs=180&uid=swg21195439
164 WebSphere Application Server V6 Problem Determination for Distributed Platforms 5
determination If users receive unexpected results in a Web browser (such as errors or incorrect information), you might have a problem with the Web components in the application. The runtime environment for Web components is called the Web container. This paper discusses techniques for diagnosing problems that can occur in the Web container. Potential symptoms of a Web container problem can include: Users are not able to access a Web component Unexpected behavior when running a Web component Errors when starting a Web component Problems with JSP compilation Errors or exceptions thrown when running a Web component Messages starting with SRVE (Web container), JSPG (JSPs), or JSFG (JSF)
Problems areas that are related to the Web container also include session management problems and character encoding problems.
by reading Approach to Problem Determination in WebSphere Application Server V6 at http://www.redbooks.ibm.com/redpapers/pdfs/redp4073.pdf.
5.1 Introduction The Web container is a runtime environment for Web applications. It processes servlets, JSP files, and other types of server-side includes. The Web container also provides session management, static content processing and an inbound transport chain for HTTP requests. Each application server runtime has one logical Web container, which can be modified but not created or removed.
Figure 5-1 illustrates the Web container and its place within the application server.
Application Server
Session Manager
Servlet JSP
EJB Container
JCA Services
Security Server
Messaging
Web Services
Web container transport chains Requests are directed to the Web container using the Web container inbound transport chain. The chain consists of a TCP inbound channel that provides the connection to the network, an HTTP inbound channel that serves HTTP 1.0 and 1.1 requests, and a Web container channel over which requests for servlets and JSPs are sent to the Web container for processing.
Servlet processing When handling servlets, the Web container creates a request object and a response object and then invokes the servlet service method. The Web container invokes the servlet's destroy method when appropriate and unloads the servlet, after which the JVM performs garbage collection. HTML and other static content processing Requests for HTML and other static content that are directed to the Web container are served by the Web container inbound chain. However, in most cases, using an external Web server and Web server plug-in as a front-end to a Web container is more appropriate for a production environment. Session management Support is provided for the javax.servlet.http.HttpSession interface as described in the Servlet API specification.
Figure 5-2 on page 168 illustrates the directory structure of a Web application.
Rational Application Developer Assembly root
ibm-web-ext.xmi ibm-web-bnd.xmi faces-config.xml Server-side class files Library JAR files
A problem that occurs in the Web container usually causes unexpected results to be displayed in the Web browser. Three common indications are addressed here: HTTP 404 errors HTTP 500 errors Incorrect information
Figure 5-3 on page 169 shows a flow chart of the high-level symptoms and the potential problem areas that might apply to each.
Browser displays
Static resource
You begin the problem determination process by collecting the appropriate data that is required to diagnose the problem. We give you a list of all the documentation that might be required and how to collect it. If you have limited ability to recreate the problem, you might want to collect all the documentation at once, before starting the problem determination process.
And lastly, we provide guidance on the next step to take for resolution, whether it be a support site, contacting IBM, information about configuration, or some other suggestion as to how to proceed.
The following data will help you with the problem determination process in a Web container execution environment: WebSphere Application Server JVM logs: SystemOut and SystemErr files WebSphere Application Server process logs: native_stderr.log and native_stdout.log log files Web server log files
For information about collecting the JVM and Process logs, see WebSphere Application Server V6: Diagnostic Data at: http://www.redbooks.ibm.com/redpapers/pdfs/redp4085.pdf
The Web server log files names and locations are specific to the product (IBM HTTP Server, Apache, SunOne, IIS, and Domino®) that is used for this function.
"The next step" on page 209 for a complete list of documentation to collect before continuing. In particular, you should review the MustGather documents for a complete list of documentation that is required by IBM support.
Consider the following error types. If you do not find the description of your problem here, go to "The next step" on page 209.
Many indicators of a Web container problem first appear to a user as an unexpected Web browser page or page contents. The unexpected information can be an error page that results from an HTTP error or a page that has incorrect or incomplete information displayed.
HTTP 404 errors can have different underlying causes. Table 5-1 shows the HTTP 404 errors that we address in this paper.
If the errors include Then go to
Page can't be found or displayed "Symptom: HTTP 404 error - The page cannot be displayed" on page 172
JSPG0036E: Failed to find resource "Symptom: HTTP 404 error - Failed to find message resource" on page 175
WebGroup/virtual host not defined" on page 177
HTTP 500 errorsHTTP 500 errors can also have different underlying causes, which include thefollowing: If the errors indicate JSP processor errors or incorrect syntax in JSPs, go to "Symptom: HTTP 500 error - JSP processing error" on page 179. If the symptoms include an HTTP 500 error page on the browser and in the SystemOut.log file you see an IllegalStateException, go to "Symptom: HTTP 500 error - IllegalStateException" on page 181.
If the users are receiving the correct page on the Web browser but the pagecontains invalid information, check these symptoms: Users report that pages appear with missing elements. If a Web page appears but is missing static resources such as text, images, or file segments, go to "Static resources not displayed" on page 190. Users report seeing an old version of application pages. If a JSP file has been modified and deployed to the server but the changes do not appear to the browser interface, you might need to ensure that the application has been enabled for JSP reloading. Go to "Web resources not reloading" on page 192. Users report that pages contain invalid information such as multiple question marks (???) or garbage characters. It is possible that the application uses DBCS characters (Japanese, Chinese, Korean languages) or specific characters for other languages that are not included in the default character encoding. In these cases, a correct character encoding configuration is necessary to display and process this information without problems. For more details related to encoding configurations, go to "Encoding and internationalization issues" on page 195. Users report problems such as lost data during a session. Users repeatedly have to enter information that should be saved during the session, lose shopping cart information, or have other short-term data loss. This data loss might be caused by a problem with HTTP sessions. In a clustered environment, the problem could be with the Web server plug-in. However, there are other possible causes, such as configuration or application errors. To further analyze this problem, go to "HTTP session management" on page 202.
5.2.3 Symptom: HTTP 404 error - The page cannot be displayed Figure 5-4 shows an overview of the steps that you can take to find the cause of an HTTP 404 error caused by a page cannot be displayed condition.
no Web Module starts sucessfully? Review JVM logs problems
yes no Web Server OK? HTTP Server support
yes
Web container but not Web server? HTTP plug-in problems
no
Review application development
no
Check the JVM logs for the application server, and look for messages in the SystemOut.log file (Example 5-1) to verify that the Web container and Web module started successfully.
ApplicationMg A WSVR0200I: Starting application: [application_name] WebContainer A SRVE0161I: IBM WebSphere Application Server - Web Container. Copyright IBM Corp. 1998-2004 WebContainer A SRVE0162I: Servlet Specification Level: 2.4 WebContainer A SRVE0163I: Supported JSP Specification Level: 2.0 WebGroup A SRVE0169I: Loading Web Module: [web_module_name] ApplicationMg A WSVR0221I: Application started: [application_name]
One possible reason that would keep the Web container from starting is aproblem with the session manager. To determine if this is the case: Look for any errors or exceptions containing a package name of com.ibm.ws.webcontainer.httpsession. You will normally find these errors between the starting application message and the application started message. Look in the logs for session manager related messages. These messages will be in the format SESNxxxxE for errors and SESNxxxxW for warnings. You might also see transaction messages that indicate a problem with session data (see Example 5-2).
Error "SRVE0054E: An error occurred while loading session context and Webapplication.
If you find a session manager error but the explanation is not sufficient to solvethe problem, go to "The next step" on page 209.
Verify that the Web server is healthy by accessing the URL http://server_namefrom a browser and seeing whether the Welcome page appears. This actionindicates whether the Web server is up and running, regardless of the state ofWebSphere Application Server.
If you have recently updated or installed the application, ensure that the Webserver plug-in was regenerated and propagated to the Web server. Also, ensurethat the Web server is using the new plug-in configuration file.
If you have regenerated the plug-in and are sure it is in use but still have aproblem, you can bypass the Web server and access the application directly
from the application server. This is not the recommended method of serving a production Web site. However, it provides a good diagnostic tool when it is not clear whether a problem resides in the Web server, WebSphere Application Server, or the Web server plug-in.
If you cannot access the page directly from the application server, verify that the URL being used to access the page is correct. For more information, see "Application URL specification" on page 186.
If the URL appears to be correct but you cannot access the resource directly through the application server, verify the health of the hosting application server and Web module by doing the following: 1. View the hosting application server and Web module in the administrative console to verify that they are up and running. In a single server environment, you can check the application server process to see if it is running, or you can use the serverStatus command as follows: c:\cd WebSphere\AppServer\profiles\AppSrv01\bin serverStatus server1
2. Copy a simple HTML or JSP file to the Web module document root and try to access it. The Web module document root is located in: /profiles//installedApps// .ear\.war If successful, the problem is with the resource. View the SystemOut.log file for the application server to discover why the resource cannot be found or served.
If none of these steps fixes your problem, check to see if the problem has been identified and documented by looking at the available online support (hints and tips, tech notes, and fixes) that are related to Web container problems: http://www.ibm.com/support/search.wss?rs=180&tc=SSEQTP&tc1=SSCMPDF
If the searches fail to find the problem, then go to "The next step" on page 209 for information about gathering the MustGather documentation for HTTP status code 404 "Not Found" problems.
If the browser displays a JSPG0036E: Failed to find resource message, the JSP processor cannot find the specified JSP page in the Web module, as shown in Figure 5-5 on page 176.
Figure 5-5 JSPG0036E: Failed to find resource
Follow the steps described in "Application URL specification" on page 186 to see if the URL being specified is the correct URL.
If the URL appears to be correct, it is possible that this page does not exist in the Web module deployment unit (WAR file). Verify that the requested page is located in the Web module directory structure:
/profiles//installedApps// .ear\.war
If these steps have failed to identify the problem, see "The next step" on page 209.
5.2.5 Symptom: HTTP 404 error - WebGroup/virtual host not defined The error message SRVE0017W: WebGroup/Virtual Host has not been defined can appear in the SystemOut.log file for many reasons. Check the following to correct the problem.
Verify that the Web module has started successfully (see "Verify that the Web module has started successfully" on page 172).
A duplicate host alias that is defined in multiple virtual hosts can cause this type of problem. For example, in Example 5-3, the host alias test:80 is duplicated in both virtual hosts because a URI that contains test:80 would match aliases in both virtual hosts (*:80 and test:80).
1. In the WebSphere administrative console, click Environment → Virtual Hosts. 2. Select the target virtual host, then click Host Aliases, and select the *:80 host alias. 3. Change the Host Name field to a specific alias instead of * alias. 4. Restart the application server.
If localhost is used as an alias entry, check the etc/hosts file to ensure that all host names (aliases) associated with the loop back address (127.0.0.1) are part of the same virtual host grouping (for example, default_host).
Verify that the URL is correct See "Application URL specification" on page 186 to determine that the URL is correct.
If you receive this error message, refer to Technote # 1193379 at the following Web address for the cause and solution details: http://www.ibm.com/support/docview.wss?uid=swg21193379
If none of these steps fixes your problem, check to see if the problem has been identified and documented by looking at the available online support (hints and tips, technotes, and fixes) that are related to Web container problems: http://www.ibm.com/support/search.wss?rs=180&tc=SSEQTP&tc1=SSCMPDF
If your searches fail to find the problem, then go to "The next step" on page 209 for information about gathering the MustGather documentation for servlet engine and Web container problems.
5.2.6 Symptom: HTTP 500 error - JSP processing error Figure 5-6 shows the diagnostic steps to take when the symptoms of the problem include an HTTP 500 - JSP processing error.
no HTML or servlet displays correctly HTTP 404 - Page not found steps
yes
no JSP Processor starts normally
You should determine whether other resource types such as HTML files or servlets are being requested and displayed correctly. If the failure does not appear to be limited to one JSP file, go to "Symptom: HTTP 404 error - The page cannot be displayed" on page 172.
If other resources are being displayed correctly, determine whether the JSP processor has started normally. Browse the SystemOut.log file of the server that hosts the JSP files that you are trying to access. The messages shown in Example 5-4 indicate that the JSP processor has started normally.
WebContainer A SRVE0239I: Extension Factory [class com.ibm.ws.jsp.webcontainerext.JSPExtensionFactory] was registered successfully. WebContainer A SRVE0240I: Extension Factory [class com.ibm.ws.jsp.webcontainerext.JSPExtensionFactory] has been associated with patterns [*.jsp *.jspx *.jsw *.jsv ].
If the JSP processor fails to load, go to "The next step" on page 209 for information about gathering the MustGather documentation for JSP exceptions.
Look for JSP code errors If the JSP processor starts normally, the problem might be with the JSP file itself. The JSP might have invalid JSP syntax that cannot be processed by the JSP processor. To look for JSP code errors: Examine the SystemOut.log file of the target application server for invalid JSP directive syntax messages. Errors similar to that shown in Example 5-5 indicate this kind of problem.
Message: /test.jsp(2,1)JSPG0076E: Missing required attribute page for jsp element jsp:include
Examine the SystemOut.log file for problems with invalid Java syntax. Errors that contain text similar to "failed to compile (shown in Example 5-6) indicate this kind of problem.
com.ibm.ws.jsp.JspCoreException: JSPG0049E: /test.jsp failed to compile : JSPG0091E: An error occurred at line: 16 in the file: /test.jsp JSPG0093E: Generated servlet error from file: /test.jsp
http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/ com.ibm.websphere.messages.doc/doc/JSPG.html Correct the error in the JSP file and retry the file.
If you have not identified the problem, check to see if the problem has been documented by looking at the available online support (hints and tips, technotes, and fixes) for JSP problems at: http://www-1.ibm.com/support/search.wss?rs=180&tc=SSEQTP&tc1= SSCC2GL&rankprofile=8&dc=DB520+D800+D900+DA900+DA800&dtm
If the searches fail to resolve your problem, go to "The next step" on page 209 for information about gathering the MustGather documentation for JSP exceptions.
5.2.7 Symptom: HTTP 500 error - IllegalStateException If you have an illegal state exception error, the exception should give you an indication of the illegal state causing the problem. This section addresses the following: Invalid session object Response generation problems
The session manager component uses the HttpSession interface to create a session between an HTTP client and the server. When a new session object is created, a unique session ID is assigned to it. The session ID, which is then passed as part of every request, matches the user with the session object. Session tracking gives servlets the ability to maintain state and user information across multiple requests.
"Session timeout interval" on page 207) or can be ended explicitly by application code. The HttpSession interface provides the following method to terminate a session explicitly: public void invalidate();
When a session terminates, the session object and the information that is stored in it are lost permanently. The session manager unbinds any objects bound to the session before it destroys the session.
no Invalid session object ? Remove object from the session yes Invalidate session object Throw IllegalStateException
When the application tries to use an invalidated session object, the IllegalStateException occurs, as shown in Example 5-7.
[7/7/05 16:41:30:627 ART] 00000028 ServletWrappe E SRVE0068E: Could not invoke the service() method on servlet TestServlet. Exception thrown : java.lang.IllegalStateException: Session Object Internals: id : pi55X7syi-ExTjyyhFn5Cu7 hashCode : 1449590567 create time : Thu Jul 07 16:24:54 ART 2005 last access : Thu Jul 07 16:41:30 ART 2005 max inactive interval : 1800 user name : anonymous valid session : false new session : true overflowed : false non-serializable app specific session data : {} serializable app specific session data : {} at com.ibm.ws.webcontainer.httpsession.SessionData.getValueGuts(SessionData.java(C ompiled Code)) at com.ibm.ws.webcontainer.httpsession.SessionData.getValue(SessionData.java(Inlin ed Compiled Code)) at com.ibm.ws.webcontainer.httpsession.SessionData.getAttribute(SessionData.java(I nlined Compiled Code)) at com.ibm.ws.webcontainer.httpsession.HttpSessionFacade.getAttribute(HttpSessionF acade.java(Compiled Code))
Check the servlet or JSP code that threw the exception to make sure the session is not being invalidated too early. If that is not the case, check the session timeout interval to ensure it is not too short.
If these steps do not identify the problem, the following resources can be of help: Review the Java Servlet Specification Version 2.4, Section SRV.15.1.7 related to the HttpSession interface and methods definitions, to obtain more details for IllegalStateException creation causes: http://jcp.org/aboutJava/communityprocess/final/jsr154/index.html
Look up the extended error definitions for session and user profiles messages: http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/ com.ibm.websphere.messages.doc/doc/SESN.html
If these actions do not identify the problem, go to "The next step" on page 209 forinformation about gathering the MustGather documentation for session andsession management problems.
When an HTTP request from a client is delegated to a servlet, the service()method of the HttpServlet class is invoked. The HttpServlet class adds additionalmethods, such as doGet(), doPost(), doPut() and doHead(), for HTTP-basedrequest processing. Figure 5-8 illustrates the life cycle of the service() method.
No Response already committed ? Display page in browser Response already committed ?
Yes
exception is thrown by the HttpServletResponse interface during the responsegeneration process (Example 5-8 on page 184). If the response has beencommitted, you cannot execute any method that is related to
HttpServletResponse object modification. For example, if you have written something in the response buffer, you cannot forward a page using the RequestDispacher interface methods.
[7/8/05 20:36:25:694 ART] 0000004f ServletWrappe E SRVE0068E: Could not invoke the service() method on servlet TestServlet. Exception thrown : java.lang.IllegalStateException at com.ibm.ws.webcontainer.webapp.WebAppDispatcherContext.sendRedirect(WebAppDispa tcherContext.java:486) at com.ibm.ws.webcontainer.srt.SRTServletResponse.sendRedirect(SRTServletResponse. java:810) at web.TestServlet.doGet(TestServlet.java:56) at javax.servlet.http.HttpServlet.service(HttpServlet.java:743) at javax.servlet.http.HttpServlet.service(HttpServlet.java:856)
rolled back due to setRollbackOnly() being called. [7/8/05 20:36:25:774 ART] 0000004f WebApp E SRVE0026E: [Servlet Error]-[TestServlet]: java.lang.IllegalStateException at com.ibm.ws.webcontainer.webapp.WebAppDispatcherContext.sendRedirect(WebAppDispa tcherContext.java:486) at com.ibm.ws.webcontainer.srt.SRTServletResponse.sendRedirect(SRTServletResponse. java:810) at web.TestServlet.doGet(TestServlet.java:56) at javax.servlet.http.HttpServlet.service(HttpServlet.java:743) at javax.servlet.http.HttpServlet.service(HttpServlet.java:856)
status. Response already committed.
java.lang.IllegalStateException are the following calls when the response has already been committed: Calling setBufferSize() Calling ServletResponse.reset() or ServletResponse.resetBuffer() Calling either HttpServletResponse.sendError() or HttpServletResponse.sendRedirect(). Calling RequestDispatcher.forward(), which includes performing a jsp:forward
Note: Remember that if you call forward() or sendRedirect() in your code, any lines of code following these will still execute.
java.lang.IllegalStateException: Header already sent One or more headers have been committed to the client, so you cannot set that header again. java.lang.IllegalStateException: Cannot forward as Output Stream or Writer has already been obtained The calling servlet has called response.getWriter() or response.getOutputStream(). Because the response has been written, it is unsuitable for forwarding.
If you still have not identified the cause of the problem, see the Java ServletSpecification Version 2.4 at the following URL to obtain more details forIllegalStateException generation causes in response generation process:http://jcp.org/aboutJava/communityprocess/final/jsr154/index.html
SRV.14.2.5 RequestDispatcher interface SRV.14.2.16 ServletRequest interface SRV.14.2.22 ServletResponse interface
If these steps do not resolve your problem, go to "The next step" on page 209 forinformation about gathering the MustGather documentation for servlet engineand Web container problems.
5.3 Analyzing problem areas Your analysis of the data you gathered will most likely lead you to one of the following areas. If not, please see "The next step" on page 209.
In a new installation or new application, it is possible that the URL of the application is not being specified correctly. To determine the URL of the installed application, you need to use the administrative console to view the configuration of a number of items.
To find the host and port numbers that are valid for the virtual host:1. Select Environment → Virtual Hosts.2. Choose the virtual host, and then under Additional Properties, click Host Aliases.3. The list contains the host name and port combinations that can be used to access this virtual host (Figure 5-10). The host column should contain values that are registered in a DNS server as a host name for the WebSphere server. An asterisk (*) in the host column indicates that any name can be used. In this case, use the server host name. If port 80 is listed, usually the request is being forwarded from a Web server. (The user specified the URL of the Web server, which is normally 80.) The WC_defaulthost (for example, 9080) and WC_defaulthost_secure (for example, 9443) ports for the application server should also be listed. You can see the corresponding ports by looking at the list of ports in the Communications section of the application server. Use either the host/port combination that accesses the virtual host through the Web server or the host/port combination for using WC_* ports to access the application directly through the application server.
Figure 5-10 Finding the alias and ports for the virtual host
If you need SSL, you can check the Web container transport chains for the corresponding ports to see if SSL has been enabled by doing the following: a. Select Servers → Application servers. b. Select the application server name. c. Under Container Settings, open the Web Container Settings list. d. Click Web container transport chains and check the status and SSL enablement for the port that is specified in the virtual host alias.
Important: Remember that the Web container context root value "/" is used by the sample application named DefaultApplication. For this reason, if you need use this value and this sample application is installed, first uninstall the sample application, install your application, and regenerate the Web server plug-in.
If the browser displays text output that is related to a JSP or servlet Web page but images or HTML files are not displayed, you could have a problem in the Web module packaging or in how the files are referenced within the application. Another possibility is that the file serving enablement feature needs to be turned on for your application.
The first step is to verify that your files are in the right place and that the document root directory of the Web application module follows the J2EE standard, that is that the document root is in the .war directory of the deployed application ear file. Typically this directory is in this location: /profiles//.ear/ .war/
If the image files are in a subdirectory of the document root, verify that the reference to the image reflects that, as shown in Example 5-11.
File Location: /images/test.gif Correct HTML tag:Incorrect HTML tag:
Note: Do not place files to be served to the client in the WEB-INF directory.
File serving allows a Web application to serve static file types (HTML, images,and style sheets) using the enable file servlet, also known as the file servingservlet or file serving enabler. This servlet serves up any resource file that ispackaged in the WAR file, and the file serving enabled attribute is set to true bydefault. For more information about this feature, see:http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com.ibm.websphere.base.doc/info/aes/ae/cweb_flserv.html
true when configuring the Web module. If it is set to false, the Web serverplug-in does not send requests for static content to the application server butleaves it up to the Web server to serve them. This is sometimes done forperformance reasons to ensure that the content is served by the Web server.Serving the page from the Web server provides a shorter path to the page andusually provides more customization options than the file servlet can offer.
Customizing SimpleFileServlet to disable file serving at: http://www.ibm.com/support/docview.wss?uid=swg21116838 Supported assembly tools in WebSphere Application Server V6: http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/ com.ibm.websphere.base.doc/info/aes/ae/catk_assemblytools.html
file at:/profiles//installedApps//.ear\.war/WEB-INF/ibm-web-ext.xmi
Developer tool:1. Go to the J2EE Hierarchy view (Project Explorer in Rational), and select the target Web application module.2. Double-click the Web deployment descriptor (web.xml), and select the Extensions tab to see the IBM Web module extensions (Figure 5-12 on page 192).3. Under the General section, select or deselect File serving enabled to enable or disable the static file serving.
4. Save the Web deployment descriptor file. 5. Redeploy the Web module. 6. Regenerate the Web server plug-in, and propagate it to the Web server. 7. Stop and restart the Web server or allow enough time for the new plug-in configuration to be reloaded. 8. Retry the Web request.
If, after modifying and saving a servlet or JSP file, the change does not show up in the browser, you need to check the reload settings in the Web module configuration and the JSP runtime reload settings.
Web module reloadingFor Web resources such as servlets and JSPs, the Web container reloads a Webmodule only when the IBM extension reloadingEnabled in the ibm-web-ext.xmifile is set to true. You can set reloadingEnabled to true when editing the Webmodule's extended deployment descriptors in an assembly tool.
Application Developer:1. Go to the J2EE Hierarchy view (Project Explorer for Rational) and select the target Web application module.2. Double-click the Web deployment descriptor (web.xml), and select the Extensions tab to see the IBM Web module extensions (Figure 5-12 on page 192).3. In the General section, check the Reloading Enabled flag and the Reload Interval value.4. Select Reloading Enabled, or if it is already selected, then set the Reload Interval lower.5. Save the Web deployment descriptor file.6. Redeploy the Web module.7. Regenerate the Web server plug-in and propagate it to the Web server.8. Stop and restart the Web server or wait for the new plug-in file to take effect.9. Retry the Web request.
You have the ability to modify the JSP processor behavior for different JSPstages (such as development, testing, or production environments). This done byconfiguring specific attributes in the IBM Web module extensions that affect theJSP runtime reload behavior.
To review these settings with the Application Server Toolkit or Rational Application Developer: 1. Go to the J2EE Hierarchy view (Project Explorer for Rational), and select the target Web application module. 2. Double-click in Web deployment descriptor (web.xml), and select the Extensions tab. 3. Under the JSP Attributes section (Figure 5-13), click Add. 4. Enter reloadEnabled in the Name field, true in the Value field, and click Finish. 5. Save the Web deployment descriptor file. 6. Redeploy the Web module. 7. Regenerate the Web server plug-in and verify it is working (see "Verify that the Web server plug-in is working" on page 173). 8. Stop and restart the Web server. 9. Retry the Web request.
The JSP engine settings are stored in ibm-web.ext.xmi, as shown in Example 5-13.
Encoding and internationalization problems usually surface as garbage characters in Web pages (Figure 5-14 on page 196). Another common symptom is that input from the user is interpreted incorrectly.
Figure 5-14 Web page with invalid character encoding settings
Web components usually use java.io.PrintWriter object to produce responses; PrintWriter automatically encodes using ISO 8859-1 character encoding. Servlets can also output binary data using java.io.OutputStream objects, which perform no encoding. An application that cannot use the default encoding must explicitly set a different encoding.
Request encodingRequest encoding is the character encoding in which parameters in an incomingrequest are interpreted. Currently, many browsers do not send a requestencoding qualifier with the content-type HTTP header. In such cases, a Webcontainer uses the default encoding: ISO-8859-1 to parse request data. If theclient has not set character encoding and the request data is encoded with adifferent encoding from the default, the data will not be interpreted correctly.
method to override the character encoding supplied by the container, as shownin Example 5-14.
public class TestServlet extends HttpServlet {
throws ServletException, IOException {
You must call the method before parsing any request parameters or reading anyinput from the request. Calling the method or tag after data has been read will notaffect the encoding.
encoding of only the file that physically contains the page directive.
encoding. Calls made after the getWriter() method has been called or after the response is committed have no effect on the character encoding.
When using a Model-View-Controller architecture, including the use of Struts, consider the following for encoding.
Request character encodingIn a Struts environment, call the setCharacterEncoding() method in theActionForm, as shown in Example 5-16.
.public class PostMessageForm extends ActionForm { public void reset(ActionMapping mapping, HttpServletRequest request) { try { request.setCharacterEncoding("UTF-8"); }catch (UnsupportedEncodingException e) { e.printStackTrace(); } }}.
Specify the response encoding using the contentType attribute with a charsetvalue in the JSP page directive.
WebSphere determines the character encoding used for request/response byparsing the client input values in getParameter() and writing the output per thevalue in the accept-language header of the incoming request.
associated in the encoding.properties file, which is located in the/properties directory (Example 5-17).
There might be cases where you want to override the definition in encoding.properties. For example, when you want to use UTF-8 for the entire application server, use the JVM command line argument client.encoding.override for the selected application server.
field in the Java Virtual Machine section.
http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/ com.ibm.websphere.base.doc/info/aes/ae/trun_svr_utf.html
Starting with WebSphere Application Server Version 5, the Web container no longer sets request and response encodings and response content types automatically. The default value for both extensions is false, then the request and response character encoding is set to the Servlet 2.4 Specification default, which is ISO-8859-1.
Use an assembly tool (Figure 5-12 on page 192) to change the default values for the autoRequestEncoding and autoResponseEncoding extensions.
examples for a description of Web container behavior when these values are set to true: http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/ com.ibm.websphere.base.doc/info/aes/ae/rweb_autoreq.html
SummaryFigure 5-15 shows a summary of how the encoding values are determined.
methods for character encoding yes
autoResponseEncoding values Servlet 2.4 methods JSP 2.0 attributes true
setCharacterEncoding() contextType autoRequestEncoding procedure
setContextType() pageEncoding autoResponseEncoding procedure
If none of these steps fixes your problem, the following resources might behelpful: Globalize your On Demand Business for tips on character encoding: http://www-306.ibm.com/software/globalization/j2ee/encoding.jsp The Java Servlet Specification Version 2.4 for details about the character encoding available methods: http://jcp.org/aboutJava/communityprocess/final/jsr154/index.html The JavaServer Pages Specification Version2.0, section JSP.4 that are related to internationalization issues: http://jcp.org/aboutJava/communityprocess/final/jsr152/index.html Internationalization: Resources for learning http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/ com.ibm.websphere.base.doc/info/aes/ae/rin_resources.html
For current information available from IBM Support on known issues and resolutions that are related to encoding, see: http://www.ibm.com/support/search.wss?rs=180&tc=SSEQTP&tc
PREV: About basic forms - Power Apps | Microsoft Docs
NEXT: Client-Server Architecture - an overview | ScienceDirect Topics
![[Solved] Can't connect to MySql server on localhost (10061) (View topic) * Apache OpenOffice Community Forum](http://website-google-hk.oss-cn-hongkong.aliyuncs.com/drawing/26/2022-3-1/7051.jpeg "[Solved] Can't connect to MySql server on localhost (10061) (View topic) * Apache OpenOffice Community Forum")
