1. WAS 버전 : WAS All
2. OS : All
3. Error 발생 시점 : WAS 를 시작하면 애플리케이션 시작 부분에서 Hang 처럼 멈춰 있다 OOM 발생 후 시작 실패
(Dmgr 에서도 동일현상 발생, Heap 을 늘려도 계속적으로 OOM 발생)
4. Error message :
Hang 시점에 javacore 을 연속적으로 생성해보면 하단과 같이 FileSystem 을 읽는 Thread stack 이 계속적으로 증가
3XMTHREADINFO "server.startup : 0" J9VMThread:0x00000000022AF200, j9thread_t:0x00007F0BFA9D81C0, java/lang/Thread:0x00000001026A8170, state:CW, prio=5
3XMTHREADINFO1 (native thread ID:0x6F87, native priority:0x5, native policy:UNKNOWN)
3XMTHREADINFO2 (native stack address range from:0x00007F0B754FD000, to:0x00007F0B7553E000, size:0x41000)
3XMTHREADINFO3 Java callstack:
4XESTACKTRACE at java/io/UnixFileSystem.getBooleanAttributes0(Native Method)
4XESTACKTRACE at java/io/UnixFileSystem.getBooleanAttributes(UnixFileSystem.java:239(Compiled Code))
4XESTACKTRACE at java/io/File.isDirectory(File.java:765(Compiled Code))
4XESTACKTRACE at org/eclipse/jst/j2ee/commonarchivecore/internal/strategy/DirectoryLoadStrategyImpl.addNextFromBinariesDirectory(DirectoryLoadStrategyImpl.java:1369(Compiled Code))
4XESTACKTRACE at org/eclipse/jst/j2ee/commonarchivecore/internal/strategy/DirectoryLoadStrategyImpl.addDirectoryFromBinariesDirectory(DirectoryLoadStrategyImpl.java:1415(Compiled Code))
4XESTACKTRACE at org/eclipse/jst/j2ee/commonarchivecore/internal/strategy/DirectoryLoadStrategyImpl.addNextFromBinariesDirectory(DirectoryLoadStrategyImpl.java:1370(Compiled Code))
4XESTACKTRACE at org/eclipse/jst/j2ee/commonarchivecore/internal/strategy/DirectoryLoadStrategyImpl.addDirectoryFromBinariesDirectory(DirectoryLoadStrategyImpl.java:1415(Compiled Code))
.........................
4XESTACKTRACE at org/eclipse/jst/j2ee/commonarchivecore/internal/strategy/DirectoryLoadStrategyImpl.addNextFromBinariesDirectory(DirectoryLoadStrategyImpl.java:1370)
4XESTACKTRACE at org/eclipse/jst/j2ee/commonarchivecore/internal/strategy/DirectoryLoadStrategyImpl.collectFilesFromBinaries(DirectoryLoadStrategyImpl.java:1287)
4XESTACKTRACE at org/eclipse/jst/j2ee/commonarchivecore/internal/strategy/LoadStrategyImpl.collectFiles(LoadStrategyImpl.java:989)
4XESTACKTRACE at org/eclipse/jst/j2ee/commonarchivecore/internal/impl/ContainerImpl.getFiles(ContainerImpl.java:443)
4XESTACKTRACE at org/eclipse/jst/j2ee/commonarchivecore/internal/impl/ContainerImpl.getFileNoException(ContainerImpl.java:910)
4XESTACKTRACE at org/eclipse/jst/j2ee/commonarchivecore/internal/impl/ArchiveImpl.maybeRemoveFile(ArchiveImpl.java:4864)
4XESTACKTRACE at com/ibm/etools/commonarchive/impl/WARFileImpl.patchXMIFiles(WARFileImpl.java:2326)
4XESTACKTRACE at com/ibm/etools/commonarchive/impl/WARFileImpl.patchFiles(WARFileImpl.java:2311)
4XESTACKTRACE at org/eclipse/jst/j2ee/commonarchivecore/internal/impl/ArchiveImpl.initializeAfterOpen(ArchiveImpl.java:2475)
4XESTACKTRACE at org/eclipse/jst/j2ee/commonarchivecore/internal/helpers/ArchiveTypeDiscriminatorImpl.openSpecificArchive(ArchiveTypeDiscriminatorImpl.java:445)
4XESTACKTRACE at org/eclipse/jst/j2ee/commonarchivecore/internal/impl/RootWarDescriminatorImpl.openSpecificArchive(RootWarDescriminatorImpl.java:57)
4XESTACKTRACE at org/eclipse/jst/j2ee/commonarchivecore/internal/impl/RootWarDescriminatorImpl.openSpecificArchive(RootWarDescriminatorImpl.java:31)
4XESTACKTRACE at org/eclipse/jst/j2ee/commonarchivecore/internal/helpers/ArchiveTypeDiscriminatorImpl.fullOpenArchive(ArchiveTypeDiscriminatorImpl.java:413)
4XESTACKTRACE at org/eclipse/jst/j2ee/commonarchivecore/internal/impl/RootWarDescriminatorImpl.fullOpenArchive(RootWarDescriminatorImpl.java:52)
4XESTACKTRACE at org/eclipse/jst/j2ee/commonarchivecore/internal/impl/CommonarchiveFactoryImpl.openWARFile(CommonarchiveFactoryImpl.java:1663)
4XESTACKTRACE at org/eclipse/jst/j2ee/commonarchivecore/internal/impl/EARFileImpl.openNestedArchive(EARFileImpl.java:1448)
4XESTACKTRACE at org/eclipse/jst/j2ee/commonarchivecore/internal/impl/EARFileImpl.openNestedArchive(EARFileImpl.java:1388)
4XESTACKTRACE at org/eclipse/jst/j2ee/commonarchivecore/internal/strategy/LoadStrategyImpl.openNestedArchive(LoadStrategyImpl.java:1461)
4XESTACKTRACE at org/eclipse/jst/j2ee/commonarchivecore/internal/strategy/LoadStrategyImpl.createFile(LoadStrategyImpl.java:492)
4XESTACKTRACE at org/eclipse/jst/j2ee/commonarchivecore/internal/strategy/DirectoryLoadStrategyImpl.addFile(DirectoryLoadStrategyImpl.java:665)
4XESTACKTRACE at org/eclipse/jst/j2ee/commonarchivecore/internal/strategy/DirectoryLoadStrategyImpl.addFiles(DirectoryLoadStrategyImpl.java:738)
4XESTACKTRACE at org/eclipse/jst/j2ee/commonarchivecore/internal/strategy/DirectoryLoadStrategyImpl.addFiles(DirectoryLoadStrategyImpl.java:677)
4XESTACKTRACE at org/eclipse/jst/j2ee/commonarchivecore/internal/strategy/DirectoryLoadStrategyImpl.getFiles(DirectoryLoadStrategyImpl.java:935)
4XESTACKTRACE at org/eclipse/jst/j2ee/commonarchivecore/internal/strategy/LoadStrategyImpl.collectFiles(LoadStrategyImpl.java:961)
4XESTACKTRACE at org/eclipse/jst/j2ee/commonarchivecore/internal/impl/ContainerImpl.getFiles(ContainerImpl.java:443)
4XESTACKTRACE at org/eclipse/jst/j2ee/commonarchivecore/internal/impl/EARFileImpl.getFiles(EARFileImpl.java:2693)
4XESTACKTRACE at org/eclipse/jst/j2ee/commonarchivecore/internal/impl/ContainerImpl.getFileNoException(ContainerImpl.java:910)
4XESTACKTRACE at org/eclipse/jst/j2ee/commonarchivecore/internal/impl/ArchiveImpl.maybeRemoveFile(ArchiveImpl.java:4864)
4XESTACKTRACE at com/ibm/etools/commonarchive/impl/EARFileImpl.patchXMIFiles(EARFileImpl.java:1021)
4XESTACKTRACE at com/ibm/etools/commonarchive/impl/EARFileImpl.patchFiles(EARFileImpl.java:1004)
4XESTACKTRACE at org/eclipse/jst/j2ee/commonarchivecore/internal/impl/ArchiveImpl.initializeAfterOpen(ArchiveImpl.java:2475)
4XESTACKTRACE at org/eclipse/jst/j2ee/commonarchivecore/internal/impl/EARFileImpl.initializeAfterOpen(EARFileImpl.java:1250)
4XESTACKTRACE at org/eclipse/jst/j2ee/commonarchivecore/internal/helpers/ArchiveTypeDiscriminatorImpl.openSpecificArchive(ArchiveTypeDiscriminatorImpl.java:445)
4XESTACKTRACE at org/eclipse/jst/j2ee/commonarchivecore/internal/impl/RootEARFileDescriminatorImpl.openSpecificArchive(RootEARFileDescriminatorImpl.java:64)
4XESTACKTRACE at org/eclipse/jst/j2ee/commonarchivecore/internal/impl/RootEARFileDescriminatorImpl.openSpecificArchive(RootEARFileDescriminatorImpl.java:31)
4XESTACKTRACE at org/eclipse/jst/j2ee/commonarchivecore/internal/helpers/ArchiveTypeDiscriminatorImpl.fullOpenArchive(ArchiveTypeDiscriminatorImpl.java:413)
4XESTACKTRACE at org/eclipse/jst/j2ee/commonarchivecore/internal/impl/RootEARFileDescriminatorImpl.fullOpenArchive(RootEARFileDescriminatorImpl.java:53)
4XESTACKTRACE at org/eclipse/jst/j2ee/commonarchivecore/internal/impl/CommonarchiveFactoryImpl.openEARFile(CommonarchiveFactoryImpl.java:1599)
4XESTACKTRACE at org/eclipse/jst/j2ee/commonarchivecore/internal/impl/CommonarchiveFactoryImpl.openEARFile(CommonarchiveFactoryImpl.java:1573)
4XESTACKTRACE at com/ibm/etools/commonarchive/impl/CommonarchiveFactoryImpl.openEARFile(CommonarchiveFactoryImpl.java:617)
4XESTACKTRACE at com/ibm/ws/runtime/component/DeployedApplicationImpl$MappableContentsArchive.open(DeployedApplicationImpl.java:1662)
4XESTACKTRACE at com/ibm/ws/runtime/component/DeployedApplicationImpl$MappableContentsArchive.access$000(DeployedApplicationImpl.java:1584)
4XESTACKTRACE at com/ibm/ws/runtime/component/DeployedApplicationImpl.initialize(DeployedApplicationImpl.java:604)
4XESTACKTRACE at com/ibm/ws/runtime/component/DeployedApplicationImpl.initialize(DeployedApplicationImpl.java:432)
4XESTACKTRACE at com/ibm/ws/runtime/component/ApplicationMgrImpl.initializeApplication(ApplicationMgrImpl.java:395)
5XESTACKTRACE (entered lock: com/ibm/ws/runtime/component/ApplicationMgrImpl@0x000000010152CF30, entry count: 1)
4XESTACKTRACE at com/ibm/ws/runtime/component/ApplicationMgrImpl.start(ApplicationMgrImpl.java:2169)
4XESTACKTRACE at com/ibm/ws/runtime/component/CompositionUnitMgrImpl.start(CompositionUnitMgrImpl.java:445)
4XESTACKTRACE at com/ibm/ws/runtime/component/CompositionUnitImpl.start(CompositionUnitImpl.java:123)
4XESTACKTRACE at com/ibm/ws/runtime/component/CompositionUnitMgrImpl.start(CompositionUnitMgrImpl.java:388)
4XESTACKTRACE at com/ibm/ws/runtime/component/CompositionUnitMgrImpl.access$500(CompositionUnitMgrImpl.java:116)
4XESTACKTRACE at com/ibm/ws/runtime/component/CompositionUnitMgrImpl$CUInitializer.run(CompositionUnitMgrImpl.java:994)
4XESTACKTRACE at com/ibm/wsspi/runtime/component/WsComponentImpl$_AsynchInitializer.run(WsComponentImpl.java:349)
4XESTACKTRACE at com/ibm/ws/util/ThreadPool$Worker.run(ThreadPool.java:1814)
5. 예상 이유 :
해당 Application 이 설치된 WAR 폴더에 특정 목적으로 여러 개의 symbolic link 를 사용하면서 recursive 형태로 loop 형태가 나올때
A -> B -> C -> A -> B -> C .....
WAS 가 Application 을 load 하기 위해 directory 를 타고 search 하는데 loop 에 걸려 빠져나오지 못함
6. 조치 방안 :
문제가 되는 loop 형태의 symbolic link 를 삭제
- 참고 : IBM WAS 는 WAR 내에 symbolic link 사용을 권장하지 않으며 Extended Document Root 를 사용하길 권고합니다.
http://www-01.ibm.com/support/docview.wss?uid=swg1PK87049
IBM - old/WAS 문제&해결
댓글