HANA Statisticsserver Down

Hi,

The statistics server in my HANA instance seems to be down. The overall status of my HANA server stays red as well.

What could be the possible reason?

Trace:

12: 0x00007f3d11c80408 in PersistenceLayer::UnbufferedFileNewDb::UnbufferedFileNewDb(PersistenceLayer::StorageLocation const&, PersistenceLayer::STORAGE_TYPE, ltt::basic_string_arg<char, ltt::char_traits<char> > const&, PersistenceLayer::OPEN_MODE, PersistenceLayer::SYNC_MODE)+0x2f4 at NewDbUnbufferedFile.cpp:112 (libhdbpersistence.so)

13: 0x00007f3d11c5dff0 in PersistenceLayer::UnbufferedFile::getFile(PersistenceLayer::StorageLocation const&, PersistenceLayer::STORAGE_TYPE, ltt::basic_string_arg<char, ltt::char_traits<char> > const&, PersistenceLayer::OPEN_MODE, PersistenceLayer::SYNC_MODE)+0x150 at UnbufferedFile.cpp:139 (libhdbpersistence.so)

14: 0x00007f3d11c5e5c2 in PersistenceLayer::UnbufferedFile::writeFile(PersistenceLayer::StorageLocation const&, PersistenceLayer::STORAGE_TYPE, ltt::basic_string_arg<char, ltt::char_traits<char> > const&, ltt::basic_string_arg<char, ltt::char_traits<char> > const&, PersistenceLayer::SYNC_MODE)+0x20 at UnbufferedFile.h:368 (libhdbpersistence.so)

15: 0x00007f3d16c8a7da in TRexConfig::RuntimeData::write(PersistenceLayer::StorageLocation const&, bool, bool)+0x726 at RuntimeData.cpp:305 (libhdbcsmd.so)

16: 0x00007f3d16c8ad95 in TRexConfig::RuntimeData::persist(bool, bool)+0x1d1 at RuntimeData.cpp:265 (libhdbcsmd.so)

17: 0x00007f3d13e2a437 in TRexAPI::TableLoad::updateAndSaveRuntimeData(TrexBase::IndexName const&, TRexConfig::Table const&, TRexConfig::IndexHandle&, bool, TRexCommonObjects::TRexApiError&, PersistenceLayer::StorageLocation const*)+0x703 at TableLoad.cpp:459 (libhdbcsapi.so)

18: 0x00007f3d13e2ab36 in TRexAPI::TableLoad::unloadLocal(TRexConfig::IndexHandle&, TRexCommonObjects::TRexApiError&, bool)+0x272 at TableLoad.cpp:613 (libhdbcsapi.so)

19: 0x00007f3d13f5e90b in TRexAPI::TRexApiSystem::unassign(TRexAPI::EUnassigneMode, TrexBase::IndexName const&, TRexConfig::IndexHandle&, bool)+0x297 at TRexApiSystem.cpp:248 (libhdbcsapi.so)

20: 0x00000000004a6f8f in TRexAPI::TREXIndexServer::unassignAll()+0x12b at TREXIndexServer.cpp:3450 (hdbstatisticsserver)

21: 0x00000000004a7886 in TRexAPI::TREXIndexServer::prepareToShutdown()+0x592 at TREXIndexServer.cpp:1644 (hdbstatisticsserver)

[12186]{0}[-11939026] 2012-11-26 11:07:18.433853 e tableload TableLoad.cpp(00460) : persisting runtimedata failed for _SYS_STATISTICS:_SYS_SPLIT_HOST_VOLUME_IO_STATISTICS~1 (86)

[12186]{0}[-11939026] 2012-11-26 11:07:18.433894 e SERVER_TRACE TRexApiSystem.cpp(00250) : unassign: can't unload index _SYS_STATISTICS:_SYS_SPLIT_HOST_VOLUME_IO_STATISTICS~1 (86): An unknown exception occurred;object=_SYS_STATISTICS:_SYS_SPLIT_HOST_VOLUME_IO_STATISTICS~1en,persisting runtimedata failed(1972)

[12186]{0}[-11939026] 2012-11-26 11:07:18.456922 e RuntimeData RuntimeData.cpp(00309) : _SYS_STATISTICS:_SYS_SPLIT_HOST_VOLUME_IO_STATISTICS~3 (86)persist(): UnbufferedFileException for index "" exception 1: no.70029020 (PersistenceLayer/NewDbUnbufferedFile.cpp:127)

ltt::exception caught while operating on I _SYS_STATISTICS _SYS_SPLIT_HOST_VOLUME_IO_STATISTICS~3 D RuntimeData

exception throw location:

1: 0x00007f3d11c593a1 in PersistenceLayer::UnbufferedFileException::UnbufferedFileException(char const*, int, ltt::basic_string_arg<char, ltt::char_traits<char> > const&, ltt::exception const&)+0x10 at UnbufferedFile.cpp:85 (libhdbpersistence.so)

2: 0x00007f3d11c808cf in PersistenceLayer::UnbufferedFileNewDb::UnbufferedFileNewDb(PersistenceLayer::StorageLocation const&, PersistenceLayer::STORAGE_TYPE, ltt::basic_string_arg<char, ltt::char_traits<char> > const&, PersistenceLayer::OPEN_MODE, PersistenceLayer::SYNC_MODE)+0x7bb at NewDbUnbufferedFile.cpp:127 (libhdbpersistence.so)

3: 0x00007f3d11c5dff0 in PersistenceLayer::UnbufferedFile::getFile(PersistenceLayer::StorageLocation const&, PersistenceLayer::STORAGE_TYPE, ltt::basic_string_arg<char, ltt::char_traits<char> > const&, PersistenceLayer::OPEN_MODE, PersistenceLayer::SYNC_MODE)+0x150 at UnbufferedFile.cpp:139 (libhdbpersistence.so)

4: 0x00007f3d11c5e5c2 in PersistenceLayer::UnbufferedFile::writeFile(PersistenceLayer::StorageLocation const&, PersistenceLayer::STORAGE_TYPE, ltt::basic_string_arg<char, ltt::char_traits<char> > const&, ltt::basic_string_arg<char, ltt::char_traits<char> > const&, PersistenceLayer::SYNC_MODE)+0x20 at UnbufferedFile.h:368 (libhdbpersistence.so)

5: 0x00007f3d16c8a7da in TRexConfig::RuntimeData::write(PersistenceLayer::StorageLocation const&, bool, bool)+0x726 at RuntimeData.cpp:305 (libhdbcsmd.so)

6: 0x00007f3d16c8ad95 in TRexConfig::RuntimeData::persist(bool, bool)+0x1d1 at RuntimeData.cpp:265 (libhdbcsmd.so)

7: 0x00007f3d13e2a437 in TRexAPI::TableLoad::updateAndSaveRuntimeData(TrexBase::IndexName const&, TRexConfig::Table const&, TRexConfig::IndexHandle&, bool, TRexCommonObjects::TRexApiError&, PersistenceLayer::StorageLocation const*)+0x703 at TableLoad.cpp:459 (libhdbcsapi.so)

8: 0x00007f3d13e2ab36 in TRexAPI::TableLoad::unloadLocal(TRexConfig::IndexHandle&, TRexCommonObjects::TRexApiError&, bool)+0x272 at TableLoad.cpp:613 (libhdbcsapi.so)

9: 0x00007f3d13f5e90b in TRexAPI::TRexApiSystem::unassign(TRexAPI::EUnassigneMode, TrexBase::IndexName const&, TRexConfig::IndexHandle&, bool)+0x297 at TRexApiSystem.cpp:248 (libhdbcsapi.so)

10: 0x00000000004a6f8f in TRexAPI::TREXIndexServer::unassignAll()+0x12b at TREXIndexServer.cpp:3450 (hdbstatisticsserver)

11: 0x00000000004a7886 in TRexAPI::TREXIndexServer::prepareToShutdown()+0x592 at TREXIndexServer.cpp:1644 (hdbstatisticsserver)

12: 0x000000000055a19b in PrepareToShutdownThread::run(void*)+0x27 at TrexService.cpp:5175 (hdbstatisticsserver)

13: 0x00007f3d1247385e in TrexThreads::PoolThread::run()+0xb9a at PoolThread.cpp:254 (libhdbbasement.so)

14: 0x00007f3d12474e98 in TrexThreads::PoolThread::run(void*&)+0x14 at PoolThread.cpp:105 (libhdbbasement.so)

15: 0x00007f3d066adac2 in Execution::Thread::staticMainImp(void**)+0x640 at Thread.cpp:448 (libhdbbasis.so)

16: 0x00007f3d066adc8d in Execution::Thread::staticMain(void*)+0x39 at Thread.cpp:507 (libhdbbasis.so)

exception 2: no.1000002 (ltt/impl/memory.cpp:91)

Out of memory ; $size$=262144; $name$=Page; $type$=pool; $inuse_count$=15; $allocated_size$=1376256

exception throw location:

1: 0x00007f3d0514b694 in PageAccess::LogicalPageAccessImpl::loadPage(PageAccess::PageNo const&, ResourceManager::Disposition, ResourceManager::HandleMissingResourceMode)+0x1330 at LogicalPageAccessImpl.cpp:671 (libhdbdataaccess.so)

2: 0x00007f3d0514ccaa in PageAccess::LogicalPageAccess::loadPage(PageAccess::PageNo const&, ResourceManager::Disposition, ResourceManager::HandleMissingResourceMode)+0x16 at LogicalPageAccessImpl.cpp:200 (libhdbdataaccess.so)

3: 0x00007f3d04f216f9 in DataContainer::PageChainContainerSPI::loadPage(PageAccess::PageNo, ResourceManager::Disposition, ResourceManager::HandleMissingResourceMode) const+0x95 at PageChainContainerImpl.cpp:1480 (libhdbdataaccess.so)

4: 0x00007f3d04ef494c in DataContainer::VarSizeEntryContainerImpl::getEntryImpl(DataContainer::VarSizeEntryID const&, DataContainer::VarSizeEntryAddress&, ResourceManager::Disposition, ResourceManager::HandleMissingResourceMode)+0x58 at VarSizeEntryContainerImpl.cpp:621 (libhdbdataaccess.so)

5: 0x00007f3d04eab06b in DataContainer::ContainerDirectoryImpl::getInternal(DataAccess::PersistenceSession&, DataContainer::ContainerID const&, ResourceManager::Disposition, ResourceManager::HandleMissingResourceMode, bool)+0x2b7 at ContainerDirectoryImpl.cpp:2789 (libhdbdataaccess.so)

6: 0x00007f3d04eafba7 in DataContainer::ContainerDirectoryImpl::get(DataAccess::PersistenceSession&, DataContainer::ContainerID const&, ResourceManager::Disposition, ResourceManager::HandleMissingResourceMode)+0x93 at ContainerDirectoryImpl.cpp:955 (libhdbdataaccess.so)

7: 0x00007f3d04eb00a9 in DataContainer::ContainerDirectory::get(DataAccess::PersistenceSession&, DataContainer::ContainerID const&, ResourceManager::Disposition, ResourceManager::HandleMissingResourceMode)+0x15 at ContainerDirectoryImpl.cpp:152 (libhdbdataaccess.so)

8: 0x00007f3d04ec803b in DataContainer::ContainerNameDirectoryImpl::FullEntry::getContainer(DataAccess::PersistenceSession&, ResourceManager::Disposition, ResourceManager::HandleMissingResourceMode)+0x27 at ContainerDirectory.hpp:370 (libhdbdataaccess.so)

9: 0x00007f3d04eda977 in DataContainer::ContainerNameDirectoryImpl::getOrCreate(DataAccess::PersistenceSession&, ltt::basic_string<char, ltt::char_traits<char> > const&, ltt::basic_string<char, ltt::char_traits<char> > const&, DataContainer::ContainerType, ResourceManager::Disposition, bool, void const*, bool)+0x173 at ContainerNameDirectoryImpl.cpp:1166 (libhdbdataaccess.so)

10: 0x00007f3d04edb3b8 in DataContainer::ContainerNameDirectory::getOrCreate(DataAccess::PersistenceSession&, ltt::basic_string<char, ltt::char_traits<char> > const&, ltt::basic_string<char, ltt::char_traits<char> > const&, DataContainer::ContainerType, ResourceManager::Disposition, bool, void const*, bool)+0x34 at ContainerNameDirectoryImpl.cpp:388 (libhdbdataaccess.so)

11: 0x00007f3d11c82d8a in DataContainer::VirtualFile::Handle DataContainer::ContainerNameDirectory::getOrCreate<DataContainer::VirtualFile>(DataAccess::PersistenceSession&, ltt::basic_string<char, ltt::char_traits<char> > const&, ltt::basic_string<char, ltt::char_traits<char> > const&, DataContainer::ContainerType, ResourceManager::Disposition)+0x36 at ContainerNameDirectory.hpp:295 (libhdbpersistence.so)

15: 0x00007f3d16c8a7da in TRexConfig::RuntimeData::write(PersistenceLayer::StorageLocation const&, bool, bool)+0x726 at RuntimeData.cpp:305 (libhdbcsmd.so)

16: 0x00007f3d16c8ad95 in TRexConfig::RuntimeData::persist(bool, bool)+0x1d1 at RuntimeData.cpp:265 (libhdbcsmd.so)

18: 0x00007f3d13e2ab36 in TRexAPI::TableLoad::unloadLocal(TRexConfig::IndexHandle&, TRexCommonObjects::TRexApiError&, bool)+0x272 at TableLoad.cpp:613 (libhdbcsapi.so)

19: 0x00007f3d13f5e90b in TRexAPI::TRexApiSystem::unassign(TRexAPI::EUnassigneMode, TrexBase::IndexName const&, TRexConfig::IndexHandle&, bool)+0x297 at TRexApiSystem.cpp:248 (libhdbcsapi.so)

20: 0x00000000004a6f8f in TRexAPI::TREXIndexServer::unassignAll()+0x12b at TREXIndexServer.cpp:3450 (hdbstatisticsserver)

21: 0x00000000004a7886 in TRexAPI::TREXIndexServer::prepareToShutdown()+0x592 at TREXIndexServer.cpp:1644 (hdbstatisticsserver)

[12186]{0}[-11939026] 2012-11-26 11:07:18.508026 e tableload TableLoad.cpp(00460) : persisting runtimedata failed for _SYS_STATISTICS:_SYS_SPLIT_HOST_VOLUME_IO_STATISTICS~3 (86)

[12186]{0}[-11939026] 2012-11-26 11:07:18.508069 e SERVER_TRACE TRexApiSystem.cpp(00250) : unassign: can't unload index _SYS_STATISTICS:_SYS_SPLIT_HOST_VOLUME_IO_STATISTICS~3 (86): An unknown exception occurred;object=_SYS_STATISTICS:_SYS_SPLIT_HOST_VOLUME_IO_STATISTICS~3en,persisting runtimedata failed(1972)

[12186]{0}[-11939026] 2012-11-26 11:07:18.531200 e RuntimeData RuntimeData.cpp(00309) : _SYS_STATISTICS:_SYS_SPLIT_HOST_VOLUME_IO_STATISTICS~5 (86)persist(): UnbufferedFileException for index "" exception 1: no.70029020 (PersistenceLayer/NewDbUnbufferedFile.cpp:127)

ltt::exception caught while operating on I _SYS_STATISTICS _SYS_SPLIT_HOST_VOLUME_IO_STATISTICS~5 D RuntimeData