Wednesday, November 8, 2017

Accessing the last element of an Intrusive list

I always thought that retrieving the last element from an intrusive list just as easy as one two three. In fact, it is not. Let's dive into the story. Here I have the intrusive list declaration:
/***** FileNode.h *****/

class FileNode : public boost::intrusive::list_base_hook<link_mode<auto_unlink> >
{
...
...

};


/***** FileBot.h *****/

typedef boost::intrusive::list<FileNode, base_hook<list_base_hook<link_mode<auto_unlink> > >, constant_time_size<false> > FileNodeListType;

class FileBot
{
private:
    FileNodeListType fileList;

...
...
};
And then I'll use the iterator as shown below go straight to access the element located at the end of the list, but somehow this a hit segmentation fault error.
        FileNodeListType::iterator it(fileList.end());

        // following piece basically doing some data manipulation
        // on the element retrieve from the list.
        FileNode *p = &*it;
        (*it).sibling.push_back(*node);

        cout << p << endl; // memory address shows 0x7ffedff8e808
The error is due to the wrong memory address is being accessed. As I verify on the element being inserted into the list (only one element is inserted in this test case) is having an address different from the one mention in the code above. Thus, according to the expert, in order to access the correct element located at the end of the list is by doing this:
        FileNodeListType::iterator it(fileList.end());
        FileNode *p = &*(--it);
        cout << p << endl; // memory address shows 0xa01bc0

Monday, October 30, 2017

Remember to clear the memory hold in Boost recursive intrusive

While I'm studying on making an intrusive container right inside an object which which is also an intrusive container. Some sort of nested intrusive container I might think of. Well, I never thought Boost really made one for us, this feature is called recursive intrusive container.

There is some issue with this container. It happens only on Windows, no problem in Linux at all. When removing an object from an intrusive container, make sure the nested intrusive container is clear off before the object is removed. I have run some test case about it.

Here is the declaration of the object for the test case, the sibling member is the recursive intrusive container:
class FileNode : public boost::intrusive::list_base_hook<link_mode<auto_unlink> >
{
public:
    boost::intrusive::list<FileNode, base_hook<list_base_hook<link_mode<auto_unlink> > >, constant_time_size<false> > sibling;

...
...
};
This is the intrusive list declaration:
typedef boost::intrusive::list<FileNode, base_hook<list_base_hook<link_mode<auto_unlink> > >, constant_time_size<false> > FileNodeListType;

class FileBot
{
public:
    FileNodeListType fileList;
 
First, an object was pumped into the intrusive container, fileList in this case. Then allocate another object into recursive list, sibling. I loop through every object in the fileList and put in another object into the recursive list.
void FileBot::addFileNode()
{
    FileNode *f1 = new FileNode();
    f1->setName("ASD");
    fileList.push_back(*f1);
}

void FileBot::addSubFileNode()
{
    FileNode *f2 = NULL;

    for(FileNode &T : fileList)
    {
        f2 = new FileNode();
        f2->setName("QWE");
        T.sibling.push_back(*f2);
    }
}
Now I want to clear everything in the fileList, without knowing there is any other object has been saved into the recursive intrusive container yet.
void FileBot::clearMemory()
{
   fileList.erase_and_dispose(fileList.begin(), fileList.end(), DisposeFileNode());
}
When this method is called, memory leaks will be detected. This happens only on Windows. How sure am I? I run through every single line of the code to find out the root cause. So in order to play safe in this game, I'll do this:
void FileBot::clearMemory()
{
    for(FileNode &T : fileList) {
        T.sibling.erase_and_dispose(T.sibling.begin(), T.sibling.end(), DisposeFileNode());
    }

    fileList.erase_and_dispose(fileList.begin(), fileList.end(), DisposeFileNode());
}
Clean off the memory in the recursive intrusive container before cleaning the memory in the parent intrusive container. This should be the strategy for this game.

Sunday, October 29, 2017

Removing an item from the Boost Intrusive list

Over the pass few weeks, I was reading the a lot of articles about the intrusive list, its benefits of using it, and its use case example on game development. This brings me a strong message that there are so many peoples on Earth promoting this container. What was so special about this container? Due to the limitation on the container provided by STL and intrusive container's low memory footprint which could benefit to the application's performance.

Does Boost provide this container. Yes! During this great moment with a sudden impulse, I begin my journey to develop a demo program using the Boost Intrusive container. Tadaa~

So far the journey was up and down, a lot of road pump, and feel tired to continue. When trying to remove an item from the list using following method will not work. Before that, let's presume I have the following declaration in the header file:
typedef boost::intrusive::list<FileNode, base_hook<list_base_hook<link_mode<auto_unlink> > >, constant_time_size<false> > FileNodeListType;

class FileBot
{
private:
    FileNodeListType fileList;

    ...
    ...
};
Then I begin to insert something into the list, assume the object type is a FileNode, do take note that I do this purely in one method:
void FileBot::addFileNode()
{
    FileNode *f1 = new FileNode();
    f1->setName("ASD");
    fileList.push_back(*f1);

    cout << "addFileNode(): memory address of insert object: " << f1 << endl;
}
Then I have another method to remove an object from the list:
void FileBot::clearMemoryByName(string filename)
{
    cout << "before delete: " << fileList.size() << endl;

    FileNode *p = new FileNode();
    p->setName("ASD");

    fileList.erase(FileNodeListType::s_iterator_to(*p));
    delete p;

    cout << "clearMemoryByName(): memory address of remove object: " << p << endl;

    cout << "after delete: " << fileList.size() << endl;
}
But somehow this method will cause a memory leak due to the following assertion failed in boost/intrusive/list.hpp, line 1270:
Assertion failed: !node_algorithms::inited(value_traits::to_node_ptr(value)), file D:\tool\boost_1_61_0\msvc\boost/intrusive/list.hpp, line 1270
Detected memory leaks!
Dumping objects ->
...
...
...
Object dump complete.
Press <return> to close this window...
A million hours were spent on studying the weak spot of this creature has finally found a solution. The clue was lying on the memory address of the object. The memory address of the object inserted into the list in addFileNode() was different from the memory object created in the clearMemoryByName().
...

addFileNode(): memory address of insert object: 001180C8
clearMemoryByName(): memory address of remove object: 00111DE0

...
In the revise's version, the object was retrieved from the list and then perform some filtering (e.g. match the property field's value), then only perform the deletion.
void FileBot::clearMemoryByName(string filename)
{
    FileNode *p = NULL;
    FileNodeListType::iterator it(fileList.begin());

    for(; it != fileList.end(); it++)
    {
        cout << &*it << endl;

        // look for particular node
        string curName = (*it).getName().c_str();
        if( filename == curName) {
            p = &*it;
            break;
        }
    }

    cout << "claerMemoryByName(): memory address of remove object: " << p << endl;

    fileList.erase(FileNodeListType::s_iterator_to(*p));
    delete p;
The result was stunning! The object was removed from the list. And here is the output of the memory address:
...

addFileNode(): memory address of insert object: 00E2DAF0
claerMemoryByName(): memory address of remove object: 00E2DAF0

...
To conclude this, in order to remove an object from the list, O(n) algorithm is the only way to filter the object, then only come to deletion.

Saturday, October 14, 2017

Convert Windows file separator into Linux format

I wrote a function to extract the parent path for a given path. An interesting thing happens here; Windows's file system was used (\) backslash as file separator, whereas Linux's using (/) slash as file separator. Thus, this will end up I have following code produce:
void FileBot::constructParentPath()
{
   ...

   string sep = string(1, boost::filesystem::path::preferred_separator);
   if( sep == "/" )
      boost::replace_last(filePath, "/", "|");
   else
      boost::replace_last(filePath, "\", "|");

   ...
}
When handling file separator with Boost library, the library will able to convert the Windows's file separator into Linux format. Therefore, my code will reduce into one line from the previous version:
void FileBot::constructParentPath()
{
   ...

   string filePath = fileNode.getName().generic_string();
   boost::replace_last(filePath, "/", "|");

   ...
}

Sunday, September 24, 2017

A problem with the SOAP logical tree format

Just one small step will complete my finding. It was the web service development on Message Broker. Ever wonder how things done and I always got an excuse to delay my finding.

Say I have a WSDL, which consist of following:
...

<wsdl:types>
  <xsd:schema xmlns:xsd="http://www.w3.org/2001/XMLSchema" targetNamespace="http://helloWorld.huahsin.org/">
    <xsd:element name="Input">
      <xsd:complexType>
        <xsd:sequence>
          <xsd:element name="name" type="xsd:string" minOccurs="1"/>
          <xsd:element name="age" type="xsd:integer" minOccurs="0"/>
        </xsd:sequence>
      </xsd:complexType>
    </xsd:element>
    <xsd:element name="Output">
      <xsd:complexType>
        <xsd:sequence>
          <xsd:element name="greetings" type="xsd:string" minOccurs="1"/>
        </xsd:sequence>
      </xsd:complexType>
    </xsd:element>
  </xsd:schema>
</wsdl:types>

...
Ignore those unnecessary stuff, let's focus only on the input and output. The above WSDL shows the following request input:
<soap:Envelope xmlns:soap="http://www.w3.org/2003/05/soap-envelope" xmlns:hel="http://helloWorld.huahsin.org/">
   <soap:Header/>
   <soap:Body>
      <hel:Input>
         <name>LKJ</name>
         <!--Optional:-->
         <age>0</age>
      </hel:Input>
   </soap:Body>
</soap:Envelope>
And the expected response message:
<soap:Envelope xmlns:soap="http://www.w3.org/2003/05/soap-envelope" xmlns:hel="http://helloWorld.huahsin.org/">
   <soap:Body>
      <hel:Output>
         <greetings>Hello World</greetings>
      </hel:Output>
   </soap:Body>
</soap:Envelope>
That's all about it. Somehow, things always out of expectation. I got this:
<soapenv:Envelope xmlns:soapenv="http://www.w3.org/2003/05/soap-envelope">
   <soapenv:Body>
      <soapenv:Fault>
         <soapenv:Code>
            <soapenv:Value>soapenv:Receiver</soapenv:Value>
         </soapenv:Code>
         <soapenv:Reason>
            <soapenv:Text xml:lang="en">BIP3113E: Exception detected in message flow Main.SOAP Input (broker brk2)</soapenv:Text>
         </soapenv:Reason>
         <soapenv:Node>://localhost:7080/HelloWorld</soapenv:Node>
         <soapenv:Detail>
            <Text>BIP3752E: The SOAP Reply node 'Main.SOAP Reply' encountered an error while processing a reply message. 
An error occurred during reply message processing. 
See previous error messages to determine the cause of the error. : F:\build\slot1\S800_P\src\WebServices\WSLibrary\ImbSOAPReplyNode.cpp: 397: ImbSOAPReplyNode::evaluate: ComIbmSOAPReplyNode: Main#FCMComposite_1_2
BIP3605E: The SOAP logical tree cannot be serialized. 
There is a problem with the SOAP logical tree format. 
Review further error messages for an indication to the cause of the error. Check that the SOAP logical supplied is correctly formatted. : F:\build\slot1\S800_P\src\WebServices\WSLibrary\ImbSOAPParser.cpp: 1360: ImbSOAPParser::refreshBitStreamFromElementsInner: : 
BIP3602E: The Web service payload ''{http://helloWorld.huahsin.org/}Input'' does not match an operation described by WSDL binding ''HelloWorldBinding'' in file ''huahsin.wsdl''. 
The first child of the SOAP Body does not correspond to any of the operations defined in the specified WSDL definition. 
Check that the correct WSDL definition was deployed. : F:\build\slot1\S800_P\src\WebServices\WSLibrary\ImbSOAPParser.cpp: 790: ImbSOAPParser::refreshBitStreamFromElementsInner: :</Text>
         </soapenv:Detail>
      </soapenv:Fault>
   </soapenv:Body>
</soapenv:Envelope>
At first I was so struggling to work on this error, luckily I got the solution during my work, The more I work, the more knowledge I learn. This was due to the message construction happened in Transform Node:
CREATE COMPUTE MODULE Main_Compute
  CREATE FUNCTION Main() RETURNS BOOLEAN
  BEGIN
    -- CALL CopyMessageHeaders();
    CALL CopyEntireMessage();

    DECLARE hel NAMESPACE 'http://helloWorld.huahsin.org/';

    SET OutputRoot.SOAP.Body.hel:Output.greetings = 'Hello World';

    RETURN TRUE;
  END;

  ...
END MODULE;
The error is from the OutputRoot. OutputRoot is the ultimate destination that the message will be reach at the end of the flow. During the message construction, any message that going to send out a response are advisable to clear them out. Like this:
    ...

    DELETE FIELD OutputRoot.SOAP.Body;
    SET OutputRoot.SOAP.Body.hel:Output.greetings = 'Hello World';

    ...
Leave the SOAP tree empty before any new message is constructed.

Never forget qmake

Oh come on! Error again! I encounter a weird error on QT compiler today. I though I had the thing fixed on last week, wonder why the fix on last week is not working on today? What a joke. Now the error is different. The error mentions that I have multiple definition found. The root cause is coming from the Boost library.
13:12:11: Starting: "D:\tool\Qt\Tools\QtCreator\bin\jom.exe" 
 D:\tool\Qt\Tools\QtCreator\bin\jom.exe -f Makefile.Debug
 cl -c -nologo -Zc:wchar_t -FS -Zc:strictStrings -Zc:throwingNew -Zi -MDd -GR -W3 -w34100 -w34189 -w44996 -w44456 -w44457 -w44458 -wd4577 -wd4467 -EHsc /Fddebug\wqt1.vc.pdb -DUNICODE -DWIN32 -DQT_QML_DEBUG -DQT_CORE_LIB -I..\wqt1 -I. -I..\..\..\tool\boost_1_61_0 -I..\..\tool\Qt\5.7\msvc2015\include -I..\..\tool\Qt\5.7\msvc2015\include\QtCore -Idebug -I..\..\tool\Qt\5.7\msvc2015\mkspecs\win32-msvc2015 -Fodebug\ @C:\Users\HUAHS_~1\AppData\Local\Temp\main.obj.4132.31.jom
main.cpp
 link /NOLOGO /DYNAMICBASE /NXCOMPAT /DEBUG /SUBSYSTEM:CONSOLE "/MANIFESTDEPENDENCY:type='win32' name='Microsoft.Windows.Common-Controls' version='6.0.0.0' publicKeyToken='6595b64144ccf1df' language='*' processorArchitecture='*'" /MANIFEST:embed /OUT:debug\wqt1.exe @C:\Users\HUAHS_~1\AppData\Local\Temp\wqt1.exe.4132.8453.jom
libboost_filesystem-vc140-mt-gd-1_61.lib(path_traits.obj) : error LNK2005: "void __cdecl boost::filesystem::path_traits::convert(char const *,char const *,class std::basic_string<wchar_t std::char_traits="" struct="" wchar_t="">,class std::allocator<wchar_t> > &,class std::codecvt<wchar_t _mbstatet="" char="" struct=""> const &)" (?convert@path_traits@filesystem@boost@@YAXPBD0AAV?$basic_string@_WU?$char_traits@_W@std@@V?$allocator@_W@2@@std@@ABV?$codecvt@_WDU_Mbstatet@@@5@@Z) already defined in boost_filesystem-vc140-mt-gd-1_61.lib(boost_filesystem-vc140-mt-gd-1_61.dll)
libboost_filesystem-vc140-mt-gd-1_61.lib(path.obj) : error LNK2005: "public: static class std::codecvt<wchar_t _mbstatet="" char="" struct=""> const & __cdecl boost::filesystem::path::codecvt(void)" (?codecvt@path@filesystem@boost@@SAABV?$codecvt@_WDU_Mbstatet@@@std@@XZ) already defined in boost_filesystem-vc140-mt-gd-1_61.lib(boost_filesystem-vc140-mt-gd-1_61.dll)
debug\wqt1.exe : fatal error LNK1169: one or more multiply defined symbols found
jom: D:\workspaceqt\build-wqt1-Desktop_Qt_5_7_1_MSVC2015_32bit-Debug\Makefile.Debug [debug\wqt1.exe] Error 1169
jom: D:\workspaceqt\build-wqt1-Desktop_Qt_5_7_1_MSVC2015_32bit-Debug\Makefile [debug] Error 2
How could I resolve them? I am spending hours experiment this error, by making a clean build or start a new project from scratch just to simulate this error have no result. Until late night only found out there is a very important step was missing during the build.

Whenever the PRO file is updated, I'm required to run qmake, because this will allow to update the Makefile generate by the QT. No wonder why sometimes I get an error on undefined reference on blah_blah_blah_functionA() when I add a new library in PRO file.

Sunday, September 10, 2017

Restart is required after switching to broker-wide listener

Just to share some though when I am using mqsicreateexecutiongroup command. First thing first, this command was so powerful that it will create a whole new world in the broker. Each world has a separate memory allocation that wouldn't affect to others when it is failing to operate. This is some sort of web instances in JAVA sense. When it was first created, I did a quit checked on its status, it shows following details.
>mqsireportproperties brk -e ceres -o ExecutionGroup -r

ExecutionGroup
  uuid='6f8ad861-5e01-0000-0080-cb62575bf054'
  userTraceLevel='none'
  traceLevel='none'
  userTraceFilter='none'
  traceFilter='none'
  label='ceres'
  unnamedUserTraceLevel='none'
  unnamedTraceLevel='none'
  processorArchitecture='64'
  consoleMode='off'
  traceNodeLevel='on'
  activeUserExitList=''
  inactiveUserExitList=''
  shortDesc=''
  longDesc=''
  httpNodesUseEmbeddedListener='false'
  soapNodesUseEmbeddedListener='true'

BIP8071I: Successful command completion.
Assuming I have a perfect program without any error being deployed, I can load up the WSDL through HTTP. But this time I receive a page cannot be found error, this indicate that the execution group is still in the embedded listener mode. In order to receive the request through HTTP, I am required to switch to broker-wide listener from embedded listener. Otherwise, my program could not be reached by the consuming system.
>mqsichangeproperties brk -e ceres -o ExecutionGroup -n soapNodesUseEmbeddedListener -v false
BIP8071I: Successful command completion.

>mqsireportproperties brk -e ceres -o ExecutionGroup -r

ExecutionGroup
  uuid='6f8ad861-5e01-0000-0080-cb62575bf054'
  userTraceLevel='none'
  traceLevel='none'
  userTraceFilter='none'
  traceFilter='none'
  label='ceres'
  unnamedUserTraceLevel='none'
  unnamedTraceLevel='none'
  processorArchitecture='64'
  consoleMode='off'
  traceNodeLevel='on'
  activeUserExitList=''
  inactiveUserExitList=''
  shortDesc=''
  longDesc=''
  httpNodesUseEmbeddedListener='false'
  soapNodesUseEmbeddedListener='false'

BIP8071I: Successful command completion.
I did a quick check on its status right after the configuration. Notice that the soapNodesUseEmbeddedListener have a value false. This is good ,but not there yet. I still will get the page cannot be found error when I send in the URL request. One trick is to restart the execution group, by now the WSDL should load up in the browser.