A Success Story of some software integration: Turbo Disaster, or When failure was Not an option...

[SergeyKostrov]

Hello everybody, My name is Sergey Kostrov. I'm an Intel Black Belt Software Developer and am Not an Intel employee. I'd like to share my experience ( a Success Story ) on how some very complex software integration was done a couple of years ago on the ScaLib project. So, It is possible that you've heard a very famous expression Failure is Not an option. Even if the expression was never told by real Mission Control personnel during Apollo 13 mission it is related to those dramatic events. Project managers, flight engineers and controllers never gave up when trying to find solutions of different problems with the spacecraft. In October 2010 I had to deal with a very complex software integration. Shortly speaking, a fate of the project was on my shoulders and Failure was Not an option for me. In modern Information Technology industry software systems are very complex and even a little problem could create lots of troubles for a software developer or a team of developers. There is another famous expression and you've also heard it: Google It! What does it actually mean? It means, that a software developer thinks that a problem he/she actually deals with is not a unique one and there is already a solution. That is possible! However, for me it means that the developer is not doing the job in a right way and doesn't try to work hard to find a solution by himself/herself. I'm not going to convince anybody that in order to become a real Software Developer ( can you see that I stressed with capital S and capital D? ) an attitude of a Strong Problem Solver needs to be grown up. Here is some set of quotes from a Technical Report I've created in 2010 related to how I solved lots of technical problems, actually overwhelming for just one C++ developer responsible for the integration of a legacy C++ compiler with the ScaLib project and 4 weeks time frame, without asking for help from anybody else due to project constraints: ... DevNote #0026: Detected problems and issues over the course of integration Turbo C++ compiler v3.00 with the ScaLib project This is an outlook on what happened with the ScaLib Project over the course of integration Borland's International Turbo C++ compiler v3.00 in October 2010. At the end of 2009 a GNU compiler MinGW v3.4.2 was integrated into the Visual Studio 2005 development environment and it significantly helped to clean the C++ codes of the ScaLib library. In the 2nd part of 2010 year a state of the project was considered as a very stable with a very good portability between different Platforms. A compilation process looked like this: Every time when the Build was started it has to be done for at least three Platforms, one by one, using two different compilers: - First step, Visual Studio 2005 compiler ( for Desktop ) - Second step, Visual Studio 2005 compiler ( for Embedded ) - Third step, MinGW v3.4.2 compiler ( for Desktop ) But, I was feeling that two compilers are not enough in order to prove that a quality of portable C++ codes is high. Unfortunately, both compilers, that is, Visual C++ and MinGW v3.4.2, are targeted for Windows Operating Systems. I suggected to integrate another C++ compiler that doesn't have any Microsoft's or Windows header files. The best candidate for that portability verification task was a compiler from Borland International for an MS-DOS Operating System and I decided to try Turbo C++ version 1.01. An attempt with Borland International Turbo C++ v1.01 stoped almost instantly. It was unsuccessful because version 1.01 doesn't support C++ templates! But, I was able to compile many pieces of generic C++ codes in the ScaLib library and it was clear that I'm working in a right direction. So, it was just a matter of time and when Borland International Turbo C++ compiler v3.00 was found on one of my archive CD I decided to start a second attempt. The key features of the compiler included support of C++ standard 2.1 and version 3.0 templates. In the morning on October 11th, 2010 I've finally started the integration of Turbo C++ compiler v3.00 and half an hour later the situation could be described with just two words: TURBO DISASTER! I've stepped back in time for almost 20 years and the number of issues and problems was OVERWHELMING. I was simply forced to make records on a clean piece of paper and it was filled on both sides completely in about one hour. Here is an initial list of problems detected during first day of integration: - A project configuration problems for a new MS-DOS based Platform - A problem with long file names because in MS-DOS only 8.3 file names are supported ( 12 characters in total ) - There is no support of '_stdcall' key word - There are no 'BOOL' or 'bool' data types - There is no '_complex' data type - An issue with a native data type 'int' because by default it is a 16-bit based - An issue with a native data type 'long double' - Usage of MAX_PATH macro constant ( Non-Portability Leak ) - Usage of MB_OK macro constant which is used in famous MessageBox function of Win32 API ( Non-Portability Leak ) - Incompatibility of #pragma directives between Visual C+, MinGW and Turbo C++ compilers - A problem with forward declarations of template functions - A problem with declaration constructors of a template class as 'inline' - A problem with declaration of a parameter based on 'typedef enum' type in a constructor of a template class - A problem with a data type declared as 'typedef enum' because by default it is a 16-bit based - A problem with default value declaration for a parameter, like ...param = T()..., in a method of a template class - A problem with global C++ operators 'new' and 'delete' - A problem with declaration of global C++ operators 'new' and 'delete' inside a '__cplusplus' block: #ifdef __cplusplus extern "C" { #endif ... ... #ifdef __cplusplus }; #endif - A problem with ';' at the end of a macro with '()' - An issue with 'strncpy' CRT-function - A problem with 'reinterpret_cast', 'dynamic_cast' and 'static_cast' C++ operators ( key-words ) which are not supported by Turbo C++ compiler v3.00 - A problem related to data types alignment because it is supported on a global scale by Turbo C++ compiler v3.00 as a command line option and as a #pragma option directive - A problem with Turbo C++ compiler v3.00 when compiling sources in RELEASE configuration when a big array of data is created locally, that is on the stack - Turbo C++ compiler v3.00 creates binaries for a Real-Mode Operating System, that is MS-DOS, and there is no direct support for a Protected-Mode or Flat Memory model - Turbo C++ compiler v3.00 ( command line version ) couldn't find Turbo Link executable if the installation was done in a different from '..\TC30' folder Note: All search paths were added! It is possible that this is an almost 20-year old bug - Issues with MS-DOS Real-Mode total memory and Turbo C++ compiler v3.00 Memory models constraints - A problem with the Turbo Link if a Tiny or Compact Memory models are selected: ... Turbo Link Version 5.0 Copyright (c) 1992 Borland International Error: Group DGROUP exceeds 64K ... - Turbo Link was unable to generate COM-executable: ... Fatal: Cannot generate COM file : segment-relocatable items present ... - Turbo C++ compiler v3.00 fails to compile some C++ codes and exits with an error message: "FATAL ERROR: GP FAULT" - A problem with some protected mode instructions generated by Turbo C++ compiler v3.00 in case of Large or Huge Memory models of an application - The Windows NTVDM crashed when an illegal CPU instruction was executed in a test application - Math related problems: ... Floating point error: Divide by 0. Abnormal program termination ... - A complitely unexplained error messages "Multi-Bit ECC Error System Halted" and "Divide Error" when executing an application compiled for a Huge Memory model - Dynamic memory management related problems with a Memory Tracing subsystem in case of selecting different Memory models for a test application And many, many another issues and problems appearing over the course of firsttwo weeks of integration. I would split these problems into 3 groups: - Project configuration due to 8.3 file names in MS-DOS - Not Portable "Leaks" in the C++ source codes of the ScaLib library - Turbo C++ compiler v3.00 related At the end of the second day, when almost 99 percent of all source codes were commented out (!), I had a plan on how to move ahead and about 4 weeks later Turbo C++ compiler v3.00 integration was completed. And what's the point? It is simple: If I would try to follow the Google It! approach the integration of the C++ compiler would be never completed and the project would be possibly canceled. As of today 5 different C++ compilers ( 14 versions ) are integrated with the ScaLib project and in terms of portability this is a really unique one. I also would like to address my message to modern software developers: Guys, you need to fight problems and you should not expect that somebody will give you a solution! Good Luck to everybody! Best regards, Sergey Kostrov Senior C++ Software Developer Intel Black Belt Software Developer ( since September 2012 ) Calgary, Alberta, Canada Note: ScaLib stands for Set Of Common Algorithms Library

[zalia64]

Hello Sergey,

Could you pls elaborate on How You Solved the problem? that is something I could learn from. I am not faint hearted myself, but the problem described is a programer's nightmare.

The wish to integrate good, proven, compact DOS-based programs and tools into gigantic, super-feature-rich, megabyte-length Windows environment is not so uncommon.

[SergeyKostrov]

>>...Could you pls elaborate on How You Solved the problem? that is something I could learn from... I will provide a couple of examples. However, a complete list of all solved problems ( and solutions ) is very long and it doesn't make sense to post it. Let me know if you need a Turbo C++ v3.x compiler for tests, etc.

[zalia64]

>>...Could you pls elaborate on How You Solved the problem? that is something I could learn from.

I meant the strategy of attack. The 'list of problems' as a starter is self-evident.

Given the fact that each patch may insert new hidden bugs, a long list of patches ensures troubles. Obviously, you did not do a lengthy point-by-point patch through. Obviously, you had a better strategy. 

It was proved to me on my first year, that a normal person (me) can't calculate a 5x5 determinant, using pen and paper in the straight-forward way. How? I checked and re-checked my results, and never got two successive equal numbers. That lesson helped me avoid long lists of patches.

[David_M_23]

I too am interested in how you tackled this!

By the way, why did you choose Turbo C++ 3, an ancient DOS C++ compiler?  It's very impressive that you got it working, but I'm still puzzled why this would be a target.  Does ScaLib have DOS users?

If you want a non-GCC, non-MS compiler, especially Borland, why not a modern version of Borland / Embarcadero C++ Builder?  If you used, say, Embarcadero C++Builder XE3 (two versions old now) you could have a non-MS but modern 32 and 64-bit C++ compiler.  In fact the 64-bit version would be quite useful since it's a brand new compiler for the company based on CLANG.  The 32-bit one is decent but would give you some useful compatibility checks; it's not CLANG-based but is the updated descendant of the Borland C++ compiler chain.  In addition, you could use it for cross-platform checks, since it compiles for OSX and iOS as well.

This isn't a criticism, just curiosity about why you chose the target you did, and perhaps also a suggestion for a compile target that users of your library might find useful.  (I think you'll find more people using C++Builder for Windows, OSX, or iOS than using Turbo C++ for DOS.)

[SergeyKostrov]

>>...why did you choose Turbo C++ 3, an ancient DOS C++ compiler?.. In order to break ALL dependencies on Microsoft APIs, like Win32 API, and deviated CRT functions ( there are lots of them and many Windows developers simply Do Not know about it. Please believe me! ). That is why the codes are 99.99% portable. Consider Turbo C++ version 3.x as some kind of Supreme C++ Normalizer. When codes are compiled with Turbo C++ it instantly detects any deviations in codes from a 20-year-old C++ standard 2.1 and version 3.0 templates. At the same time Turbo C++ is Not used for any performance evaluations. >>...Does ScaLib have DOS users?.. There are No any MD-DOS customers however Turbo C++ version 3.x forces a developer to implement codes in as clean as possible way. For example, an executable just 24KB in size ( Not MB! ) could be created. The executable of a such small size doesn't do too much and it is a demonstration of efficient and highly optimized internal solutions. For example, the library has a set of linear algebra algorithms and one of them is Strassen Heap Based matrix multiplication algorithm. The same codes could be compiled for 16-bit MS-DOS, 32-bit Desktop and Embedded, and 64-bit Desktop operating systems with five ( 5 ) different major versions of C++ compilers ( and already 15 different editions / updates ): - Turbo C++ v3.x - Borland C++ v5.x - MinGW v3.4.2 & v4.8.1 - Microsoft Visual C++ v6.x ( SP5 ) - Microsoft Visual Studios 2005, 2008, 2010, 2012 - Intel C++ v7.x - Intel C++ v8.x - Intel C++ v12.x ( two updates are supported ) - Intel C++ v13.x ( two updates are supported )

[SergeyKostrov]

I was asked already by a couple of IDZ users about what exactly I did to complete that integration. That's a really big subject and I could write a book with many chapters about it. However, here are a couple of very simple examples which could help you to understand what kind of problems I had to deal with...

[SergeyKostrov]

[ Example 1 - Part A - MS compatible C++ compilers ] // MSC compilers Compatible Data Types ( size in bytes ): RTbool - 1 RTchar - 1 RTuchar - 1 RTbyte - 1 RTubyte - 1 RTlong - 4 RTulong - 4 RTint8 - 1 RTuint8 - 1 RTint16 - 2 RTuint16 - 2 RTint32 - 4 RTuint32 - 4 RTfloat - 4 RTm128float - 4 RTdouble - 8 RTm128double - 8 RTint - 4 RTuint - 4 RTint64 - 8 RTuint64 - 8 RTclock_t - 4 RTm128int - 4 Incompatible Data Types ( size in bytes ): RTtime_t - 4 RTtchar - 2 RTlongdouble - 8 // ICC compilers Compatible Data Types ( size in bytes ): RTbool - 1 RTchar - 1 RTuchar - 1 RTbyte - 1 RTubyte - 1 RTlong - 4 RTulong - 4 RTint8 - 1 RTuint8 - 1 RTint16 - 2 RTuint16 - 2 RTint32 - 4 RTuint32 - 4 RTfloat - 4 RTm128float - 4 RTdouble - 8 RTm128double - 8 RTint - 4 RTuint - 4 RTint64 - 8 RTuint64 - 8 RTclock_t - 4 RTm128int - 4 Incompatible Data Types ( size in bytes ): RTtime_t - 4 RTtchar - 2 RTlongdouble - 8

[SergeyKostrov]

[ Example 1 - Part B - GCC compatible C++ compilers ] // MGW compilers Compatible Data Types ( size in bytes ): RTbool - 1 RTchar - 1 RTuchar - 1 RTbyte - 1 RTubyte - 1 RTlong - 4 RTulong - 4 RTint8 - 1 RTuint8 - 1 RTint16 - 2 RTuint16 - 2 RTint32 - 4 RTuint32 - 4 RTfloat - 4 RTm128float - 4 RTdouble - 8 RTm128double - 8 RTint - 4 RTuint - 4 RTint64 - 8 RTuint64 - 8 RTclock_t - 4 RTm128int - 4 Incompatible Data Types ( size in bytes ): RTtime_t - 4 RTtchar - 2 RTlongdouble - 12

[SergeyKostrov]

[ Example 1 - Part C - Borland compatible C++ compilers ] // BCC compiler Compatible Data Types ( size in bytes ): RTbool - 1 RTchar - 1 RTuchar - 1 RTbyte - 1 RTubyte - 1 RTlong - 4 RTulong - 4 RTint8 - 1 RTuint8 - 1 RTint16 - 2 RTuint16 - 2 RTint32 - 4 RTuint32 - 4 RTfloat - 4 RTm128float - 4 RTdouble - 8 RTm128double - 8 RTint - 4 RTuint - 4 RTint64 - 8 RTuint64 - 8 RTclock_t - 4 RTm128int - 4 Incompatible Data Types ( size in bytes ): RTtime_t - 4 RTtchar - 2 RTlongdouble - 10 // TCC compiler Compatible Data Types ( size in bytes ): RTbool - 1 RTchar - 1 RTuchar - 1 RTbyte - 1 RTubyte - 1 RTlong - 4 RTulong - 4 RTint8 - 1 RTuint8 - 1 RTint16 - 2 RTuint16 - 2 RTint32 - 4 RTuint32 - 4 RTfloat - 4 RTm128float - 4 RTdouble - 8 RTm128double - 8 RTint - 4 RTuint - 4 RTint64 - 8 RTuint64 - 8 RTclock_t - 4 RTm128int - 4 Incompatible Data Types ( size in bytes ): RTtime_t - 4 RTtchar - 1 RTlongdouble - 10

[SergeyKostrov]

[ Example 1 - Part D ] There is a small subset of some data types which create Binary Compatibility problems. So, take a look at a section with Incompatible Data Types: // MSC compilers ... Incompatible Data Types ( size in bytes ): RTtime_t - 4 RTtchar - 2 RTlongdouble - 8 // ICC compilers ... Incompatible Data Types ( size in bytes ): RTtime_t - 4 RTtchar - 2 RTlongdouble - 8 // MGW compilers ... Incompatible Data Types ( size in bytes ): RTtime_t - 4 RTtchar - 2 RTlongdouble - 12 // BCC compiler ... Incompatible Data Types ( size in bytes ): RTtime_t - 4 RTtchar - 2 RTlongdouble - 10 // TCC compiler ... Incompatible Data Types ( size in bytes ): RTtime_t - 4 RTtchar - 1 RTlongdouble - 10 Even if RTtime_t equals to 4 in all these cases this is Not Compatible (!) data type and take a look at MSDN to read a topic about what is wrong with the data type.

[SergeyKostrov]

This is how a log-file looks like when a Binary Compatibility of Data Types is Passed, for example on a 16-bit platform: ... *********************************** Configuration - COS16_TCC - Debug CTestSet::InitTestEnv - Passed * CCommonSet Start * > CCommonSet Methods < GetPlatformName - COS16_TCC GetPlatformName - Passed GetPlatformType - 16-bit GetPlatformType - Passed VerifyDataTypes - Data Types Verified VerifyDataTypes - Passed GetHardwareParams - Passed GetSoftwareParams - Passed > CCommonSet Algorithms < * CCommonSet End * Memory Blocks Allocated : 1 Memory Blocks Released : 1 Memory Blocks NOT Released: 0 ...

[SergeyKostrov]

[ Example 2 ] There are some issues with measurements of time intervals and a workaround needs to be used. Here is a simple demonstration of how results look like on 32-bit and 16-bit platforms when a portable version of a CrtRdtcs function is used: [ MinGW C++ compiler ] ... > Test1144 Start < Sub-Test 5 - [ CrtRdtsc ] Sub-Test 5 - 1 sec delay between calls to [ CrtRdtcs ] - 1586180388 clock cycles Test Completed in 1000 ticks > Test1144 End < ... [ Turbo C++ compiler ] ... > Test1144 Start < Sub-Test 5 - [ CrtRdtsc ] Sub-Test 5 - 1 sec delay between calls to [ CrtRdtcs ] - 18 clock ticks Test Completed in 1000 ticks > Test1144 End < ... Note: There are 18.2 ticks in a second on a 16-bit platform. This is a limitation of a Timer Interrupt ( INT 8 ) and how programmable controller of interrupts is initialized by default, for example in MS-DOS, PhysTech-DOS, DR-DOS and similar operating systems.

[SergeyKostrov]

[ Example 3 ] Support for some simple Intel intrinsic functions on a 16-bit platform when Turbo C++ compiler is used, like _mm_prefetch or _mm_sfence. That's the problem because inline assembler is Not supported by a Turbo C++ compiler.

[SergeyKostrov]

[ Example 4 ] There is No support for __restrict__ keyword by Turbo C++ compiler and a macro-based workaround needs to be used.

[SergeyKostrov]

[ Example 5 ] There are internal problems with Turbo C++ compiler. For example, it fails to build an executable in Debug configuration when a copy constructor is declared in some class. At the same time, in Release configuration an executable could be successfully built.

[SergeyKostrov]

[ Example 6 ] Exact Binary Compatibility of Data Types breaks if data transferred between platforms using TCP/IP or RS-232. If a union, like typedef union tagSomeUnion { char A; char B; char C; size_t D; } SomeUnion; is used then there is No Binary Compatibility and this is why: 16-bit platform: sizeof( SomeUnion ) = 2 bytes 32-bit platform: sizeof( SomeUnion ) = 4 bytes 64-bit platform: sizeof( SomeUnion ) = 8 bytes and so on. Note: Microsoft's RPC manages Binary Compatibility very good but I didn't have a Microsoft C++ compiler version 1.5 ( for MS DOS ). Next, Turbo C++ compiler doesn't provide any support for RPC on a 16-bit platforms. Also, it doesn't have a 16-bit Interface Definition Language ( IDL ) compiler to compile the IDL files with declared interfaces. Tools from competitors, like CORBA or DCE, do not help as well since I don't think they were ever released for 16-bit platforms.

[SergeyKostrov]

[ Example 7 ] Here is a 8.3-Problem with files names on 16-bit platforms and this is how it affects sources: ... #if ( defined ( _MSC_COMPILER_A_ ) || defined ( _ICC_COMPILER_B_ ) || defined ( _GCC_COMPILER_C_ ) ) #include "../../Include/SomeFileWithLongName.h" #endif #if ( defined ( _ Turbo C++ Compiler _ ) ) #include "../../Include/SomeFi~1.h" #endif ...

[SergeyKostrov]

[ Example 8 ] Here is a Configuration Params Problem and a simple solution looks like: ... g_iParam1 = atoi( getenv( "PARAM_1" ) ); g_iParam2 = atoi( getenv( "PARAM_2" ) ); g_iParam3 = atoi( getenv( "PARAM_3" ) ); ... Of course, number of parameters defined as Environment Variables needs to be limited but this is a one-line solution compared to multi-line implemntation of different ini-file, cfg-file, or xml-file readers, etc.

[SergeyKostrov]

[ Example 9 ] There is No any support for 64-bit integer types on 16-bit platforms and a very simple solution based on a Single Precision Floating Point data type ( float ) is used. But, there is a limitation and, for example, a valid range of unsigned 64-bit integers values is from 0 to 16777216 ( that is, from 0 to 2^24 ).

[SergeyKostrov]

And, please let me know if you have some questions related to implementations of highly portable software systems because I've been doing it since 1994.